Audio encoding method and apparatus, and audio decoding method and apparatus

Abstract

An audio encoding method and apparatus, and an audio decoding method and apparatus. The encoding method comprises: acquiring a first code stream, which is obtained on the basis of a first audio signal (701); acquiring side information, which is obtained on the basis of a second audio signal (702); acquiring a second code stream on the basis of the side information (703); and transmitting the code streams, which comprise the first code stream and the second code stream (704).

Description

Audio encoding and decoding method and device Technical Field

[0001] The present application relates to audio processing technology, and in particular to an audio encoding and decoding method and device. Background Art

[0002] With the upgrade of communication systems, the quality of audio transmission has been greatly improved. However, there are still some scenarios where the transmission bandwidth is still limited, and the limited transmission bits cannot represent the full-bandwidth signal. To solve this problem, the relevant technology divides the audio into low-frequency components and high-frequency components, processes the low-frequency components through encoding and decoding, and allocates most of the transmission bits to the code stream of the low-frequency components to ensure the encoding quality of the low-frequency components; then, using the correlation between high and low frequencies, the high-frequency components are restored through band expansion on the decoding side. This method of extending the frequency band is called Bandwidth Extension (BWE). Compared with narrowband audio, the listening experience of the audio obtained using BWE will be significantly improved.

[0003] However, the BWE method described above may cause audio distortion due to the weak correlation between the spectrum of the high-frequency component and the spectrum of the low-frequency component.

[0004] Summary of the Invention

[0005] The present application provides an audio coding and decoding method and apparatus to reconstruct more high-frequency details, thereby avoiding the problem of audio distortion.

[0006] In a first aspect, the present application provides an audio encoding method, comprising: obtaining a first bitstream, the first bitstream being obtained based on a first audio signal, the first audio signal being an audio signal having a frequency lower than a preset threshold in a current frame; obtaining side information, the side information being obtained based on a second audio signal, the second audio signal including an audio signal having a frequency higher than or equal to the preset threshold in the current frame; obtaining a second bitstream based on the side information; and transmitting a bitstream, the bitstream comprising the first bitstream and the second bitstream.

[0007] In this embodiment, side information is extracted from audio through a machine learning model to guide the decoding end to implement non-blind BWE and reconstruct high-frequency components, especially high-frequency components that are weakly correlated with low-frequency components. This implementation method can reflect more details of the high-frequency components in the current frame in the feature information. These details, such as the characteristics of the high-frequency components, the correlation between the high-frequency components and the low-frequency components, etc., help to reconstruct more high-frequency details, thereby avoiding the problem of audio distortion.

[0008] The original audio is sampled to obtain an audio frame sequence, which includes multiple audio frames, which can be compressed frame by frame at the encoding end. For the convenience of description, in the embodiment of the present application, the audio frame currently being processed is referred to as the current frame.

[0009] The characteristics of the audio signal contained in the current frame can be characterized by frequency, amplitude and phase. The embodiment of the present application extracts the first audio signal and the second audio signal from the current frame according to the frequency, wherein the first audio signal is an audio signal in the current frame whose frequency is lower than a preset threshold (in the embodiment of the present application, the audio signal that meets this condition can also be referred to as a low-frequency signal or a low-frequency component), and the second audio signal includes an audio signal in the current frame whose frequency is higher than or equal to the preset threshold (in the embodiment of the present application, the audio signal that meets this condition can also be referred to as a high-frequency signal or a high-frequency component). The aforementioned preset threshold can be 16kHz, for example, and the embodiment of the present application does not make specific restrictions on this. It should be noted that the foregoing classifies the audio signal in the current frame whose frequency is equal to the preset threshold into the second audio signal. Optionally, the embodiment of the present application can also classify the audio signal in the current frame whose frequency is equal to the preset threshold into the first audio signal, and there is no specific restriction on this.

[0010] In some possible implementations, a low-pass filter may be used to extract the first audio signal from the current frame.

[0011] In some possible implementations, a core encoder may be used to encode the first audio signal. The core encoder may be, for example, an AVS3 core encoder, which is not specifically limited in the embodiments of the present application.

[0012] In this embodiment of the present application, the second audio signal may exist in the following three situations:

[0013] 1) The second audio signal includes only high-frequency components. In this case, the second audio signal is equivalent to the high-frequency components of the current frame. Correspondingly, the side information may only include characteristic information of the high-frequency components.

[0014] 2) The second audio signal includes not only the high-frequency component but also a portion of the low-frequency component. That is, the second audio signal and the first audio signal share some of the same audio signal. In this case, the second audio signal is equivalent to the high-frequency component plus some of the low-frequency component of the current frame. Accordingly, the side information may include not only the characteristic information of the high-frequency component but also the correlation information between the high-frequency component and the aforementioned low-frequency component.

[0015] 3) The second audio signal includes all low-frequency components in addition to the high-frequency components. That is, the second audio signal includes the first audio signal. In this case, the second audio signal is equivalent to the current frame. Accordingly, the side information may include not only the feature information of the high-frequency components but also the correlation information between the high-frequency and low-frequency components.

[0016] It can be seen from this that side information can only occupy a small number of transmission bits to guide the decoding end to implement non-blind BWE to reconstruct high-frequency components, especially the mapping relationship between high and low frequencies to build complex signals.

[0017] In some possible implementations, obtaining the side information may include inputting the second audio signal into a pre-trained second machine learning model to obtain the side information, where the second machine learning model includes at least one sub-model. Each of the at least one sub-models may focus on different characteristics. For example, constraints may be set for each sub-model, including input and output constraints, model structure, and processing rules, so that each sub-model can focus on extracting side information related to a corresponding characteristic.

[0018] In the embodiment of the present application, the second machine learning model can refer to the neural network, deep neural network, convolutional neural network, recurrent neural network, etc. mentioned above. The machine learning model can reflect more details of the high-frequency components in the current frame in the feature information. These details, such as the characteristics of the high-frequency components and the relationship between the high-frequency components and the low-frequency components, help reconstruct more high-frequency details.

[0019] In some possible implementations, the second machine learning model may include multiple sub-models, and the input of each of the multiple sub-models includes an audio signal in the current frame whose frequency is higher than or equal to a preset threshold, that is, the input of each of the multiple sub-models is part or all of the audio signal in the second audio signal.

[0020] In the embodiments of the present application, if the input of a sub-model contains only high-frequency components, the sub-model is primarily used to compress the high-frequency components into features, or to represent the high-frequency components through features. If the input of the sub-model contains low-frequency components in addition to high-frequency components, the sub-model is also used to extract the mapping relationship between high and low frequencies. Using multiple sub-models can obtain more diverse features in the extraction of side information. Therefore, this application does not specifically limit the number, type, function, etc. of sub-models.

[0021] The outputs of each of the n sub-models, including feature information 1, feature information 2, ..., feature information n, are merged to obtain side information. That is, the side information includes the feature information output by each of the n sub-models.

[0022] In some possible embodiments, the first sub-model and at least one other sub-model among the multiple sub-models include the same neurons, and the same neurons are located in the input stage, output stage or intermediate stage of the first sub-model, and the first sub-model is any one of the multiple sub-models.

[0023] The first sub-model may have the same neurons (also referred to as shared neurons) as one or more of the multiple sub-models included in the second machine learning model. The same neurons may have one or more layers, and the one or more layers of neurons may be located in the input stage, output stage or intermediate stage of the sub-model, wherein the input stage refers to the first layer of the input end of the sub-model and the layer or layers thereafter, the output stage refers to the last layer of the output end of the sub-model and the layer or layers before it, and the intermediate stage refers to the layer or layers excluding the first and last layers of the sub-model. It should be noted that the embodiments of the present application do not specifically limit the number, type, etc. of the aforementioned same neurons, nor do they specifically limit the positions of the input stage, output stage and intermediate stage.

[0024] In some possible implementations, the input of the first sub-model also includes the first input; when the second machine learning model includes multiple sub-models, the first input includes the output of at least one other sub-model among the multiple sub-models, or the first input includes the output of at least one sub-rule method; the first sub-model is any one of the multiple sub-models.

[0025] In some possible implementations, the input of the first sub-model also includes a second input, which is an audio signal obtained after redundant information processing of the first input; when the second machine learning model includes multiple sub-models, the first input includes the output of at least one other sub-model among the multiple sub-models, or the first input includes the output of at least one sub-rule method; the first sub-model is any one of the multiple sub-models.

[0026] It should be noted that the above two examples are illustrated using two sub-models as an example, but this does not constitute a limitation on the series relationship between sub-models or between sub-models and sub-rule methods. In addition to its own input, the input of any sub-model may also include the output of one or more other sub-models or the output after redundant processing, and / or the output of one or more sub-rule methods or the output after redundant processing. The embodiments of the present application do not limit the number of series connections or the series connection method.

[0027] In some possible implementations, obtaining the side information may further include processing the second audio signal according to a preset rule to obtain the side information, where the preset rule includes at least one sub-rule.

[0028] In an embodiment of the present application, a method for extracting features according to a pre-set rule may include, for example, extracting a spectrum envelope and extracting features based on AVS3. The spectrum envelope represents the energy of each frequency band in the spectrum of the current frame, which is used to recover the energy of the corresponding frequency bands for the decoding side. The features extracted by AVS3 include the frequency band energy and the identification of the harmonic and noise characteristics of each frequency band, which are used to recover the BWE basis for the decoding side. Compared to machine learning methods, rule-based methods can retain information such as spectrum energy, thereby avoiding the impact on the auditory experience.

[0029] In an embodiment of the present application, based on the second machine learning model, a rule method can also be set to process the second audio signal according to the aforementioned rule method to obtain the corresponding output (feature information).

[0030] The feature information 1 output by the second machine learning model and the feature information 2 output by the rule method are merged to obtain side information, that is, the side information includes the feature information output by the second machine learning model and the feature information output by the rule method.

[0031] In some possible implementations, the preset rule method may include multiple sub-rule methods, and the input of each of the multiple sub-rule methods includes an audio signal in the current frame whose frequency is higher than or equal to a preset threshold, that is, the input of each of the multiple sub-rule methods is part or all of the audio signal in the second audio signal.

[0032] The outputs of the n sub-models and m regular methods include feature information 1, feature information 2, ..., feature information n, feature information n+1, feature information n+2, ..., feature information n+m. After merging, side information is obtained, that is, the side information includes the feature information output by the n sub-models and the feature information output according to the m regular methods.

[0033] In some possible implementations, the input of the first sub-rule method also includes a third input; when the pre-set rule method includes multiple sub-rule methods, the third input includes the output of at least one other sub-rule method among the multiple sub-rule methods, or the third input includes the output of at least one sub-model; the first sub-rule method is any one of the multiple sub-rule methods.

[0034] In some possible implementations, the input of the first sub-rule method also includes a fourth input, which is an audio signal obtained after redundant information processing of the third input; when the pre-set rule method includes multiple sub-rule methods, the third input includes the output of at least one other sub-rule method among the multiple sub-rule methods, or the third input includes the output of at least one sub-model; the first sub-rule method is any one of the multiple sub-rule methods.

[0035] It should be noted that the above example uses two sub-rule methods as an example, but this does not limit the series relationship between sub-rule methods and between sub-rule methods and sub-models. In addition to its own input, the input of any sub-rule method can also include the output of one or more other sub-rule methods or the output after redundant processing, and / or, the output of one or more sub-models or the output after redundant processing. The embodiment of the present application does not limit the number of series connections or the series connection method.

[0036] In some possible implementations, the second machine learning model and / or rule method for obtaining side information can be jointly trained. That is, by jointly training multiple mechanisms for obtaining side information (which can be the above-mentioned sub-models or sub-rules), the information redundancy between feature groups can be reduced or eliminated; a joint loss function is set for the multiple mechanisms for obtaining side information, and the weights of the multiple mechanisms for obtaining side information are corrected through error back propagation to achieve the overall optimal effect of multiple feature information, thereby reducing or eliminating the information redundancy between feature groups; each mechanism for obtaining side information can have its own loss function to retain the uniqueness of the feature information it extracts.

[0037] In the embodiment of the present application, the side information may be encoded by a side information encoder. The side information encoder may use the same encoding algorithm as the core encoder described above, or may use a different encoding algorithm, which is not specifically limited.

[0038] The code stream generated by the encoder consists of two parts: one part (the first code stream) corresponds to the low-frequency components of the current frame, and the other part (the second code stream) corresponds to the side information of the current frame. The encoder transmits the code stream to the decoder, which reconstructs the low-frequency components and side information based on the code stream, thereby obtaining a reconstructed frame corresponding to the current frame.

[0039] In a second aspect, the present application provides an audio decoding method, comprising: receiving a bitstream, the bitstream comprising a first bitstream and a second bitstream; obtaining a first audio signal based on the first bitstream, wherein the frequency of the first audio signal is lower than a preset threshold; obtaining side information based on the second bitstream, wherein the side information is used to characterize characteristics of a second audio signal, wherein the frequency of the second audio signal is higher than or equal to the preset threshold; and obtaining a reconstructed frame based on the first audio signal and the side information.

[0040] In this embodiment, a machine learning model is used to obtain reconstructed frames based on low-frequency components under the guidance of side information, especially reconstructed frames in scenarios where the correlation between high-frequency components and low-frequency components is weak or the relationship between high- and low-frequency components is close but the mapping relationship is complex. This implementation method can reflect more details of the high-frequency components in the feature information. These details, such as the characteristics of the high-frequency components, the correlation between the high-frequency components and the low-frequency components, etc., help to reconstruct more high-frequency details, thereby avoiding the problem of audio distortion.

[0041] In the embodiment of the present application, a code stream is received from an encoding end. Corresponding to the code stream generated by the encoding end, the received code stream also includes a first code stream and a second code stream.

[0042] In the embodiment of the present application, the first code stream can be parsed by the core decoder to obtain the first audio signal.

[0043] The reconstructed frame to be obtained by the decoding end corresponds to the current frame of the encoding end. The characteristics of the audio signal contained in the reconstructed frame can also be characterized by frequency, amplitude and phase. According to the frequency division, the reconstructed frame also includes a first audio signal and a second audio signal, wherein the frequency of the first audio signal is lower than the preset threshold audio signal (in the embodiment of the present application, the audio signal that meets this condition can also be called a low-frequency signal or a low-frequency component), and the frequency of the second audio signal is higher than or equal to the preset threshold audio signal (in the embodiment of the present application, the audio signal that meets this condition can also be called a high-frequency signal or a high-frequency component), that is, the reconstructed frame includes a low-frequency component and a high-frequency component. The aforementioned preset threshold can be, for example, 16kHz, and the embodiment of the present application does not make specific restrictions on this. It should be noted that the foregoing classifies the audio signal with a frequency equal to the preset threshold as the second audio signal. Optionally, the embodiment of the present application can also classify the audio signal with a frequency equal to the preset threshold as the first audio signal, and this is not specifically limited.

[0044] Based on this, the first audio signal obtained according to the first bitstream may be the low-frequency component of the reconstructed frame. That is, the decoding end may directly obtain the low-frequency component of the reconstructed frame by parsing the bitstream.

[0045] The side information is used to characterize the characteristics of the second audio signal. Furthermore, the side information can also be used to characterize the mapping relationship between high and low frequencies. In the embodiments of the present application, the side information only occupies a small number of transmission bits and can guide the decoder to implement non-blind BWE to reconstruct high-frequency components, especially high-frequency components that are closely related to low-frequency components but have a complex mapping relationship.

[0046] In some possible implementations, obtaining the side information based on the second bitstream can be performed by directly parsing the second bitstream to obtain the side information. That is, the second bitstream is obtained by the encoder processing the side information, so the decoder can directly parse the second bitstream to obtain the side information.

[0047] Optionally, the second code stream is parsed to first obtain the processed front side information; then the processed front side information is subjected to side information processing to obtain side information, wherein the side information processing includes inputting a pre-trained third machine learning model or processing according to a pre-set third rule.

[0048] In an embodiment of the present application, the information carried by the second bitstream may be pre-processing side information associated with the side information. This information may be obtained by the encoder based on the side information to further conserve bitstream, and then encoded into the bitstream. Accordingly, to obtain the side information, the decoder may first parse the pre-processing side information obtained from the second bitstream, and then obtain the side information based on the pre-processing side information. The aforementioned method for obtaining the side information may include obtaining the side information through a third machine learning model or according to a pre-set third rule.

[0049] In some possible implementations, obtaining the reconstructed frame can be performed using a pre-trained first machine learning model based on the first audio signal and the side information. Because obtaining the reconstructed frame references the side information to achieve expansion from low-frequency components to high-frequency components, obtaining the reconstructed frame can also be referred to as non-blind BWE.

[0050] In the embodiment of the present application, the first machine learning model can refer to the above-mentioned neural network, deep neural network, convolutional neural network, recurrent neural network, etc.

[0051] Optionally, the reconstructed frame is directly output by the first machine learning model.

[0052] In an embodiment of the present application, a non-blind BWE can use a machine learning model (a first machine learning model) to directly output a reconstructed frame based on the decoded low-frequency component (the first audio signal) under the guidance of the side information. In addition, the first machine learning model can perform codec distortion correction for the low-frequency component and possible audio rendering, which is not specifically limited in the embodiment of the present application.

[0053] Optionally, the reconstructed frame is obtained based on the second audio signal output by the first machine learning model and the first audio signal.

[0054] In an embodiment of the present application, the non-blind BWE can adopt a machine learning model (a first machine learning model) to output a reconstructed high-frequency component based on the decoded low-frequency component (the first audio signal) under the guidance of the side information, and merge or add the reconstructed high-frequency component with the decoded low-frequency component to obtain reconstructed audio. In addition, the first machine learning model can complete the codec distortion correction of the low-frequency component and possible audio rendering, which is not specifically limited in the embodiment of the present application.

[0055] In some possible implementations, the first machine learning model may include a first sub-model and a second sub-model, and the non-blind BWE may input the first audio signal and the first side information into the first sub-model to obtain a third audio signal; and then input the third audio signal and the second side information into the second sub-model to obtain a reconstructed frame; the first side information and the second side information are respectively part or all of the information in the side information.

[0056] In some possible implementations, the non-blind BWE may process the first audio signal and the third side information according to a pre-set first rule to obtain a fourth audio signal; then input the fourth audio signal and the fourth side information into the first machine learning model to obtain a reconstructed frame; the third side information and the fourth side information are respectively part or all of the information in the side information.

[0057] In some possible implementations, the non-blind BWE can obtain a fifth audio signal based on the first audio signal and the fifth side information through a first machine learning model obtained by pre-training; and then process the fifth audio signal and the sixth side information according to a pre-set second rule to obtain a reconstructed frame; the fifth side information and the sixth side information are respectively part or all of the information in the side information.

[0058] It should be noted that the multiple side information may be part or all of the side information, and the multiple side information may be completely the same or completely different or partially the same and partially different, and there is no specific limitation on this.

[0059] In a third aspect, the present application provides an encoding device, comprising: an acquisition module, configured to acquire a first bitstream, the first bitstream being obtained based on a first audio signal, the first audio signal being a low-frequency signal in a current frame having a frequency lower than a preset threshold; an extraction module, configured to acquire side information, the side information being obtained based on a second audio signal, the second audio signal comprising a high-frequency signal in the current frame having a frequency higher than or equal to the preset threshold; an encoding module, configured to acquire a second bitstream based on the side information; and a transmission module, configured to transmit bitstreams, the bitstreams comprising the first bitstream and the second bitstream.

[0060] In a possible implementation, the extraction module is specifically configured to input the second audio signal into a pre-trained second machine learning model to obtain the side information, where the second machine learning model includes at least one sub-model.

[0061] In one possible implementation, when the second machine learning model includes multiple sub-models, the input of each of the multiple sub-models includes part of the audio signal or all of the audio signal in the second audio signal, wherein the part of the audio signal includes the high-frequency signal.

[0062] In one possible implementation, the first sub-model and at least one other sub-model among the multiple sub-models include the same neurons, and the same neurons are located in the input stage, output stage or intermediate stage of the first sub-model, and the first sub-model is any one of the multiple sub-models.

[0063] In a possible implementation, the side information includes feature information outputted respectively by the multiple sub-models.

[0064] In one possible implementation, when the second machine learning model includes a sub-model, the side information includes feature information output by the sub-model.

[0065] In a possible implementation, the extraction module is specifically configured to process the second audio signal according to a preset rule to obtain the side information, where the preset rule includes at least one sub-rule.

[0066] In a possible implementation, when the preset rule includes multiple sub-rule methods, inputs of each of the multiple sub-rule methods include part of or all of the second audio signal, wherein the part of the audio signal includes the high-frequency signal.

[0067] In a possible implementation, the side information includes feature information outputted respectively according to the multiple sub-rules.

[0068] In a possible implementation, when the preset rule mode includes a sub-rule mode, the side information includes feature information output according to the sub-rule mode.

[0069] In one possible implementation, the input of the first sub-model also includes a first input; when the second machine learning model includes multiple sub-models, the first input includes the output of at least one other sub-model among the multiple sub-models, or the first input includes the output of the at least one sub-rule method; the first sub-model is any one of the multiple sub-models.

[0070] In one possible implementation, the input of the first sub-model also includes a second input, which is an audio signal obtained after the first input is processed with redundant information; when the second machine learning model includes multiple sub-models, the first input includes the output of at least one other sub-model among the multiple sub-models, or the first input includes the output of the at least one sub-rule method; the first sub-model is any one of the multiple sub-models.

[0071] In one possible implementation, the input of the first sub-rule method also includes a third input; when the preset rule method includes multiple sub-rule methods, the third input includes the output of at least one other sub-rule method among the multiple sub-rule methods, or the third input includes the output of the at least one sub-model; the first sub-rule method is any one of the multiple sub-rule methods.

[0072] In one possible implementation, the input of the first sub-rule method also includes a fourth input, and the fourth input is an audio signal obtained after the third input is processed with redundant information; when the preset rule method includes multiple sub-rule methods, the third input includes the output of at least one other sub-rule method among the multiple sub-rule methods, or the third input includes the output of the at least one sub-model; the first sub-rule method is any one of the multiple sub-rule methods.

[0073] In a possible implementation, the method further includes: a filtering module configured to obtain the first audio signal from the current frame by low-pass filtering.

[0074] In a possible implementation manner, the second audio signal further includes part or all of the audio signal in the first audio signal.

[0075] In a fourth aspect, the present application provides a decoding device, comprising: a receiving module for receiving a bitstream, the bitstream comprising a first bitstream and a second bitstream; an acquisition module for acquiring a first audio signal based on the first bitstream, the frequency of the first audio signal being lower than a preset threshold; an extraction module for acquiring side information based on the second bitstream, the side information being used to characterize characteristics of a second audio signal, the frequency of the second audio signal being higher than or equal to the preset threshold; and a reconstruction module for acquiring a reconstructed frame based on the first audio signal and the side information.

[0076] In a possible implementation, the reconstruction module is specifically configured to obtain the reconstructed frame based on a first machine learning model obtained through pre-training according to the first audio signal and the side information.

[0077] In one possible implementation, the reconstructed frame is directly output by the first machine learning model; or, the reconstructed frame is obtained based on the second audio signal and the first audio signal output by the first machine learning model.

[0078] In one possible implementation, when the first machine learning model includes a first sub-model and a second sub-model, the reconstruction module is specifically used to input the first audio signal and the first side information into the first sub-model to obtain a third audio signal; and input the third audio signal and the second side information into the second sub-model to obtain the reconstructed frame; the first side information and the second side information are respectively part or all of the information in the side information.

[0079] In one possible implementation, the reconstruction module is specifically configured to process the first audio signal and the third side information according to a preset first rule to obtain a fourth audio signal; input the fourth audio signal and the fourth side information into the first machine learning model to obtain the reconstructed frame; and the third side information and the fourth side information are respectively part or all of the side information.

[0080] In one possible implementation, the reconstruction module is specifically configured to obtain a fifth audio signal based on a first machine learning model obtained through pre-training according to the first audio signal and the five-side information; and to process the fifth audio signal and the sixth side information according to a preset second rule to obtain the reconstructed frame; the fifth side information and the sixth side information are respectively part or all of the side information.

[0081] In a possible implementation, the extraction module is specifically configured to parse the second code stream to obtain the side information.

[0082] In one possible implementation, the extraction module is specifically configured to parse the second bitstream to obtain pre-processing side information; perform side information processing on the pre-processing side information to obtain the side information, wherein the side information processing includes inputting a pre-trained third machine learning model or processing according to a pre-set third rule.

[0083] In a fifth aspect, the present application provides a device comprising: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of the first to second aspects above.

[0084] In a sixth aspect, the present application provides a computer-readable storage medium comprising a computer program, which, when executed on a computer, enables the computer to perform the method described in any one of the first to second aspects above.

[0085] In a seventh aspect, the present application provides a computer program product, which includes a computer program code. When the computer program code is run on a computer, the computer executes the method described in any one of the first to second aspects above. BRIEF DESCRIPTION OF THE DRAWINGS

[0086] FIG1 shows a schematic diagram of an application scenario 100 according to an embodiment of the present application;

[0087] FIG2 shows a schematic diagram of another application scenario 200 of an embodiment of the present application;

[0088] FIG3 shows a schematic diagram of another application scenario 300 of an embodiment of the present application;

[0089] FIG4 shows a schematic diagram of another application scenario 400 of an embodiment of the present application;

[0090] FIG5 is a schematic structural diagram of a terminal device 500 provided in this application;

[0091] FIG6 is a schematic structural diagram of an access network device 600 provided in an embodiment of the present application;

[0092] FIG7 is a flowchart of a process 700 of an audio encoding method according to an embodiment of the present application;

[0093] FIG8a is a schematic diagram of obtaining side information according to an embodiment of the present application;

[0094] FIG8b is a schematic diagram of obtaining side information according to an embodiment of the present application;

[0095] FIG8c is a schematic diagram of obtaining side information according to an embodiment of the present application;

[0096] FIG8 d is a schematic diagram of obtaining side information according to an embodiment of the present application;

[0097] FIG9 a is a schematic diagram of the same parts of two sub-models according to an embodiment of the present application;

[0098] FIG9 b is a schematic diagram of the same parts of two sub-models according to an embodiment of the present application;

[0099] FIG9 c is a schematic diagram of the same parts of two sub-models according to an embodiment of the present application;

[0100] FIG10 a is a schematic diagram of a series connection of sub-models according to an embodiment of the present application;

[0101] FIG10 b is a schematic diagram of the series connection of sub-models according to an embodiment of the present application;

[0102] FIG11a is a schematic diagram of the serial connection of sub-rules in an embodiment of the present application;

[0103] FIG11b is a schematic diagram of the serial connection of sub-rules in an embodiment of the present application;

[0104] FIG12 is a schematic diagram of joint training according to an embodiment of the present application;

[0105] FIG13 is a flowchart of a process 1300 of an audio decoding method provided in an embodiment of the present application;

[0106] FIG14 a is a schematic diagram of a non-blind BWE according to an embodiment of the present application;

[0107] FIG14 b is a schematic diagram of a non-blind BWE according to an embodiment of the present application;

[0108] FIG14c is a schematic diagram of a non-blind BWE according to an embodiment of the present application;

[0109] FIG14 d is a schematic diagram of a non-blind BWE according to an embodiment of the present application;

[0110] FIG14e is a schematic diagram of a non-blind BWE according to an embodiment of the present application;

[0111] FIG14f is a schematic diagram of a non-blind BWE according to an embodiment of the present application;

[0112] FIG15a is a schematic diagram of obtaining a reconstructed frame according to an embodiment of the present application;

[0113] FIG15b is a schematic diagram of obtaining a reconstructed frame according to an embodiment of the present application;

[0114] FIG16 is a schematic diagram of a coding and decoding framework of a non-blind BWE including AI according to an embodiment of the present application;

[0115] FIG17 is a schematic diagram of a coding and decoding framework of a non-blind BWE including AI according to an embodiment of the present application;

[0116] FIG18 is a schematic diagram of a coding and decoding framework of a non-blind BWE including AI according to an embodiment of the present application;

[0117] FIG19 is a flow chart of the side information extraction step of an embodiment of the present application;

[0118] FIG20 is a flow chart of a non-blind BWE process according to an embodiment of the present application;

[0119] Figures 21 and 22 are schematic diagrams comparing the audio spectra obtained by the three BWE methods;

[0120] FIG23 is a schematic diagram of a coding and decoding framework of a non-blind BWE including AI according to an embodiment of the present application;

[0121] FIG24 is a flow chart of the side information extraction step of an embodiment of the present application;

[0122] FIG25 is a flow chart of a non-blind BWE process according to an embodiment of the present application;

[0123] FIG26 is a schematic structural diagram of an encoding device 2600 according to an embodiment of the present application;

[0124] FIG27 is a schematic structural diagram of a decoding device 2700 according to an embodiment of the present application. DETAILED DESCRIPTION

[0125] To make the objectives, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments of this application, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this application.

[0126] The terms "first," "second," and the like in the description, embodiments, claims, and drawings of this application are used solely for descriptive purposes and are not to be construed as indicating or implying relative importance or order. Furthermore, the terms "including," "having," and any variations thereof are intended to cover non-exclusive inclusions, such as, for example, inclusion of a series of steps or units. A method, system, product, or apparatus is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0127] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0128] Explanation of relevant terms involved in this application:

[0129] Audio frame: Audio data is streamed. In practical applications, to facilitate audio processing and transmission, a frame of audio is usually composed of a certain amount of audio data within a certain time period. This time period is called the "sampling time" and can be determined based on the requirements of the codec and specific application. For example, the time period can be 2.5ms to 60ms, where ms stands for milliseconds.

[0130] Audio signals: Audio signals are carriers of information, including the regular frequency and amplitude variations of sound waves that carry speech, music, and sound effects. Audio is a continuously varying analog signal that can be represented by a continuous curve, called a sound wave. Audio signals are digital signals generated through analog-to-digital conversion or computer generation. Sound waves have three key parameters: frequency, amplitude, and phase, which determine the characteristics of an audio signal.

[0131] In addition, since the embodiments of the present application involve the application of artificial intelligence (AI), for ease of understanding, some relevant nouns or terms used in the embodiments of the present application are explained below.

[0132] (1) Neural Network

[0133] A neural network (NN) is a machine learning model. A neural network can be composed of neural units. A neural unit can refer to an operation unit with xs and intercept 1 as input. The output of the operation unit can be:

[0134] Where, s = 1, 2, ... n, n is a natural number greater than 1, W s is x s The weight of the neural unit, b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolutional layer. The activation function can be a sigmoid function. A neural network is a network formed by connecting many of the above-mentioned single neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be an area composed of several neural units.

[0135] (2) Deep Neural Networks

[0136] A deep neural network (DNN), also known as a multi-layer neural network, can be understood as a neural network with many hidden layers. The "many" here does not have a specific metric. Based on the position of different layers in a DNN, the neural network inside the DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are all hidden layers. The layers are fully connected, that is, any neuron in the i-th layer must be connected to any neuron in the i+1-th layer. Although DNN looks complicated, the work of each layer is actually not complicated. Simply put, it is the following linear relationship expression: in, is the input vector, is the output vector, is the offset vector, W is the weight matrix (also called coefficient), and α() is the activation function. Each layer is just an input vector After such a simple operation, the output vector Since there are many DNN layers, the coefficient W and the offset vector The definition of these parameters in DNN is as follows: Take the coefficient W as an example: Assume that in a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as The superscript 3 represents the layer number of the coefficient W, while the subscript corresponds to the output of the third layer index 2 and the input of the second layer index 4. In summary, the coefficient from the kth neuron in the L-1th layer to the jth neuron in the Lth layer is defined as It's important to note that the input layer has no W parameter. In deep neural networks, more hidden layers allow the network to better capture complex real-world situations. Theoretically, a model with more parameters has higher complexity and greater "capacity," meaning it can handle more complex learning tasks. Training a deep neural network is essentially the process of learning the weight matrix, with the ultimate goal of obtaining the weight matrices for all layers of a trained deep neural network (a weight matrix formed by the vectors W across many layers).

[0137] (3) Convolutional Neural Network

[0138] A convolutional neural network (CNN) is a deep neural network with a convolutional structure and a deep learning architecture. A deep learning architecture involves multiple levels of learning at different levels of abstraction using machine learning algorithms. As a deep learning architecture, a CNN is a feed-forward artificial neural network in which individual neurons respond to input images. A CNN consists of a feature extractor consisting of convolutional and pooling layers. The feature extractor can be thought of as a filter, and the convolution process can be thought of as convolving an input image or feature map with a trainable filter.

[0139] A convolutional layer is a layer of neurons in a convolutional neural network that performs convolution on the input signal. A convolutional layer can include multiple convolution operators, also known as kernels. In image processing, a convolution operator acts as a filter that extracts specific information from the input image matrix. A convolution operator is essentially a weight matrix, which is usually predefined. During the convolution operation, the weight matrix is ​​typically applied horizontally to the input image, pixel by pixel (or two pixels by two pixels, depending on the stride), to extract specific features from the image. The size of the weight matrix should be proportional to the image size. It is important to note that the depth dimension of the weight matrix is ​​the same as the depth dimension of the input image; during the convolution operation, the weight matrix extends across the entire depth of the input image. Therefore, convolution with a single weight matrix produces a convolved output with a single depth dimension. However, in most cases, multiple weight matrices of the same size (row × column) are applied instead. This is known as multiple homogeneous matrices. The outputs of each weight matrix are stacked to form the depth dimension of the convolutional image. The dimension here can be understood as being determined by the "multiple" mentioned above. Different weight matrices can be used to extract different features from the image. For example, one weight matrix is ​​used to extract image edge information, another weight matrix is ​​used to extract specific colors in the image, and yet another weight matrix is ​​used to blur unwanted noise in the image. The multiple weight matrices have the same size (rows × columns), and the feature maps extracted by these multiple weight matrices of the same size are also the same size. The extracted feature maps of the same size are then merged to form the output of the convolution operation. In practical applications, the weight values ​​in these weight matrices require extensive training. The weight matrices formed by the trained weight values ​​can be used to extract information from the input image, allowing the convolutional neural network to make accurate predictions. When a convolutional neural network has multiple convolutional layers, the initial convolutional layers often extract more general features, which can also be called low-level features. As the depth of the convolutional neural network increases, the features extracted by subsequent convolutional layers become increasingly complex, such as high-level semantic features. Features with higher semantics are more applicable to the problem being solved.

[0140] Because it's often necessary to reduce the number of trainable parameters, pooling layers are often periodically introduced after convolutional layers. This can be done in a single convolutional layer followed by a pooling layer, or in a multi-layered system followed by one or more pooling layers. In image processing, the sole purpose of a pooling layer is to reduce the spatial size of an image. Pooling layers can include average pooling and / or max pooling operators, which are used to downsample the input image to produce a smaller image. The average pooling operator calculates the average value of pixel values ​​within a specific range, producing the average pooling result. The max pooling operator takes the pixel with the largest value within a specific range as the max pooling result. Furthermore, just as the size of the weight matrix used in a convolutional layer should be related to the image size, the operators in a pooling layer should also be related to the image size. The output image size after processing by a pooling layer can be smaller than the size of the image input to the pooling layer. Each pixel in the output image represents the average or maximum value of the corresponding subregion of the image input to the pooling layer.

[0141] After being processed by the convolution layer / pooling layer, the convolutional neural network is still not sufficient to output the required output information. Because as mentioned above, the convolution layer / pooling layer only extracts features and reduces the parameters brought by the input image. However, in order to generate the final output information (the required class information or other related information), the convolutional neural network needs to use the neural network layer to generate one or a group of outputs of the required number of classes. Therefore, the neural network layer may include multiple hidden layers, and the parameters contained in the multiple hidden layers can be pre-trained based on relevant training data of a specific task type. For example, the task type may include image recognition, image classification, image super-resolution reconstruction, etc.

[0142] Optionally, after the multiple hidden layers in the neural network layer, an output layer of the entire convolutional neural network is also included. The output layer has a loss function similar to the classification cross entropy, which is specifically used to calculate the prediction error. Once the forward propagation of the entire convolutional neural network is completed, the backpropagation will begin to update the weight values ​​and biases of the aforementioned layers to reduce the loss of the convolutional neural network and the error between the result output by the convolutional neural network through the output layer and the ideal result.

[0143] (4) Recurrent Neural Network

[0144] Recurrent neural networks (RNNs) are designed to process sequential data. In traditional neural network models, layers are fully connected, from the input layer to the hidden layer to the output layer, while nodes within each layer are disconnected. While these conventional neural networks have solved many difficult problems, they are still inadequate for many others. For example, to predict the next word in a sentence, you generally need to use the previous words, as the previous and next words in a sentence are not independent. RNNs are called recurrent neural networks because the current output of a sequence is dependent on the previous output. Specifically, the network memorizes previous information and applies it to the calculation of the current output. This means that nodes within the hidden layer are no longer disconnected but connected, and the input to a hidden layer includes not only the output of the input layer but also the output of the previous hidden layer. In theory, RNNs can process sequence data of any length. Training an RNN is similar to training a traditional CNN or DNN. This approach also uses the backpropagation algorithm, but with one key difference: if the RNN is expanded, its parameters, such as W, are shared; this is not the case with traditional neural networks, as in the example above. Furthermore, when using gradient descent, the output of each step depends not only on the state of the network at the current step but also on the state of the network at several previous steps. This learning algorithm is called backpropagation through time (BPTT).

[0145] Given the existence of convolutional neural networks, why do we still need recurrent neural networks? The reason is simple. Convolutional neural networks assume that elements are independent of each other, and that inputs and outputs are also independent, such as cats and dogs. However, in the real world, many elements are interconnected, such as the changes in stock prices over time. Or, for example, someone says, "I love traveling, and my favorite place is Yunnan. I must visit it someday." Humans should know to fill in the blank with "Yunnan." This is because humans make inferences based on context, but how can machines do this? RNNs were invented. RNNs are designed to give machines the ability to remember, like humans. Therefore, the output of an RNN depends on both the current input and historical memory.

[0146] (5) Loss function

[0147] During the training of a deep neural network, because we want the output of the deep neural network to be as close as possible to the desired predicted value, we can compare the current network's predicted value with the desired target value and then update the weight vector of each layer of the neural network based on the difference between the two. (Of course, before the first update, there is usually an initialization process, which pre-configures the parameters for each layer in the deep neural network.) For example, if the network's predicted value is too high, the weight vector is adjusted to make it predict a lower value. This adjustment is continued until the deep neural network can predict the desired target value or a value very close to the desired target value. Therefore, it is necessary to predefine "how to compare the difference between the predicted value and the target value." This is the loss function (or objective function), which is an important equation used to measure the difference between the predicted value and the target value. For example, the loss function output value (loss) indicates a greater difference, so training a deep neural network becomes a process of minimizing this loss.

[0148] (6) Backpropagation algorithm

[0149] Convolutional neural networks can use the back propagation (BP) algorithm to correct the size of the parameters in the initial super-resolution model during training, reducing the reconstruction error loss of the super-resolution model. Specifically, the forward propagation of the input signal to the output generates an error loss. This error loss information is then backpropagated to update the parameters of the initial super-resolution model, thereby converging the error loss. The BP algorithm is a backward propagation movement dominated by the error loss, aiming to obtain the optimal super-resolution model parameters, such as the weight matrix.

[0150] (7) Generative Adversarial Networks

[0151] Generative adversarial networks (GANs) are a type of deep learning model. They consist of at least two modules: a generative model and a discriminative model. These two modules learn from each other through interaction to produce better outputs. Both the generative and discriminative models can be neural networks, specifically deep neural networks or convolutional neural networks. The basic principle of a GAN is as follows: For example, consider a GAN that generates images. Suppose there are two networks, G (Generator) and D (Discriminator). G is the image generator network, which receives random noise z and generates an image from it, denoted as G(z). D is the discriminator network, which determines whether an image is "real." Its input parameter is x, representing an image. Its output, D(x), represents the probability that x is real. A value of 1 indicates a 100% probability of authenticity, while a value of 0 indicates a high probability of non-authenticity. During the training of this generative adversarial network, the goal of the generative network G is to generate realistic images as much as possible to deceive the discriminative network D, while the goal of the discriminative network D is to distinguish the images generated by G from real images as much as possible. This creates a dynamic "game" between G and D, which is the "game" in "generative adversarial network." Ultimately, under ideal conditions, G can generate images G(z) that are sufficiently realistic, while D has difficulty determining whether the images generated by G are real, i.e., D(G(z)) = 0.5. This results in an excellent generative model G that can be used to generate images.

[0152] Based on this, the technical solutions of the embodiments of the present application are described below in conjunction with the accompanying drawings.

[0153] The embodiments of the present application can be applied to audio codecs of terminal devices, wireless access network devices, core network devices, etc.

[0154] Figure 1 shows a schematic diagram of an application scenario 100 of an embodiment of the present application. Figure 1 shows a schematic diagram of the system architecture of an embodiment of the present application applied on the terminal device side. As shown in Figure 1, it includes a first terminal device 110, a second terminal device 120, a wireless or wired network communication device 130 and a wireless or wired network communication device 140. Among them, the first terminal device 110 and the second terminal device 120 can be a sending end device or a receiving end device. As an example, the first terminal device 110 is a sending end device and the second terminal device 120 is a receiving end device.

[0155] In audio communication, the audio acquisition module in the first terminal device 110 is used to collect audio, the audio encoder performs audio encoding on the collected audio signal, the channel encoding module performs channel encoding to obtain a code stream, and then transmits the signal in a digital channel through the wireless or wired network communication device 130 on the sending side.

[0156] The wireless or wired network communication device 140 on the receiving side obtains the signal sent by the first terminal device 110 through a digital channel and transmits the signal to the second terminal device 120. The second terminal device 120 performs channel decoding in the channel decoding module according to the received signal, and then decodes the audio signal through the audio decoder, and then performs audio playback in the audio playback module according to the decoded audio signal.

[0157] It should be understood that when the second terminal device 120 is a sending device and the first terminal device 110 is a receiving device, the first terminal device 110 can be understood as a sending device and the second terminal device 120 is a receiving device, and no further details will be given here.

[0158] It should be understood that the wireless or wired network communication device 130 and the wireless or wired network communication device 140 may also be core network devices.

[0159] FIG2 shows a schematic diagram of another application scenario 200 of an embodiment of the present application. FIG2 shows a schematic diagram of the system architecture of the transcoding application of an embodiment of the present application in a wireless access network device or a core network device. As shown in FIG2 , the wireless access network device or the core network device includes a channel decoding module, other audio decoders, an audio encoder, and a channel coding module. Among them, corresponding audio encoding and decoding processing is required during transcoding. The wireless access network device or the core network device performs channel decoding on the received signal in the channel decoding module, and then decodes the audio code stream through other audio decoders to obtain an audio code stream. The audio encoder re-encodes the audio code stream, and then performs channel coding to transmit the audio signal.

[0160] FIG3 shows a schematic diagram of another application scenario 300 of an embodiment of the present application. FIG3 shows a schematic diagram of the system architecture of the transcoding application of the embodiment of the present application in a wireless access network device or a core network device. As shown in FIG3, the wireless access network device or the core network device includes a channel decoding module, an audio decoder, other audio encoders, and a channel coding module. Among them, corresponding audio encoding and decoding processing is required during transcoding. The wireless access network device or the core network device performs channel decoding on the received signal in the channel decoding module, and then decodes the audio code stream through the audio decoder to obtain an audio code stream. Other audio encoders re-encode the audio code stream, and then perform channel coding to transmit the audio signal.

[0161] The embodiment of the present application can also be applied to the audio encoding module (Audio Encoding) and audio decoding module (Audio Decoding) in the virtual reality (VR) streaming service. FIG4 shows a schematic diagram of another application scenario 400 of the embodiment of the present application, as shown in the dashed box in FIG4.

[0162] The end-to-end audio signal processing process is as follows: After the audio and video signals are processed by the acquisition module (Acquisition), the transmitter separates them into audio and video signals. The audio signal undergoes audio preprocessing, which involves filtering out low-frequency components, typically at a cutoff of 20Hz or 50Hz, extracting directional information from the signal, and then performing audio encoding. The video signal undergoes visual stitching, projection, and mapping, followed by video and image encoding. The audio, video, and image streams are packaged (file / segment encapsulation) and delivered to the decoder.

[0163] The decoding end first performs file / segment decapsulation, then performs audio decoding, video decoding, and image decoding. The decoded audio signal is processed by audio binaural rendering. The rendered signal is mapped to the listener's headphones (which can be independent headphones or headphones on glasses devices such as HTC VIVE). The decoded video signal and image signal are processed by video binaural rendering, and the rendered signal is mapped to the display.

[0164] The terminal device in the embodiments of the present application may also be referred to as: user equipment (UE), mobile station (MS), mobile terminal (MT), access terminal, user unit, user station, mobile station, mobile station, remote station, remote terminal, mobile device, user terminal, terminal, wireless communication equipment, user agent or user device, etc.

[0165] Terminal devices can be either wireless or wired. Wireless terminals can be devices that provide voice and / or other service data connectivity to users, handheld devices with wireless connectivity, or other processing devices connected to a wireless modem. Wireless terminals can communicate with one or more core networks via a Radio Access Network (RAN). Wireless terminals can be mobile terminals, such as mobile phones (also known as "cellular" phones) and computers with mobile terminals. For example, they can be portable, pocket-sized, handheld, built-in computers, or in-vehicle mobile devices that exchange voice and / or data with the RAN. Examples include Personal Communication Service (PCS) phones, cordless phones, Session Initiation Protocol (SIP) phones, Wireless Local Loop (WLL) stations, and Personal Digital Assistants (PDAs).A wireless terminal may also be referred to as a system, a subscriber unit, a subscriber station, a mobile station, a mobile station, a remote station, a remote terminal, an access terminal, a user terminal, a user agent, a user device or user equipment, a mobile internet device (MID), a wearable device, a virtual reality (VR) device, an augmented reality (AR) device, a wireless terminal in industrial control, a wireless terminal in self-driving, a wireless terminal in remote medical surgery, a wireless terminal in a smart grid, a wireless terminal in transportation safety, a wireless terminal in a smart city, a wireless terminal in a smart home, an in-vehicle device, a wearable device, a terminal device in a 5G network, or a future-evolved public land mobile communication network. The terminal equipment in the network (PLMN), etc., is not limited to this in the embodiments of the present application.

[0166] As an example and not a limitation, in the embodiments of the present application, wearable devices may also be referred to as wearable smart devices, which are a general term for wearable devices that are intelligently designed and developed using wearable technology for daily wear, such as glasses, gloves, watches, clothing, and shoes. A wearable device is a portable device that is worn directly on the body or integrated into the user's clothes or accessories. Wearable devices are not just hardware devices, but also achieve powerful functions through software support, data interaction, and cloud interaction. Broadly speaking, wearable smart devices include those that are fully functional, large in size, and can achieve complete or partial functions without relying on smartphones, such as smart watches or smart glasses, as well as those that only focus on a certain type of application function and need to be used in conjunction with other devices such as smartphones, such as various smart bracelets and smart jewelry for vital sign monitoring.

[0167] In addition, in the embodiment of the present application, the terminal device can also be a terminal device in the Internet of Things (IoT) system. IoT is an important part of the future development of information technology. Its main technical feature is to connect objects to the network through communication technology, thereby realizing an intelligent network of human-machine interconnection and object-to-object interconnection.

[0168] The various terminal devices introduced above, if located on a vehicle (eg, placed in or installed in a vehicle), can be considered as vehicle-mounted terminal devices, which are also called on-board units (OBUs).

[0169] In the embodiment of the present application, the terminal device may further include a relay. Alternatively, it can be understood that anything that can communicate data with a base station can be considered a terminal device.

[0170] 5 is a schematic diagram of the structure of a terminal device 500 provided by the present application. The terminal device 500 can execute the actions executed by the terminal device in the above embodiments.

[0171] For ease of explanation, only the main components of the terminal device are shown in Figure 5. As shown in Figure 5, the terminal device 500 includes a processor, a memory, a control circuit, an antenna, and an input and output device.

[0172] The processor is primarily used to process communication protocols and communication data, control the entire terminal device, execute software programs, and process software program data, such as supporting the terminal device in executing the actions described in the embodiment of the method for indicating a transmission precoding matrix. The memory is primarily used to store software programs and data, such as the codebook described in the above embodiment. The control circuit is primarily used to convert baseband signals into radio frequency signals and process radio frequency signals. The control circuit and antenna together are also called a transceiver, which is primarily used to transmit and receive radio frequency signals in the form of electromagnetic waves. Input and output devices, such as touch screens, displays, and keyboards, are primarily used to receive user input and output data to the user.

[0173] When the terminal device is powered on, the processor reads the software program stored in the storage unit, interprets and executes the program's instructions, and processes the program's data. When data needs to be transmitted wirelessly, the processor performs baseband processing on the data to be transmitted and outputs the baseband signal to the RF circuit. The RF circuit then performs RF processing on the baseband signal and transmits it via the antenna as electromagnetic waves. When data is sent to the terminal device, the RF circuit receives the RF signal via the antenna, converts it into a baseband signal, and outputs the baseband signal to the processor, which converts the baseband signal into data and processes it.

[0174] Those skilled in the art will appreciate that, for ease of explanation, FIG5 shows only one memory and processor. In an actual terminal device, there may be multiple processors and memories. The memory may also be referred to as a storage medium or storage device, etc., which is not limited in the present embodiment.

[0175] For example, a processor may include a baseband processor and a central processing unit (CPU). The baseband processor is primarily responsible for processing communication protocols and communication data, while the CPU is primarily responsible for controlling the entire terminal device, executing software programs, and processing data from software programs. The processor in Figure 5 integrates the functions of both the baseband processor and the CPU. Those skilled in the art will appreciate that the baseband processor and the CPU may also be independent processors interconnected via a bus or other technology. Those skilled in the art will appreciate that a terminal device may include multiple baseband processors to accommodate different network standards, multiple CPUs to enhance its processing capabilities, and that the various components of the terminal device may be connected via various buses. The baseband processor may also be referred to as a baseband processing circuit or a baseband processing chip. The CPU may also be referred to as a central processing circuit or a central processing chip. The functionality for processing communication protocols and communication data may be built into the processor or stored as a software program in a storage unit, with the processor executing the software program to implement the baseband processing functionality.

[0176] For example, in the embodiment of the present application, the antenna and control circuit with transceiver functions can be regarded as the transceiver unit 510 of the terminal device 500, and the processor with processing function can be regarded as the processing unit 520 of the terminal device 500. The processing unit 520 can also realize the function of the acquisition unit. As shown in Figure 5, the terminal device 500 includes a transceiver unit 510 and a processing unit 520. The transceiver unit can also be referred to as a transceiver, a transceiver, a transceiver device, etc. Optionally, the device for realizing the receiving function in the transceiver unit 510 can be regarded as a receiving unit, and the device for realizing the transmitting function in the transceiver unit 510 can be regarded as a transmitting unit, that is, the transceiver unit includes a receiving unit and a transmitting unit. For example, the receiving unit can also be referred to as a receiver, a receiver, a receiving circuit, etc., and the transmitting unit can be referred to as a transmitter, a transmitter, or a transmitting circuit, etc.

[0177] The access network device in the embodiments of the present application may be a device for communicating with a terminal device, and may be a base station, an access point, or a network device, or may refer to a device in an access network that communicates with a wireless terminal through one or more sectors on an air interface. The network device may be used to convert received air frames to and from IP packets, acting as a router between the wireless terminal and the rest of the access network, which may include an Internet Protocol (IP) network. The network device may also coordinate attribute management of the air interface. For example, the access network device can be a base station (Base Transceiver Station, BTS) in Global System of Mobile communication (GSM) or Code Division Multiple Access (CDMA), a base station (NodeB, NB) in Wideband Code Division Multiple Access (WCDMA), an evolved NodeB (eNB or eNodeB) in the LTE system, or a wireless controller in a cloud radio access network (CRAN) scenario, or the access device can be a relay station, an access point, a vehicle-mounted device, a wearable device, an access device in a 5G network, or a network device in a future evolved PLMN network, etc., an access point (AP) in a WLAN, or a gNB in ​​a new radio system (NR). The embodiments of this application are not limited. It should be noted that for the 5G system, there may be one or more transmission reception points (Transmission Reception Point, TRP) under one base station, and all TRPs belong to the same cell, wherein each TRP and terminal can use the measurement reporting method described in the embodiment of the present application. In another scenario, the network device can also be divided into a control unit (Control Unit, CU) and a data unit (Data Unit, DU). Under one CU, there may be multiple DUs, wherein each DU and terminal can use the measurement reporting method described in the embodiment of the present application. The difference between the CU-DU separation scenario and the multi-TRP scenario is that the TRP is just a radio frequency unit or an antenna device, and the protocol stack function can be implemented in the DU, for example, the physical layer function can be implemented in the DU.

[0178] In addition, in the embodiments of the present application, an access network device is a device in an access network (RAN), or in other words, a RAN node that connects a terminal device to a wireless network. For example, as an example and not a limitation, examples of access network devices include: gNB, transmission reception point (TRP), evolved Node B (eNB), radio network controller (RNC), Node B (NB), base station controller (BSC), base transceiver station (BTS), home base station (e.g., home evolved Node B, or home Node B, HNB), base band unit (BBU), or wireless fidelity (Wifi) access point (AP).

[0179] The access network equipment provides services for the cell, and the terminal device communicates with the access network equipment through the transmission resources used by the cell (for example, frequency domain resources, or spectrum resources). The cell can be the cell corresponding to the access network equipment (for example, a base station). The cell can belong to a macro base station or a base station corresponding to a small cell. The small cells here can include: metro cells, micro cells, pico cells, femto cells, etc. These small cells have the characteristics of small coverage and low transmission power, and are suitable for providing high-speed data transmission services.

[0180] For example, Figure 6 is a structural diagram of an access network device 600 provided in an embodiment of the present application, which can be used to implement the functions of the wireless access network device in the above embodiment. The access network device 600 includes one or more radio frequency units, such as a remote radio unit (RRU) 66 and one or more baseband units (BBU) (also called digital units, digital units, DU) 620. The RRU 66 can be called a transceiver unit, a transceiver, a transceiver circuit, or a transceiver, etc., and may include at least one antenna 611 and a radio frequency unit 612. The RRU 66 part is mainly used for receiving and transmitting radio frequency signals and converting radio frequency signals into baseband signals, for example, for sending the signaling messages described in the above embodiment to the terminal device. The BBU 620 part is mainly used for baseband processing, controlling the base station, etc. The RRU 66 and the BBU 620 can be physically set together or physically separated, that is, a distributed base station.

[0181] The BBU 620 is the control center of the base station, which can also be called a processing unit. It is mainly used to perform baseband processing functions such as channel coding, multiplexing, modulation, spread spectrum, etc. For example, the BBU (processing unit) 620 can be used to control the access network device to execute the operation process of the access network device in the above method embodiment.

[0182] In one example, the BBU620 may be composed of one or more single boards, and multiple single boards may jointly support a wireless access network of a single access standard (such as an LTE system, or a 5G system, or a future wireless access network system, etc.), or may respectively support wireless access networks of different access standards. The BBU620 also includes a memory 621 and a processor 622. The memory 621 is used to store necessary instructions and data. For example, the memory 621 stores the codebook in the above embodiment, etc. The processor 622 is used to control the base station to perform necessary actions, such as controlling the base station to execute the operation process of the network device in the above method embodiment. The memory 621 and the processor 622 can serve one or more single boards. That is, a memory and a processor can be set separately on each single board. Alternatively, multiple single boards may share the same memory and processor. In addition, necessary circuits may be set on each single board.

[0183] In one possible implementation, with the development of system-on-chip (SoC) technology, all or part of the functions of parts 620 and 66 can be implemented using SoC technology. For example, they can be implemented using a base station function chip that integrates a processor, memory, antenna interface, and other components. Programs for base station-related functions are stored in the memory, and the processor executes the programs to implement the base station-related functions. Optionally, the base station function chip can also read memory external to the chip to implement the base station-related functions.

[0184] It should be understood that the structure of the access network device illustrated in FIG6 is only one possible form and should not constitute any limitation to the embodiments of the present application. The present application does not exclude the possibility of other forms of base station structures that may appear in the future.

[0185] It should be understood that in the embodiments of the present application, the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0186] It should also be understood that the memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0187] The core network equipment can be a core network network element, such as an Access and Mobility Management Function (AMF) entity, a Session Management Function (SMF) entity, a User Plane Function (UPF) entity, a Policy Control Function (PCF) entity, etc. The AMF entity provides mobility management functions in the core network and is mainly responsible for access and mobility control, including registration management (RM) and connection management (CM), access authentication and access authorization, reachability management, and mobility management. The SMF entity is the session management function in the core network. In addition to performing mobility management on terminal devices, the AMF entity is also responsible for forwarding session management-related messages between terminal devices and SMF entities. The PCF entity is the policy management function in the core network and is responsible for formulating policies related to mobility management, session management, billing, etc. for terminal devices. The UPF entity is the user plane function in the core network. It transmits data with the external data network through the interface and with the access network equipment through the interface. It mainly provides user plane support, including the connection point between PDU session and data network, packet routing and forwarding, packet detection and user plane policy execution, QoS processing for the user plane, downlink packet caching and triggering of downlink data notification, etc.

[0188] It should be understood that the functional units of the above-mentioned core network can work independently or be combined to implement certain control functions. For example, AMF, SMF and PCF can be combined together as a management device to complete access control and mobility management functions such as access authentication, security encryption, and location registration of terminal devices, as well as session management functions such as recording, releasing and changing user-side transmission paths, as well as functions for analyzing some slice-related data (such as congestion) and terminal-related data. UPF, as a gateway device, mainly completes functions such as routing and forwarding of user-side data, such as: responsible for filtering data packets of terminal devices, data transmission / forwarding, rate control, and generating billing information.

[0189] The technical solutions of the embodiments of the present application can be applied to various communication systems, such as: global system for mobile communications (GSM) system, code division multiple access (CDMA) system, wideband code division multiple access (WCDMA) system, general packet radio service (GPRS), long term evolution (LTE) system, LTE frequency division duplex (FDD) system, LTE time division duplex (TDD), universal mobile telecommunication system (UMTS), world-wide interoperability for microwave access (WiMAX) communication system, fifth generation (5G) system or new radio (NR). In addition, it can also be applicable to subsequent evolution systems, such as the sixth generation (6G) communication system and even the more advanced seventh generation (7G) communication system.

[0190] In the core encoder, correlation between signals in different frequency bands can be exploited to reduce the encoding bit rate. The basic principle of this approach is to use low-frequency signals to generate high-frequency signals through a method called bandwidth extension (BWE). Whether it's the latest 3GPP Enhanced Voice Services (EVS) audio codec, or the Moving Picture Experts Group High-Efficiency Advanced Audio Coding (MPEG HE-AAC) and Unified Speech and Audio Coding (USAC) audio codecs, all exploit the correlation between signals in different frequency bands and use bandwidth extension technology to encode high-frequency signals. This allows them to be encoded with fewer bits, thereby reducing the encoder's encoding bit rate. However, in real audio signals, the high-frequency spectrum often contains spectra that are dissimilar or weakly correlated with the low-frequency spectrum, resulting in distortion in the reconstruction of the high-frequency signals.

[0191] In order to solve the above technical problems, the present application proposes an audio encoding and decoding method and device.

[0192] It should be noted that the following method embodiments can be applied in the application scenarios shown in Figures 1 to 4, and of course can also be applied in other scenarios, and the embodiments of the present application are not limited here. It should also be understood that in the embodiments of the present application, the execution subject of the method can be a terminal device, an access network device, and a core network device. As an example and not a limitation, the execution subject of the method can also be a chip, a chip system, or a processor applied to a terminal device, an access network device, and a core network device. Among them, the terminal device, the access network device, and the core network device have encoding and decoding functions and can also be called encoding and decoding devices.

[0193] FIG7 is a flow chart of process 700 of the audio encoding method provided by an embodiment of the present application. Process 700 is described as a series of steps or operations. It should be understood that process 700 can be performed in various orders and / or occur simultaneously, and is not limited to the execution order shown in FIG7. Process 700 may include:

[0194] Step 701: Acquire a first code stream, where the first code stream is obtained based on a first audio signal.

[0195] The original audio is sampled to obtain an audio frame sequence, which includes multiple audio frames, which can be compressed frame by frame at the encoding end. For the convenience of description, in the embodiment of the present application, the audio frame currently being processed is referred to as the current frame.

[0196] The characteristics of the audio signal contained in the current frame can be characterized by frequency, amplitude and phase. The embodiment of the present application extracts the first audio signal and the second audio signal from the current frame according to the frequency, wherein the first audio signal is an audio signal in the current frame whose frequency is lower than a preset threshold (in the embodiment of the present application, the audio signal that meets this condition can also be referred to as a low-frequency signal or a low-frequency component), and the second audio signal includes an audio signal in the current frame whose frequency is higher than or equal to the preset threshold (in the embodiment of the present application, the audio signal that meets this condition can also be referred to as a high-frequency signal or a high-frequency component). The aforementioned preset threshold can be 16kHz, for example, and the embodiment of the present application does not make specific restrictions on this. It should be noted that the foregoing classifies the audio signal in the current frame whose frequency is equal to the preset threshold into the second audio signal. Optionally, the embodiment of the present application can also classify the audio signal in the current frame whose frequency is equal to the preset threshold into the first audio signal, and there is no specific restriction on this.

[0197] In some possible implementations, a low-pass filter may be used to extract the first audio signal from the current frame.

[0198] In some possible implementations, a core encoder may be used to encode the first audio signal. The core encoder may be, for example, an AVS3 core encoder, which is not specifically limited in the embodiments of the present application.

[0199] Step 702: Obtain side information, where the side information is obtained based on the second audio signal.

[0200] In this embodiment of the present application, the second audio signal may exist in the following three situations:

[0201] 1) The second audio signal includes only high-frequency components. In this case, the second audio signal is equivalent to the high-frequency components of the current frame. Correspondingly, the side information may only include characteristic information of the high-frequency components.

[0202] 2) The second audio signal includes not only the high-frequency component but also a portion of the low-frequency component. That is, the second audio signal and the first audio signal share some of the same audio signal. In this case, the second audio signal is equivalent to the high-frequency component plus some of the low-frequency component of the current frame. Accordingly, the side information may include not only the characteristic information of the high-frequency component but also the correlation information between the high-frequency component and the aforementioned low-frequency component.

[0203] 3) The second audio signal includes all low-frequency components in addition to the high-frequency components. That is, the second audio signal includes the first audio signal. In this case, the second audio signal is equivalent to the current frame. Accordingly, the side information may include not only the feature information of the high-frequency components but also the correlation information between the high-frequency and low-frequency components.

[0204] It can be seen from this that side information can only occupy a small number of transmission bits to guide the decoding end to implement non-blind BWE to reconstruct high-frequency components, especially signals with complex mapping relationships between high and low frequencies.

[0205] In some possible implementations, obtaining the side information may include inputting the second audio signal into a pre-trained second machine learning model to obtain the side information, where the second machine learning model includes at least one sub-model. Each of the at least one sub-models may focus on different characteristics. For example, constraints may be set for each sub-model, including input and output constraints, model structure, and processing rules, so that each sub-model can focus on extracting side information related to a corresponding characteristic.

[0206] In the embodiment of the present application, the second machine learning model can refer to the neural network, deep neural network, convolutional neural network, recurrent neural network, etc. mentioned above. The machine learning model can reflect more details of the high-frequency components in the current frame in the feature information. These details, such as the characteristics of the high-frequency components and the relationship between the high-frequency components and the low-frequency components, help reconstruct more high-frequency details.

[0207] Exemplarily, Figure 8a is a schematic diagram of obtaining side information in an embodiment of the present application. As shown in Figure 8a, obtaining side information is achieved through a second machine learning model, which includes a sub-model, namely the second machine learning model, that is, the sub-model, whose input is the second audio signal and output is side information.

[0208] In some possible implementations, the second machine learning model may include multiple sub-models, and the input of each of the multiple sub-models includes an audio signal in the current frame whose frequency is higher than or equal to a preset threshold, that is, the input of each of the multiple sub-models is part or all of the audio signal in the second audio signal.

[0209] Exemplarily, Figure 8b is a schematic diagram of obtaining side information in an embodiment of the present application. As shown in Figure 8b, obtaining side information is implemented through a second machine learning model, which includes multiple sub-models (n sub-models are taken as an example in Figure 8b, n>1), namely sub-model 1, sub-model 2,..., sub-model n, where the input of sub-model 1 is input 1 and the output is feature information 1, the input of sub-model 2 is input 2 and the output is feature information 2,..., the input of sub-model n is input n and the output is feature information n.

[0210] Input 1, input 2, ..., input n all come from the second audio signal, and their common point is that they all include the high-frequency components in the second audio signal. In addition, input 1, input 2, ..., input n may also include or exclude the low-frequency components in the second audio signal. When the low-frequency components in the second audio signal are included, input 1, input 2, ..., input n respectively include part or all of the low-frequency components in the second audio signal. For example, input 1 includes the high-frequency components in the second audio signal, input 2 includes the high-frequency components and part of the low-frequency components in the second audio signal, and input 3 includes the high-frequency components and all of the low-frequency components in the second audio signal. It should be noted that in the embodiment of the present application, input 1, input 2, ..., input n can be the same, or completely different, or partially the same, and there is no specific limitation on this.

[0211] In the embodiments of the present application, if the input of a sub-model contains only high-frequency components, the sub-model is primarily used to compress the high-frequency components into features, or to represent the high-frequency components through features. If the input of the sub-model contains low-frequency components in addition to high-frequency components, the sub-model is also used to extract the mapping relationship between high and low frequencies. Using multiple sub-models can obtain more diverse features in the extraction of side information. Therefore, this application does not specifically limit the number, type, function, etc. of sub-models.

[0212] The outputs of the n sub-models, including feature information 1, feature information 2, ..., feature information n, are merged to obtain side information, that is, the side information includes the feature information output by the n sub-models.

[0213] In some possible embodiments, the first sub-model and at least one other sub-model among the multiple sub-models include the same neurons, and the same neurons are located in the input stage, output stage or intermediate stage of the first sub-model, and the first sub-model is any one of the multiple sub-models.

[0214] The first sub-model may have the same neurons (also referred to as shared neurons) as one or more of the multiple sub-models included in the second machine learning model. The same neurons may have one or more layers, and the one or more layers of neurons may be located in the input stage, output stage or intermediate stage of the sub-model, wherein the input stage refers to the first layer of the input end of the sub-model and the layer or layers thereafter, the output stage refers to the last layer of the output end of the sub-model and the layer or layers before it, and the intermediate stage refers to the layer or layers excluding the first and last layers of the sub-model. It should be noted that the embodiments of the present application do not specifically limit the number, type, etc. of the aforementioned same neurons, nor do they specifically limit the positions of the input stage, output stage and intermediate stage.

[0215] Exemplarily, Figure 9a is a schematic diagram of the same parts of two sub-models of an embodiment of the present application. As shown in Figure 9a, taking two sub-models as an example, the two sub-models have the same neurons (i.e., the same parts) and each has different neurons (i.e., the different parts).

[0216] The same part is located at the input stage of sub-model 1 and sub-model 2, and the difference part 1 of sub-model 1 and the difference part 2 of sub-model 2 are respectively located after the aforementioned same part, and output feature information 1 and feature information 2 respectively.

[0217] Exemplarily, Figure 9b is a schematic diagram of the same parts of two sub-models of an embodiment of the present application. As shown in Figure 9b, taking two sub-models as an example, the two sub-models have the same neurons (i.e., the same parts) and each has different neurons (i.e., the different parts).

[0218] The same part is located at the output stage of sub-model 1 and sub-model 2, the difference part 1 of sub-model 1 and the difference part 2 of sub-model 2 are respectively located before the aforementioned same part, and the two sub-models output feature information 1 and feature information 2 respectively.

[0219] Exemplarily, Figure 9c is a schematic diagram of the same parts of two sub-models of an embodiment of the present application. As shown in Figure 9c, taking two sub-models as an example, the two sub-models have the same neurons (i.e., the same parts) and each has different neurons (i.e., the different parts).

[0220] The same parts are located in the input stage and output stage of sub-model 1 and sub-model 2, the difference part 1 of sub-model 1 and the difference part 2 of sub-model 2 are respectively located between the two same parts, and the two sub-models output feature information 1 and feature information 2 respectively.

[0221] It should be noted that the above two examples are illustrated using two sub-models as an example, but this does not limit the same parts between multiple sub-models. The embodiments of this application do not make specific limitations on the number of multiple sub-models with the same parts, the specific implementation methods of the same parts (including the position of the same parts, the layers included in the same parts, etc.).

[0222] In some possible implementations, the input of the first sub-model also includes the first input; when the second machine learning model includes multiple sub-models, the first input includes the output of at least one other sub-model among the multiple sub-models, or the first input includes the output of at least one sub-rule method; the first sub-model is any one of the multiple sub-models.

[0223] For example, FIG10a is a schematic diagram of a sub-model series connection in an embodiment of the present application. As shown in FIG10a , taking two sub-models as an example, the output of sub-model 2 can be used as an input to sub-model 1, so that sub-model 1 includes input 1 and the output of sub-model 2. In addition, the output of the sub-rule method can also be used as an input to sub-model 1, so that sub-model 1 includes input 1 and the output of the sub-rule method, and can even add the output of sub-model 2, without specific limitation. This can be a series connection of sub-models or sub-rule methods.

[0224] In some possible implementations, the input of the first sub-model also includes a second input, which is an audio signal obtained after redundant information processing of the first input; when the second machine learning model includes multiple sub-models, the first input includes the output of at least one other sub-model among the multiple sub-models, or the first input includes the output of at least one sub-rule method; the first sub-model is any one of the multiple sub-models.

[0225] For example, FIG10b is a schematic diagram of the sub-model series connection of an embodiment of the present application. As shown in FIG10b, taking two sub-models as an example, in the two sub-models, the output of sub-model 2 is first processed for redundant information and then can be used as an input of sub-model 1. In this way, sub-model 1 includes input 1 and the output of sub-model 2 after redundant information processing. In addition, the output of the sub-rule method can also be first processed for redundant information and then used as an input of sub-model 1. In this way, sub-model 1 includes input 1 and the output of the sub-rule method after redundant information processing, and can even add the output of sub-model 2 after redundant information processing, without specific limitation. This can be a series connection of sub-models and sub-models or sub-rule methods.

[0226] It should be noted that the above two examples are illustrated using two sub-models as an example, but this does not constitute a limitation on the series relationship between sub-models or between sub-models and sub-rule methods. In addition to its own input, the input of any sub-model may also include the output of one or more other sub-models or the output after redundant processing, and / or the output of one or more sub-rule methods or the output after redundant processing. The embodiment of the present application does not limit the number of series connections or the series connection method.

[0227] In some possible implementations, obtaining the side information may further include processing the second audio signal according to a preset rule to obtain the side information, where the preset rule includes at least one sub-rule.

[0228] In an embodiment of the present application, a method for extracting features according to a pre-set rule may include, for example, extracting a spectrum envelope and extracting features based on AVS3. The spectrum envelope represents the energy of each frequency band in the spectrum of the current frame, which is used to recover the energy of the corresponding frequency bands for the decoding side. The features extracted by AVS3 include the frequency band energy and the identification of the harmonic and noise characteristics of each frequency band, which are used to recover the BWE basis for the decoding side. Compared to machine learning methods, rule-based methods can retain information such as spectrum energy, thereby avoiding the impact on the auditory experience.

[0229] In an embodiment of the present application, based on the second machine learning model, a rule method can also be set to process the second audio signal according to the aforementioned rule method to obtain the corresponding output (feature information).

[0230] Exemplarily, Figure 8c is a schematic diagram of obtaining side information in an embodiment of the present application. As shown in Figure 8c, obtaining side information is achieved through a second machine learning model and a pre-set rule method. The second machine learning model includes a sub-model, that is, the second machine learning model, also known as the sub-model, whose input is input 1 and output is feature information 1. The rule method includes a sub-rule method, that is, the pre-set rule method, also known as the sub-rule method, whose input is input 2 and output is feature information 2.

[0231] Input 1 and input 2 both come from the second audio signal, and their common point is that they both include the high-frequency components in the second audio signal. In addition, input 1 and input 2 may also include or exclude the low-frequency components in the second audio signal, respectively. When the low-frequency components in the second audio signal are included, input 1 and input 2 respectively include part or all of the low-frequency components in the second audio signal. For example, input 1 includes the high-frequency components in the second audio signal, and input 2 includes the high-frequency components and part of the low-frequency components in the second audio signal. It should be noted that in the embodiment of the present application, input 1 and input 2 can be the same, or completely different, or partially the same and partially different, and there is no specific limitation on this.

[0232] The feature information 1 output by the second machine learning model and the feature information 2 output by the rule method are merged to obtain side information, that is, the side information includes the feature information output by the second machine learning model and the feature information output by the rule method.

[0233] In some possible implementations, the preset rule method may include multiple sub-rule methods, and the input of each of the multiple sub-rule methods includes an audio signal in the current frame whose frequency is higher than or equal to a preset threshold, that is, the input of each of the multiple sub-rule methods is part or all of the audio signal in the second audio signal.

[0234] Exemplarily, FIG8d is a schematic diagram of obtaining side information in an embodiment of the present application. As shown in FIG8d , obtaining side information is achieved through a second machine learning model and a pre-set rule method. The second machine learning model includes multiple sub-models ( FIG8d takes n sub-models as an example, n>1), namely sub-model 1, sub-model 2, ..., sub-model n, wherein the input of sub-model 1 is input 1 and the output is feature information 1, the input of sub-model 2 is input 2 and the output is feature information 2, ..., the input of sub-model n is input n and the output is feature information n. The rule method includes multiple sub-rule methods ( FIG8d takes m sub-rule methods as an example, m>1), namely, the input of sub-rule method 1 is input n+1 and the output is feature information n+1, the input of sub-rule method 2 is input n+2 and the output is feature information n+2, ..., the input of sub-rule method m is input n+m and the output is feature information n+m.

[0235] Input 1, input 2, ..., input n, input n+1, input n+2, ..., input n+m all come from the second audio signal, and their common feature is that they all include high-frequency components in the second audio signal. In addition, input 1, input 2, ..., input n, input n+1, input n+2, ..., input n+m may also include or exclude low-frequency components in the second audio signal. When including low-frequency components in the second audio signal, input 1, input 2, ..., input n, input n+1, input n+2, ..., input n+m may each include some or all of the low-frequency components in the second audio signal. For example, input 1 includes the high-frequency components in the second audio signal, input 2 includes the high-frequency components and some of the low-frequency components in the second audio signal, and input 3 includes the high-frequency components and all of the low-frequency components in the second audio signal. It should be noted that in the embodiments of the present application, input 1, input 2, ..., input n, input n+1, input n+2, ..., input n+m may be the same, completely different, or partially the same and partially different, and this is not specifically limited.

[0236] The outputs of the above-mentioned n sub-models and m regular methods include feature information 1, feature information 2,..., feature information n, feature information n+1, feature information n+2,..., feature information n+m. After merging, side information is obtained, that is, the side information includes the feature information output by the n sub-models respectively and the feature information output according to the m regular methods.

[0237] In some possible implementations, the input of the first sub-rule method also includes a third input; when the pre-set rule method includes multiple sub-rule methods, the third input includes the output of at least one other sub-rule method among the multiple sub-rule methods, or the third input includes the output of at least one sub-model; the first sub-rule method is any one of the multiple sub-rule methods.

[0238] For example, FIG11a is a schematic diagram of the serial connection of sub-rule methods in an embodiment of the present application. As shown in FIG11a, taking two sub-rule methods as an example, in the two sub-rule methods, the output of sub-rule method 2 can be used as an input of sub-rule method 1, so that sub-rule method 1 includes input 1 and the output of sub-rule method 2. In addition, the output of the sub-model can also be used as an input of sub-rule method 1, so that sub-rule method 1 includes input 1 and the output of the sub-model, and can even add the output of sub-rule method 2, without specific limitation. This can be the serial connection of a sub-rule method and a sub-rule method or a sub-model.

[0239] In some possible implementations, the input of the first sub-rule method also includes a fourth input, which is an audio signal obtained after redundant information processing of the third input; when the pre-set rule method includes multiple sub-rule methods, the third input includes the output of at least one other sub-rule method among the multiple sub-rule methods, or the third input includes the output of at least one sub-model; the first sub-rule method is any one of the multiple sub-rule methods.

[0240] Exemplarily, FIG11b is a schematic diagram of the sub-rule method of the embodiment of the present application connected in series. As shown in FIG11b, taking two sub-rule methods as an example, in the two sub-rule methods, the output of sub-rule method 2 is first processed with redundant information, and then can be used as an input of sub-rule method 1, so that sub-rule method 1 includes input 1 and the output of sub-rule method 2 after redundant information processing. In addition, the output of the sub-model can also be processed with redundant information first, and then used as an input of sub-rule method 1, so that sub-rule method 1 includes input 1 and the output of the sub-model after redundant information processing, and can even add the output of sub-rule method 2 after redundant information processing, without specific limitation. This can be the connection of a sub-rule method with a sub-rule method or a sub-model.

[0241] It should be noted that the above example uses two sub-rule methods as an example, but this does not limit the series relationship between sub-rule methods and between sub-rule methods and sub-models. In addition to its own input, the input of any sub-rule method can also include the output of one or more other sub-rule methods or the output after redundant processing, and / or, the output of one or more sub-models or the output after redundant processing. The embodiment of the present application does not limit the number of series connections or the series connection method.

[0242] In some possible implementations, the second machine learning model and / or rule method for obtaining side information can be jointly trained. That is, by jointly training multiple mechanisms for obtaining side information (which can be the above-mentioned sub-models or sub-rules), the information redundancy between feature groups can be reduced or eliminated; a joint loss function is set for the multiple mechanisms for obtaining side information, and the weights of the multiple mechanisms for obtaining side information are corrected through error back propagation to achieve the overall optimal effect of multiple feature information, thereby reducing or eliminating the information redundancy between feature groups; each mechanism for obtaining side information can have its own loss function to retain the uniqueness of the feature information it extracts.

[0243] Exemplarily, FIG12 is a schematic diagram of the joint training of an embodiment of the present application. As shown in FIG12, taking two mechanisms for obtaining side information (the mechanism can be the above-mentioned sub-model or sub-rule method) as an example, each mechanism for obtaining side information can have its own loss function (loss function 1 and loss function 2), and a joint loss function is set for the two mechanisms for obtaining side information. These loss functions correct the weights of the two mechanisms for obtaining side information through error back propagation to achieve the overall optimal effect of the two feature information. On the one hand, the uniqueness of the extracted feature information can be retained, and on the other hand, the information redundancy between the two feature groups can be reduced or eliminated.

[0244] Step 703: Obtain a second bitstream according to the side information.

[0245] In the embodiment of the present application, the side information may be encoded by a side information encoder. The side information encoder may use the same encoding algorithm as the core encoder described above, or may use a different encoding algorithm, which is not specifically limited.

[0246] Step 704: Transmit a code stream, where the code stream includes a first code stream and a second code stream.

[0247] The code stream generated by the encoder consists of two parts: one part (the first code stream) corresponds to the low-frequency components of the current frame, and the other part (the second code stream) corresponds to the side information of the current frame. The encoder transmits the code stream to the decoder, which reconstructs the low-frequency components and side information based on the code stream, thereby obtaining a reconstructed frame corresponding to the current frame.

[0248] In this embodiment, side information is extracted from audio through a machine learning model to guide the decoding end to implement non-blind BWE and reconstruct high-frequency components, especially high-frequency components that are weakly correlated with low-frequency components. This implementation method can reflect more details of the high-frequency components in the current frame in the feature information. These details, such as the characteristics of the high-frequency components, the correlation between the high-frequency components and the low-frequency components, etc., help to reconstruct more high-frequency details, thereby avoiding the problem of audio distortion.

[0249] FIG13 is a flow chart of process 1300 of the audio decoding method provided in an embodiment of the present application. Process 1300 is described as a series of steps or operations. It should be understood that process 1300 can be performed in various orders and / or occur simultaneously, and is not limited to the execution order shown in FIG13. Process 1300 may include:

[0250] Step 1301: Receive a code stream, which includes a first code stream and a second code stream.

[0251] In the embodiment of the present application, a code stream is received from an encoding end. Corresponding to the code stream generated by the encoding end, the received code stream also includes a first code stream and a second code stream.

[0252] Step 1302: Acquire a first audio signal according to the first code stream, where the frequency of the first audio signal is lower than a preset threshold.

[0253] In the embodiment of the present application, the first code stream can be parsed by the core decoder to obtain the first audio signal.

[0254] The reconstructed frame to be obtained by the decoding end corresponds to the current frame of the encoding end. The characteristics of the audio signal contained in the reconstructed frame can also be characterized by frequency, amplitude and phase. According to the frequency division, the reconstructed frame also includes a first audio signal and a second audio signal, wherein the frequency of the first audio signal is lower than the preset threshold audio signal (in the embodiment of the present application, the audio signal that meets this condition can also be called a low-frequency signal or a low-frequency component), and the frequency of the second audio signal is higher than or equal to the preset threshold audio signal (in the embodiment of the present application, the audio signal that meets this condition can also be called a high-frequency signal or a high-frequency component), that is, the reconstructed frame includes a low-frequency component and a high-frequency component. The aforementioned preset threshold can be, for example, 16kHz, and the embodiment of the present application does not make specific restrictions on this. It should be noted that the foregoing classifies the audio signal with a frequency equal to the preset threshold as the second audio signal. Optionally, the embodiment of the present application can also classify the audio signal with a frequency equal to the preset threshold as the first audio signal, and this is not specifically limited.

[0255] Based on this, the first audio signal obtained according to the first bitstream may be the low-frequency component of the reconstructed frame. That is, the decoding end may directly obtain the low-frequency component of the reconstructed frame by parsing the bitstream.

[0256] Step 1303: Obtain side information according to the second bitstream.

[0257] The side information is used to characterize the characteristics of the second audio signal. Furthermore, the side information can also be used to characterize the mapping relationship between high and low frequencies. In the embodiments of the present application, the side information only occupies a small number of transmission bits and can guide the decoder to implement non-blind BWE to reconstruct high-frequency components, especially high-frequency components that are closely related to low-frequency components but have a complex mapping relationship.

[0258] In some possible implementations, obtaining the side information based on the second bitstream can be performed by directly parsing the second bitstream to obtain the side information. That is, the second bitstream is obtained by the encoder processing the side information, so the decoder can directly parse the second bitstream to obtain the side information.

[0259] Optionally, the second code stream is parsed to first obtain the processed front side information; then the processed front side information is subjected to side information processing to obtain side information, wherein the side information processing includes inputting a pre-trained third machine learning model or processing according to a pre-set third rule.

[0260] In an embodiment of the present application, the information carried by the second bitstream may be pre-processing side information associated with the side information. This information may be obtained by the encoder based on the side information to further conserve bitstream, and then encoded into the bitstream. Accordingly, to obtain the side information, the decoder may first parse the pre-processing side information obtained from the second bitstream, and then obtain the side information based on the pre-processing side information. The aforementioned method for obtaining the side information may include obtaining the side information through a third machine learning model or according to a pre-set third rule.

[0261] Step 1304: Obtain a reconstructed frame according to the first audio signal and the side information.

[0262] In some possible implementations, obtaining the reconstructed frame can be performed using a pre-trained first machine learning model based on the first audio signal and the side information. Because obtaining the reconstructed frame references the side information to achieve expansion from low-frequency components to high-frequency components, obtaining the reconstructed frame can also be referred to as non-blind BWE.

[0263] In the embodiment of the present application, the first machine learning model can refer to the above-mentioned neural network, deep neural network, convolutional neural network, recurrent neural network, etc.

[0264] Exemplarily, Figure 14a is a schematic diagram of the non-blind BWE of an embodiment of the present application. As shown in Figure 14a, the non-blind BWE is implemented by a first machine learning model, which includes a sub-model, namely the first machine learning model, that is, the sub-model, whose input is the first audio signal and side information, and the output is a reconstructed frame.

[0265] Optionally, the reconstructed frame is directly output by the first machine learning model.

[0266] In an embodiment of the present application, a non-blind BWE can use a machine learning model (a first machine learning model) to directly output a reconstructed frame based on the decoded low-frequency component (the first audio signal) under the guidance of the side information. In addition, the first machine learning model can perform codec distortion correction for the low-frequency component and possible audio rendering, which is not specifically limited in the embodiment of the present application.

[0267] Exemplarily, FIG15a is a schematic diagram of obtaining a reconstructed frame according to an embodiment of the present application. As shown in FIG15a , the input of the non-blind BWE includes low-frequency components and side information, and the output is a reconstructed frame.

[0268] Optionally, the reconstructed frame is obtained based on the second audio signal output by the first machine learning model and the first audio signal.

[0269] In an embodiment of the present application, the non-blind BWE can adopt a machine learning model (a first machine learning model) to output a reconstructed high-frequency component based on the decoded low-frequency component (the first audio signal) under the guidance of the side information, and merge or add the reconstructed high-frequency component with the decoded low-frequency component to obtain reconstructed audio. In addition, the first machine learning model can complete the codec distortion correction of the low-frequency component and possible audio rendering, which is not specifically limited in the embodiment of the present application.

[0270] Exemplarily, Figure 15b is a schematic diagram of obtaining a reconstructed frame in an embodiment of the present application. As shown in Figure 15b, the input of the non-blind BWE includes low-frequency components and side information, and the output is a reconstructed high-frequency component. The high-frequency component and the low-frequency component are then merged at high and low frequencies to obtain a reconstructed frame.

[0271] In some possible implementations, the first machine learning model may include a first sub-model and a second sub-model, and the non-blind BWE may input the first audio signal and the first side information into the first sub-model to obtain a third audio signal; and then input the third audio signal and the second side information into the second sub-model to obtain a reconstructed frame; the first side information and the second side information are respectively part or all of the information in the side information.

[0272] For example, FIG14b is a schematic diagram of a non-blind BWE according to an embodiment of the present application. As shown in FIG14b , the first machine learning model used in the non-blind BWE can be divided into a first sub-model (corresponding to low-frequency feature extraction) and a second sub-model (corresponding to high-frequency component reconstruction). The two sub-models are connected in series, that is, the output of the low-frequency feature extraction (the third audio signal, i.e., the low-frequency feature) can be used as the input for high-frequency component reconstruction; the input for the low-frequency feature extraction includes the first audio signal and the first side information, and the input for the high-frequency component reconstruction includes the third audio signal and the second side information; the low-frequency feature extraction is guided by the first side information, and the high-frequency component reconstruction is guided by the second side information; the output of the high-frequency component reconstruction is a reconstructed frame. It should be noted that in the embodiment of the present application, the first side information and the second side information can be part or all of the information in the side information, respectively, and the first side information and the second side information can be the same or different, without specific limitation.

[0273] In some possible implementations, the non-blind BWE may process the first audio signal and the third side information according to a pre-set first rule to obtain a fourth audio signal; then input the fourth audio signal and the fourth side information into the first machine learning model to obtain a reconstructed frame; the third side information and the fourth side information are respectively part or all of the information in the side information.

[0274] Exemplarily, FIG14c is a schematic diagram of a non-blind BWE according to an embodiment of the present application. As shown in FIG14c, corresponding to the encoding end, when the encoding end obtains side information in a regular manner, the decoding end can also adopt a regular manner in the non-blind BWE accordingly. The regular manner can be set before machine learning, that is, first processed according to a first regular manner (corresponding to rule-based pre-processing), with the first audio signal and the third side information as input and the fourth audio signal as output; then through the first machine learning model, the fourth audio signal and the fourth side information as input, and the reconstructed frame as output. It should be noted that in the embodiment of the present application, the third side information and the fourth side information can be part or all of the information in the side information, respectively, and the third side information and the fourth side information can be the same or different, and there is no specific limitation on this.

[0275] In some possible implementations, the non-blind BWE can obtain a fifth audio signal based on the first audio signal and the fifth side information through a first machine learning model obtained by pre-training; and then process the fifth audio signal and the sixth side information according to a pre-set second rule to obtain a reconstructed frame; the fifth side information and the sixth side information are respectively part or all of the information in the side information.

[0276] For example, FIG14d is a schematic diagram of a non-blind BWE according to an embodiment of the present application. As shown in FIG14d , corresponding to the encoding end, when the encoding end obtains side information in a regular manner, the decoding end can also correspondingly adopt a regular manner in the non-blind BWE. This regular manner can be set after machine learning, that is, first through a first machine learning model, whose input is the first audio signal and the fifth side information, and the output is the fifth audio signal; then processed according to a second regular manner, whose input is the fifth audio signal and the sixth side information, and the output is a reconstructed frame. It should be noted that in the embodiment of the present application, the fifth side information and the sixth side information can be part or all of the information in the side information, respectively, and the fifth side information and the sixth side information can be the same or different, and there is no specific limitation on this.

[0277] It should be noted that in the embodiments shown in Figures 14a-14d, the non-blind BWE directly outputs the reconstructed frame (refer to Figure 15a) as an example. It should be understood that in the embodiments of the present application, the method shown in the embodiment of Figure 15b can also be used to obtain the reconstructed frame. In addition, the first machine learning model used by the non-blind BWE in the embodiments shown in Figures 14c and 14d can also refer to the embodiment shown in Figure 14b.

[0278] Exemplarily, Figure 14e is a schematic diagram of the non-blind BWE of an embodiment of the present application. As shown in Figure 14e, the non-blind BWE first performs rule-based pre-processing, whose input is the low-frequency component and side information 1; then the first machine learning model is adopted, whose input is the output of the rule-based pre-processing and side information 2; then rule-based post-processing is performed, whose input is the output of the first machine learning model and side information 3, and the output is the high-frequency component; finally, the low-frequency component and the high-frequency component are merged to output the reconstructed frame.

[0279] It should be noted that the above-mentioned side information 1 to side information 3 can be part or all of the information in the side information respectively, and side information 1 to side information 3 can be completely the same or completely different or partially the same and partially different, and there is no specific limitation on this.

[0280] Exemplarily, Figure 14f is a schematic diagram of the non-blind BWE of an embodiment of the present application. As shown in Figure 14f, the non-blind BWE first performs rule-based pre-processing, whose input is the low-frequency component and side information 1; then the low-frequency feature extraction in the first machine learning model is adopted, whose input is the output of the rule-based pre-processing and side information 2, and the output is the low-frequency feature; then the high-frequency component reconstruction in the first machine learning model is adopted, whose input is the low-frequency feature and side information 3, and the output is the high-frequency feature; then rule-based post-processing is performed, whose input is the high-frequency feature and side information 4, and the output is the high-frequency component; finally, the low-frequency component and the high-frequency component are merged for high and low frequency, and the reconstructed frame is output.

[0281] It should be noted that the above-mentioned side information 1 to side information 4 can be part or all of the information in the side information respectively, and side information 1 to side information 4 can be completely the same or completely different or partially the same and partially different, and there is no specific limitation on this.

[0282] In this embodiment, a machine learning model is used to obtain reconstructed frames based on low-frequency components under the guidance of side information, especially reconstructed frames in scenarios where the correlation between high-frequency components and low-frequency components is weak or the relationship between high- and low-frequency components is close but the mapping relationship is complex. This implementation method can reflect more details of the high-frequency components in the feature information. These details, such as the characteristics of the high-frequency components, the correlation between the high-frequency components and the low-frequency components, etc., help to reconstruct more high-frequency details, thereby avoiding the problem of audio distortion.

[0283] The following uses several specific embodiments to explain in detail the technical solutions of the method embodiments shown in FIG. 7 and FIG. 13 .

[0284] FIG16 is a schematic diagram of a coding and decoding framework of a non-blind BWE including AI according to an embodiment of the present application. As shown in FIG16 , the framework includes the following steps:

[0285] 1. The encoder converts the original audio into a bit stream.

[0286] 1.1. Input the low-frequency components of the original audio into the core encoder, which encodes the low-frequency components into a low-frequency bitstream (corresponding to the first bitstream described above). Refer to step 701 above.

[0287] 1.2. The side information extraction step uses a machine learning model to extract features that are helpful for BWE from the audio containing high-frequency components of the original audio, which is called side information. See step 702 above.

[0288] 1.3. The side information is input into the side information encoder, which encodes the side information into a side information code stream (corresponding to the second code stream described above). Refer to step 703 above.

[0289] 1.4. The side information code stream and the low-frequency code stream are combined to obtain a code stream, which is transmitted to the decoding end through the transmission network.

[0290] 2. The decoding end reconstructs the audio based on the bit stream.

[0291] 2.1. After receiving the code stream, parse the code stream into a low-frequency code stream and a side information code stream. Refer to step 1301 above.

[0292] 2.2. Input the low-frequency code stream into the core decoder, and the core decoder decodes the low-frequency code stream into low-frequency components. Refer to step 1302 above.

[0293] 2.3. Input the side information code stream into the side information decoder, and the side information decoder decodes the side information code stream into side information (refer to step 1303 above).

[0294] 2.4 Non-blind BWE uses a machine learning model. Guided by side information, the machine learning model outputs reconstructed audio based on the decoded low-frequency components. Furthermore, the machine learning model can correct encoding and decoding distortion of the low-frequency components and potentially perform audio rendering.

[0295] Optionally, step 2.4 may also refer to FIG15b to first obtain the reconstructed high-frequency components and then obtain the reconstructed audio.

[0296] FIG17 is a schematic diagram of the encoding and decoding framework of the non-blind BWE including AI in an embodiment of the present application. As shown in FIG17 , based on the embodiment shown in FIG16 , the side information extraction link at the encoding end of this embodiment, in addition to adopting the machine learning model, also adopts a rule-based approach, so that the side information output by the side information extraction link includes the feature information output by the machine learning model and the feature information output according to the rule-based approach. The non-blind BWE link at the decoding end, in addition to adopting the machine learning model, also adopts a rule-based approach, so that the reconstructed audio output by the non-blind BWE link combines the output of the machine learning model and the output according to the rule-based approach.

[0297] Several implementations of the side information extraction step of this embodiment can be referred to the embodiments shown in Figures 8b to 8d, Figures 9a to 9c, Figures 10a to 10b, and Figures 11a to 11b.

[0298] Figure 18 is a schematic diagram of the codec framework of the non-blind BWE including AI in an embodiment of the present application. As shown in Figure 18, this embodiment is built on the AVS3 codec framework and implements 48kHz audio codec. Among them, the core codec is responsible for encoding and reconstructing low-frequency components below 16kHz through codec, the side information extraction link is responsible for extracting side information, and the non-blind BWE link is responsible for reconstructing high-frequency components from 16kHz to 48kHz. The framework includes the following steps:

[0299] 1. The encoder converts the original 48kHz audio into a bitstream.

[0300] 1.1. Frame the original 48kHz audio. Each frame is 2048 sampling points long. 1024 sampling points are overlapped between frames. These signal frames are processed sequentially.

[0301] 1.2. The low-frequency channel encodes the low-frequency components below 16kHz in the signal frame.

[0302] 1.2.1. Use a low-pass filter to extract the audio signal below 16kHz from the signal frame, which is the low-frequency component.

[0303] 1.2.2. Encode the low-frequency components below 16kHz into a low-frequency bit stream through the AVS3 core encoder.

[0304] 1.3. The side information channel encodes the high-frequency components from 16kHz to 48kHz in the signal frame.

[0305] 1.3.1. The side information extraction step extracts side information from the signal frame, as shown in FIG19 ( FIG19 is a flow chart of the side information extraction step of an embodiment of the present application):

[0306] 1.3.1.1. Convert the signal frame into a spectrum through Fourier transform.

[0307] 1.3.1.2. Divide the high-frequency region of the spectrum into multiple frequency bands and perform envelope calculation, that is, calculate the average value of the spectrum energy of each frequency band as the frequency band envelope (feature group 1).

[0308] 1.3.1.3. Divide the spectrum of each high-frequency band by the corresponding band envelope, reducing the spectral energy of these bands to 1 and eliminating the energy envelope information contained therein. The spectrum after band normalization is called the normalized spectrum.

[0309] 1.3.1.4. The normalized spectrum is input to a neural network, which extracts features from the normalized spectrum (feature group 2).

[0310] 1.3.1.5. Merge feature group 1 and feature group 2 as side information.

[0311] 1.3.2. The quantization encoder quantizes the side information into a side information code stream.

[0312] 1.4. Combine the low-frequency code stream and the side information code stream into a code stream and transmit it to the decoding end.

[0313] 2. The decoder reconstructs the original 48kHz audio based on the bit stream.

[0314] 2.1. Receive the code stream transmitted by the encoder and split it into a low-frequency code stream and a side information code stream.

[0315] 2.2. Use the AVS3 core decoder to decode the low-frequency components below 16kHz from the low-frequency bit stream.

[0316] 2.3. Decode the side information from the side information bitstream through a quantized decoder.

[0317] 2.4. The non-blind BWE link is responsible for reconstructing the high-frequency signal based on the decoded low-frequency signal below 16 kHz under the guidance of the decoded side information, as shown in Figure 20 (Figure 20 is a flowchart of the non-blind BWE link of an embodiment of the present application):

[0318] 2.4.1. The decoded low-frequency components are transformed into low-frequency spectrum after Fourier transformation.

[0319] 2.4.2. Extract low-frequency features from the low-frequency spectrum based on the low-frequency feature extraction model.

[0320] 2.4.3. The side information processing model maps feature group 2 into high-frequency features.

[0321] 2.4.4. High-frequency component reconstruction model Under the guidance of high-frequency features, the high-frequency components are reconstructed according to the low-frequency features to obtain the reconstructed spectrum.

[0322] 2.4.5. Adjust the spectrum energy of the corresponding frequency band according to feature group 1 to ensure that the spectrum energy of each frequency band is the same as the actual spectrum, and replace the spectrum below 16 kHz with the low-frequency spectrum to obtain the adjusted spectrum.

[0323] 2.4.6. Perform inverse Fourier transform on the adjusted spectrum to obtain reconstructed audio.

[0324] 2.5. Add the decoded low-frequency components and the reconstructed high-frequency components, and then perform splicing to obtain 48kHz reconstructed audio.

[0325] Under the same experimental conditions, the results of rule-based BWE, AI-based fully blind BWE, and AI-based non-blind BWE (the technical solution of the embodiment of the present application) were compared with the real target. The experimental results are shown in Figures 21 and 22 (Figures 21 and 22 are schematic diagrams comparing the audio spectra obtained by the three BWE methods, respectively):

[0326] Rule-based BWE copies the low-frequency spectrum to the high-frequency spectrum and adjusts the energy of each frequency band based on side information. Therefore, this method cannot extend the texture that does not exist in the low-frequency components. The spectrum of the framed areas in Figures 21 and 22 is copied from the low-frequency spectrum and differs significantly from the actual spectrum, which in turn causes a difference in listening experience.

[0327] The AI-based, fully blind BWE can reconstruct some high-frequency components from low-frequency components. Since higher-frequency spectra have weaker correlations with lower-frequency spectra, the resulting spectrum distortion increases. This also results in the high-frequency energy reconstructed by the model gradually decreasing with increasing frequency. Furthermore, some high-frequency components exist independently. Without the guidance of side information, the model cannot determine the existence of these components and cannot reconstruct them.

[0328] Compared to rule-based BWE and AI-based fully blind BWE, the high-frequency spectrum reconstructed by AI-based non-blind BWE is significantly closer to reality and more consistent with the original audio. Compared to rule-based BWE, the high- and low-frequency mapping relationships discovered by the AI ​​model are more complex and comprehensive. Therefore, the side information extracted by the AI-based side information extraction module can contain richer spectral features, allowing the AI-based non-blind BWE module to reconstruct more and more accurate high-frequency details. The rule-based pre-processing module, guided by the side information, constructs a high-frequency floor, simplifying subsequent AI calculations. Introducing side information into the low-frequency feature extraction and high-frequency component reconstruction modules, respectively, extracts low-frequency features consistent with the side information and reconstructs high-frequency components consistent with the side information. Rule-based post-processing further adjusts the reconstructed high-frequency components based on the side information to ensure that they meet a priori characteristics. For example, the spectral envelope energy of each frequency band can be adjusted based on the side information to ensure the accuracy of the envelope energy.

[0329] Figure 23 is a schematic diagram of the codec framework of the non-blind BWE including AI in an embodiment of the present application. As shown in Figure 23, this embodiment is built on the AVS3 codec framework and implements 48kHz audio codec. Among them, the core codec is responsible for encoding and reconstructing the low-frequency components below the specified frequency f through the codec method, the side information extraction link is responsible for extracting the side information, and the non-blind BWE link is responsible for reconstructing the high-frequency components from f to 48kHz. The framework includes the following steps:

[0330] 1. The encoder converts the 48kHz raw audio into a bitstream.

[0331] 1.1. Frame the original 48kHz audio. Each frame is 2048 sampling points long. 1024 sampling points are overlapped between frames. These signal frames are processed sequentially.

[0332] 1.2. Specify the frequency boundary f, and the low-frequency channel encodes the low-frequency components below f in the signal frame.

[0333] 1.2.1. Use a low-pass filter to extract low-frequency signals below f from the original 48kHz audio, which are the low-frequency components.

[0334] 1.2.2. Encode the low-frequency components below f into a low-frequency bit stream through the AVS3 core encoder.

[0335] 1.3. The side information channel encodes the high frequency components from f to 48kHz.

[0336] 1.3.1. The side information extraction step extracts side information from the original 48 kHz audio, as shown in FIG. 24 ( FIG. 24 is a flow chart of the side information extraction step of an embodiment of the present application).

[0337] 1.3.1.1. Convert the signal frame into a spectrum through Fourier transform.

[0338] 1.3.1.2. Use the side information extraction module provided by AVS3 to divide the high-frequency region of the spectrum into multiple frequency bands and perform envelope calculation. That is, calculate the average spectral energy of each frequency band as the band envelope, and calculate the band replication method identifier used for each frequency band. Combine these two types of information into feature group 1.

[0339] 1.3.1.3. Divide the spectrum of each high-frequency band by the corresponding band envelope, reducing the spectral energy of these bands to 1 and eliminating the energy envelope information contained therein. The spectrum after band normalization is called the normalized spectrum.

[0340] 1.3.1.4. The normalized spectrum is input to a neural network, which extracts features from the normalized spectrum (feature group 2).

[0341] 1.3.1.5. Merge feature group 1 and feature group 2 as side information.

[0342] 1.3.2. The quantization encoder quantizes the side information into a side information code stream.

[0343] 1.4. Combine the low-frequency code stream, the side information code stream, and f into a code stream and transmit it to the decoding end.

[0344] 2. The decoder reconstructs the original 48kHz audio based on the bit stream.

[0345] 2.1. Receive the code stream transmitted by the encoder and split it into a low-frequency code stream and a side information code stream.

[0346] 2.2. Use the AVS3 core decoder to decode the low-frequency signal below f from the low-frequency bit stream.

[0347] 2.3. Decode the side information from the side information bitstream through a quantized decoder.

[0348] 2.4. The non-blind BWE link is responsible for reconstructing the high-frequency signal based on the decoded low-frequency signal below f under the guidance of the decoded side information, as shown in Figure 25 (Figure 25 is a flowchart of the non-blind BWE link of an embodiment of the present application).

[0349] 2.4.1. Input the decoded low-frequency components and feature group 1 into the AVS3 band extension, use the rule-based BWE method to expand the high-frequency components, and add the high-frequency components to the low-frequency components to obtain the signal base.

[0350] 2.4.2. The side information processing model maps feature group 2 into high-frequency features.

[0351] 2.4.3. Under the guidance of high-frequency features, the non-blind BWE model obtains the reconstructed signal based on the signal basis.

[0352] 2.4.4. Perform a Fourier transform on the reconstructed signal to obtain a reconstructed spectrum. Adjust the spectral energy of the corresponding frequency band based on the band envelope of feature group 1 to ensure that the spectral energy of each frequency band is consistent with the true spectrum. Then perform an inverse Fourier transform on the adjusted spectrum to obtain the adjusted reconstructed signal.

[0353] 2.5. The adjusted reconstructed signal is spliced ​​to obtain a 48 kHz reconstructed audio.

[0354] FIG26 is a schematic diagram of the structure of an encoding device 2600 according to an embodiment of the present application. As shown in FIG26 , the encoding device 2600 according to this embodiment can be applied to the above encoding device. The encoding device 2600 may include: an acquisition module 2601, an extraction module 2602, an encoding module 2603, a transmission module 2604, and a filtering module 2605.

[0355] An acquisition module 2601 is configured to acquire a first bitstream, where the first bitstream is obtained based on a first audio signal, where the first audio signal is a low-frequency signal in a current frame having a frequency lower than a preset threshold. An extraction module 2602 is configured to acquire side information, where the side information is obtained based on a second audio signal, where the second audio signal includes a high-frequency signal in the current frame having a frequency higher than or equal to the preset threshold. An encoding module 2603 is configured to acquire a second bitstream based on the side information. A transmission module 2604 is configured to transmit bitstreams, where the bitstreams include the first bitstream and the second bitstream.

[0356] In one possible implementation, the extraction module 2602 is specifically configured to input the second audio signal into a pre-trained second machine learning model to obtain the side information, where the second machine learning model includes at least one sub-model.

[0357] In one possible implementation, when the second machine learning model includes multiple sub-models, the input of each of the multiple sub-models includes part of the audio signal or all of the audio signal in the second audio signal, wherein the part of the audio signal includes the high-frequency signal.

[0358] In one possible implementation, the first sub-model and at least one other sub-model among the multiple sub-models include the same neurons, and the same neurons are located in the input stage, output stage or intermediate stage of the first sub-model, and the first sub-model is any one of the multiple sub-models.

[0359] In a possible implementation, the side information includes feature information outputted respectively by the multiple sub-models.

[0360] In one possible implementation, when the second machine learning model includes a sub-model, the side information includes feature information output by the sub-model.

[0361] In a possible implementation, the extraction module 2602 is specifically configured to process the second audio signal according to a preset rule to obtain the side information, where the preset rule includes at least one sub-rule.

[0362] In a possible implementation, when the preset rule includes multiple sub-rule methods, inputs of each of the multiple sub-rule methods include part of or all of the second audio signal, wherein the part of the audio signal includes the high-frequency signal.

[0363] In a possible implementation, the side information includes feature information outputted respectively according to the multiple sub-rules.

[0364] In a possible implementation, when the preset rule mode includes a sub-rule mode, the side information includes feature information output according to the sub-rule mode.

[0365] In one possible implementation, the input of the first sub-model also includes a first input; when the second machine learning model includes multiple sub-models, the first input includes the output of at least one other sub-model among the multiple sub-models, or the first input includes the output of the at least one sub-rule method; the first sub-model is any one of the multiple sub-models.

[0366] In one possible implementation, the input of the first sub-model also includes a second input, which is an audio signal obtained after the first input is processed with redundant information; when the second machine learning model includes multiple sub-models, the first input includes the output of at least one other sub-model among the multiple sub-models, or the first input includes the output of the at least one sub-rule method; the first sub-model is any one of the multiple sub-models.

[0367] In one possible implementation, the input of the first sub-rule method also includes a third input; when the preset rule method includes multiple sub-rule methods, the third input includes the output of at least one other sub-rule method among the multiple sub-rule methods, or the third input includes the output of the at least one sub-model; the first sub-rule method is any one of the multiple sub-rule methods.

[0368] In one possible implementation, the input of the first sub-rule method also includes a fourth input, and the fourth input is an audio signal obtained after the third input is processed with redundant information; when the preset rule method includes multiple sub-rule methods, the third input includes the output of at least one other sub-rule method among the multiple sub-rule methods, or the third input includes the output of the at least one sub-model; the first sub-rule method is any one of the multiple sub-rule methods.

[0369] In a possible implementation, the filtering module 2605 is configured to obtain the first audio signal from the current frame by low-pass filtering.

[0370] In a possible implementation manner, the second audio signal further includes part or all of the audio signal in the first audio signal.

[0371] The device of this embodiment can be used to execute the technical solution of the method embodiment shown in Figure 7. Its implementation principle and technical effects are similar and will not be repeated here.

[0372] FIG27 is a schematic diagram of the structure of a decoding device 2700 according to an embodiment of the present application. As shown in FIG27 , the decoding device 2700 according to the present embodiment can be applied to the above decoding device. The decoding device 2700 may include: a receiving module 2701, an acquisition module 2702, an extraction module 2703, and a reconstruction module 2704.

[0373] A receiving module 2701 is configured to receive a bitstream, the bitstream including a first bitstream and a second bitstream. An acquiring module 2702 is configured to acquire a first audio signal based on the first bitstream, where the frequency of the first audio signal is lower than a preset threshold. An extracting module 2703 is configured to acquire side information based on the second bitstream, where the side information is used to characterize a second audio signal, where the frequency of the second audio signal is higher than or equal to the preset threshold. A reconstructing module 2704 is configured to acquire a reconstructed frame based on the first audio signal and the side information.

[0374] In a possible implementation, the reconstruction module 2704 is specifically configured to obtain the reconstructed frame based on a first machine learning model obtained through pre-training according to the first audio signal and the side information.

[0375] In one possible implementation, the reconstructed frame is directly output by the first machine learning model; or, the reconstructed frame is obtained based on the second audio signal and the first audio signal output by the first machine learning model.

[0376] In one possible implementation, when the first machine learning model includes a first sub-model and a second sub-model, the reconstruction module 2704 is specifically used to input the first audio signal and the first side information into the first sub-model to obtain a third audio signal; and input the third audio signal and the second side information into the second sub-model to obtain the reconstructed frame; the first side information and the second side information are respectively part or all of the information in the side information.

[0377] In one possible implementation, the reconstruction module 2704 is specifically configured to process the first audio signal and the third side information according to a preset first rule to obtain a fourth audio signal; input the fourth audio signal and the fourth side information into the first machine learning model to obtain the reconstructed frame; the third side information and the fourth side information are respectively part or all of the information in the side information.

[0378] In one possible implementation, the reconstruction module 2704 is specifically configured to obtain a fifth audio signal based on a first machine learning model obtained through pre-training of the first audio signal and the five-side information; and to process the fifth audio signal and the sixth side information according to a pre-set second rule to obtain the reconstructed frame; the fifth side information and the sixth side information are respectively part or all of the side information.

[0379] In a possible implementation, the extraction module 2703 is specifically configured to parse the second bitstream to obtain the side information.

[0380] In one possible implementation, the extraction module 2703 is specifically configured to parse the second bitstream to obtain pre-processing side information; perform side information processing on the pre-processing side information to obtain the side information, wherein the side information processing includes inputting a pre-trained third machine learning model or processing according to a pre-set third rule.

[0381] The device of this embodiment can be used to execute the technical solution of the method embodiment shown in Figure 13. Its implementation principle and technical effects are similar and will not be repeated here.

[0382] During implementation, each step of the above method embodiment can be completed by an integrated logic circuit of hardware in a processor or by instructions in the form of software. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application can be directly embodied as being executed by a hardware coding processor, or can be executed by a combination of hardware and software modules in the coding processor. The software module can be located in a mature storage medium in the art such as random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, registers, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0383] The memory mentioned in the above embodiments may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus RAM (DR RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0384] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0385] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0386] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0387] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of the solution of this embodiment according to actual needs.

[0388] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0389] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0390] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. An audio encoding method, characterized in that: include: Obtaining a first code stream, where the first code stream is obtained based on a first audio signal, where the first audio signal is a low-frequency signal having a frequency lower than a preset threshold in a current frame; Acquire side information, where the side information is obtained based on a second audio signal, where the second audio signal includes a high-frequency signal in the current frame whose frequency is greater than or equal to the preset threshold; Acquire a second bitstream according to the side information; A code stream is transmitted, where the code stream includes the first code stream and the second code stream.

2. The method according to claim 1, characterized in that The obtaining of side information includes: The second audio signal is input into a pre-trained second machine learning model to obtain the side information, where the second machine learning model includes at least one sub-model.

3. The method according to claim 2, characterized in that When the second machine learning model includes multiple sub-models, the input of each of the multiple sub-models includes part of the audio signal or all of the audio signal in the second audio signal, wherein the part of the audio signal includes the high-frequency signal.

4. The method according to claim 3, characterized in that The first sub-model and at least one other sub-model among the multiple sub-models include the same neurons, and the same neurons are located in the input stage, output stage or intermediate stage of the first sub-model. The first sub-model is any one of the multiple sub-models.

5. The method according to any one of claims 2 to 4, characterized in that The obtaining of side information includes: The side information is obtained by processing the second audio signal in a preset rule manner, where the preset rule manner includes at least one sub-rule manner.

6. The method according to claim 5, characterized in that When the preset rule includes multiple sub-rules, the input of each of the multiple sub-rules includes a portion of the audio signal or the entire audio signal in the second audio signal, wherein the portion of the audio signal includes the high-frequency signal.

7. The method according to claim 5 or 6, characterized in that The input of the first sub-model also includes a first input; when the second machine learning model includes multiple sub-models, the first input includes the output of at least one other sub-model among the multiple sub-models, or the first input includes the output of the at least one sub-rule method; the first sub-model is any one of the multiple sub-models.

8. The method according to claim 5 or 6, characterized in that The input of the first sub-model also includes a second input, which is an audio signal obtained after the first input is processed with redundant information; when the second machine learning model includes multiple sub-models, the first input includes the output of at least one other sub-model among the multiple sub-models, or the first input includes the output of the at least one sub-rule method; the first sub-model is any one of the multiple sub-models.

9. The method according to any one of claims 5 to 8, characterized in that The input of the first sub-rule method also includes a third input; when the preset rule method includes multiple sub-rule methods, the third input includes the output of at least one other sub-rule method among the multiple sub-rule methods, or the third input includes the output of the at least one sub-model; the first sub-rule method is any one of the multiple sub-rule methods.

10. The method according to any one of claims 5 to 8, characterized in that The input of the first sub-rule method also includes a fourth input, which is an audio signal obtained after the third input is processed with redundant information; when the preset rule method includes multiple sub-rule methods, the third input includes the output of at least one other sub-rule method among the multiple sub-rule methods, or the third input includes the output of the at least one sub-model; the first sub-rule method is any one of the multiple sub-rule methods.

11. An audio decoding method, characterized in that: include: receiving a code stream, wherein the code stream includes a first code stream and a second code stream; Acquire a first audio signal according to the first code stream, where the frequency of the first audio signal is lower than a preset threshold; Obtaining side information according to the second bitstream, where the side information is used to characterize characteristics of a second audio signal, where a frequency of the second audio signal is greater than or equal to the preset threshold; A reconstructed frame is obtained according to the first audio signal and the side information.

12. The method according to claim 11, characterized in that The obtaining a reconstructed frame according to the first audio signal and the side information includes: The reconstructed frame is obtained according to a first machine learning model obtained by pre-training the first audio signal and the side information.

13. The method according to claim 12, characterized in that The reconstructed frame is directly output by the first machine learning model; or, the reconstructed frame is obtained based on the second audio signal and the first audio signal output by the first machine learning model.

14. The method according to claim 12 or 13, characterized in that When the first machine learning model includes a first sub-model and a second sub-model, obtaining the reconstructed frame by the first machine learning model obtained by pre-training according to the first audio signal and the side information includes: Inputting the first audio signal and the first side information into the first sub-model to obtain a third audio signal; The third audio signal and the second side information are input into the second sub-model to obtain the reconstructed frame; the first side information and the second side information are part or all of the side information respectively.

15. The method according to any one of claims 12 to 14, characterized in that The obtaining the reconstructed frame by using a first machine learning model obtained by pre-training according to the first audio signal and the side information includes: Processing the first audio signal and the third side information according to a preset first rule to obtain a fourth audio signal; The fourth audio signal and the fourth side information are input into the first machine learning model to obtain the reconstructed frame; the third side information and the fourth side information are respectively part or all of the side information.

16. The method according to any one of claims 12 to 15, characterized in that The obtaining the reconstructed frame by using a first machine learning model obtained by pre-training according to the first audio signal and the side information includes: Obtaining a fifth audio signal based on a first machine learning model obtained by pre-training the first audio signal and the five-sided information; The reconstructed frame is obtained by processing the fifth audio signal and the sixth side information in a preset second rule manner; the fifth side information and the sixth side information are part or all of the side information respectively.

17. The method according to any one of claims 11 to 16, characterized in that The acquiring side information according to the second code stream includes: The second code stream is parsed to obtain the side information.

18. The method according to any one of claims 11 to 16, characterized in that The acquiring side information according to the second code stream includes: Parsing the second code stream to obtain processing front edge information; The pre-processed side information is subjected to side information processing to obtain the side information, wherein the side information processing includes inputting a pre-trained third machine learning model or processing according to a pre-set third rule.

19. An encoding device, characterized in that: include: an acquisition module, configured to acquire a first bit stream, where the first bit stream is obtained based on a first audio signal, where the first audio signal is a low-frequency signal having a frequency lower than a preset threshold in a current frame; an extraction module, configured to obtain side information, where the side information is obtained based on a second audio signal, where the second audio signal includes a high-frequency signal in the current frame whose frequency is greater than or equal to the preset threshold; an encoding module, configured to obtain a second bitstream according to the side information; The transmission module is configured to transmit a code stream, where the code stream includes the first code stream and the second code stream.

20. The device according to claim 19, characterized in that The extraction module is specifically used to input the second audio signal into a pre-trained second machine learning model to obtain the side information, where the second machine learning model includes at least one sub-model.

21. The device according to claim 20, characterized in that When the second machine learning model includes multiple sub-models, the input of each of the multiple sub-models includes part of the audio signal or all of the audio signal in the second audio signal, wherein the part of the audio signal includes the high-frequency signal.

22. The device according to claim 21, characterized in that The first sub-model and at least one other sub-model among the multiple sub-models include the same neurons, and the same neurons are located in the input stage, output stage or intermediate stage of the first sub-model. The first sub-model is any one of the multiple sub-models.

23. The device according to any one of claims 20 to 22, characterized in that The extraction module is specifically configured to process the second audio signal according to a preset rule to obtain the side information, where the preset rule includes at least one sub-rule.

24. The device according to claim 23, characterized in that When the preset rule includes multiple sub-rules, the input of each of the multiple sub-rules includes a portion of the audio signal or the entire audio signal in the second audio signal, wherein the portion of the audio signal includes the high-frequency signal.

25. The device according to claim 23 or 24, characterized in that The input of the first sub-model also includes a first input; when the second machine learning model includes multiple sub-models, the first input includes the output of at least one other sub-model among the multiple sub-models, or the first input includes the output of the at least one sub-rule method; the first sub-model is any one of the multiple sub-models.

26. The device according to claim 23 or 24, characterized in that The input of the first sub-model also includes a second input, which is an audio signal obtained after the first input is processed with redundant information; when the second machine learning model includes multiple sub-models, the first input includes the output of at least one other sub-model among the multiple sub-models, or the first input includes the output of the at least one sub-rule method; the first sub-model is any one of the multiple sub-models.

27. The device according to any one of claims 23 to 26, characterized in that The input of the first sub-rule method also includes a third input; when the preset rule method includes multiple sub-rule methods, the third input includes the output of at least one other sub-rule method among the multiple sub-rule methods, or the third input includes the output of the at least one sub-model; the first sub-rule method is any one of the multiple sub-rule methods.

28. The device according to any one of claims 23 to 26, characterized in that The input of the first sub-rule method also includes a fourth input, which is an audio signal obtained after the third input is processed with redundant information; when the preset rule method includes multiple sub-rule methods, the third input includes the output of at least one other sub-rule method among the multiple sub-rule methods, or the third input includes the output of the at least one sub-model; the first sub-rule method is any one of the multiple sub-rule methods.

29. A decoding device, characterized in that: include: A receiving module, configured to receive a code stream, wherein the code stream includes a first code stream and a second code stream; an acquisition module, configured to acquire a first audio signal according to the first code stream, wherein a frequency of the first audio signal is lower than a preset threshold; an extraction module, configured to obtain side information based on the second bitstream, wherein the side information is used to characterize characteristics of a second audio signal, wherein a frequency of the second audio signal is greater than or equal to the preset threshold; A reconstruction module is configured to obtain a reconstructed frame according to the first audio signal and the side information.

30. The device according to claim 29, characterized in that The reconstruction module is specifically configured to obtain the reconstructed frame based on a first machine learning model obtained through pre-training based on the first audio signal and the side information.

31. The device according to claim 30, characterized in that The reconstructed frame is directly output by the first machine learning model; or, the reconstructed frame is obtained based on the second audio signal and the first audio signal output by the first machine learning model.

32. The device according to claim 30 or 31, characterized in that When the first machine learning model includes a first sub-model and a second sub-model, the reconstruction module is specifically used to input the first audio signal and the first side information into the first sub-model to obtain a third audio signal; and input the third audio signal and the second side information into the second sub-model to obtain the reconstructed frame; the first side information and the second side information are respectively part or all of the information in the side information.

33. The device according to any one of claims 30 to 32, characterized in that The reconstruction module is specifically configured to process the first audio signal and the third side information according to a preset first rule to obtain a fourth audio signal; input the fourth audio signal and the fourth side information into the first machine learning model to obtain the reconstructed frame; the third side information and the fourth side information are respectively part or all of the side information.

34. The device according to any one of claims 30 to 33, characterized in that The reconstruction module is specifically configured to obtain a fifth audio signal based on a first machine learning model obtained through pre-training of the first audio signal and the five-side information; and to process the fifth audio signal and the sixth side information according to a pre-set second rule to obtain the reconstructed frame; the fifth side information and the sixth side information are respectively part or all of the side information.

35. The device according to any one of claims 29 to 34, characterized in that The extraction module is specifically configured to parse the second bitstream to obtain the side information.

36. The device according to any one of claims 29 to 34, characterized in that The extraction module is specifically used to parse the second code stream to obtain the front-end side information; perform side information processing on the front-end side information to obtain the side information, and the side information processing includes inputting a pre-trained third machine learning model or processing according to a pre-set third rule.

37. A device, characterized in that include: one or more processors; a memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 18.

38. A computer-readable storage medium, characterized in that The invention comprises a computer program which, when executed on a computer, causes the computer to perform the method according to any one of claims 1 to 18.

39. A computer program product, characterized in that The computer program product comprises a computer program code, and when the computer program code is run on a computer, the computer is caused to perform the method according to any one of claims 1 to 18.

Images