Audio encoding method and apparatus, and audio decoding method and apparatus

Abstract

An audio encoding method and apparatus, and an audio decoding method and apparatus. The encoding method comprises: acquiring a first code stream, which is obtained on the basis of a first audio signal (701); acquiring side information, which is obtained on the basis of a second audio signal (702); acquiring a second code stream on the basis of the side information (703); and transmitting the code streams, which comprise the first code stream and the second code stream (704).

Description

Audio encoding and decoding methods and apparatus Technical Field

[0001] This application relates to audio processing technology, and more particularly to an audio encoding / decoding method and apparatus. Background Technology

[0002] With the upgrading of communication systems, audio transmission quality has been greatly improved. However, in some scenarios, transmission bandwidth is still limited, and the limited transmission bits cannot represent the full bandwidth signal. To solve this problem, related technologies divide audio into low-frequency and high-frequency components. Low-frequency components are processed through encoding and decoding methods, and most of the transmission bits are allocated to the low-frequency component bitstream to ensure the encoding quality of low-frequency components. Then, utilizing the correlation between high and low frequencies, high-frequency components are recovered on the decoding side through bandwidth extension. This method of extending the bandwidth is called Bandwidth Extension (BWE). Compared to narrowband audio, the listening experience of audio obtained using BWE is significantly improved.

[0003] However, the BWE method mentioned above suffers from audio distortion due to the weak correlation between the spectra of high-frequency components and those of low-frequency components.

[0004] Summary of the Invention

[0005] This application provides an audio encoding and decoding method and apparatus to reconstruct more high-frequency details, thereby avoiding the problem of audio distortion.

[0006] In a first aspect, this application provides an audio encoding method, comprising: acquiring a first bitstream, the first bitstream being obtained based on a first audio signal, the first audio signal being an audio signal in the current frame with a frequency lower than a preset threshold; acquiring side information, the side information being obtained based on a second audio signal, the second audio signal including audio signals in the current frame with a frequency higher than or equal to the preset threshold; acquiring a second bitstream based on the side information; and transmitting a bitstream, the bitstream including the first bitstream and the second bitstream.

[0007] In this embodiment, side information is extracted from the audio through a machine learning model to guide the decoding end to implement non-blind BWE and reconstruct high-frequency components, especially high-frequency components that are weakly correlated with low-frequency components. This implementation method can reflect more details of the high-frequency components in the current frame in the feature information. These details include, for example, the characteristics of the high-frequency components and the correlation between the high-frequency components and low-frequency components, which helps to reconstruct more high-frequency details and thus avoid the problem of audio distortion.

[0008] The original audio is sampled to obtain an audio frame sequence, which includes multiple audio frames. At the encoding end, compression processing can be performed frame by frame. For ease of description, the audio frame currently being processed in this application embodiment is referred to as the current frame.

[0009] The characteristics of the audio signal contained in the current frame can be characterized by frequency, amplitude, and phase. In this embodiment, a first audio signal and a second audio signal are extracted from the current frame based on frequency. The first audio signal is an audio signal in the current frame with a frequency lower than a preset threshold (in this embodiment, an audio signal satisfying this condition can also be called a low-frequency signal or low-frequency component). The second audio signal includes audio signals in the current frame with a frequency higher than or equal to the preset threshold (in this embodiment, an audio signal satisfying this condition can also be called a high-frequency signal or high-frequency component). The aforementioned preset threshold can be, for example, 16kHz, and this embodiment does not specifically limit this. It should be noted that, while the audio signal with a frequency equal to the preset threshold in the current frame is categorized as the second audio signal, optionally, this embodiment can also categorize the audio signal with a frequency equal to the preset threshold in the current frame as the first audio signal, without specific limitation.

[0010] In some possible implementations, a low-pass filter can be used to extract the first audio signal from the current frame.

[0011] In some possible implementations, a core encoder may be used to encode the first audio signal. For example, the core encoder may be an AVS3 core encoder. This application does not specifically limit this implementation.

[0012] In this embodiment of the application, the second audio signal may exist in the following three situations:

[0013] 1) The second audio signal contains only high-frequency components. In this case, the second audio signal is equivalent to the high-frequency components of the current frame. Correspondingly, the side information can contain only the characteristic information of the high-frequency components.

[0014] 2) The second audio signal includes not only high-frequency components but also a portion of the low-frequency components. That is, the second audio signal and the first audio signal share some audio signals. In this case, the second audio signal is equivalent to the high-frequency components plus a portion of the low-frequency components of the current frame. Correspondingly, the side information, in addition to containing characteristic information of the high-frequency components, may also contain correlation information between the high-frequency components and the aforementioned portion of the low-frequency components.

[0015] 3) The second audio signal includes not only high-frequency components but also all low-frequency audio signals; that is, the second audio signal includes the first audio signal. In this case, the second audio signal is equivalent to the current frame. Correspondingly, the side information may include not only the feature information of the high-frequency components but also the correlation information between the high-frequency and low-frequency components.

[0016] This shows that side information can guide the decoding end to implement non-blind BWE with only a small number of transmission bits, in order to reconstruct high-frequency components, especially signals with complex mapping relationships between high and low frequencies.

[0017] In some possible implementations, obtaining side information may include inputting a second audio signal into a pre-trained second machine learning model to obtain side information. This second machine learning model includes at least one sub-model. Each sub-model may focus on different characteristics; for example, constraints may be set for each sub-model, including input / output, model structure, processing rules, etc., so that each sub-model can focus on extracting side information corresponding to its characteristic.

[0018] In this embodiment, the second machine learning model can refer to the neural networks, deep neural networks, convolutional neural networks, recurrent neural networks, etc., mentioned above. The machine learning model can incorporate more details of the high-frequency components in the current frame into the feature information. These details include, for example, the characteristics of the high-frequency components and the correlation between the high-frequency components and the low-frequency components, which helps to reconstruct more high-frequency details.

[0019] In some possible implementations, the second machine learning model may include multiple sub-models, each of which takes into account an audio signal in the current frame with a frequency higher than or equal to a preset threshold as its input. In other words, the input of each of the multiple sub-models is part or all of the audio signal in the second audio signal.

[0020] In this embodiment, if the input of a sub-model contains only high-frequency components, the sub-model is mainly used to compress the high-frequency components into features, or to represent the high-frequency components through features; if the input of a sub-model contains low-frequency components in addition to high-frequency components, the sub-model is also used to extract the mapping relationship between high and low frequencies. Using multiple sub-models can yield more diverse features in the extraction of side information; therefore, this application does not specifically limit the number, type, or function of the sub-models.

[0021] The outputs of each of the n sub-models, including feature information 1, feature information 2, ..., feature information n, are merged to obtain the edge information. That is, the edge information includes the feature information output by each of the n sub-models.

[0022] In some possible implementations, the first sub-model and at least one other sub-model among a plurality of sub-models include the same neurons, which are located in the input phase, output phase or intermediate phase of the first sub-model, which is any one of the plurality of sub-models.

[0023] The first sub-model may share neurons (also called common neurons) with one or more of the multiple sub-models included in the second machine learning model. These common neurons may exist in one or more layers, and these layers may be located in the input phase, output phase, or intermediate phase of the sub-model. The input phase refers to the first layer and one or more layers following the input of the sub-model; the output phase refers to the last layer and one or more layers preceding the output of the sub-model; and the intermediate phase refers to one or more layers excluding the first and last layers of the sub-model. It should be noted that this application does not specifically limit the number or type of the aforementioned common neurons, nor does it specifically limit the location of the input phase, output phase, or intermediate phase.

[0024] In some possible implementations, the input to the first sub-model also includes a first input; when the second machine learning model includes multiple sub-models, the first input includes the output of at least one other sub-model among the multiple sub-models, or the first input includes the output of at least one sub-rule; the first sub-model is any one of the multiple sub-models.

[0025] In some possible implementations, the input of the first sub-model may also include a second input, which is an audio signal obtained after the first input has undergone redundant information processing; when the second machine learning model includes multiple sub-models, the first input may include the output of at least one other sub-model among the multiple sub-models, or the first input may include the output of at least one sub-rule; the first sub-model may be any one of the multiple sub-models.

[0026] It should be noted that the above two examples are illustrated using two sub-models, but this does not constitute a limitation on the chaining relationship between sub-models or between sub-models and sub-rule methods. In addition to its own input, the input of any sub-model may include the output of one or more other sub-models or the output after redundancy processing, and / or the output of one or more sub-rule methods or the output after redundancy processing. The embodiments of this application do not limit the number of chains or the chaining method.

[0027] In some possible implementations, obtaining the side information may also include processing the second audio signal according to a pre-set rule to obtain the side information, the pre-set rule including at least one sub-rule.

[0028] In this embodiment, the method for extracting features according to pre-set rules may include, for example, extracting the spectral envelope and extracting features by referring to AVS3. The spectral envelope is the energy of each frequency band in the spectrum of the current frame, used to recover the corresponding frequency band energy on the decoding side. The features extracted by AVS3 include frequency band energy and identifiers of harmonic characteristics and noise features of each frequency band, used to recover the BWE basis on the decoding side. Compared to machine learning methods, rule-based methods can preserve information such as spectral energy, thereby avoiding any impact on the listening experience.

[0029] In this embodiment of the application, based on the second machine learning model, a rule-based approach can be set to process the second audio signal according to the aforementioned rule-based approach to obtain the corresponding output (feature information).

[0030] The feature information 1 output by the second machine learning model and the feature information 2 output by the rule method are merged to obtain the edge information. That is, the edge information includes the feature information output by the second machine learning model and the feature information output by the rule method respectively.

[0031] In some possible implementations, the pre-set rule method may include multiple sub-rule methods, each of which takes as input an audio signal in the current frame with a frequency higher than or equal to a preset threshold. That is, the input of each of the multiple sub-rule methods is part or all of the audio signal in the second audio signal.

[0032] The outputs of n sub-models and m rule methods, including feature information 1, feature information 2, ..., feature information n, feature information n+1, feature information n+2, ..., feature information n+m, are merged to obtain edge information. That is, the edge information includes the feature information output by the n sub-models and the feature information output according to the m rule methods.

[0033] In some possible implementations, the input of the first sub-rule method also includes a third input; when the preset rule method includes multiple sub-rule methods, the third input includes the output of at least one other sub-rule method among the multiple sub-rule methods, or the third input includes the output of at least one sub-model; the first sub-rule method is any one of the multiple sub-rule methods.

[0034] In some possible implementations, the input of the first sub-rule method also includes a fourth input, which is the audio signal obtained after the third input has undergone redundant information processing; when the preset rule method includes multiple sub-rule methods, the third input includes the output of at least one other sub-rule method among the multiple sub-rule methods, or the third input includes the output of at least one sub-model; the first sub-rule method is any one of the multiple sub-rule methods.

[0035] It should be noted that the above example uses two sub-rule methods for illustration, but this does not constitute a limitation on the chaining relationship between sub-rule methods or between sub-rule methods and sub-models. In addition to its own input, the input of any sub-rule method may also include the output of one or more other sub-rule methods or the output after redundancy processing, and / or the output of one or more sub-models or the output after redundancy processing. The embodiments of this application do not limit the number of chains or the chaining method.

[0036] In some possible implementations, the second machine learning model and / or rule-based approach for acquiring side information can be jointly trained. That is, by jointly training multiple mechanisms for acquiring side information (which can be the aforementioned sub-models or sub-rule-based approaches), information redundancy between feature groups can be reduced or eliminated; a joint loss function is set for the multiple side information acquisition mechanisms, and the weights of the multiple side information acquisition mechanisms are corrected through error backpropagation to achieve the overall optimal effect of multiple feature information, thereby reducing or eliminating information redundancy between feature groups; each side information acquisition mechanism can have its own loss function to preserve the uniqueness of its extracted feature information.

[0037] In this embodiment of the application, the edge information can be encoded by an edge information encoder. The edge information encoder can use the same encoding algorithm as the core encoder mentioned above, or it can use a different encoding algorithm. No specific limitation is made in this regard.

[0038] The bitstream obtained at the encoding end consists of two parts: one part (the first bitstream) corresponds to the low-frequency components of the current frame, and the other part (the second bitstream) corresponds to the side information of the current frame. The encoding end transmits the bitstream to the decoding end so that the decoding end can obtain the reconstructed low-frequency components and side information based on the bitstream, thereby obtaining the reconstructed frame corresponding to the current frame.

[0039] Secondly, this application provides an audio decoding method, comprising: receiving a bitstream, the bitstream including a first bitstream and a second bitstream; obtaining a first audio signal based on the first bitstream, the frequency of the first audio signal being lower than a preset threshold; obtaining side information based on the second bitstream, the side information being used to characterize the features of a second audio signal, the frequency of the second audio signal being higher than or equal to the preset threshold; and obtaining a reconstructed frame based on the first audio signal and the side information.

[0040] In this embodiment, a machine learning model, guided by side information, is used to obtain reconstructed frames based on low-frequency components. This is especially useful for reconstructed frames in scenarios where the correlation between high-frequency and low-frequency components is weak, or where the relationship between high and low-frequency components is close but the mapping relationship is complex. This approach can incorporate more details of high-frequency components into the feature information, such as the characteristics of high-frequency components and the correlation between high-frequency and low-frequency components. This helps to reconstruct more high-frequency details, thereby avoiding the problem of audio distortion.

[0041] In this embodiment of the application, a bitstream is received from the encoding end. Corresponding to the bitstream generated by the encoding end, the received bitstream also includes two parts: a first bitstream and a second bitstream.

[0042] In this embodiment of the application, the first bitstream can be parsed by a core decoder to obtain the first audio signal.

[0043] The reconstructed frame obtained at the decoding end corresponds to the current frame at the encoding end. The characteristics of the audio signal contained in the reconstructed frame can also be characterized by frequency, amplitude, and phase. According to frequency division, the reconstructed frame also includes a first audio signal and a second audio signal. The first audio signal is an audio signal with a frequency lower than a preset threshold (in this embodiment, the audio signal that meets this condition can also be called a low-frequency signal or low-frequency component), and the second audio signal is an audio signal with a frequency higher than or equal to the preset threshold (in this embodiment, the audio signal that meets this condition can also be called a high-frequency signal or high-frequency component). That is, the reconstructed frame includes low-frequency components and high-frequency components. The aforementioned preset threshold can be, for example, 16kHz, and this embodiment does not specifically limit it. It should be noted that the audio signal with a frequency equal to the preset threshold is classified as the second audio signal above. Optionally, this embodiment can also classify the audio signal with a frequency equal to the preset threshold as the first audio signal, and this is not specifically limited.

[0044] Based on this, the first audio signal obtained from the first bitstream can be the low-frequency component of the reconstructed frame. That is, the decoding end can directly obtain the low-frequency component of the reconstructed frame by parsing the bitstream.

[0045] Side information is used to characterize the properties of the second audio signal. Furthermore, it can also be used to characterize the mapping relationship between high and low frequencies. In this embodiment, side information occupies only a small number of transmission bits to guide the decoding end in implementing non-blind BWE (Blind Web End-to-Wide) to reconstruct high-frequency components, especially those closely related to low-frequency components but with complex mapping relationships.

[0046] In some possible implementations, the side information can be obtained by directly parsing the second bitstream. That is, the second bitstream is obtained by processing the side information at the encoding end, so the decoding end can directly parse the second bitstream to obtain the side information.

[0047] Optionally, the second bitstream is parsed to obtain the preceding information; then the preceding information is processed to obtain the side information, which includes inputting a pre-trained third machine learning model or processing according to a pre-set third rule.

[0048] In this embodiment, the information carried by the second bitstream may be pre-processing information associated with the side information. This information may be obtained by the encoding end based on the side information to further save bitstream space, and then encoded into the bitstream. Correspondingly, the decoding end can first parse the pre-processing information obtained from the second bitstream to obtain the side information, and then obtain the side information based on the pre-processing information. The aforementioned method for obtaining the side information may include obtaining the side information through a third machine learning model or according to a pre-set third rule.

[0049] In some possible implementations, the reconstructed frame can be obtained using a pre-trained first machine learning model based on the first audio signal and side information. Since the reconstructed frame is obtained with reference to side information to extend from low-frequency components to high-frequency components, this process can also be called non-blind BWE (Blind Image Wrapper).

[0050] In this embodiment, the first machine learning model can refer to the neural networks, deep neural networks, convolutional neural networks, recurrent neural networks, etc. mentioned above.

[0051] Optionally, the reconstructed frames are the direct output of the first machine learning model.

[0052] In this embodiment, the non-blind BWE can employ a machine learning model (first machine learning model) to directly output reconstructed frames based on the decoded low-frequency components (first audio signal) under the guidance of side information. Furthermore, the first machine learning model can perform encoding / decoding distortion correction for low-frequency components and possible audio rendering; this embodiment does not specifically limit its application in these aspects.

[0053] Optionally, the reconstructed frame is obtained based on the second audio signal output by the first machine learning model and the first audio signal.

[0054] In this embodiment, the non-blind BWE can employ a machine learning model (first machine learning model) that, guided by side information, outputs reconstructed high-frequency components based on the decoded low-frequency components (first audio signal). The reconstructed high-frequency components are then merged or added to the decoded low-frequency components to obtain the reconstructed audio. Furthermore, the first machine learning model can perform encoding / decoding distortion correction for low-frequency components and possible audio rendering; this embodiment does not specifically limit these aspects.

[0055] In some possible implementations, the first machine learning model may include a first sub-model and a second sub-model. The non-blind BWE may input the first audio signal and the first side information into the first sub-model to obtain the third audio signal; then input the third audio signal and the second side information into the second sub-model to obtain the reconstructed frame; the first side information and the second side information are part or all of the information in the side information.

[0056] In some possible implementations, a non-blind BWE can process the first audio signal and the third side information according to a pre-set first rule to obtain a fourth audio signal; then input the fourth audio signal and the fourth side information into a first machine learning model to obtain a reconstructed frame; the third side information and the fourth side information are part or all of the information in the side information.

[0057] In some possible implementations, a non-blind BWE can obtain a fifth audio signal based on a first machine learning model that has been pre-trained, using the first audio signal and the fifth side information; then, based on the fifth audio signal and the sixth side information, it can be processed according to a pre-set second rule to obtain a reconstructed frame; the fifth side information and the sixth side information are part or all of the side information.

[0058] It should be noted that multiple edge information can be part or all of the edge information, and multiple edge information can be completely the same or completely different, or partially the same and partially different. There are no specific restrictions on this.

[0059] Thirdly, this application provides an encoding apparatus, comprising: an acquisition module for acquiring a first bitstream, the first bitstream being obtained based on a first audio signal, the first audio signal being a low-frequency signal in the current frame with a frequency lower than a preset threshold; an extraction module for acquiring side information, the side information being obtained based on a second audio signal, the second audio signal including a high-frequency signal in the current frame with a frequency higher than or equal to the preset threshold; an encoding module for acquiring a second bitstream based on the side information; and a transmission module for transmitting a bitstream, the bitstream including the first bitstream and the second bitstream.

[0060] In one possible implementation, the extraction module is specifically used to input the second audio signal into a pre-trained second machine learning model to obtain the side information, wherein the second machine learning model includes at least one sub-model.

[0061] In one possible implementation, when the second machine learning model includes multiple sub-models, the input of each of the multiple sub-models includes a portion or all of the audio signal in the second audio signal, wherein the portion of the audio signal includes the high-frequency signal.

[0062] In one possible implementation, the first sub-model includes the same neurons as at least one other sub-model among the plurality of sub-models. The same neurons are located in the input phase, output phase, or intermediate phase of the first sub-model, and the first sub-model is any one of the plurality of sub-models.

[0063] In one possible implementation, the edge information includes feature information output by the plurality of sub-models respectively.

[0064] In one possible implementation, when the second machine learning model includes a sub-model, the edge information includes feature information output by the sub-model.

[0065] In one possible implementation, the extraction module is specifically used to process the second audio signal according to a pre-set rule to obtain the side information, wherein the pre-set rule includes at least one sub-rule.

[0066] In one possible implementation, when the pre-set rule method includes multiple sub-rule methods, the input of each of the multiple sub-rule methods includes a portion or all of the audio signal in the second audio signal, wherein the portion of the audio signal includes the high-frequency signal.

[0067] In one possible implementation, the edge information includes feature information output according to the plurality of sub-rules.

[0068] In one possible implementation, when the pre-set rule method includes a sub-rule method, the edge information includes feature information output according to the sub-rule method.

[0069] In one possible implementation, the input to the first sub-model further includes a first input; when the second machine learning model includes multiple sub-models, the first input includes the output of at least one other sub-model among the multiple sub-models, or the first input includes the output of the at least one sub-rule; the first sub-model is any one of the multiple sub-models.

[0070] In one possible implementation, the input of the first sub-model further includes a second input, which is an audio signal obtained after the first input has undergone redundant information processing; when the second machine learning model includes multiple sub-models, the first input includes the output of at least one other sub-model among the multiple sub-models, or the first input includes the output of the at least one sub-rule; the first sub-model is any one of the multiple sub-models.

[0071] In one possible implementation, the input of the first sub-rule method further includes a third input; when the preset rule method includes multiple sub-rule methods, the third input includes the output of at least one other sub-rule method among the multiple sub-rule methods, or the third input includes the output of the at least one sub-model; the first sub-rule method is any one of the multiple sub-rule methods.

[0072] In one possible implementation, the input of the first sub-rule method further includes a fourth input, which is an audio signal obtained after the third input has undergone redundant information processing; when the preset rule method includes multiple sub-rule methods, the third input includes the output of at least one other sub-rule method among the multiple sub-rule methods, or the third input includes the output of the at least one sub-model; the first sub-rule method is any one of the multiple sub-rule methods.

[0073] In one possible implementation, it further includes a filtering module for obtaining the first audio signal from the current frame by means of low-pass filtering.

[0074] In one possible implementation, the second audio signal may further include some or all of the audio signals in the first audio signal.

[0075] Fourthly, this application provides a decoding apparatus, comprising: a receiving module for receiving a bitstream, the bitstream including a first bitstream and a second bitstream; an acquisition module for acquiring a first audio signal based on the first bitstream, wherein the frequency of the first audio signal is lower than a preset threshold; an extraction module for acquiring side information based on the second bitstream, the side information being used to characterize the features of a second audio signal, wherein the frequency of the second audio signal is higher than or equal to the preset threshold; and a reconstruction module for acquiring a reconstructed frame based on the first audio signal and the side information.

[0076] In one possible implementation, the reconstruction module is specifically used to obtain the reconstructed frame based on the first audio signal and the side information using a pre-trained first machine learning model.

[0077] In one possible implementation, the reconstructed frame is directly output by the first machine learning model; or, the reconstructed frame is obtained based on the second audio signal output by the first machine learning model and the first audio signal.

[0078] In one possible implementation, when the first machine learning model includes a first sub-model and a second sub-model, the reconstruction module is specifically used to input the first audio signal and the first side information into the first sub-model to obtain a third audio signal; and to input the third audio signal and the second side information into the second sub-model to obtain the reconstructed frame; wherein the first side information and the second side information are part or all of the side information.

[0079] In one possible implementation, the reconstruction module is specifically used to process the first audio signal and the third side information according to a pre-set first rule to obtain a fourth audio signal; input the fourth audio signal and the fourth side information into the first machine learning model to obtain the reconstructed frame; the third side information and the fourth side information are part or all of the side information respectively.

[0080] In one possible implementation, the reconstruction module is specifically used to obtain a fifth audio signal based on the first audio signal and the pentagonal information through a pre-trained first machine learning model; and to process the fifth audio signal and the sixth pentagonal information according to a pre-set second rule to obtain the reconstructed frame; wherein the fifth pentagonal information and the sixth pentagonal information are part or all of the information in the pentagonal information.

[0081] In one possible implementation, the extraction module is specifically used to parse the second bitstream to obtain the side information.

[0082] In one possible implementation, the extraction module is specifically used to parse the second bitstream to obtain the preceding information; and to perform side information processing on the preceding information to obtain the side information, wherein the side information processing includes inputting a pre-trained third machine learning model or processing according to a pre-set third rule.

[0083] Fifthly, this application provides an apparatus comprising: one or more processors; a memory for storing one or more programs; wherein when the one or more programs are executed by the one or more processors, the one or more processors perform the method as described in any one of the first or second aspects above.

[0084] Sixthly, this application provides a computer-readable storage medium including a computer program that, when executed on a computer, causes the computer to perform the method described in any one of the first to second aspects above.

[0085] In a seventh aspect, this application provides a computer program product comprising computer program code, which, when executed on a computer, causes the computer to perform the method described in any one of the first to second aspects. Attached Figure Description

[0086] Figure 1 shows a schematic diagram of an application scenario 100 according to an embodiment of this application;

[0087] Figure 2 shows a schematic diagram of another application scenario 200 of the present application embodiment;

[0088] Figure 3 shows a schematic diagram of another application scenario 300 of the present application embodiment;

[0089] Figure 4 shows a schematic diagram of another application scenario 400 of the present application embodiment;

[0090] Figure 5 is a structural schematic diagram of a terminal device 500 provided in this application;

[0091] Figure 6 is a structural schematic diagram of an access network device 600 provided in an embodiment of this application;

[0092] Figure 7 is a flowchart of the audio encoding method process 700 provided in the embodiment of this application;

[0093] Figure 8a is a schematic diagram of obtaining edge information according to an embodiment of this application;

[0094] Figure 8b is a schematic diagram of obtaining edge information according to an embodiment of this application;

[0095] Figure 8c is a schematic diagram of obtaining edge information according to an embodiment of this application;

[0096] Figure 8d is a schematic diagram of obtaining edge information according to an embodiment of this application;

[0097] Figure 9a is a schematic diagram of the same parts of two sub-models in an embodiment of this application;

[0098] Figure 9b is a schematic diagram of the same parts of two sub-models in an embodiment of this application;

[0099] Figure 9c is a schematic diagram of the same parts of two sub-models in an embodiment of this application;

[0100] Figure 10a is a schematic diagram of the sub-models connected in an embodiment of this application;

[0101] Figure 10b is a schematic diagram of the sub-models connected in an embodiment of this application;

[0102] Figure 11a is a schematic diagram of the sub-rule concatenation method according to an embodiment of this application;

[0103] Figure 11b is a schematic diagram of the sub-rule concatenation method according to an embodiment of this application;

[0104] Figure 12 is a schematic diagram of joint training according to an embodiment of this application;

[0105] Figure 13 is a flowchart of process 1300 of the audio decoding method provided in an embodiment of this application;

[0106] Figure 14a is a schematic diagram of a non-blind BWE according to an embodiment of this application;

[0107] Figure 14b is a schematic diagram of a non-blind BWE according to an embodiment of this application;

[0108] Figure 14c is a schematic diagram of a non-blind BWE according to an embodiment of this application;

[0109] Figure 14d is a schematic diagram of a non-blind BWE according to an embodiment of this application;

[0110] Figure 14e is a schematic diagram of a non-blind BWE according to an embodiment of this application;

[0111] Figure 14f is a schematic diagram of a non-blind BWE according to an embodiment of this application;

[0112] Figure 15a is a schematic diagram of obtaining the reconstructed frame according to an embodiment of this application;

[0113] Figure 15b is a schematic diagram of obtaining the reconstructed frame according to an embodiment of this application;

[0114] Figure 16 is a schematic diagram of the encoding and decoding framework of a non-blind BWE containing AI according to an embodiment of this application;

[0115] Figure 17 is a schematic diagram of the encoding and decoding framework of a non-blind BWE containing AI according to an embodiment of this application;

[0116] Figure 18 is a schematic diagram of the encoding and decoding framework of a non-blind BWE containing AI according to an embodiment of this application;

[0117] Figure 19 is a flowchart of the edge information extraction step in an embodiment of this application;

[0118] Figure 20 is a flowchart of the non-blind BWE process in an embodiment of this application;

[0119] Figures 21 and 22 are comparative diagrams of the audio spectra obtained by the three BWE methods.

[0120] Figure 23 is a schematic diagram of the encoding and decoding framework of a non-blind BWE containing AI according to an embodiment of this application;

[0121] Figure 24 is a flowchart of the edge information extraction step in an embodiment of this application;

[0122] Figure 25 is a flowchart of the non-blind BWE process in an embodiment of this application;

[0123] Figure 26 is a schematic diagram of the structure of the encoding device 2600 according to an embodiment of this application;

[0124] Figure 27 is a schematic diagram of the structure of the decoding device 2700 according to an embodiment of this application. Detailed Implementation

[0125] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0126] The terms "first," "second," etc., used in the specification, embodiments, claims, and drawings of this application are for distinguishing purposes only and should not be construed as indicating or implying relative importance or order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, such as including a series of steps or units. A method, system, product, or apparatus is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or apparatuses.

[0127] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0128] Explanation of relevant terms used in this application:

[0129] Audio frame: Audio data is streamed. In practical applications, in order to facilitate audio processing and transmission, the amount of audio data within a certain duration is usually taken as an audio frame. This duration is called the "sampling time". Its value can be determined according to the codec and the specific application requirements. For example, this duration is 2.5ms to 60ms, where ms stands for millisecond.

[0130] Audio signals: Audio signals are information carriers of the frequency and amplitude variations of sound waves, containing speech, music, and sound effects. Audio is a continuously changing analog signal, which can be represented by a continuous curve called a sound wave. Audio signals are digital signals generated by analog-to-digital conversion or computers. Sound waves have three important parameters: frequency, amplitude, and phase, which determine the characteristics of audio signals.

[0131] Furthermore, since the embodiments of this application involve the application of artificial intelligence (AI), for ease of understanding, some related terms or concepts used in the embodiments of this application will be explained below.

[0132] (1) Neural Network

[0133] Neural Networks (NNs) are machine learning models. A neural network can be composed of neural units, which are computational units that take xs and an intercept of 1 as input. The output of such a computational unit can be:

[0134] Where s = 1, 2, ..., n, n is a natural number greater than 1, W s For x s The weights are denoted by b, where b is the bias of the neural unit. f is the activation function of the neural unit, used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into an output signal. The output signal of this activation function can be used as the input to the next convolutional layer. The activation function can be the sigmoid function. A neural network is a network formed by connecting many of the above-mentioned individual neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, which can be a region composed of several neural units.

[0135] (2) Deep Neural Networks

[0136] Deep neural networks (DNNs), also known as multilayer neural networks, can be understood as neural networks with many hidden layers, though there's no specific metric for "many." DNNs can be categorized into three layers based on their position: input layers, hidden layers, and output layers. Generally, the first layer is the input layer, the last layer is the output layer, and the layers in between are hidden layers. All layers are fully connected, meaning that any neuron in the i-th layer is connected to any neuron in the (i+1)-th layer. Although DNNs appear complex, the operation of each layer is actually quite simple, resembling a linear relationship as follows: in, It is the input vector. It is the output vector. α is the offset vector, W is the weight matrix (also called coefficients), and α() is the activation function. Each layer is simply an adjustment of the input vector. The output vector is obtained through such a simple operation. Because DNNs have many layers, the coefficients W and the offset vector... The number of these parameters is therefore quite large. The definitions of these parameters in a DNN are as follows: Taking the coefficient W as an example: Assuming a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as... The superscript 3 represents the layer number where coefficient W resides, while the subscript corresponds to the output third layer index 2 and the input second layer index 4. In summary, the coefficients from the k-th neuron in layer L-1 to the j-th neuron in layer L are defined as follows: It's important to note that the input layer does not have a W parameter. In deep neural networks, more hidden layers allow the network to better represent complex real-world situations. Theoretically, the more parameters a model has, the higher its complexity and "capacity," meaning it can perform more complex learning tasks. Training a deep neural network is essentially the process of learning the weight matrix, with the ultimate goal of obtaining the weight matrix of all layers in the trained deep neural network (a weight matrix formed by the vectors W from many layers).

[0137] (3) Convolutional Neural Network

[0138] A convolutional neural network (CNN) is a deep neural network with convolutional structures. It is a deep learning architecture, which refers to learning at multiple levels of abstraction using machine learning algorithms. As a deep learning architecture, CNN is a feed-forward artificial neural network, where each neuron responds to an input image. A CNN contains a feature extractor consisting of convolutional layers and pooling layers. This feature extractor can be viewed as a filter, and the convolution process can be seen as performing convolution with a trainable filter and an input image or a convolutional feature map.

[0139] A convolutional layer is a layer of neurons in a convolutional neural network that performs convolution processing on the input signal. A convolutional layer can contain multiple convolution operators, also called kernels. In image processing, these operators act as filters, extracting specific information from the input image matrix. Essentially, a convolution operator can be a weight matrix, which is usually predefined. During the convolution operation, the weight matrix typically processes the input image pixel by pixel (or two pixels by two pixels, depending on the stride) along the horizontal direction, thus extracting specific features from the image. The size of the weight matrix should be related to the image size. It's important to note that the depth dimension of the weight matrix is ​​the same as the depth dimension of the input image; during convolution, the weight matrix extends to the entire depth of the input image. Therefore, convolution with a single weight matrix produces a single-depth convolutional output. However, in most cases, multiple weight matrices of the same size (rows × columns) are used instead of a single weight matrix. The outputs of each weight matrix are stacked to form the depth dimension of the convolutional image. This dimension can be understood as being determined by the "multiple" factors mentioned above. Different weight matrices can be used to extract different features from the image. For example, one weight matrix can be used to extract edge information, another to extract specific colors, and yet another to blur unwanted noise. These multiple weight matrices have the same size (rows × columns), and the feature maps extracted by these weight matrices also have the same size. These extracted feature maps are then merged to form the output of the convolution operation. The weight values ​​in these weight matrices need to be obtained through extensive training in practical applications. The weight matrices formed by these trained weight values ​​can be used to extract information from the input image, enabling the convolutional neural network to make correct predictions. When a convolutional neural network has multiple convolutional layers, the initial convolutional layers often extract more general features, which can also be called low-level features. As the depth of the convolutional neural network increases, the features extracted by later convolutional layers become increasingly complex, such as high-level semantic features. Features with higher semantic levels are more suitable for the problem being solved.

[0140] Because it's often necessary to reduce the number of training parameters, pooling layers are frequently introduced periodically after convolutional layers. This can be a single convolutional layer followed by a pooling layer, or multiple convolutional layers followed by one or more pooling layers. In image processing, the sole purpose of pooling layers is to reduce the spatial size of the image. Pooling layers can include average pooling and / or max pooling operators to sample the input image to obtain a smaller image size. Average pooling calculates the average value of pixel values ​​within a specific range as the result of average pooling. Max pooling takes the pixel with the largest value within a specific range as the result of max pooling. Furthermore, just as the size of the weight matrix in a convolutional layer should be related to the image size, the operators in a pooling layer should also be related to the image size. The size of the output image after pooling can be smaller than the size of the input image of the pooling layer. Each pixel in the output image represents the average or maximum value of the corresponding sub-region of the input image of the pooling layer.

[0141] After processing by convolutional / pooling layers, a convolutional neural network (CNN) is still insufficient to output the required information. As mentioned earlier, convolutional / pooling layers only extract features and reduce the parameters introduced by the input image. However, to generate the final output information (the required class information or other relevant information), the CNN needs to utilize neural network layers to generate one or a set of required class numbers of outputs. Therefore, the neural network can include multiple hidden layers, the parameters of which can be pre-trained based on training data relevant to a specific task type, such as image recognition, image classification, image super-resolution reconstruction, etc.

[0142] Optionally, after the multiple hidden layers in the neural network, there is also an output layer of the entire convolutional neural network. This output layer has a loss function similar to the classification cross-entropy, which is specifically used to calculate the prediction error. Once the forward propagation of the entire convolutional neural network is completed, the backpropagation will begin to update the weight values ​​and biases of the aforementioned layers to reduce the loss of the convolutional neural network and the error between the result output by the convolutional neural network through the output layer and the ideal result.

[0143] (4) Recurrent Neural Network

[0144] Recurrent neural networks (RNNs) are used to process sequential data. In traditional neural network models, the layers from the input layer to the hidden layer and then to the output layer are fully connected, but the nodes within each layer are unconnected. While this type of neural network has solved many difficult problems, it remains inadequate for many others. For example, predicting the next word in a sentence generally requires using the preceding words because words in a sentence are not independent. RNNs are called recurrent neural networks because the current output of a sequence is related to the outputs of previous sequences. Specifically, the network memorizes previous information and applies it to the calculation of the current output; that is, nodes within the same hidden layer are no longer unconnected but connected, and the input to a hidden layer includes not only the output of the input layer but also the output of the hidden layer at the previous time step. Theoretically, RNNs can process sequential data of any length. Training an RNN is similar to training a traditional CNN or DNN. This algorithm also uses the backpropagation algorithm, but with one key difference: when an RNN is expanded, its parameters, such as W, are shared; however, this is not the case with traditional neural networks as illustrated above. Furthermore, in gradient descent, the output at each step depends not only on the network at the current step but also on the states of the network in previous steps. This learning algorithm is called Backpropagation Through Time (BPTT).

[0145] Since we already have convolutional neural networks (CNNs), why do we need recurrent neural networks (RNNs)? The reason is simple. CNNs rely on the fundamental assumption that elements are independent of each other, and that input and output are also independent—like a cat and a dog. However, in the real world, many elements are interconnected. For example, stock prices fluctuate over time. Or, imagine someone saying, "I love traveling, and my favorite place is Yunnan. I definitely want to go there someday." Humans know the answer to this question is "Yunnan." Humans can infer from context. But how can machines do this? That's where RNNs come in. RNNs aim to give machines the ability to remember, just like humans. Therefore, the output of an RNN depends on both the current input information and historical memory information.

[0146] (5) Loss Function

[0147] In training a deep neural network, to ensure the output closely approximates the desired predicted value, we compare the network's prediction with the target value. Based on the difference, we update the weight vector of each layer (usually pre-configuring parameters before the initial update). For example, if the prediction is too high, the weight vector is adjusted to predict a lower value. This adjustment continues until the deep neural network predicts the target value or a value very close to it. Therefore, we need to predefine "how to compare the difference between the predicted and target values," which is the loss function or objective function. These are important equations used to measure the difference between the predicted and target values. Taking the loss function as an example, a higher output value (loss) indicates a greater difference, and training the deep neural network becomes a process of minimizing this loss.

[0148] (6) Backpropagation algorithm

[0149] Convolutional neural networks can employ backpropagation (BP) to correct the parameters in the initial super-resolution model during training, thereby reducing the reconstruction error loss. Specifically, forward propagation of the input signal to the output generates an error loss; this error loss information is then propagated back to update the parameters in the initial super-resolution model, leading to convergence of the error loss. The backpropagation algorithm is an error-loss-driven backpropagation process aimed at obtaining the optimal parameters of the super-resolution model, such as the weight matrix.

[0150] (7) Generative Adversarial Networks

[0151] Generative adversarial networks (GANs) are a type of deep learning model. This model comprises at least two modules: a generative model and a discriminative model. These two modules learn from each other through a game-like interaction, resulting in better outputs. Both the generative and discriminative models can be neural networks, specifically deep neural networks or convolutional neural networks. The basic principle of GANs is as follows: Taking an image-generating GAN as an example, suppose there are two networks, G (Generator) and D (Discriminator). G is a network that generates images by receiving random noise z and using this noise, denoted as G(z). D is a discriminative network used to determine whether an image is "real." Its input parameter is x, representing an image, and its output D(x) represents the probability that x is a real image. A value of 1 indicates that the image is 100% real, while a value of 0 indicates that the image is impossible to be real. During the training of this generative adversarial network (GAN), the goal of the generative network G is to generate realistic images to deceive the discriminator network D, while the goal of the discriminator network D is to distinguish the images generated by G from real images as much as possible. Thus, G and D constitute a dynamic "game," which is the "adversarial" aspect of the GAN. Ideally, the game will result in G generating images G(z) that are sufficiently realistic, while D struggles to determine whether the images generated by G are real or not, i.e., D(G(z)) = 0.5. This yields a superior generative model G that can be used to generate images.

[0152] Based on this, the technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings.

[0153] The embodiments of this application can be applied to audio codecs in terminal devices, wireless access network devices, core network devices, etc.

[0154] Figure 1 illustrates a schematic diagram of an application scenario 100 according to an embodiment of this application. Figure 1 shows a schematic diagram of the system architecture of this application embodiment applied on the terminal device side. As shown in Figure 1, it includes a first terminal device 110, a second terminal device 120, a wireless or wired network communication device 130, and a wireless or wired network communication device 140. The first terminal device 110 and the second terminal device 120 can be either transmitting or receiving devices. As an example, the explanation will focus on the first terminal device 110 as the transmitting device and the second terminal device 120 as the receiving device.

[0155] In audio communication, the audio acquisition module in the first terminal device 110 is used to acquire audio, the audio encoder encodes the acquired audio signal, the channel coding module performs channel coding to obtain the bit stream, and then the signal is transmitted in the digital channel through the wireless or wired network communication device 130 on the transmitting end side.

[0156] The wireless or wired network communication device 140 on the receiving end side obtains the signal sent by the first terminal device 110 through the digital channel and transmits the signal to the second terminal device 120. The second terminal device 120 performs channel decoding in the channel decoding module according to the received signal, and then decodes the audio signal through the audio decoder. Then, it performs audio playback in the audio playback module according to the decoded audio signal.

[0157] It should be understood that when the second terminal device 120 is the transmitting device and the first terminal device 110 is the receiving device, it can be understood as the first terminal device 110 being the transmitting device and the second terminal device 120 being the receiving device, which will not be elaborated here.

[0158] It should be understood that the wireless or wired network communication device 130 and the wireless or wired network communication device 140 may also be core network devices.

[0159] Figure 2 illustrates another application scenario 200 of this application embodiment. Figure 2 shows a system architecture diagram of the transcoding application of this application embodiment in a wireless access network device or core network device. As shown in Figure 2, the wireless access network device or core network device includes a channel decoding module, other audio decoders, an audio encoder, and a channel coding module. During transcoding, corresponding audio encoding and decoding processing is required. The wireless access network device or core network device performs channel decoding on the received signal in the channel decoding module, then decodes the audio bitstream through other audio decoders to obtain the audio bitstream, the audio encoder re-encodes the audio bitstream, and then performs channel coding to transmit the audio signal.

[0160] Figure 3 illustrates another application scenario 300 of this application embodiment. Figure 3 shows a system architecture diagram of the transcoding application of this application embodiment in a wireless access network device or core network device. As shown in Figure 3, the wireless access network device or core network device includes a channel decoding module, an audio decoder, other audio encoders, and a channel coding module. During transcoding, corresponding audio encoding and decoding processing is required. The wireless access network device or core network device performs channel decoding on the received signal in the channel decoding module, then decodes the audio bitstream through the audio decoder to obtain the audio bitstream. Other audio encoders re-encode the audio bitstream, and then channel coding is performed again to transmit the audio signal.

[0161] This application embodiment can also be applied to the audio encoding and audio decoding modules in virtual reality (VR) streaming services. Figure 4 shows a schematic diagram of another application scenario 400 of this application embodiment, as shown in the dashed box in Figure 4.

[0162] The end-to-end audio signal processing flow is as follows: At the transmitting end, the audio and video signals are processed by the acquisition module, resulting in separate audio and video signals. The audio signal undergoes preprocessing, which includes filtering out low-frequency components, typically using 20Hz or 50Hz as a dividing point, extracting location information, and then performing audio encoding. The video signal undergoes visual stitching, projection, and mapping, followed by video and image encoding. The audio, video, and image streams are then packaged (file / segment encapsulation) and delivered to the decoding end.

[0163] The decoding end first performs file / segment decapsulation, followed by audio decoding, video decoding, and image decoding. The decoded audio signal undergoes audio rendering, and the rendered signal is mapped onto the listener's headphones, which can be standalone headphones or headphones from glasses devices such as HTC VIVE. The decoded video and image signals undergo video rendering, and the rendered signal is mapped onto the display.

[0164] The terminal equipment in this application embodiment can also be referred to as: user equipment (UE), mobile station (MS), mobile terminal (MT), access terminal, user unit, user station, mobile station, mobile station, remote station, remote terminal, mobile device, user terminal, terminal, wireless communication equipment, user agent, or user device, etc.

[0165] Terminal devices can be wireless or wired. Wireless terminals can be devices that provide voice and / or other service data connectivity to users, handheld devices with wireless connectivity, or other processing devices connected to a wireless modem. Wireless terminals can communicate with one or more core networks via a Radio Access Network (RAN). Wireless terminals can be mobile terminals, such as mobile phones (or "cellular" phones) and computers with mobile terminals, for example, portable, pocket-sized, handheld, computer-embedded, or vehicle-mounted mobile devices that exchange voice and / or data with the RAN. Examples include Personal Communication Service (PCS) phones, cordless phones, Session Initiation Protocol (SIP) phones, Wireless Local Loop (WLL) stations, and Personal Digital Assistants (PDAs).Wireless terminals can also be referred to as systems, subscriber units, subscriber stations, mobile stations, mobile devices, remote stations, remote terminals, access terminals, user terminals, user agents, user devices or user equipment, mobile internet devices (MID), wearable devices, virtual reality (VR) devices, augmented reality (AR) devices, wireless terminals in industrial control, wireless terminals in self-driving, wireless terminals in remote medical surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, wireless terminals in smart homes, in-vehicle equipment, wearable devices, terminal equipment in 5G networks, or future evolved public land mobile communication networks. Terminal devices in a network (PLMN), etc., are not limited to this in the embodiments of this application.

[0166] As an example and not a limitation, in this application embodiment, wearable devices can also be called wearable smart devices. This is a general term for devices that utilize wearable technology to intelligently design and develop everyday wearables, such as glasses, gloves, watches, clothing, and shoes. Wearable devices are portable devices that are worn directly on the body or integrated into the user's clothing or accessories. Wearable devices are not merely hardware devices; they achieve powerful functions through software support, data interaction, and cloud interaction. Broadly defined, wearable smart devices include those that are feature-rich, large in size, and can achieve complete or partial functionality without relying on a smartphone, such as smartwatches or smart glasses, as well as those that focus on a specific application function and require use with other devices such as smartphones, such as various smart bracelets and smart jewelry for vital sign monitoring.

[0167] Furthermore, in this embodiment of the application, the terminal device can also be a terminal device in an Internet of Things (IoT) system. IoT is an important component of the future development of information technology. Its main technical feature is to connect objects to the network through communication technology, thereby realizing an intelligent network of human-machine interconnection and object-to-object interconnection.

[0168] The various terminal devices described above, if located in a vehicle (e.g., placed inside or installed inside a vehicle), can be considered as vehicle-mounted terminal devices, also known as on-board units (OBUs).

[0169] In this embodiment, the terminal device may further include a relay. Alternatively, it can be understood that anything capable of data communication with a base station can be considered a terminal device.

[0170] For example, Figure 5 is a schematic diagram of the structure of a terminal device 500 provided in this application. The terminal device 500 can perform the actions performed by the terminal device in the above embodiments.

[0171] For ease of explanation, Figure 5 only shows the main components of the terminal device. As shown in Figure 5, the terminal device 500 includes a processor, memory, control circuitry, antenna, and input / output devices.

[0172] The processor is primarily used to process communication protocols and data, control the entire terminal device, execute software programs, and process the data within those programs. For example, it supports the terminal device in performing the actions described in the embodiments of the transmission precoding matrix instruction method. The memory is primarily used to store software programs and data, such as the codebook described in the embodiments above. The control circuit is primarily used for converting baseband signals to radio frequency signals and processing radio frequency signals. The control circuit and antenna together can also be called a transceiver, primarily used for transmitting and receiving radio frequency signals in the form of electromagnetic waves. Input / output devices, such as touchscreens, displays, and keyboards, are primarily used to receive user input data and output data to the user.

[0173] When the terminal device is powered on, the processor can read the software program from the storage unit, interpret and execute the software program's instructions, and process the software program's data. When data needs to be transmitted wirelessly, the processor performs baseband processing on the data to be transmitted and outputs the baseband signal to the radio frequency (RF) circuit. The RF circuit then processes the baseband signal and transmits the RF signal outward as electromagnetic waves through the antenna. When data is sent to the terminal device, the RF circuit receives the RF signal through the antenna, converts the RF signal into a baseband signal, and outputs the baseband signal to the processor. The processor converts the baseband signal back into data and processes the data.

[0174] Those skilled in the art will understand that, for ease of explanation, Figure 5 only shows one memory and processor. In actual terminal devices, multiple processors and memories may exist. Memory may also be referred to as storage medium or storage device, etc., and the embodiments of this application do not limit this.

[0175] For example, a processor may include a baseband processor and a central processing unit (CPU). The baseband processor is mainly used for processing communication protocols and communication data, while the CPU is mainly used for controlling the entire terminal device, executing software programs, and processing the data of the software programs. The processor in Figure 5 integrates the functions of a baseband processor and a CPU. Those skilled in the art will understand that the baseband processor and CPU can also be independent processors interconnected via technologies such as buses. Those skilled in the art will understand that a terminal device may include multiple baseband processors to adapt to different network standards, and a terminal device may include multiple CPUs to enhance its processing capabilities. The various components of the terminal device can be connected via various buses. The baseband processor can also be described as a baseband processing circuit or a baseband processing chip. The CPU can also be described as a central processing circuit or a central processing chip. The function of processing communication protocols and communication data can be built into the processor or stored in the storage unit as a software program, which is then executed by the processor to implement the baseband processing function.

[0176] For example, in this embodiment, the antenna and control circuit with transceiver functions can be regarded as the transceiver unit 510 of the terminal device 500, and the processor with processing functions can be regarded as the processing unit 520 of the terminal device 500. The processing unit 520 can also implement the function of the acquisition unit. As shown in FIG5, the terminal device 500 includes the transceiver unit 510 and the processing unit 520. The transceiver unit can also be called a transceiver, transceiver device, transceiver apparatus, etc. Optionally, the device in the transceiver unit 510 used to implement the receiving function can be regarded as the receiving unit, and the device in the transceiver unit 510 used to implement the transmitting function can be regarded as the transmitting unit, that is, the transceiver unit includes the receiving unit and the transmitting unit. For example, the receiving unit can also be called a receiver, receiver circuit, etc., and the transmitting unit can be called a transmitter, transmitter, or transmitting circuit, etc.

[0177] The access network device in this application embodiment can be a device for communicating with terminal devices, such as a base station, access point, or network device, or it can refer to a device in the access network that communicates with a wireless terminal through one or more sectors on the air interface. The network device can be used to convert received air frames and IP packets to each other, acting as a router between the wireless terminal and the rest of the access network, which may include an Internet Protocol (IP) network. The network device can also coordinate the attribute management of the air interface. For example, the access network device can be a base station (BTS) in Global System for Mobile communication (GSM) or Code Division Multiple Access (CDMA), a base station (NodeB, NB) in Wideband Code Division Multiple Access (WCDMA), an evolved base station (eNB or eNodeB) in an LTE system, a radio controller in a cloud radio access network (CRAN) scenario, or the access device can be a relay station, access point, vehicle-mounted equipment, wearable device, or access equipment in a 5G network or a network device in a future evolved PLMN network, etc. It can be an access point (AP) in a WLAN, or a gNB in ​​a new radio (NR) system. This application embodiment is not limited. It should be noted that for a 5G system, under a single base station, there may be one or more Transmission Reception Points (TRPs). All TRPs belong to the same cell, and each TRP and terminal can use the measurement reporting method described in the embodiments of this application. In another scenario, network devices can also be divided into Control Units (CUs) and Data Units (DUs). Under a CU, there can be multiple DUs, and each DU and terminal can use the measurement reporting method described in the embodiments of this application. The difference between the CU-DU separation scenario and the multi-TRP scenario is that a TRP is only a radio frequency unit or an antenna device, while a DU can implement protocol stack functions, such as physical layer functions.

[0178] Furthermore, in the embodiments of this application, the access network device is a device in the radio access network (RAN), or in other words, a RAN node that connects the terminal device to the wireless network. For example, by way of example and not limitation, the following can be listed as access network devices: gNB, transmission reception point (TRP), evolved Node B (eNB), radio network controller (RNC), Node B (NB), base station controller (BSC), base transceiver station (BTS), home base station (e.g., home evolved Node B, or home Node B, HNB), base band unit (BBU), or wireless fidelity (Wi-Fi) access point (AP), etc.

[0179] Access network equipment provides services to cells. Terminal devices communicate with access network equipment through the transmission resources (e.g., frequency domain resources, or spectrum resources) used by the cell. The cell can be the cell corresponding to the access network equipment (e.g., base station). The cell can belong to a macro base station or the base station corresponding to a small cell. The small cells here can include: metro cells, micro cells, pico cells, femto cells, etc. These small cells have the characteristics of small coverage area and low transmission power, and are suitable for providing high-speed data transmission services.

[0180] For example, Figure 6 is a schematic diagram of the structure of an access network device 600 provided in an embodiment of this application, which can be used to implement the functions of the wireless access network device in the above embodiments. The access network device 600 includes one or more radio frequency units, such as a remote radio unit (RRU) 66 and one or more baseband units (BBU) (also called digital units, DU) 620. The RRU 66 can be called a transceiver unit, transceiver, transceiver circuit, or transceiver, etc., and it may include at least one antenna 611 and a radio frequency unit 612. The RRU 66 is mainly used for transmitting and receiving radio frequency signals and converting radio frequency signals to baseband signals, for example, for sending signaling messages as described in the above embodiments to terminal devices. The BBU 620 is mainly used for baseband processing and controlling the base station, etc. The RRU 66 and BBU 620 can be physically set together or physically separated, i.e., a distributed base station.

[0181] The BBU620 is the control center of the base station, also known as a processing unit, and is mainly used to complete baseband processing functions, such as channel coding, multiplexing, modulation, spreading, etc. For example, the BBU (processing unit) 620 can be used to control the access network equipment to execute the operation procedures of the access network equipment in the above method embodiments.

[0182] In one example, the BBU620 can be composed of one or more boards. Multiple boards can collectively support a single access standard wireless access network (such as an LTE system, a 5G system, or a future wireless access network system, etc.), or they can each support wireless access networks with different access standards. The BBU620 also includes a memory 621 and a processor 622. The memory 621 is used to store necessary instructions and data. For example, the memory 621 stores the codebook as described in the above embodiments. The processor 622 is used to control the base station to perform necessary actions, such as controlling the base station to execute the operation procedures related to the network device in the above method embodiments. The memory 621 and the processor 622 can serve one or more boards. That is, each board can have its own memory and processor, or multiple boards can share the same memory and processor. Furthermore, each board can also have necessary circuitry.

[0183] In one possible implementation, with the development of system-on-chip (SoC) technology, all or part of the functions of the 620 and 66 parts can be implemented by SoC technology, for example, by a base station function chip. This base station function chip integrates a processor, memory, antenna interface, and other devices. The program for the base station-related functions is stored in the memory, and the processor executes the program to implement the relevant functions of the base station. Optionally, the base station function chip can also read external memory to implement the relevant functions of the base station.

[0184] It should be understood that the structure of the access network device illustrated in Figure 6 is only one possible configuration and should not be construed as limiting the embodiments of this application. This application does not exclude the possibility of other base station structures in the future.

[0185] It should be understood that in the embodiments of this application, the processor can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0186] It should also be understood that the memory in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0187] Core network equipment can be core network elements, such as Access and Mobility Management Function (AMF) entities, Session Management Function (SMF) entities, User Plane Function (UPF) entities, and Policy Control Function (PCF) entities. The AMF entity provides mobility management functions in the core network, primarily responsible for access and mobility control, including registration management (RM) and connection management (CM), access authentication and authorization, reachability management, and mobility management. The SMF entity is the session management function in the core network. In addition to performing mobility management on terminal devices, the AMF entity is also responsible for forwarding session management-related messages between the terminal devices and the SMF entity. The PCF entity is the policy management function in the core network, responsible for formulating policies related to mobility management, session management, and charging for terminal devices. UPF entities are user plane functions in the core network. They transmit data with external data networks and access network devices through interfaces. They mainly provide user plane support, including PDU sessions and connection points to data networks, packet routing and forwarding, packet inspection and user plane policy enforcement, QoS processing for user plane, downlink packet buffering, and triggering downlink data notifications.

[0188] It should be understood that the aforementioned core network functional units can work independently or be combined to implement certain control functions. For example, AMF, SMF, and PCF can be combined as management devices to perform access control and mobility management functions such as access authentication, security encryption, and location registration for terminal devices, as well as session management functions such as recording, releasing, and modifying user plane transmission paths, and functions for analyzing some slice-related data (such as congestion) and terminal device-related data. UPF, as a gateway device, mainly performs functions such as routing and forwarding user plane data, such as filtering data packets from terminal devices, data transmission / forwarding, rate control, and generating billing information.

[0189] The technical solutions of this application can be applied to various communication systems, such as: Global System for Mobile Communications (GSM) system, Code Division Multiple Access (CDMA) system, Wideband Code Division Multiple Access (WCDMA) system, General Packet Radio Service (GPRS), Long Term Evolution (LTE) system, LTE Frequency Division Duplex (FDD) system, LTE Time Division Duplex (TDD) system, Universal Mobile Telecommunication System (UMTS), Worldwide Interoperability for Microwave Access (WiMAX) system, 5th Generation (5G) system, or New Radio (NR). In addition, it can also be applied to subsequent evolution systems, such as 6G communication systems or even more advanced 7G communication systems.

[0190] In the core encoder, to reduce the coding bit rate, the correlation between signals from different frequency bands can be used for encoding. The basic principle of encoding using the correlation between signals from different frequency bands is to use low-frequency signals and generate high-frequency signals through bandwidth extension (BWE). Whether it's the latest 3GPP Enhanced Voice Services (EVS) audio codec, or the Moving Picture Experts Group High-Efficiency Advanced Audio Coding (MPEG HE-AAC) and Unified Speech and Audio Coding (USAC) audio codecs, they all utilize the correlation between signals from different frequency bands and employ bandwidth extension technology to encode high-frequency signals, thereby reducing the encoder's coding bit rate by encoding high-frequency signals with fewer bits. However, in real audio signals, the high-frequency spectrum often contains some spectra that are dissimilar to or weakly correlated with the low-frequency spectrum, leading to distortion in the reconstruction of the high-frequency signal.

[0191] To address the aforementioned technical problems, this application proposes an audio encoding / decoding method and apparatus.

[0192] It should be noted that the following method embodiments can be applied to the application scenarios shown in Figures 1-4, and of course, they can also be applied to other scenarios. This application embodiment does not impose any limitations on these applications. It should also be understood that in this application embodiment, the executing entity of the method can be a terminal device, an access network device, or a core network device. As an example and not a limitation, the executing entity of the method can also be a chip, chip system, or processor applied to the terminal device, access network device, and core network device. The terminal device, access network device, and core network device have encoding and decoding functions and can also be called encoding and decoding devices.

[0193] Figure 7 is a flowchart of process 700 of the audio encoding method provided in an embodiment of this application. Process 700 describes a series of steps or operations. It should be understood that process 700 can be executed in various orders and / or occur simultaneously, and is not limited to the execution order shown in Figure 7. Process 700 may include:

[0194] Step 701: Obtain the first bitstream, which is obtained based on the first audio signal.

[0195] The original audio is sampled to obtain an audio frame sequence, which includes multiple audio frames. At the encoding end, compression processing can be performed frame by frame. For ease of description, the audio frame currently being processed in this application embodiment is referred to as the current frame.

[0196] The characteristics of the audio signal contained in the current frame can be characterized by frequency, amplitude, and phase. In this embodiment, a first audio signal and a second audio signal are extracted from the current frame based on frequency. The first audio signal is an audio signal in the current frame with a frequency lower than a preset threshold (in this embodiment, an audio signal satisfying this condition can also be called a low-frequency signal or low-frequency component). The second audio signal includes audio signals in the current frame with a frequency higher than or equal to the preset threshold (in this embodiment, an audio signal satisfying this condition can also be called a high-frequency signal or high-frequency component). The aforementioned preset threshold can be, for example, 16kHz, and this embodiment does not specifically limit this. It should be noted that, while the audio signal with a frequency equal to the preset threshold in the current frame is categorized as the second audio signal, optionally, this embodiment can also categorize the audio signal with a frequency equal to the preset threshold in the current frame as the first audio signal, without specific limitation.

[0197] In some possible implementations, a low-pass filter can be used to extract the first audio signal from the current frame.

[0198] In some possible implementations, a core encoder may be used to encode the first audio signal. For example, the core encoder may be an AVS3 core encoder. This application does not specifically limit this implementation.

[0199] Step 702: Obtain edge information, which is obtained based on the second audio signal.

[0200] In this embodiment of the application, the second audio signal may exist in the following three situations:

[0201] 1) The second audio signal contains only high-frequency components. In this case, the second audio signal is equivalent to the high-frequency components of the current frame. Correspondingly, the side information can contain only the characteristic information of the high-frequency components.

[0202] 2) The second audio signal includes not only high-frequency components but also a portion of the low-frequency components. That is, the second audio signal and the first audio signal share some audio signals. In this case, the second audio signal is equivalent to the high-frequency components plus a portion of the low-frequency components of the current frame. Correspondingly, the side information, in addition to containing characteristic information of the high-frequency components, may also contain correlation information between the high-frequency components and the aforementioned portion of the low-frequency components.

[0203] 3) The second audio signal includes not only high-frequency components but also all low-frequency audio signals; that is, the second audio signal includes the first audio signal. In this case, the second audio signal is equivalent to the current frame. Correspondingly, the side information may include not only the feature information of the high-frequency components but also the correlation information between the high-frequency and low-frequency components.

[0204] This shows that side information can guide the decoding end to implement non-blind BWE with only a small number of transmission bits, in order to reconstruct high-frequency components, especially signals with complex mapping relationships between high and low frequencies.

[0205] In some possible implementations, obtaining side information may include inputting a second audio signal into a pre-trained second machine learning model to obtain side information. This second machine learning model includes at least one sub-model. Each sub-model may focus on different characteristics; for example, constraints may be set for each sub-model, including input / output, model structure, processing rules, etc., so that each sub-model can focus on extracting side information corresponding to its characteristic.

[0206] In this embodiment, the second machine learning model can refer to the neural networks, deep neural networks, convolutional neural networks, recurrent neural networks, etc., mentioned above. The machine learning model can incorporate more details of the high-frequency components in the current frame into the feature information. These details include, for example, the characteristics of the high-frequency components and the correlation between the high-frequency components and the low-frequency components, which helps to reconstruct more high-frequency details.

[0207] For example, Figure 8a is a schematic diagram of obtaining edge information according to an embodiment of this application. As shown in Figure 8a, the edge information is obtained through a second machine learning model. The second machine learning model includes a sub-model, that is, the second machine learning model is also the sub-model. Its input is the second audio signal and its output is the edge information.

[0208] In some possible implementations, the second machine learning model may include multiple sub-models, each of which takes into account an audio signal in the current frame with a frequency higher than or equal to a preset threshold as its input. In other words, the input of each of the multiple sub-models is part or all of the audio signal in the second audio signal.

[0209] For example, Figure 8b is a schematic diagram of obtaining edge information according to an embodiment of this application. As shown in Figure 8b, the edge information is obtained through a second machine learning model. The second machine learning model includes multiple sub-models (n sub-models are used as an example in Figure 8b, n>1), namely sub-model 1, sub-model 2, ..., sub-model n. The input of sub-model 1 is input 1 and the output is feature information 1. The input of sub-model 2 is input 2 and the output is feature information 2. ..., the input of sub-model n is input n and the output is feature information n.

[0210] Input 1, Input 2, ..., Input n all originate from the second audio signal, and they all share the common feature of including high-frequency components from the second audio signal. Furthermore, Input 1, Input 2, ..., Input n may or may not include low-frequency components from the second audio signal. When low-frequency components from the second audio signal are included, Input 1, Input 2, ..., Input n respectively include some or all of the low-frequency components from the second audio signal. For example, Input 1 includes high-frequency components from the second audio signal, Input 2 includes high-frequency components and some low-frequency components from the second audio signal, and Input 3 includes high-frequency components and all low-frequency components from the second audio signal. It should be noted that in the embodiments of this application, Input 1, Input 2, ..., Input n can be the same, completely different, or partially the same and partially different; no specific limitation is made in this regard.

[0211] In this embodiment, if the input of a sub-model contains only high-frequency components, the sub-model is mainly used to compress the high-frequency components into features, or to represent the high-frequency components through features; if the input of a sub-model contains low-frequency components in addition to high-frequency components, the sub-model is also used to extract the mapping relationship between high and low frequencies. Using multiple sub-models can yield more diverse features in the extraction of side information; therefore, this application does not specifically limit the number, type, or function of the sub-models.

[0212] The outputs of the above n sub-models, including feature information 1, feature information 2, ..., feature information n, are merged to obtain the edge information. That is, the edge information includes the feature information output by each of the n sub-models.

[0213] In some possible implementations, the first sub-model and at least one other sub-model among a plurality of sub-models include the same neurons, which are located in the input phase, output phase or intermediate phase of the first sub-model, which is any one of the plurality of sub-models.

[0214] The first sub-model may share neurons (also called common neurons) with one or more of the multiple sub-models included in the second machine learning model. These common neurons may exist in one or more layers, and these layers may be located in the input phase, output phase, or intermediate phase of the sub-model. The input phase refers to the first layer and one or more layers following the input of the sub-model; the output phase refers to the last layer and one or more layers preceding the output of the sub-model; and the intermediate phase refers to one or more layers excluding the first and last layers of the sub-model. It should be noted that this application does not specifically limit the number or type of the aforementioned common neurons, nor does it specifically limit the location of the input phase, output phase, or intermediate phase.

[0215] For example, Figure 9a is a schematic diagram of the same part of two sub-models in an embodiment of this application. As shown in Figure 9a, taking two sub-models as an example, the two sub-models have the same neurons (i.e., the same part), and each has different neurons (i.e., the different part).

[0216] The common parts are located in the input stage of sub-model 1 and sub-model 2. The difference part 1 of sub-model 1 and the difference part 2 of sub-model 2 are located after the aforementioned common parts, and feature information 1 and feature information 2 are output respectively.

[0217] For example, Figure 9b is a schematic diagram of the same part of two sub-models in an embodiment of this application. As shown in Figure 9b, taking two sub-models as an example, the two sub-models have the same neurons (i.e., the same part), and each has different neurons (i.e., the different part).

[0218] The common parts are located in the output stage of sub-model 1 and sub-model 2. The difference part 1 of sub-model 1 and the difference part 2 of sub-model 2 are located before the aforementioned common parts. The two sub-models output feature information 1 and feature information 2 respectively.

[0219] For example, Figure 9c is a schematic diagram of the same part of two sub-models in an embodiment of this application. As shown in Figure 9c, taking two sub-models as an example, the two sub-models have the same neurons (i.e., the same part), and each has different neurons (i.e., the different part).

[0220] The common parts are located in the input and output stages of sub-model 1 and sub-model 2, respectively. The difference part 1 of sub-model 1 and the difference part 2 of sub-model 2 are located between the two common parts. The two sub-models output feature information 1 and feature information 2 respectively.

[0221] It should be noted that the above two examples are illustrated using two sub-models, but this is not a limitation on the common parts among multiple sub-models. The embodiments of this application do not specifically limit the number of multiple sub-models with common parts, or the specific implementation of the common parts (including the position of the common parts, the layers included in the common parts, etc.).

[0222] In some possible implementations, the input to the first sub-model also includes a first input; when the second machine learning model includes multiple sub-models, the first input includes the output of at least one other sub-model among the multiple sub-models, or the first input includes the output of at least one sub-rule; the first sub-model is any one of the multiple sub-models.

[0223] For example, Figure 10a is a schematic diagram of the concatenation of sub-models according to an embodiment of this application. As shown in Figure 10a, taking two sub-models as an example, the output of sub-model 2 can be used as an input of sub-model 1, so sub-model 1 includes input 1 and the output of sub-model 2. In addition, the output of the sub-rule method can also be used as an input of sub-model 1, so sub-model 1 includes input 1 and the output of the sub-rule method, and even the output of sub-model 2 can be added, without specific limitation. This can be a concatenation of sub-models with sub-models or sub-rule methods.

[0224] In some possible implementations, the input of the first sub-model may also include a second input, which is an audio signal obtained after the first input has undergone redundant information processing; when the second machine learning model includes multiple sub-models, the first input may include the output of at least one other sub-model among the multiple sub-models, or the first input may include the output of at least one sub-rule; the first sub-model may be any one of the multiple sub-models.

[0225] For example, Figure 10b is a schematic diagram of the sub-model concatenation of an embodiment of this application. As shown in Figure 10b, taking two sub-models as an example, the output of sub-model 2 is first processed with redundant information and then can be used as an input to sub-model 1. Thus, sub-model 1 includes input 1 and the output of sub-model 2 after redundant information processing. Alternatively, the output of the sub-rule method can also be processed with redundant information and then used as an input to sub-model 1. Thus, sub-model 1 includes input 1 and the output of sub-rule method after redundant information processing. It is even possible to add the output of sub-model 2 after redundant information processing; no specific limitation is made. This can be a concatenation of sub-models or sub-rule methods.

[0226] It should be noted that the above two examples are illustrated using two sub-models, but this does not limit the chaining relationship between sub-models or between sub-models and sub-rule methods. In addition to its own input, the input of any sub-model may include the output of one or more other sub-models or the output after redundancy processing, and / or the output of one or more sub-rule methods or the output after redundancy processing. The embodiments of this application do not limit the number of chains or the chaining method.

[0227] In some possible implementations, obtaining the side information may also include processing the second audio signal according to a pre-set rule to obtain the side information, the pre-set rule including at least one sub-rule.

[0228] In this embodiment, the method for extracting features according to pre-set rules may include, for example, extracting the spectral envelope and extracting features by referring to AVS3. The spectral envelope is the energy of each frequency band in the spectrum of the current frame, used to recover the corresponding frequency band energy on the decoding side. The features extracted by AVS3 include frequency band energy and identifiers of harmonic characteristics and noise features of each frequency band, used to recover the BWE basis on the decoding side. Compared to machine learning methods, rule-based methods can preserve information such as spectral energy, thereby avoiding any impact on the listening experience.

[0229] In this embodiment of the application, based on the second machine learning model, a rule-based approach can be set to process the second audio signal according to the aforementioned rule-based approach to obtain the corresponding output (feature information).

[0230] For example, Figure 8c is a schematic diagram of obtaining edge information according to an embodiment of this application. As shown in Figure 8c, the edge information is obtained through a second machine learning model and a pre-set rule method. The second machine learning model includes a sub-model, that is, the second machine learning model is also the sub-model, whose input is input 1 and output is feature information 1. The rule method includes a sub-rule method, that is, the pre-set rule method is also the sub-rule method, whose input is input 2 and output is feature information 2.

[0231] Both input 1 and input 2 originate from the second audio signal, and they share the common feature of including high-frequency components from the second audio signal. Furthermore, input 1 and input 2 may or may not include low-frequency components from the second audio signal. When low-frequency components from the second audio signal are included, input 1 and input 2 respectively include some or all of the low-frequency components. For example, input 1 includes high-frequency components from the second audio signal, and input 2 includes high-frequency components and some low-frequency components from the second audio signal. It should be noted that in the embodiments of this application, input 1 and input 2 can be the same, completely different, or partially the same and partially different; no specific limitation is made in this regard.

[0232] The feature information 1 output by the second machine learning model and the feature information 2 output by the rule method are merged to obtain the edge information. That is, the edge information includes the feature information output by the second machine learning model and the feature information output by the rule method respectively.

[0233] In some possible implementations, the pre-set rule method may include multiple sub-rule methods, each of which takes as input an audio signal in the current frame with a frequency higher than or equal to a preset threshold. That is, the input of each of the multiple sub-rule methods is part or all of the audio signal in the second audio signal.

[0234] For example, Figure 8d is a schematic diagram of obtaining edge information according to an embodiment of this application. As shown in Figure 8d, the edge information is obtained through a second machine learning model and a pre-set rule method. The second machine learning model includes multiple sub-models (n sub-models are used as an example in Figure 8d, n>1), namely sub-model 1, sub-model 2, ..., sub-model n. The input of sub-model 1 is input 1 and the output is feature information 1; the input of sub-model 2 is input 2 and the output is feature information 2, ...; the input of sub-model n is input n and the output is feature information n. The rule method includes multiple sub-rule methods (m sub-rule methods are used as an example in Figure 8d, m>1), namely, the input of sub-rule method 1 is input n+1 and the output is feature information n+1; the input of sub-rule method 2 is input n+2 and the output is feature information n+2, ...; the input of sub-rule method m is input n+m and the output is feature information n+m.

[0235] Input 1, Input 2, ..., Input n, Input n+1, Input n+2, ..., Input n+m all originate from the second audio signal, and they all share the common feature of including high-frequency components from the second audio signal. Furthermore, Input 1, Input 2, ..., Input n, Input n+1, Input n+2, ..., Input n+m may or may not include low-frequency components from the second audio signal. When low-frequency components from the second audio signal are included, Input 1, Input 2, ..., Input n, Input n+1, Input n+2, ..., Input n+m respectively include some or all of the low-frequency components from the second audio signal. For example, Input 1 includes high-frequency components from the second audio signal, Input 2 includes high-frequency components and some low-frequency components from the second audio signal, and Input 3 includes high-frequency components and all low-frequency components from the second audio signal. It should be noted that in the embodiments of this application, Input 1, Input 2, ..., Input n, Input n+1, Input n+2, ..., Input n+m can be the same, completely different, or partially the same and partially different; no specific limitation is made in this regard.

[0236] The outputs of the above n sub-models and m rule methods, including feature information 1, feature information 2, ..., feature information n, feature information n+1, feature information n+2, ..., feature information n+m, are merged to obtain edge information. That is, the edge information includes the feature information output by the n sub-models and the feature information output according to the m rule methods.

[0237] In some possible implementations, the input of the first sub-rule method also includes a third input; when the preset rule method includes multiple sub-rule methods, the third input includes the output of at least one other sub-rule method among the multiple sub-rule methods, or the third input includes the output of at least one sub-model; the first sub-rule method is any one of the multiple sub-rule methods.

[0238] For example, Figure 11a is a schematic diagram of the concatenation of sub-rule methods according to an embodiment of this application. As shown in Figure 11a, taking two sub-rule methods as an example, the output of sub-rule method 2 can be used as an input of sub-rule method 1, so sub-rule method 1 includes input 1 and the output of sub-rule method 2. In addition, the output of the sub-model can also be used as an input of sub-rule method 1, so sub-rule method 1 includes input 1 and the output of the sub-model, and even the output of sub-rule method 2 can be added, without specific limitation. This can be a concatenation of sub-rule methods with sub-rule methods or sub-models.

[0239] In some possible implementations, the input of the first sub-rule method also includes a fourth input, which is the audio signal obtained after the third input has undergone redundant information processing; when the preset rule method includes multiple sub-rule methods, the third input includes the output of at least one other sub-rule method among the multiple sub-rule methods, or the third input includes the output of at least one sub-model; the first sub-rule method is any one of the multiple sub-rule methods.

[0240] For example, Figure 11b is a schematic diagram of the concatenation of sub-rule methods according to an embodiment of this application. As shown in Figure 11b, taking two sub-rule methods as an example, the output of sub-rule method 2 is first processed with redundant information and then can be used as an input to sub-rule method 1. Thus, sub-rule method 1 includes input 1 and the output of sub-rule method 2 after redundant information processing. Alternatively, the output of the sub-model can also be processed with redundant information and then used as an input to sub-rule method 1. Thus, sub-rule method 1 includes input 1 and the output of the sub-model after redundant information processing. It is even possible to add the output of sub-rule method 2 after redundant information processing; no specific limitation is made. This can be a concatenation of sub-rule methods with sub-rule methods or sub-models.

[0241] It should be noted that the above example uses two sub-rule methods for illustration, but this does not constitute a limitation on the chaining relationship between sub-rule methods or between sub-rule methods and sub-models. In addition to its own input, the input of any sub-rule method may also include the output of one or more other sub-rule methods or the output after redundancy processing, and / or the output of one or more sub-models or the output after redundancy processing. The embodiments of this application do not limit the number of chains or the chaining method.

[0242] In some possible implementations, the second machine learning model and / or rule-based approach for acquiring side information can be jointly trained. That is, by jointly training multiple mechanisms for acquiring side information (which can be the aforementioned sub-models or sub-rule-based approaches), information redundancy between feature groups can be reduced or eliminated; a joint loss function is set for the multiple side information acquisition mechanisms, and the weights of the multiple side information acquisition mechanisms are corrected through error backpropagation to achieve the overall optimal effect of multiple feature information, thereby reducing or eliminating information redundancy between feature groups; each side information acquisition mechanism can have its own loss function to preserve the uniqueness of its extracted feature information.

[0243] For example, Figure 12 is a schematic diagram of joint training in an embodiment of this application. As shown in Figure 12, taking two mechanisms for acquiring side information (which can be the sub-model or sub-rule method described above) as an example, each mechanism for acquiring side information can have its own loss function (loss function 1 and loss function 2), and a joint loss function is set for the two mechanisms for acquiring side information. These loss functions correct the weights of the two mechanisms for acquiring side information through error backpropagation to achieve the overall optimal effect of the two feature information. On the one hand, it can preserve the uniqueness of the extracted feature information, and on the other hand, it can reduce or eliminate information redundancy between the two feature groups.

[0244] Step 703: Obtain the second bitstream based on the edge information.

[0245] In this embodiment of the application, the edge information can be encoded by an edge information encoder. The edge information encoder can use the same encoding algorithm as the core encoder mentioned above, or it can use a different encoding algorithm. No specific limitation is made in this regard.

[0246] Step 704: Transmit the bitstream, which includes a first bitstream and a second bitstream.

[0247] The bitstream obtained at the encoding end consists of two parts: one part (the first bitstream) corresponds to the low-frequency components of the current frame, and the other part (the second bitstream) corresponds to the side information of the current frame. The encoding end transmits the bitstream to the decoding end so that the decoding end can obtain the reconstructed low-frequency components and side information based on the bitstream, thereby obtaining the reconstructed frame corresponding to the current frame.

[0248] In this embodiment, side information is extracted from the audio through a machine learning model to guide the decoding end to implement non-blind BWE and reconstruct high-frequency components, especially high-frequency components that are weakly correlated with low-frequency components. This implementation method can reflect more details of the high-frequency components in the current frame in the feature information. These details include, for example, the characteristics of the high-frequency components and the correlation between the high-frequency components and low-frequency components, which helps to reconstruct more high-frequency details and thus avoid the problem of audio distortion.

[0249] Figure 13 is a flowchart of process 1300 of the audio decoding method provided in an embodiment of this application. Process 1300 describes a series of steps or operations. It should be understood that process 1300 can be executed in various orders and / or occur simultaneously, and is not limited to the execution order shown in Figure 13. Process 1300 may include:

[0250] Step 1301: Receive the bitstream, which includes a first bitstream and a second bitstream.

[0251] In this embodiment of the application, a bitstream is received from the encoding end. Corresponding to the bitstream generated by the encoding end, the received bitstream also includes two parts: a first bitstream and a second bitstream.

[0252] Step 1302: Obtain a first audio signal based on the first bitstream, wherein the frequency of the first audio signal is lower than a preset threshold.

[0253] In this embodiment of the application, the first bitstream can be parsed by a core decoder to obtain the first audio signal.

[0254] The reconstructed frame obtained at the decoding end corresponds to the current frame at the encoding end. The characteristics of the audio signal contained in the reconstructed frame can also be characterized by frequency, amplitude, and phase. According to frequency division, the reconstructed frame also includes a first audio signal and a second audio signal. The first audio signal is an audio signal with a frequency lower than a preset threshold (in this embodiment, the audio signal that meets this condition can also be called a low-frequency signal or low-frequency component), and the second audio signal is an audio signal with a frequency higher than or equal to the preset threshold (in this embodiment, the audio signal that meets this condition can also be called a high-frequency signal or high-frequency component). That is, the reconstructed frame includes low-frequency components and high-frequency components. The aforementioned preset threshold can be, for example, 16kHz, and this embodiment does not specifically limit it. It should be noted that the audio signal with a frequency equal to the preset threshold is classified as the second audio signal above. Optionally, this embodiment can also classify the audio signal with a frequency equal to the preset threshold as the first audio signal, and this is not specifically limited.

[0255] Based on this, the first audio signal obtained from the first bitstream can be the low-frequency component of the reconstructed frame. That is, the decoding end can directly obtain the low-frequency component of the reconstructed frame by parsing the bitstream.

[0256] Step 1303: Obtain side information based on the second bitstream.

[0257] Side information is used to characterize the properties of the second audio signal. Furthermore, it can also be used to characterize the mapping relationship between high and low frequencies. In this embodiment, side information occupies only a small number of transmission bits to guide the decoding end in implementing non-blind BWE (Blind Web End-to-Wide) to reconstruct high-frequency components, especially those closely related to low-frequency components but with complex mapping relationships.

[0258] In some possible implementations, the side information can be obtained by directly parsing the second bitstream. That is, the second bitstream is obtained by processing the side information at the encoding end, so the decoding end can directly parse the second bitstream to obtain the side information.

[0259] Optionally, the second bitstream is parsed to obtain the preceding information; then the preceding information is processed to obtain the side information, which includes inputting a pre-trained third machine learning model or processing according to a pre-set third rule.

[0260] In this embodiment, the information carried by the second bitstream may be pre-processing information associated with the side information. This information may be obtained by the encoding end based on the side information to further save bitstream space, and then encoded into the bitstream. Correspondingly, the decoding end can first parse the pre-processing information obtained from the second bitstream to obtain the side information, and then obtain the side information based on the pre-processing information. The aforementioned method for obtaining the side information may include obtaining the side information through a third machine learning model or according to a pre-set third rule.

[0261] Step 1304: Obtain the reconstructed frame based on the first audio signal and side information.

[0262] In some possible implementations, the reconstructed frame can be obtained using a pre-trained first machine learning model based on the first audio signal and side information. Since the reconstructed frame is obtained with reference to side information to extend from low-frequency components to high-frequency components, this process can also be called non-blind BWE (Blind Image Wrapper).

[0263] In this embodiment, the first machine learning model can refer to the neural networks, deep neural networks, convolutional neural networks, recurrent neural networks, etc. mentioned above.

[0264] For example, Figure 14a is a schematic diagram of a non-blind BWE according to an embodiment of the present application. As shown in Figure 14a, the non-blind BWE is implemented through a first machine learning model, which includes a sub-model, namely the first machine learning model is also the sub-model. Its input is a first audio signal and side information, and its output is a reconstructed frame.

[0265] Optionally, the reconstructed frames are the direct output of the first machine learning model.

[0266] In this embodiment, the non-blind BWE can employ a machine learning model (first machine learning model) to directly output reconstructed frames based on the decoded low-frequency components (first audio signal) under the guidance of side information. Furthermore, the first machine learning model can perform encoding / decoding distortion correction for low-frequency components and possible audio rendering; this embodiment does not specifically limit its application in these aspects.

[0267] For example, Figure 15a is a schematic diagram of obtaining a reconstructed frame according to an embodiment of this application. As shown in Figure 15a, the input of the non-blind BWE includes low-frequency components and side information, and the output is a reconstructed frame.

[0268] Optionally, the reconstructed frame is obtained based on the second audio signal output by the first machine learning model and the first audio signal.

[0269] In this embodiment, the non-blind BWE can employ a machine learning model (first machine learning model) that, guided by side information, outputs reconstructed high-frequency components based on the decoded low-frequency components (first audio signal). The reconstructed high-frequency components are then merged or added to the decoded low-frequency components to obtain the reconstructed audio. Furthermore, the first machine learning model can perform encoding / decoding distortion correction for low-frequency components and possible audio rendering; this embodiment does not specifically limit these aspects.

[0270] For example, Figure 15b is a schematic diagram of obtaining the reconstructed frame according to an embodiment of this application. As shown in Figure 15b, the input of the non-blind BWE includes low-frequency components and side information, and the output is the reconstructed high-frequency components. Then, the high-frequency components and low-frequency components are merged to obtain the reconstructed frame.

[0271] In some possible implementations, the first machine learning model may include a first sub-model and a second sub-model. The non-blind BWE may input the first audio signal and the first side information into the first sub-model to obtain the third audio signal; then input the third audio signal and the second side information into the second sub-model to obtain the reconstructed frame; the first side information and the second side information are part or all of the information in the side information.

[0272] For example, Figure 14b is a schematic diagram of a non-blind BWE according to an embodiment of this application. As shown in Figure 14b, the first machine learning model used in the non-blind BWE can be divided into a first sub-model (corresponding to low-frequency feature extraction) and a second sub-model (corresponding to high-frequency component reconstruction). These two sub-models are connected in series, that is, the output of low-frequency feature extraction (the third audio signal, i.e., low-frequency features) can be used as the input of high-frequency component reconstruction. The input of low-frequency feature extraction includes the first audio signal and the first side information, and the input of high-frequency component reconstruction includes the third audio signal and the second side information. Low-frequency feature extraction is guided by the first side information, and high-frequency component reconstruction is guided by the second side information. The output of high-frequency component reconstruction is a reconstructed frame. It should be noted that in the embodiments of this application, the first side information and the second side information can be part or all of the information in the side information, and the first side information and the second side information can be the same or different, which is not specifically limited.

[0273] In some possible implementations, a non-blind BWE can process the first audio signal and the third side information according to a pre-set first rule to obtain a fourth audio signal; then input the fourth audio signal and the fourth side information into a first machine learning model to obtain a reconstructed frame; the third side information and the fourth side information are part or all of the information in the side information.

[0274] For example, Figure 14c is a schematic diagram of a non-blind BWE according to an embodiment of this application. As shown in Figure 14c, corresponding to the encoding end, when the encoding end obtains the side information according to the rule method, the decoding end can also adopt the rule method in the non-blind BWE accordingly. This rule method can be set before machine learning, that is, it is first processed according to the first rule method (corresponding to rule-based preprocessing), the input of which is the first audio signal and the third side information, and the output is the fourth audio signal; then it is passed through the first machine learning model, the input of which is the fourth audio signal and the fourth side information, and the output is the reconstructed frame. It should be noted that in the embodiments of this application, the third side information and the fourth side information can be part or all of the information in the side information, and the third side information and the fourth side information can be the same or different, which is not specifically limited.

[0275] In some possible implementations, a non-blind BWE can obtain a fifth audio signal based on a first machine learning model that has been pre-trained, using the first audio signal and the fifth side information; then, based on the fifth audio signal and the sixth side information, it can be processed according to a pre-set second rule to obtain a reconstructed frame; the fifth side information and the sixth side information are part or all of the side information.

[0276] For example, Figure 14d is a schematic diagram of a non-blind BWE according to an embodiment of this application. As shown in Figure 14d, corresponding to the encoding end, when the encoding end obtains the side information according to the rule method, the decoding end can also adopt the rule method in the non-blind BWE accordingly. This rule method can be set after machine learning, that is, firstly, it is processed by a first machine learning model, whose input is a first audio signal and the fifth side information, and the output is a fifth audio signal; then, it is processed according to a second rule method, whose input is the fifth audio signal and the sixth side information, and the output is a reconstructed frame. It should be noted that in the embodiments of this application, the fifth side information and the sixth side information can be part or all of the side information, and the fifth side information and the sixth side information can be the same or different, which is not specifically limited.

[0277] It should be noted that in the embodiments shown in Figures 14a-14d, the reconstructed frame is directly output by a non-blind BWE (see Figure 15a) as an example. It should be understood that in the embodiments of this application, the reconstructed frame can also be obtained using the method shown in the embodiment of Figure 15b. In addition, in the embodiments shown in Figures 14c and 14d, the first machine learning model used by the non-blind BWE can also refer to the embodiment shown in Figure 14b.

[0278] For example, Figure 14e is a schematic diagram of the non-blind BWE according to an embodiment of this application. As shown in Figure 14e, the non-blind BWE first performs rule-based preprocessing, the input of which is low-frequency components and side information 1; then it uses a first machine learning model, the input of which is the output of the rule-based preprocessing and side information 2; then it performs rule-based postprocessing, the input of which is the output of the first machine learning model and side information 3, and the output is high-frequency components; finally, the low-frequency components and high-frequency components are merged into high and low frequencies, and the reconstructed frame is output.

[0279] It should be noted that the aforementioned edge information 1 to edge information 3 may be part or all of the edge information, and edge information 1 to edge information 3 may be completely the same or completely different, or partially the same or partially different, without specific limitations.

[0280] For example, Figure 14f is a schematic diagram of the non-blind BWE according to an embodiment of this application. As shown in Figure 14f, the non-blind BWE first performs rule-based preprocessing, with low-frequency components and side information 1 as inputs; then, low-frequency feature extraction is performed in the first machine learning model, with the output of the rule-based preprocessing and side information 2 as inputs, and low-frequency features as outputs; then, high-frequency component reconstruction is performed in the first machine learning model, with low-frequency features and side information 3 as inputs, and high-frequency features as outputs; then, rule-based postprocessing is performed, with high-frequency features and side information 4 as inputs, and high-frequency components as outputs; finally, the low-frequency components and high-frequency components are merged to output a reconstructed frame.

[0281] It should be noted that the aforementioned edge information 1 to edge information 4 may be part or all of the edge information, and edge information 1 to edge information 4 may be completely the same or completely different, or partially the same or partially different, without specific limitations.

[0282] In this embodiment, a machine learning model, guided by side information, is used to obtain reconstructed frames based on low-frequency components. This is especially useful for reconstructed frames in scenarios where the correlation between high-frequency and low-frequency components is weak, or where the relationship between high and low-frequency components is close but the mapping relationship is complex. This approach can incorporate more details of high-frequency components into the feature information, such as the characteristics of high-frequency components and the correlation between high-frequency and low-frequency components. This helps to reconstruct more high-frequency details, thereby avoiding the problem of audio distortion.

[0283] The technical solutions of the method embodiments shown in Figures 7 and 13 will be described in detail below using several specific examples.

[0284] Figure 16 is a schematic diagram of the encoding and decoding framework of a non-blind BWE containing AI according to an embodiment of this application. As shown in Figure 16, the framework includes the following steps:

[0285] 1. The encoding end converts the original audio into a bitstream.

[0286] 1.1 Input the low-frequency components of the original audio into the core encoder, which encodes the low-frequency components into a low-frequency bitstream (corresponding to the first bitstream above). Refer to step 701 above.

[0287] 1.2 The side information extraction stage employs a machine learning model. This model extracts features that contribute to the BWE from the audio containing high-frequency components of the original audio; these features are called side information. Refer to step 702 above.

[0288] 1.3. The edge information is input into the edge information encoder, which encodes the edge information into an edge information bitstream (corresponding to the second bitstream above). Refer to step 703 above.

[0289] 1.4 The side information bitstream and the low-frequency bitstream are merged to obtain a bitstream, which is then transmitted to the decoding end via the transmission network.

[0290] 2. The decoding end reconstructs the audio based on the bitstream.

[0291] 2.1 After receiving the bitstream, parse it into a low-frequency bitstream and a side information bitstream. Refer to step 1301 above.

[0292] 2.2 Input the low-frequency bitstream into the core decoder, which decodes the low-frequency bitstream into low-frequency components. Refer to step 1302 above.

[0293] 2.3 Input the side information bitstream into the side information decoder, which decodes the side information bitstream into side information. Refer to step 1303 above.

[0294] 2.4. Non-blind BWE employs a machine learning model. Guided by side information, the machine learning model outputs reconstructed audio based on the decoded low-frequency components. Furthermore, the machine learning model can also perform encoding / decoding distortion correction for low-frequency components and, if possible, audio rendering.

[0295] Optionally, step 2.4 can also refer to Figure 15b to first obtain the reconstructed high-frequency components and then obtain the reconstructed audio.

[0296] Figure 17 is a schematic diagram of the AI-integrated non-blind BWE encoding and decoding framework according to an embodiment of this application. As shown in Figure 17, based on the embodiment shown in Figure 16, the side information extraction stage at the encoding end employs both a machine learning model and a rule-based approach. Thus, the side information output by the side information extraction stage includes feature information output by the machine learning model and feature information output according to the rule-based approach. Similarly, the non-blind BWE stage at the decoding end employs both a machine learning model and a rule-based approach. Therefore, the reconstructed audio output by the non-blind BWE stage combines the output of the machine learning model and the output according to the rule-based approach.

[0297] The implementation methods of the edge information extraction stage in this embodiment can be referred to the embodiments shown in Figures 8b-8d, 9a-9c, 10a-10b, and 11a-11b.

[0298] Figure 18 is a schematic diagram of the codec framework for a non-blind BWE (Browser-Wide Interface) incorporating AI according to an embodiment of this application. As shown in Figure 18, this embodiment is built on the AVS3 codec framework and implements 48kHz audio codec. The core codec is responsible for encoding and reconstructing low-frequency components below 16kHz through a codec method, the side information extraction stage is responsible for extracting side information, and the non-blind BWE stage is responsible for reconstructing high-frequency components from 16kHz to 48kHz. The framework includes the following steps:

[0299] 1. The encoding end converts the original 48kHz audio into a bitstream.

[0300] 1.1 Perform frame-by-frame processing on the original 48kHz audio, with each frame having a length of 2048 sampling points and 1024 sampling points superimposed between frames, and process these signal frames sequentially.

[0301] 1.2 Low-frequency components below 16kHz in the low-frequency channel encoded signal frame.

[0302] 1.2.1. Use a low-pass filter to extract audio signals below 16kHz from the signal frame, which are the low-frequency components.

[0303] 1.2.2 Encode low-frequency components below 16kHz into a low-frequency bitstream using the AVS3 core encoder.

[0304] 1.3 High-frequency components from 16kHz to 48kHz in the side information channel encoded signal frame.

[0305] 1.3.1 The edge information extraction step extracts edge information from the signal frame, as shown in Figure 19 (Figure 19 is a flowchart of the edge information extraction step in an embodiment of this application):

[0306] 1.3.1.1 Transform the signal frame into a spectrum using Fourier transform.

[0307] 1.3.1.2 Divide the high-frequency region of the spectrum into multiple frequency bands and perform envelope calculation, that is, calculate the average value of the spectral energy of each frequency band as the frequency band envelope (characteristic group 1).

[0308] 1.3.1.3 Divide the spectrum of each frequency band in the high-frequency region by the corresponding frequency band envelope to set the spectral energy of these frequency bands to 1, eliminating the energy envelope information contained therein. The spectrum after frequency band normalization is called the normalized spectrum.

[0309] 1.3.1.4 Input the normalized spectrum into the neural network, which extracts features (feature group 2) from the normalized spectrum.

[0310] 1.3.1.5 Merge feature group 1 and feature group 2 as edge information.

[0311] 1.3.2 The quantization encoder quantizes the side information into a side information bitstream.

[0312] 1.4. Merge the low-frequency bitstream and the side information bitstream into a single bitstream and transmit it to the decoding end.

[0313] 2. The decoding end reconstructs the original 48kHz audio from the bitstream.

[0314] 2.1 Receive the bitstream transmitted from the encoding end and split it into low-frequency bitstream and side information bitstream.

[0315] 2.2. Use the AVS3 core decoder to decode low-frequency components below 16kHz from the low-frequency bitstream.

[0316] 2.3 Decode the side information from the side information bitstream using a quantization decoder.

[0317] 2.4 The non-blind BWE (Blind Web Edge) stage is responsible for reconstructing the high-frequency signal from the decoded low-frequency signal below 16kHz, guided by the decoded side information, as shown in Figure 20 (Figure 20 is a flowchart of the non-blind BWE stage in an embodiment of this application).

[0318] 2.4.1 The decoded low-frequency components are transformed into a low-frequency spectrum after Fourier transform.

[0319] 2.4.2 Extracting low-frequency features from the low-frequency spectrum based on the low-frequency feature extraction model.

[0320] 2.4.3 The edge information processing model maps feature group 2 into high-frequency features.

[0321] 2.4.4 The high-frequency component reconstruction model reconstructs high-frequency components based on low-frequency characteristics under the guidance of high-frequency features, thus obtaining the reconstructed spectrum.

[0322] 2.4.5. Adjust the spectral energy of the corresponding frequency band according to feature group 1 to ensure that the spectral energy of each frequency band is the same as the real one, and replace the spectrum below 16kHz with the low-frequency spectrum to obtain the adjusted spectrum.

[0323] 2.4.6 Perform an inverse Fourier transform on the adjusted spectrum to obtain the reconstructed audio.

[0324] 2.5. Add the decoded low-frequency components and the reconstructed high-frequency components together, and then perform a superposition process to obtain the reconstructed audio at 48kHz.

[0325] Under the same experimental conditions, the results of rule-based BWE, AI-based fully blind BWE, and AI-based non-blind BWE (the technical solutions of the embodiments of this application) and the real target were compared. The experimental results are shown in Figures 21 and 22 (Figures 21 and 22 are schematic diagrams comparing the audio spectra obtained by the three BWE methods).

[0326] Rule-based BWE copies the low-frequency spectrum to the high-frequency spectrum and adjusts the energy of each frequency band based on side information. Therefore, this method cannot extend the texture that is not present in the low-frequency components. As shown in Figures 21 and 22, the spectrum of the region outlined is copied from the low frequency and differs significantly from the true spectrum, thus causing differences in the perceived sound.

[0327] AI-based fully blind BWE can reconstruct some high-frequency components from low-frequency components. However, because the correlation between higher frequency spectra and lower frequency spectra is weaker, the reconstructed spectrum suffers greater distortion. This also leads to a gradual decrease in the high-frequency energy reconstructed by the model as the frequency increases. Furthermore, some high-frequency components exist independently. Without the guidance of side information, the model cannot determine the existence of these components, nor can it reconstruct them.

[0328] Compared to rule-based BWE and AI-based fully blind BWE, the high-frequency spectrum reconstructed by AI-based non-blind BWE is significantly closer to reality and more consistent with the listening experience of the original audio. Compared to rule-based BWE, the high-low frequency mapping relationships discovered by the AI ​​model are more complex and comprehensive. Therefore, the side information extracted by the AI-based side information extraction module contains richer spectral features, enabling the AI-based non-blind BWE module to reconstruct more accurate high-frequency details. Specifically, the rule-based preprocessing module can construct a high-frequency basis under the guidance of side information, simplifying subsequent AI calculations. Introducing side information into the low-frequency feature extraction and high-frequency component reconstruction modules can respectively achieve the effects of extracting low-frequency features that conform to the side information and reconstructing high-frequency components that conform to the side information. The rule-based post-processing further adjusts the reconstructed high-frequency components based on the side information to ensure that the high-frequency components satisfy prior characteristics. For example, the spectral envelope energy of each frequency band can be adjusted according to the side information to ensure the accuracy of the envelope energy.

[0329] Figure 23 is a schematic diagram of the codec framework for a non-blind BWE (Browser-Wide Web Interface) incorporating AI according to an embodiment of this application. As shown in Figure 23, this embodiment is built on the AVS3 codec framework and implements audio codec for 48kHz. The core codec is responsible for encoding and reconstructing low-frequency components below a specified frequency f through a specific encoding / decoding method. The side information extraction stage is responsible for extracting side information, and the non-blind BWE stage is responsible for reconstructing high-frequency components from f to 48kHz. The framework includes the following steps:

[0330] 1. The encoding end converts the original 48kHz audio into a bitstream.

[0331] 1.1 Perform frame-by-frame processing on the original 48kHz audio, with each frame having a length of 2048 sampling points and 1024 sampling points superimposed between frames, and process these signal frames sequentially.

[0332] 1.2. Specify the frequency boundary f, which represents the low-frequency components below f in the low-frequency channel encoded signal frame.

[0333] 1.2.1. Use a low-pass filter to extract low-frequency signals below f from the original 48kHz audio, which are the low-frequency components.

[0334] 1.2.2 Encode low-frequency components below f into a low-frequency bitstream using the AVS3 core encoder.

[0335] 1.3. Side information channel encoding of high-frequency components from f to 48kHz.

[0336] 1.3.1 Side information extraction step: Side information is extracted from the original 48kHz audio, as shown in Figure 24 (Figure 24 is a flowchart of the side information extraction step in this application embodiment).

[0337] 1.3.1.1 Transform the signal frame into a spectrum using Fourier transform.

[0338] 1.3.1.2 Using the side information extraction module built into AVS3, the high-frequency region of the spectrum is divided into multiple frequency bands, and envelope calculation is performed. That is, the average value of the spectral energy of each frequency band is calculated as the frequency band envelope, and the frequency band replication method identifier used by each frequency band is calculated. The two types of information are merged into feature group 1.

[0339] 1.3.1.3 Divide the spectrum of each frequency band in the high-frequency region by the corresponding frequency band envelope to set the spectral energy of these frequency bands to 1, eliminating the energy envelope information contained therein. The spectrum after frequency band normalization is called the normalized spectrum.

[0340] 1.3.1.4 Input the normalized spectrum into the neural network, which extracts features (feature group 2) from the normalized spectrum.

[0341] 1.3.1.5 Merge feature group 1 and feature group 2 as edge information.

[0342] 1.3.2 The quantization encoder quantizes the side information into a side information bitstream.

[0343] 1.4. Merge the low-frequency bitstream, side information bitstream, and f into a single bitstream and transmit it to the decoding end.

[0344] 2. The decoding end reconstructs the original 48kHz audio from the bitstream.

[0345] 2.1 Receive the bitstream transmitted from the encoding end and split it into low-frequency bitstream and side information bitstream.

[0346] 2.2. Use the AVS3 core decoder to decode low-frequency signals below f from the low-frequency bitstream.

[0347] 2.3 Decode the side information from the side information bitstream using a quantization decoder.

[0348] 2.4 The non-blind BWE process is responsible for reconstructing the high-frequency signal based on the decoded low-frequency signal below f, guided by the decoded side information, as shown in Figure 25 (Figure 25 is a flowchart of the non-blind BWE process in an embodiment of this application).

[0349] 2.4.1 Input the decoded low-frequency components and feature group 1 into the AVS3 band extension, use the rule-based BWE method to extend the high-frequency components, and add the high-frequency components with the low-frequency components to obtain the signal basis.

[0350] 2.4.2 The edge information processing model maps feature group 2 into high-frequency features.

[0351] 2.4.3. Guided by high-frequency features, the non-blind BWE model obtains the reconstructed signal based on the signal basis.

[0352] 2.4.4 Perform a Fourier transform on the reconstructed signal to obtain the reconstructed spectrum. Adjust the spectral energy of the corresponding frequency bands according to the frequency band envelope of feature group 1 to ensure that the spectral energy of each frequency band is the same as the real one. Then perform an inverse Fourier transform on the adjusted spectrum to obtain the adjusted reconstructed signal.

[0353] 2.5. The reconstructed signal after adjustment is overlaid to obtain the reconstructed audio at 48kHz.

[0354] Figure 26 is a schematic diagram of the structure of the encoding device 2600 according to an embodiment of this application. As shown in Figure 26, the encoding device 2600 of this embodiment can be applied to the above-mentioned encoding device. The encoding device 2600 may include: an acquisition module 2601, an extraction module 2602, an encoding module 2603, a transmission module 2604, and a filtering module 2605.

[0355] The acquisition module 2601 is used to acquire a first bitstream, which is obtained based on a first audio signal, and the first audio signal is a low-frequency signal with a frequency lower than a preset threshold in the current frame; the extraction module 2602 is used to acquire side information, which is obtained based on a second audio signal, and the second audio signal includes a high-frequency signal with a frequency higher than or equal to the preset threshold in the current frame; the encoding module 2603 is used to acquire a second bitstream according to the side information; and the transmission module 2604 is used to transmit the bitstream, which includes the first bitstream and the second bitstream.

[0356] In one possible implementation, the extraction module 2602 is specifically used to input the second audio signal into a pre-trained second machine learning model to obtain the side information, wherein the second machine learning model includes at least one sub-model.

[0357] In one possible implementation, when the second machine learning model includes multiple sub-models, the input of each of the multiple sub-models includes a portion or all of the audio signal in the second audio signal, wherein the portion of the audio signal includes the high-frequency signal.

[0358] In one possible implementation, the first sub-model includes the same neurons as at least one other sub-model among the plurality of sub-models. The same neurons are located in the input phase, output phase, or intermediate phase of the first sub-model, and the first sub-model is any one of the plurality of sub-models.

[0359] In one possible implementation, the edge information includes feature information output by the plurality of sub-models respectively.

[0360] In one possible implementation, when the second machine learning model includes a sub-model, the edge information includes feature information output by the sub-model.

[0361] In one possible implementation, the extraction module 2602 is specifically used to process the second audio signal according to a preset rule to obtain the side information, wherein the preset rule includes at least one sub-rule.

[0362] In one possible implementation, when the pre-set rule method includes multiple sub-rule methods, the input of each of the multiple sub-rule methods includes a portion or all of the audio signal in the second audio signal, wherein the portion of the audio signal includes the high-frequency signal.

[0363] In one possible implementation, the edge information includes feature information output according to the plurality of sub-rules.

[0364] In one possible implementation, when the pre-set rule method includes a sub-rule method, the edge information includes feature information output according to the sub-rule method.

[0365] In one possible implementation, the input to the first sub-model further includes a first input; when the second machine learning model includes multiple sub-models, the first input includes the output of at least one other sub-model among the multiple sub-models, or the first input includes the output of the at least one sub-rule; the first sub-model is any one of the multiple sub-models.

[0366] In one possible implementation, the input of the first sub-model further includes a second input, which is an audio signal obtained after the first input has undergone redundant information processing; when the second machine learning model includes multiple sub-models, the first input includes the output of at least one other sub-model among the multiple sub-models, or the first input includes the output of the at least one sub-rule; the first sub-model is any one of the multiple sub-models.

[0367] In one possible implementation, the input of the first sub-rule method further includes a third input; when the preset rule method includes multiple sub-rule methods, the third input includes the output of at least one other sub-rule method among the multiple sub-rule methods, or the third input includes the output of the at least one sub-model; the first sub-rule method is any one of the multiple sub-rule methods.

[0368] In one possible implementation, the input of the first sub-rule method further includes a fourth input, which is an audio signal obtained after the third input has undergone redundant information processing; when the preset rule method includes multiple sub-rule methods, the third input includes the output of at least one other sub-rule method among the multiple sub-rule methods, or the third input includes the output of the at least one sub-model; the first sub-rule method is any one of the multiple sub-rule methods.

[0369] In one possible implementation, the filtering module 2605 is used to obtain the first audio signal from the current frame by means of low-pass filtering.

[0370] In one possible implementation, the second audio signal may further include some or all of the audio signals in the first audio signal.

[0371] The apparatus in this embodiment can be used to execute the technical solution of the method embodiment shown in FIG7. Its implementation principle and technical effect are similar, and will not be described again here.

[0372] Figure 27 is a schematic diagram of the structure of the decoding device 2700 according to an embodiment of this application. As shown in Figure 27, the decoding device 2700 of this embodiment can be applied to the decoding device mentioned above. The decoding device 2700 may include: a receiving module 2701, an acquiring module 2702, an extracting module 2703, and a reconstruction module 2704.

[0373] The receiving module 2701 is used to receive a bitstream, the bitstream including a first bitstream and a second bitstream; the acquiring module 2702 is used to acquire a first audio signal based on the first bitstream, the frequency of the first audio signal being lower than a preset threshold; the extracting module 2703 is used to acquire side information based on the second bitstream, the side information being used to characterize the features of a second audio signal, the frequency of the second audio signal being higher than or equal to the preset threshold; and the reconstruction module 2704 is used to acquire a reconstructed frame based on the first audio signal and the side information.

[0374] In one possible implementation, the reconstruction module 2704 is specifically used to obtain the reconstructed frame based on the first audio signal and the side information through a pre-trained first machine learning model.

[0375] In one possible implementation, the reconstructed frame is directly output by the first machine learning model; or, the reconstructed frame is obtained based on the second audio signal output by the first machine learning model and the first audio signal.

[0376] In one possible implementation, when the first machine learning model includes a first sub-model and a second sub-model, the reconstruction module 2704 is specifically used to input the first audio signal and the first side information into the first sub-model to obtain a third audio signal; and to input the third audio signal and the second side information into the second sub-model to obtain the reconstructed frame; wherein the first side information and the second side information are part or all of the information in the side information.

[0377] In one possible implementation, the reconstruction module 2704 is specifically used to process the first audio signal and the third side information according to a pre-set first rule to obtain a fourth audio signal; input the fourth audio signal and the fourth side information into the first machine learning model to obtain the reconstructed frame; the third side information and the fourth side information are part or all of the side information respectively.

[0378] In one possible implementation, the reconstruction module 2704 is specifically used to obtain a fifth audio signal based on the first audio signal and the pentagonal information through a pre-trained first machine learning model; and to process the fifth audio signal and the sixth pentagonal information according to a pre-set second rule to obtain the reconstructed frame; wherein the fifth pentagonal information and the sixth pentagonal information are part or all of the information in the pentagonal information.

[0379] In one possible implementation, the extraction module 2703 is specifically used to parse the second bitstream to obtain the side information.

[0380] In one possible implementation, the extraction module 2703 is specifically used to parse the second bitstream to obtain the preceding information; and to perform side information processing on the preceding information to obtain the side information, wherein the side information processing includes inputting a pre-trained third machine learning model or processing according to a pre-set third rule.

[0381] The apparatus in this embodiment can be used to execute the technical solution of the method embodiment shown in FIG13. Its implementation principle and technical effect are similar, and will not be described again here.

[0382] In implementation, each step of the above method embodiments can be completed by integrated logic circuits in the processor hardware or by instructions in software form. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this application can be directly implemented by a hardware encoding processor, or by a combination of hardware and software modules in the encoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0383] The memory mentioned in the above embodiments can be volatile memory or non-volatile memory, or may include both. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM). It should be noted that the memory used in the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0384] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0385] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0386] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0387] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0388] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0389] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0390] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An audio encoding method, characterized in that, include: Acquire a first bitstream, which is obtained based on a first audio signal, which is a low-frequency signal with a frequency lower than a preset threshold in the current frame; Obtain edge information, which is obtained based on a second audio signal, the second audio signal including high-frequency signals in the current frame with frequencies higher than or equal to the preset threshold; The second bitstream is obtained based on the edge information; Transmitted bitstream, the bitstream comprising the first bitstream and the second bitstream.

2. The method according to claim 1, characterized in that, The acquisition of edge information includes: The second audio signal is input into a pre-trained second machine learning model to obtain the side information, wherein the second machine learning model includes at least one sub-model.

3. The method according to claim 2, characterized in that, When the second machine learning model includes multiple sub-models, the input of each of the multiple sub-models includes a portion or all of the audio signal in the second audio signal, wherein the portion of the audio signal includes the high-frequency signal.

4. The method according to claim 3, characterized in that, The first sub-model and at least one other sub-model among the plurality of sub-models include the same neurons, which are located in the input phase, output phase or intermediate phase of the first sub-model, and the first sub-model is any one of the plurality of sub-models.

5. The method according to any one of claims 2-4, characterized in that, The acquisition of edge information includes: The second audio signal is processed according to a pre-set rule to obtain the side information, wherein the pre-set rule includes at least one sub-rule.

6. The method according to claim 5, characterized in that, When the preset rule method includes multiple sub-rule methods, the input of each of the multiple sub-rule methods includes a portion or all of the audio signal in the second audio signal, wherein the portion of the audio signal includes the high-frequency signal.

7. The method according to claim 5 or 6, characterized in that, The input to the first sub-model also includes a first input; when the second machine learning model includes multiple sub-models, the first input includes the output of at least one other sub-model among the multiple sub-models, or the first input includes the output of the at least one sub-rule; the first sub-model is any one of the multiple sub-models.

8. The method according to claim 5 or 6, characterized in that, The input to the first sub-model also includes a second input, which is the audio signal obtained after the first input has undergone redundant information processing; when the second machine learning model includes multiple sub-models, the first input includes the output of at least one other sub-model among the multiple sub-models, or the first input includes the output of the at least one sub-rule method; the first sub-model is any one of the multiple sub-models.

9. The method according to any one of claims 5-8, characterized in that, The input of the first sub-rule method also includes a third input; when the preset rule method includes multiple sub-rule methods, the third input includes the output of at least one other sub-rule method among the multiple sub-rule methods, or the third input includes the output of the at least one sub-model; the first sub-rule method is any one of the multiple sub-rule methods.

10. The method according to any one of claims 5-8, characterized in that, The input of the first sub-rule method also includes a fourth input, which is the audio signal obtained after the third input has undergone redundant information processing; when the preset rule method includes multiple sub-rule methods, the third input includes the output of at least one other sub-rule method among the multiple sub-rule methods, or the third input includes the output of the at least one sub-model; the first sub-rule method is any one of the multiple sub-rule methods.

11. An audio decoding method, characterized in that, include: Receive a bitstream, the bitstream including a first bitstream and a second bitstream; A first audio signal is obtained based on the first bitstream, wherein the frequency of the first audio signal is lower than a preset threshold. Side information is obtained based on the second bitstream, and the side information is used to characterize the characteristics of the second audio signal, wherein the frequency of the second audio signal is higher than or equal to the preset threshold. The reconstructed frame is obtained based on the first audio signal and the side information.

12. The method according to claim 11, characterized in that, The step of obtaining the reconstructed frame based on the first audio signal and the side information includes: The reconstructed frame is obtained by using a first machine learning model that has been pre-trained based on the first audio signal and the side information.

13. The method according to claim 12, characterized in that, The reconstructed frame is directly output by the first machine learning model; or, the reconstructed frame is obtained based on the second audio signal output by the first machine learning model and the first audio signal.

14. The method according to claim 12 or 13, characterized in that, When the first machine learning model includes a first sub-model and a second sub-model, obtaining the reconstructed frame based on the first audio signal and the side information obtained through the pre-trained first machine learning model includes: The first audio signal and the first side information are input into the first sub-model to obtain the third audio signal; The third audio signal and the second side information are input into the second sub-model to obtain the reconstructed frame; the first side information and the second side information are part or all of the side information, respectively.

15. The method according to any one of claims 12-14, characterized in that, The step of obtaining the reconstructed frame using a pre-trained first machine learning model based on the first audio signal and the side information includes: The first audio signal and the third side information are processed according to a pre-set first rule to obtain the fourth audio signal; The fourth audio signal and the fourth side information are input into the first machine learning model to obtain the reconstructed frame; the third side information and the fourth side information are part or all of the side information, respectively.

16. The method according to any one of claims 12-15, characterized in that, The step of obtaining the reconstructed frame using a pre-trained first machine learning model based on the first audio signal and the side information includes: Based on the first audio signal and the pentagonal information, a fifth audio signal is obtained through a pre-trained first machine learning model. The reconstructed frame is obtained by processing the fifth audio signal and the sixth side information according to a pre-set second rule; the fifth side information and the sixth side information are part or all of the side information.

17. The method according to any one of claims 11-16, characterized in that, The step of obtaining side information based on the second bitstream includes: The second bitstream is parsed to obtain the side information.

18. The method according to any one of claims 11-16, characterized in that, The step of obtaining side information based on the second bitstream includes: The second bitstream is parsed to obtain the information from the previous processing. The preceding information is processed to obtain the side information. The side information processing includes inputting a pre-trained third machine learning model or processing it according to a pre-set third rule.

19. An encoding device, characterized in that, include: The acquisition module is used to acquire a first bitstream, which is obtained based on a first audio signal, and the first audio signal is a low-frequency signal with a frequency lower than a preset threshold in the current frame. The extraction module is used to obtain edge information, which is obtained based on a second audio signal, the second audio signal including high-frequency signals in the current frame whose frequency is higher than or equal to the preset threshold; The encoding module is used to obtain the second bitstream based on the side information; A transmission module is used to transmit a bitstream, the bitstream including a first bitstream and a second bitstream.

20. The apparatus according to claim 19, characterized in that, The extraction module is specifically used to input the second audio signal into a pre-trained second machine learning model to obtain the side information. The second machine learning model includes at least one sub-model.

21. The apparatus according to claim 20, characterized in that, When the second machine learning model includes multiple sub-models, the input of each of the multiple sub-models includes a portion or all of the audio signal in the second audio signal, wherein the portion of the audio signal includes the high-frequency signal.

22. The apparatus according to claim 21, characterized in that, The first sub-model and at least one other sub-model among the plurality of sub-models include the same neurons, which are located in the input phase, output phase or intermediate phase of the first sub-model, and the first sub-model is any one of the plurality of sub-models.

23. The apparatus according to any one of claims 20-22, characterized in that, The extraction module is specifically used to process the second audio signal according to a pre-set rule to obtain the side information, wherein the pre-set rule includes at least one sub-rule.

24. The apparatus according to claim 23, characterized in that, When the preset rule method includes multiple sub-rule methods, the input of each of the multiple sub-rule methods includes a portion or all of the audio signal in the second audio signal, wherein the portion of the audio signal includes the high-frequency signal.

25. The apparatus according to claim 23 or 24, characterized in that, The input to the first sub-model also includes a first input; when the second machine learning model includes multiple sub-models, the first input includes the output of at least one other sub-model among the multiple sub-models, or the first input includes the output of the at least one sub-rule; the first sub-model is any one of the multiple sub-models.

26. The apparatus according to claim 23 or 24, characterized in that, The input to the first sub-model also includes a second input, which is the audio signal obtained after the first input has undergone redundant information processing; when the second machine learning model includes multiple sub-models, the first input includes the output of at least one other sub-model among the multiple sub-models, or the first input includes the output of the at least one sub-rule method; the first sub-model is any one of the multiple sub-models.

27. The apparatus according to any one of claims 23-26, characterized in that, The input of the first sub-rule method also includes a third input; when the preset rule method includes multiple sub-rule methods, the third input includes the output of at least one other sub-rule method among the multiple sub-rule methods, or the third input includes the output of the at least one sub-model; the first sub-rule method is any one of the multiple sub-rule methods.

28. The apparatus according to any one of claims 23-26, characterized in that, The input of the first sub-rule method also includes a fourth input, which is the audio signal obtained after the third input has undergone redundant information processing; when the preset rule method includes multiple sub-rule methods, the third input includes the output of at least one other sub-rule method among the multiple sub-rule methods, or the third input includes the output of the at least one sub-model; the first sub-rule method is any one of the multiple sub-rule methods.

29. A decoding device, characterized in that, include: A receiving module is used to receive a bitstream, the bitstream including a first bitstream and a second bitstream; The acquisition module is used to acquire a first audio signal based on the first bitstream, wherein the frequency of the first audio signal is lower than a preset threshold. An extraction module is used to obtain side information based on the second bitstream, wherein the side information is used to characterize the characteristics of the second audio signal, and the frequency of the second audio signal is higher than or equal to the preset threshold. The reconstruction module is used to obtain a reconstruction frame based on the first audio signal and the side information.

30. The apparatus according to claim 29, characterized in that, The reconstruction module is specifically used to obtain the reconstructed frame based on the first audio signal and the side information through a pre-trained first machine learning model.

31. The apparatus according to claim 30, characterized in that, The reconstructed frame is directly output by the first machine learning model; or, the reconstructed frame is obtained based on the second audio signal output by the first machine learning model and the first audio signal.

32. The apparatus according to claim 30 or 31, characterized in that, When the first machine learning model includes a first sub-model and a second sub-model, the reconstruction module is specifically used to input the first audio signal and the first side information into the first sub-model to obtain a third audio signal; and to input the third audio signal and the second side information into the second sub-model to obtain the reconstructed frame; wherein the first side information and the second side information are part or all of the information in the side information.

33. The apparatus according to any one of claims 30-32, characterized in that, The reconstruction module is specifically used to process the first audio signal and the third side information according to a pre-set first rule to obtain a fourth audio signal; input the fourth audio signal and the fourth side information into the first machine learning model to obtain the reconstructed frame; the third side information and the fourth side information are part or all of the side information respectively.

34. The apparatus according to any one of claims 30-33, characterized in that, The reconstruction module is specifically used to obtain a fifth audio signal based on the first audio signal and the pentagonal information through a pre-trained first machine learning model; and to process the fifth audio signal and the sixth pentagonal information according to a pre-set second rule to obtain the reconstructed frame; wherein the fifth pentagonal information and the sixth pentagonal information are part or all of the information in the pentagonal information.

35. The apparatus according to any one of claims 29-34, characterized in that, The extraction module is specifically used to parse the second bitstream to obtain the side information.

36. The apparatus according to any one of claims 29-34, characterized in that, The extraction module is specifically used to parse the second bitstream to obtain the preceding information; and to perform side information processing on the preceding information to obtain the side information, wherein the side information processing includes inputting a pre-trained third machine learning model or processing according to a pre-set third rule.

37. A device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-18.

38. A computer-readable storage medium, characterized in that, Includes a computer program, which, when executed on a computer, causes the computer to perform the method of any one of claims 1-18.

39. A computer program product, characterized in that, The computer program product includes computer program code that, when run on a computer, causes the computer to perform the method of any one of claims 1-18.

Images