Coding and decoding method, equipment, system, storage medium and program product

Abstract

The invention discloses a coding and decoding method, equipment, a system, a storage medium and a program product. The method executed by the coding equipment comprises the following steps: obtaining at least one category of audio frames according to the complexity of each audio frame in a plurality of audio frames; the audio frames of each category are coded to obtain first coded data, and the coding bit rates of the audio frames of different categories are different. By adopting the mode, the audio processing effect and efficiency can be improved.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of communication, and particularly relates to a coding and decoding method, device, system, storage medium and program product. BACKGROUND

[0002] With the continuous change of hardware performance, network conditions and application requirements, audio coding technology is rapidly developing towards high efficiency, low delay and high quality. From the traditional Pulse Code Modulation (PCM) to the efficient lossy compression technology, and then to the Artificial Intelligence (AI) driven intelligent coding technology, audio coding technology has undergone significant changes. Traditional technology focuses on compression efficiency and bandwidth optimization, while AI technology provides unprecedented improvement in audio quality, low delay, noise suppression and other aspects. SUMMARY

[0003] In order to further improve the effect of audio processing, the embodiment of the present disclosure proposes a coding and decoding method, device, system, storage medium and program product.

[0004] According to a first aspect of the embodiment of the present disclosure, a coding method is provided, which is executed by a coding device, and the method comprises: obtaining at least one category of audio frames according to the complexity of each audio frame in a plurality of audio frames; and respectively coding each category of audio frames to obtain first coding data, wherein the coding bit rate of different categories of audio frames is different.

[0005] According to a second aspect of the embodiment of the present disclosure, a decoding method is provided, which is executed by a decoding device, and the method comprises: receiving first coding data sent by a coding device, wherein the first coding data is obtained by the coding device according to the complexity of each audio frame in a plurality of audio frames to determine at least one category of audio frames, and respectively coding each category of audio frames to obtain, wherein the coding bit rate of different categories of audio frames is different; and decoding the first coding data to obtain a plurality of decoded audio frames.

[0006] According to a third aspect of the embodiments of the present disclosure, a model training method is provided, the model comprising an encoding device and a decoding device, the method comprising: obtaining, by the encoding device, sample audio frames of at least one category according to the complexity of each sample audio frame in a plurality of sample audio frames; encoding, by the encoding device, the sample audio frames of each category respectively to obtain sample first encoding data, wherein the encoding bit rates of the sample audio frames of different categories are different; decoding, by the decoding device, the sample first encoding data to obtain a plurality of sample decoded audio frames; determining a loss value according to the difference between the plurality of sample audio frames and the plurality of sample decoded audio frames and the bit number of the sample first encoding data; and updating trainable parameters in the model according to the loss value.

[0007] According to a fourth aspect of the embodiments of the present disclosure, an encoding device is provided, which is configured to implement the encoding method of the first aspect.

[0008] According to a fifth aspect of the embodiments of the present disclosure, a decoding device is provided, which is configured to implement the decoding method of the second aspect.

[0009] According to a sixth aspect of the embodiments of the present disclosure, a codec system is provided, comprising an encoding device and a decoding device, wherein the encoding device is configured to implement the encoding method of the first aspect, and the decoding device is configured to implement the decoding method of the second aspect.

[0010] According to a seventh aspect of the embodiments of the present disclosure, a storage medium is provided, which stores instructions, when the instructions are run on a communication device, causing the communication device to execute the method of the first aspect, the second aspect or the third aspect.

[0011] According to an eighth aspect of the embodiments of the present disclosure, a program product is provided, comprising at least one of a program and instructions, which are executed by a communication device to implement the method of the first aspect, the second aspect or the third aspect.

[0012] With the above technical solutions, the following beneficial technical effects can be achieved: through classification based on audio frame complexity, content-adaptive variable bit rate (VBR) encoding is achieved. This differentiated treatment of the encoding of audio frames of different categories optimizes the overall bit allocation strategy, thereby being able to significantly improve the audio processing effect and efficiency while reducing the average code rate. BRIEF DESCRIPTION OF DRAWINGS

[0013] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the following describes the drawings required for the embodiments, and the following drawings are only some embodiments of the present disclosure, and do not specifically limit the protection scope of the present disclosure.

[0014] Figure 1A is an architecture schematic diagram of a coding system according to an embodiment of the present disclosure.

[0015] Figure 1B is a model training schematic diagram according to an embodiment of the present disclosure.

[0016] Figure 1C is a model application schematic diagram according to an embodiment of the present disclosure.

[0017] Figure 2A is an interaction schematic diagram of a coding method according to an embodiment of the present disclosure.

[0018] Figure 2B is an interaction schematic diagram of a coding method according to an embodiment of the present disclosure.

[0019] Figure 2C is a flow schematic diagram of a model training method according to an embodiment of the present disclosure.

[0020] Figure 3A is a flow schematic diagram of an encoding method according to an embodiment of the present disclosure.

[0021] Figure 3B is a flow schematic diagram of a decoding method according to an embodiment of the present disclosure.

[0022] Figure 4 is an architecture schematic diagram of a coding system according to an embodiment of the present disclosure.

[0023] Figure 5A is a structure schematic diagram of an encoding device according to an embodiment of the present disclosure.

[0024] Figure 5B is a structure schematic diagram of a decoding device according to an embodiment of the present disclosure.

[0025] Figure 6A is a structure schematic diagram of a communication device according to an embodiment of the present disclosure.

[0026] Figure 6B is a structure schematic diagram of a chip according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0027] The present disclosure provides a coding method, device, system, storage medium and program product.

[0028] In a first aspect, the embodiments of the present disclosure provide an encoding method, performed by an encoding device, the method comprising: obtaining at least one category of audio frames according to the complexity of each audio frame in a plurality of audio frames; and encoding each category of audio frames respectively to obtain first encoding data, wherein the encoding bit rates of different categories of audio frames are different.

[0029] In the above embodiment, the frame classification is performed based on the complexity of each audio frame, and different encoding bit rates are used to encode different categories of audio frames, which realizes adaptive variable bit rate encoding based on the content of the audio frames, optimizes the global bit allocation, and improves the audio processing effect while reducing the average code rate.

[0030] In some embodiments in combination with the first aspect, the at least one category of audio frames includes first category audio frames and second category audio frames, and the complexity of the first category audio frames is higher than that of the second category audio frames.

[0031] In the above embodiment, the plurality of audio frames are divided into two categories with different complexities, which lays a foundation for subsequent differential processing of high complexity and low complexity audio frames, so that fine bit allocation can be realized.

[0032] In some embodiments in combination with the first aspect, in a case where the complexity of the audio frame is a first value, the audio frame belongs to the first category of audio frames; and in a case where the complexity of the audio frame is a second value, the audio frame belongs to the second category of audio frames.

[0033] In the above embodiment, each audio frame is assigned a first value or a second value to clearly distinguish its category, which provides a clear basis for accurate classification and subsequent differential layered encoding, and enhances the controllability and accuracy of the encoding process.

[0034] In some embodiments in combination with the first aspect, in a case where the complexity of the audio frame is greater than a threshold value, the audio frame is indicated as the first category of audio frames; and in a case where the complexity of the audio frame is less than or equal to the threshold value, the audio frame is indicated as the second category of audio frames, wherein the complexity of the audio frame is a value between 0 and 1.

[0035] In the above embodiment, the complexity of each audio frame is quantified using a value between 0 and 1, and classified according to a threshold value, which provides a continuous and adjustable standard for audio frame classification, making the classification more flexible and adaptive, and further optimizing the implementation of the layered encoding strategy.

[0036] In some embodiments of the first aspect, the encoding the audio frames of each category respectively to obtain the first encoded data comprises: performing full-amount encoding on the first type of audio frames to obtain second encoded data; and performing non-full-amount encoding on the second type of audio frames to obtain third encoded data, wherein the full-amount encoding has a higher encoding bit rate than the non-full-amount encoding; and the first encoded data is determined according to the second encoded data and the third encoded data.

[0037] In the above embodiments, by assigning high bit rates for high-complexity audio frames for full-amount encoding and low bit rates for low-complexity audio frames for non-full-amount encoding, non-uniform and content-adaptive bit allocation is achieved. This strategy concentrates limited bit resources on key frames (i.e., high-complexity audio frames), significantly improving overall sound quality at extremely low average bit rates, while optimizing encoding efficiency. Moreover, only the classified first type of audio frames are subjected to computationally complex full-amount encoding, and the second type of audio frames are subjected to extremely lightweight non-full-amount encoding. This dynamic computing mechanism significantly reduces average encoding delay and power consumption, making real-time applications of high-performance neural codecs on edge devices such as mobile phones and Internet of Things devices possible.

[0038] In some embodiments of the first aspect, the method further comprises: determining a total bit budget of the plurality of audio frames according to the complexity of each audio frame in the plurality of audio frames, the total bit budget being a maximum value for constraining the length of the first encoded data.

[0039] In the above embodiments, the total bit budget is determined according to the complexity of each audio frame before encoding, setting an upper limit on the length of the encoding process, which enables more accurate allocation of bit resources during encoding, ensuring more efficient encoding and better sound quality performance while meeting data length constraints.

[0040] In some embodiments of the first aspect, the method further comprises: determining a bit budget of each audio frame according to the total bit budget and the complexity of each audio frame, wherein the complexity of an audio frame is positively correlated with the corresponding allocated bit budget, and the bit budget of an audio frame is a maximum value for constraining the length of the encoded data of the audio frame.

[0041] In the above embodiments, the bit allocation strategy is further refined, with each audio frame being allocated a corresponding bit budget according to the total bit budget and the complexity of each audio frame, ensuring that audio frames with higher complexity are allocated more bit resources, thereby achieving more refined and accurate resource allocation while meeting overall data length constraints.

[0042] In some embodiments of the first aspect, the method further comprises: determining an audio feature of the plurality of audio frames; and determining the complexity of each audio frame based on the audio feature.

[0043] In the above embodiments, the audio features of the plurality of audio frames are extracted before the plurality of audio frames are classified, and the complexity of each audio frame is determined based on the features, which provides a reliable basis for subsequent accurate classification and encoding, thereby improving the audio processing effect.

[0044] In some embodiments of the first aspect, the method further comprises: sending the first encoded data to a decoding device.

[0045] In the above embodiments, the first encoded data of the plurality of audio frames is sent to a decoding device, which ensures that the encoded data can be effectively transmitted to the decoding end for decoding and restoration, and realizes a complete audio encoding and decoding process.

[0046] Furthermore, the multi-frame packaging technology combines multiple frames into one packet for transmission, which greatly reduces the proportion of metadata. At the same time, global bit allocation optimization is performed in the packet to improve the payload ratio and overall compression efficiency.

[0047] In the second aspect, the disclosure embodiments provide a decoding method, which is performed by a decoding device and comprises: receiving first encoded data sent by an encoding device, wherein the first encoded data is obtained by the encoding device based on the complexity of each audio frame in a plurality of audio frames to determine at least one category of audio frames, and each category of audio frames is encoded to obtain the first encoded data, wherein the encoding bit rate of different categories of audio frames is different; and decoding the first encoded data to obtain a plurality of decoded audio frames.

[0048] In some embodiments of the second aspect, the first encoded data comprises second encoded data and third encoded data, wherein the second encoded data is obtained by the encoding device by full-amount encoding of first category audio frames, and the third encoded data is obtained by the encoding device by non-full-amount encoding of second category audio frames, and the complexity of the first category audio frames is higher than that of the second category audio frames.

[0049] The decoding of the first encoded data to obtain a plurality of decoded audio frames comprises: full-amount decoding of the second encoded data to obtain first category decoded audio frames; and non-full-amount decoding of the third encoded data to obtain second category decoded audio frames based on the non-full-amount decoding result.

[0050] In the above embodiments, a hierarchical decoding strategy of "full-quantity decoding + non-full-quantity decoding (including prediction)" is adopted. The full-quantity decoding ensures perfect reconstruction of details of important frames (i.e., audio frames with high complexity), and the non-full-quantity decoding is specially optimized for intelligent recovery of high-frequency information and psychoacoustic details of simple frames (i.e., audio frames with low complexity). The two work together to improve the reconstruction fidelity of all types of audio (speech, music, environmental sound).

[0051] In combination with some embodiments of the second aspect, in some embodiments, obtaining the second type of decoded audio frame based on the non-full-quantity decoding result comprises: predicting non-full-quantity encoding loss information based on the non-full-quantity decoding result, and obtaining the second type of decoded audio frame based on the non-full-quantity decoding result and the non-full-quantity encoding loss information.

[0052] In a third aspect, the embodiments of the present disclosure provide a model training method, the model comprising an encoding device and a decoding device, the method comprising: the encoding device obtaining at least one category of sample audio frames according to the complexity of each sample audio frame in a plurality of sample audio frames; the encoding device encoding each category of sample audio frames respectively to obtain sample first encoding data, wherein the encoding bit rates of sample audio frames of different categories are different; the decoding device decoding the sample first encoding data to obtain a plurality of sample decoded audio frames; determining a loss value based on the difference between the plurality of sample audio frames and the plurality of sample decoded audio frames and the bit number of the sample first encoding data; and updating trainable parameters in the model based on the loss value.

[0053] In a fourth aspect, the embodiments of the present disclosure provide an encoding device, which can comprise at least one of a transceiver module and a processing module; wherein the encoding device can be configured to perform the optional implementation manners of the first aspect.

[0054] In a fifth aspect, the embodiments of the present disclosure provide a decoding device, which can comprise at least one of a transceiver module and a processing module; wherein the decoding device can be configured to perform the optional implementation manners of the second aspect.

[0055] In a sixth aspect, the embodiments of the present disclosure provide an encoding device, which can comprise one or more processors; wherein the encoding device can be configured to perform the optional implementation manners of the first aspect.

[0056] In a seventh aspect, the embodiments of the present disclosure provide a decoding device, which can comprise one or more processors; wherein the decoding device can be configured to perform the optional implementation manners of the second aspect.

[0057] In an eighth aspect, the embodiments of the present disclosure provide a coding system, which can include an encoding device and a decoding device. The encoding device is configured to perform the method described in the optional implementation manner of the first aspect, and the decoding device is configured to perform the method described in the optional implementation manner of the second aspect.

[0058] In a ninth aspect, the embodiments of the present disclosure provide a storage medium storing instructions that, when executed on a communication device, cause the communication device to perform the method described in the optional implementation manner of at least one of the first aspect, the second aspect, and the third aspect.

[0059] In a tenth aspect, the embodiments of the present disclosure provide a program product that, when executed by a communication device, causes the communication device to perform the method described in the optional implementation manner of at least one of the first aspect, the second aspect, and the third aspect.

[0060] In an eleventh aspect, the embodiments of the present disclosure provide a computer program that, when executed on a computer, causes the computer to perform the method described in the optional implementation manner of at least one of the first aspect, the second aspect, and the third aspect.

[0061] In a twelfth aspect, the embodiments of the present disclosure provide a chip or chip system. The chip or chip system includes processing circuitry configured to perform the method described in the optional implementation manner of at least one of the first aspect, the second aspect, and the third aspect.

[0062] It can be understood that the above-mentioned encoding device, decoding device, storage medium, program product, computer program, chip or chip system can be used to perform the method proposed in the embodiments of the present disclosure. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method, which will not be described here.

[0063] The embodiments of the present disclosure provide a coding method, device, system, storage medium, and program product. In some embodiments, the coding method and the terms such as multi-frame adaptive coding method and communication method implemented based on a neural network can be replaced with each other.

[0064] The embodiments of the present disclosure are not exhaustive, but only illustrate some embodiments, and are not specific limitations on the protection scope of the present disclosure. In the case of no contradiction, each step in an embodiment can be implemented as an independent embodiment, and the steps can be combined arbitrarily, for example, the scheme after removing some steps in an embodiment can also be implemented as an independent embodiment, and the order of the steps in an embodiment can be exchanged arbitrarily, in addition, the optional implementation manners in an embodiment can be combined arbitrarily; in addition, the embodiments can be combined arbitrarily, for example, the steps of different embodiments can be combined arbitrarily, an embodiment can be combined with the optional implementation manners of other embodiments. In each embodiment of the present disclosure, the terms and / or descriptions between the embodiments are consistent and can be referred to each other if there is no special description and logical conflict, and the technical features in different embodiments can be combined to form a new embodiment according to their inherent logical relationship.

[0065] The terms used in the embodiments of the present disclosure are only for the purpose of describing the specific embodiments, and not as a limitation on the present disclosure.

[0066] In the embodiments of the present disclosure, unless otherwise specified, the elements expressed in singular form, such as "one", "a", "the", "above", "said", "preceding", "this" and the like, can represent "one and only one", and can also represent "one or more", "at least one" and the like. For example, in the case of using articles such as "a", "an", "the" and the like in English, the noun after the article can be understood as singular expression, and can also be understood as plural expression.

[0067] In the embodiments of the present disclosure, "a plurality of" means two or more.

[0068] In some embodiments, the terms "at least one of A or B", "at least one of A and B", "one or more", "a plurality of", "multiple" and the like can be replaced with each other.

[0069] In some embodiments, the description of "at least one of A, B", "A and / or B", "A in one case and B in another", "A in response to one case and B in response to another", etc. can include the following technical solutions according to the case: in some embodiments, A (A is executed regardless of whether B branch exists); in some embodiments, B (B is executed regardless of whether A branch exists); in some embodiments, A and B are selectively executed (A and B are selectively executed); in some embodiments, A and B (A and B are executed). When there are more branches of A, B, C, etc., the above is similar.

[0070] In some embodiments, the description of "A or B" and the like can include the following technical solutions according to the case: in some embodiments, A (A is executed regardless of whether B branch exists); in some embodiments, B (B is executed regardless of whether A branch exists); in some embodiments, A and B are selectively executed (A and B are selectively executed). When there are more branches of A, B, C, etc., the above is similar.

[0071] The prefix words "first", "second" and the like in the embodiments of the present disclosure are only used to distinguish different description objects, and do not constitute a limitation on the position, order, priority, quantity or content of the description objects. The description of the description objects should refer to the description in the context of the claims or embodiments, and should not constitute an additional limitation because of the use of the prefix words. For example, the description object is "field", and the ordinal words before "field" in "first field" and "second field" do not limit the position or order between "fields", and "first" and "second" do not limit whether the "fields" modified by them are in the same message or not, nor limit the order of "first field" and "second field". For another example, the description object is "level", and the ordinal words before "level" in "first level" and "second level" do not limit the priority between "levels". For another example, the quantity of the description object is not limited by the ordinal words, and can be one or more. For example, "first device", where the quantity of "device" can be one or more. In addition, the objects modified by different prefix words can be the same or different, for example, the description object is "device", and "first device" and "second device" can be the same device or different devices, and their types can be the same or different; for another example, the description object is "information", and "first information" and "second information" can be the same information or different information, and their contents can be the same or different.

[0072] In some embodiments, "including A", "containing A", "for indicating A", "carrying A" can be interpreted as directly carrying A, or indirectly indicating A.

[0073] In some embodiments, the terms "time / frequency", "time / frequency domain" and the like refer to time domain and / or frequency domain.

[0074] In some embodiments, the terms "in response to", "in response to determining", "in the case of", "when", "if", "if", "if", etc. can be replaced with each other, which means that the device will make corresponding processing under certain objective circumstances, not necessarily limited to time, and does not require the device to have a judgment action when implemented, nor does it mean that there must be other limitations.

[0075] In some embodiments, the terms "greater than", "greater than or equal to", "not less than", "more than", "more than or equal to", "not less than", "higher than", "higher than or equal to", "not lower than", "above", etc. can be replaced with each other, and the terms "less than", "less than or equal to", "not greater than", "less than", "less than or equal to", "not more than", "lower than", "lower than or equal to", "not higher than", "below", etc. can be replaced with each other.

[0076] In some embodiments, the device, etc. can be interpreted as physical or virtual, and its name is not limited to the name recorded in the embodiment. The terms "device", "equipment", "device", "circuit", "network element", "network function", "network device", "function", "node", "unit", "section", "system", "network", "chip", "chip system", "entity", "subject" can be replaced with each other.

[0077] In some embodiments, "network" can be interpreted as a device (for example, access network device, core network device, etc.) contained in the network.

[0078] In some embodiments, the terms “access network device (AN device),” “radio access network device (RAN device),” “base station (BS),” “radio base station,” “fixed station,” “node,” “access point,” “transmission point (TP),” “reception point (RP),” “transmission / reception point (TRP),” “panel,” “antenna panel,” “antenna array,” “cell,” “macro cell,” “small cell,” “femto cell,” “pico cell,” “sector,” “cell group,” “serving cell,” “carrier,” “component carrier,” “bandwidth part (BWP),” and the like can be used interchangeably.

[0079] In some embodiments, the terms "terminal," "terminal device," "user equipment (UE)," "user terminal," "mobile station (MS)," "mobile terminal (MT)," "subscriber station," "mobile unit," "subscriber unit," "wireless unit," "remote unit," "mobile device," "wireless device," "wireless communication device," "remote device," "mobile subscriber station," "access terminal," "mobile terminal," "wireless terminal," "remote terminal," "handset," "user agent," "mobile client," "client," and so on can be replaced with each other.

[0080] In some embodiments, the access network device, the core network device, or the network device can be replaced with a terminal. For example, the embodiments of the present disclosure can also be applied to a structure in which communication between the access network device, the core network device, or the network device and the terminal is replaced with communication between a plurality of terminals (e.g., device-to-device (D2D), vehicle-to-everything (V2X), etc.). In this case, the terminal can also be configured to have all or part of the functions of the access network device. In addition, the terms "uplink," "downlink," and the like can also be replaced with terms corresponding to the inter-terminal communication (e.g., "side"). For example, the uplink channel, the downlink channel, and the like can be replaced with the side channel, and the uplink, the downlink, and the like can be replaced with the sidelink.

[0081] In some embodiments, the terminal can be replaced with the access network device, the core network device, or the network device. In this case, the access network device, the core network device, or the network device can also be configured to have all or part of the functions of the terminal.

[0082] In some embodiments, the data, information, etc. can be obtained in compliance with the laws and regulations of the country where the location is situated.

[0083] In some embodiments, the data, information, etc. can be obtained after obtaining the consent of the user.

[0084] In addition, each element, each row, or each column in the table of the embodiments of the present disclosure can be implemented as an independent embodiment, and any combination of any element, any row, or any column can also be implemented as an independent embodiment.

[0085] Figure 1A is a schematic diagram of a coding system according to an embodiment of the present disclosure. As shown in the figure, the coding system 100 can include an encoding device 101 and a decoding device 102. It should be noted that the coding system 100 can also include other devices, and the present disclosure does not limit the devices included in the coding system 100. Figure 1A

[0086] In some embodiments, the encoding device can be referred to as an encoder, an encoding apparatus, etc. The decoding device can be referred to as a decoder, a decoding apparatus, etc.

[0087] In some embodiments, the encoding device 101 and / or the decoding device 102 can be a terminal or be arranged in a terminal.

[0088] In some embodiments, the terminal can include at least one of a mobile phone, a wearable device, an Internet of Things device, a car with communication function, a smart car, a tablet computer (Pad), a computer with wireless transceiver function, a virtual reality (VR) terminal device, an augmented reality (AR) terminal device, a wireless terminal device in industrial control, a wireless terminal device in self-driving, a wireless terminal device in remote medical surgery, a wireless terminal device in smart grid, a wireless terminal device in transportation safety, a wireless terminal device in smart city, and a wireless terminal device in smart home, but is not limited thereto.

[0089] In some embodiments, the encoding device 101 and / or the decoding device 102 can be a network device or be arranged in a network device.

[0090] In some embodiments, the network device can include at least one of an access network device and a core network device. In some embodiments, the network device can include at least one of an access network device and a core network device.

[0091] In some embodiments, the access network device is, for example, at least one of a node or a device that accesses a terminal to a wireless network, and the access network device can include at least one of an evolved node B (eNB) in a 5G communication system, a next generation eNB (ng-eNB), a next generation node B (gNB), a node B (NB), a home node B (HNB), a home evolved node B (HeNB), a wireless backhaul device, a radio network controller (RNC), a base station controller (BSC), a base transceiver station (BTS), a base band unit (BBU), a mobile switching center, a base station in a 6G communication system, an open base station (Open RAN), a cloud base station (Cloud RAN), a base station in other communication systems, an access node in a Wi-Fi system, but is not limited thereto.

[0092] In some embodiments, the access network device is a base station. Optionally, the base station is, for example, a macro base station, a micro base station (also known as a small station), a relay station, an access point, a 5 / 6G base station or a future base station, a satellite, a transmitting and receiving point (TRP), a transmitting point (TP), a mobile switching center, or other devices that perform the functions of a base station in a communication system, etc., and the embodiments of the present disclosure do not make specific limitations thereto. For the convenience of description, in all embodiments of the present disclosure, the apparatuses that provide wireless communication functions for terminal devices are collectively referred to as network devices or base stations.

[0093] In some embodiments, the core network device can be one device including all or part of the first network element, the second network element, etc., or can be a plurality of devices or device groups including all or part of the first network element, the second network element, etc. The network element can be virtual or physical. The core network includes, for example, at least one of an evolved packet core (EPC), a 5 / 6G core network (5G / 6G Core Network, 5G CN / 6G CN), a next generation core (NGC).

[0094] In some embodiments, the technical solutions of the present disclosure can be applied to an Open RAN architecture, at this time, the interfaces between or within the access network devices involved in the embodiments of the present disclosure can become internal interfaces of the Open RAN, and the processes and information interactions between these internal interfaces can be implemented through software or programs.

[0095] In some embodiments, the access network device can be composed of a central unit (CU) and a distributed unit (DU), where the CU can also be referred to as a control unit. The CU-DU structure can split the protocol layers of the access network device, and some of the protocol layers are controlled by the CU, and the rest or all of the protocol layers are distributed in the DU and controlled by the CU. However, the present disclosure is not limited thereto.

[0096] It can be understood that the codec system described in the embodiments of the present disclosure is for more clearly illustrating the technical solutions of the embodiments of the present disclosure, and does not constitute a limitation on the technical solutions proposed by the embodiments of the present disclosure. Those skilled in the art can know that, with the evolution of system architecture and the appearance of new business scenarios, the technical solutions proposed by the embodiments of the present disclosure are also applicable to similar technical problems.

[0097] The following embodiments of the present disclosure can be applied to Figure 1A The codec system 100 shown or part of the main body, but not limited thereto. Figure 1A The main body shown is an example, and the codec system can include Figure 1A All or part of the main body in Figure 1A Other main bodies, the number and form of each main body is arbitrary, each main body can be physical or virtual, the connection relationship between each main body is an example, each main body can not be connected or can be connected, the connection can be any way, can be direct connection or indirect connection, can be wired connection or wireless connection.

[0098] Embodiments of the present disclosure can be applied to Long Term Evolution (LTE), LTE-Advanced (LTE-A), LTE-Beyond (LTE-B), SUPER 3G, IMT-Advanced, 4th generation mobile communication system (4G), 5th generation mobile communication system (5G), 5G New Radio (NR), Future Radio Access (FRA), New-Radio Access Technology (RAT), New Radio (NR), New Radio Access (NX), Future generation radio access (FX), Global System for Mobile communications (GSM (registered trademark)), CDMA2000, Ultra Mobile Broadband (UMB), IEEE 802.11 (Wi-Fi (registered trademark)), IEEE 802.16 (WiMAX (registered trademark)), IEEE 802.20, Ultra-WideBand (UWB), Bluetooth (Bluetooth (registered trademark)), Public Land Mobile Network (PLMN) network, Device-to-Device (D2D) system, Machine to Machine (M2M) system, Internet of Things (IoT) system, Vehicle-to-Everything (V2X), system using other communication methods, next-generation system expanded based thereon, and the like. Further, a plurality of systems can be applied in combination (for example, combination of LTE or LTE-A and 5G, and the like).

[0099] In some embodiments, the present disclosure relates to the field of audio compression and coding, particularly the field of improving audio quality under bandwidth constraints. It belongs to the cross-field of audio coding, bandwidth extension, and multimedia signal processing. Specifically, embodiments of the present disclosure can be applied to real-time voice interaction in scenarios such as Voice over Internet Protocol (VoIP), satellite phones, Bluetooth headsets, audio streaming platforms, video conferencing systems, in-vehicle voice interaction systems, telemedicine, and hearing aid devices. In addition, the technical solutions of the present disclosure are not only applicable to traditional communication products, but can also be extended to the following technical fields:

[0100] Audio streaming and online platforms: can improve audio quality and smoothness, ensuring a good listening experience.

[0101] In-vehicle audio systems: can optimize in-vehicle voice assistants and audio playback effects, improving voice recognition accuracy.

[0102] Augmented Reality / Virtual Reality (AR / VR): can enhance immersion and improve audio experience through bandwidth optimization.

[0103] Smart home and Internet of Things (IoT) field: can improve voice control of smart devices and audio performance of home entertainment systems.

[0104] Speech recognition and audio enhancement field: can improve audio recognition accuracy and reduce background noise.

[0105] Hearing aid and telemedicine field: can ensure audio quality in telemedicine and hearing enhancement devices, improving diagnosis and treatment effects.

[0106] In some embodiments, with the continuous changes in hardware performance, network conditions, and application requirements, audio coding technology is rapidly developing towards high efficiency, low latency, and high quality, undergoing a transformation from traditional PCM to efficient lossy compression technology, and then to AI-driven intelligent coding technology. Traditional technologies focus on compression efficiency and bandwidth optimization, while AI technology provides unprecedented improvements in audio quality, low latency, and noise suppression. Currently, common audio coding technologies mainly include the following categories:

[0107] Traditional codecs: The first audio codec technology is based on PCM, which is a lossless encoding method that directly converts analog audio signals into digital signals. However, due to its large bandwidth occupation, it limits its use in low-bandwidth environments. In order to more efficiently transmit and store audio data, various lossy compression algorithms have emerged. For example, the emergence of MP3 format marks the revolution of digital audio compression. It uses efficient lossy compression technology to greatly reduce the size of audio files while retaining high sound quality, becoming the most popular audio format in the world. After MP3, AAC became another widely used audio codec format. It uses more advanced compression algorithms to provide higher sound quality at lower bit rates. Opus, as a codec designed for low-latency communication, is particularly suitable for VoIP, video conferencing, and real-time communication. It integrates multiple audio encoding technologies, supports audio transmission from low bit rates to high bit rates, and has excellent speech clarity.

[0108] AI codecs: With the rise of artificial intelligence and deep learning technologies, audio codec technology has also undergone fundamental changes. By combining cutting-edge technologies such as convolutional neural networks (CNN), recurrent neural networks (RNN), variational autoencoders (VAE), generative adversarial networks (GAN), and diffusion models, multiple end-to-end AI codec models have emerged. These models learn the underlying features of data, not only improving encoding efficiency, but also significantly improving the quality of compressed data. At the same time, they can provide efficient compression and decompression solutions in multiple fields, especially in the fields of speech, audio, and video processing, showing great application potential.

[0109] In some embodiments, a high-fidelity universal neural audio compression algorithm based on an improved residual vector quantization generative adversarial network (RVQGAN) is generated, which achieves 90 times compression of 44.1KHz audio to only 8kbps bandwidth.

[0110] In some embodiments, RVQGAN is an improvement of VQGAN, and the training of VQGAN is as shown in Figure 1B and the inference of VQGAN is as shown in Figure 1CThe full convolutional encoder receives time-domain waveforms as input and generates a latent representation of the input audio samples, which is quantized using a variable number of residual vector quantizers (RVQ). An optional conditioning input (denoising on / off) can be used to indicate whether background noise must be removed from the audio. When deploying the model, the encoder and quantizers on the sender send the compressed bitstream to the receiver client, which then receives the quantized embeddings with the full convolutional decoder to reconstruct the original waveforms. At training time, the model is trained end-to-end using reconstruction and adversarial losses.

[0111] In some embodiments, improvements are made based on this model:

[0112] Optimizing the problem of underutilized bandwidth due to codebook collapse (part of the codebook is not used) in the model: introducing factorized code and L2 normalized code, thereby improving codebook utilization.

[0113] Optimizing the problem of audio quality loss caused by the technology that allows a single model to support variable bit rates: quantizers are discarded with a certain probability.

[0114] Introducing periodic inductive bias (using snake activation function), multi-scale STFT discriminator, multi-scale Mel loss.

[0115] However, the improved model described above still has difficulty in reconstructing some challenging audio. By slicing the results by domain (speech, music, environment), it can be found that it performs best in speech, but has more problems in environmental sounds. And it cannot perfectly simulate some musical instruments, such as piano or synthesizer sounds. And whether it is the quantizer discard in VQGAN or the improved quantizer discard with a certain probability, it is essentially a passive and uniform degradation. For complex frames, this can cause a serious drop in sound quality. In addition, in order to be able to output high code rates, the encoder must always calculate the most complex path (all RVQ layers), even in low code rate mode, resulting in a waste of computing resources.

[0116] In some embodiments, the neural audio codec may have the following core shortcomings and deficiencies when pursuing high compression rate and variable bit rate:

[0117] 1. Blindness and uniformity of timbre reconstruction: The VBR (Variable Bitrate or Dynamic Bitrate) technology in related technologies (such as quantizer dropping) is essentially a passive and uniform quality reduction strategy. It treats all audio frames equally and drops information to the same extent. This results in significant loss of details and significant decrease in reconstruction quality for important frames in "challenging audio" such as music and complex environmental sounds, while there is bit waste for silent frames or simple frames. This "one-size-fits-all" approach cannot achieve perceptually optimal bit allocation.

[0118] 2. Inefficiency and waste of computing resources: In order to support high bit rate output, the encoder must perform all calculations (such as calculating all RVQ layers) completely at any bit rate, even if most of the calculation results will be discarded in low bit rate mode. This design results in unnecessary computing energy consumption, making the model difficult to deploy on resource-constrained edge sides such as mobile devices and IoT devices, and making it difficult to use.

[0119] 3. Insufficient modeling capability for complex audio content: The coding and decoding model in related technologies processes all types of audio (speech, music, environmental sound) through a single architecture, lacking fine-grained perceptual ability for audio content. This results in problems such as blurring, metallic artifacts, or loss of details when reconstructing certain high-frequency rich, transient characteristic sounds (such as piano, synthesizer, metal impact sound) or complex environmental sound, with limited fidelity.

[0120] Therefore, the present disclosure proposes a coding method, device, system, storage medium and program product to improve the effect and efficiency of audio processing.

[0121] Figure 2A is an interaction diagram of a coding method according to an embodiment of the present disclosure. The method can be performed by the coding system 100 described above. As shown in Figure 2A , the method includes at least one of the following steps:

[0122] Step S2101, the encoding device 101 determines the audio features of a plurality of audio frames.

[0123] In some embodiments, the plurality of audio frames are audio frames within a time window.

[0124] In some embodiments, the plurality of audio frames are a plurality of consecutive audio frames, i.e. a plurality of consecutive audio frames. For example, assuming that a sequence of audio frames corresponding to a piece of speech is A, B, C, D, E, F, then the plurality of audio frames can refer to B, C, D, or can refer to C, D, E, F, or can refer to C, D, or can refer to A, B, C, D, etc.

[0125] In some embodiments, the terms "multiple audio frames", "multiple frames of audio data", "multiple frames of data", "multiple frames of audio", "audio signal within a time window", and the like can be used interchangeably.

[0126] In some embodiments, the multiple audio frames can be collected by the encoding device or sent to the encoding device after being collected by a collection device.

[0127] In some embodiments, the encoding device can obtain an audio frame at a time interval. Alternatively, the time point at which the encoding device obtains an audio frame can be set by the encoding device itself or agreed upon by a protocol, and the embodiments of the present disclosure do not limit this.

[0128] In some embodiments, the multiple audio frames can be multiple frames of continuous audio waveform signals with an arbitrary sampling rate (for example, 16 kilohertz (kHz), 14 kHz, or 10 kHz, etc.).

[0129] In some embodiments, the encoding device can include a feature extraction unit. The feature extraction unit is configured to convert the multiple audio frames from a time-domain waveform to a high-dimensional feature representation. The high-dimensional feature representation is, for example, a high-dimensional feature vector. The name of the feature extraction unit is not limited.

[0130] In some embodiments, the feature extraction unit can include a backbone network including multiple layers of one-dimensional convolution (1D-Convolution) and / or Gated Recurrent Unit (GRU). The convolution layer can be used to quickly extract local spectral features, and the GRU can be used to model the dependency on the time series.

[0131] For example, the multiple audio frames are input into the feature extraction unit, and after being processed by the backbone network, audio features are output. The audio features are the high-dimensional feature representation of the multiple audio frames.

[0132] It should be noted that the terms "audio features", "feature vectors", "abstract representations", "high-dimensional vectors", "abstract vectors", and "latent vectors" can be used interchangeably.

[0133] In step S2102, the encoding device 101 determines the complexity of each audio frame according to the audio features.

[0134] In some embodiments, the complexity of an audio frame can be understood as perceptual importance, audio complexity, semantic importance, etc. In some embodiments, the terms "complexity", "classification parameter", and "importance" can be used interchangeably.

[0135] It should be explained that perceptual importance can be understood as analyzing the audio frame (of the feature vector) based on the characteristics of the human auditory system to determine the degree of influence of the audio frame on auditory perception. For example, assuming that the human auditory system is more sensitive to sounds in certain frequency ranges (such as low, medium, and high frequencies), the perceptual importance of the audio frames in these frequency ranges is higher.

[0136] In some embodiments, perceptual importance can be understood as analyzing the audio frame according to psychoacoustic principles to determine the importance of the audio frame at the psychological level. For example, assuming that sudden changes in volume, sudden appearance of high-frequency sounds, etc. are more likely to attract human attention, the perceptual importance of these audio frames is higher.

[0137] In some embodiments, the audio complexity can be understood as at least one of spectral complexity, time-domain complexity, and energy distribution complexity.

[0138] Among them, the spectral complexity refers to the complexity of the audio signal in the frequency domain, which reflects the distribution and change of the audio signal at different frequency points.

[0139] The time-domain complexity refers to the complexity of the audio signal in the time domain, which reflects the change and structural characteristics of the audio signal at different time points. For example, the time-domain complexity of the audio frame is higher in the shorter time. For example, the time-domain complexity of the audio frame of transient change (such as percussion) is high.

[0140] The energy distribution complexity refers to the complexity of the energy distribution of the audio signal in different frequency or time regions.

[0141] In some embodiments, semantic importance can be understood as the key degree of the content of the audio frame in conveying specific information or meaning. It evaluates the importance of the audio frame at the semantic level, that is, the contribution of the content of the audio frame to understanding the overall information, emotion or intention, such as the contribution degree to speech recognition, audio classification, music information retrieval.

[0142] In some embodiments, the complexity of each audio frame in the plurality of audio frames can be determined according to the audio features of the plurality of audio frames. Or, the audio features of the plurality of audio frames represent the complexity of each audio frame in the plurality of audio frames.

[0143] In some embodiments, the encoding device can include an importance evaluation head. The importance evaluation head is used to determine the complexity of the audio frame according to the audio features.

[0144] In some embodiments, the importance evaluation head can include a fully connected layer (FC) + activation function (such as Softmax / Sigmoid function) connected after the main network.

[0145] For example, the audio features of the plurality of audio frames are input into the importance evaluation head for processing, and the importance evaluation head outputs the complexity of each audio frame.

[0146] In some embodiments, the complexity of each audio frame can be represented by a first value or a second value. For example, assuming that the first value and the second value are 0 and 1, the output of the importance evaluation head can be [0, 1, …, 0].

[0147] In some embodiments, the complexity of each audio frame can be represented by a value between 0 and 1. For example, the output of the importance evaluation head can be [0.1, 0.05, 0.3, …, 0.15].

[0148] It should be noted that the complexity of each audio frame can be represented by a value greater than or equal to 1, or the complexity of each audio frame can be represented by symbols, characters, etc., which are not limited by the present disclosure.

[0149] In step S2103, the encoding device 101 divides the plurality of audio frames into first type audio frames and second type audio frames according to the complexity of each audio frame in the plurality of audio frames.

[0150] In some embodiments, after determining the complexity of each audio frame, the plurality of audio frames can be divided into first type audio frames and second type audio frames according to the complexity of each audio frame. The complexity of the first type audio frames is different from the complexity of the second type audio frames. Alternatively, the complexity of the first type audio frames is higher than the complexity of the second type audio frames. Of course, in some embodiments, the complexity of the first type audio frames can be lower than the complexity of the second type audio frames.

[0151] In some embodiments, if the complexity of an audio frame is a first value or a second value, and the first value indicates that the audio frame is a first type audio frame, and the second value indicates that the audio frame is a second type audio frame. Then the implementation of the encoding device for dividing the plurality of audio frames into first type audio frames and second type audio frames according to the complexity of each audio frame includes: taking the audio frame with the first value as the first type audio frame, and taking the audio frame with the second value as the second type audio frame.

[0152] For example, in the case where the complexity of an audio frame is a first value, it indicates that the audio frame belongs to the first type audio frame. In the case where the complexity of an audio frame is a second value, it indicates that the audio frame belongs to the second type audio frame.

[0153] In some embodiments, the first value is 0 and the second value is 1. In some embodiments, the first value is 1 and the second value is 0.

[0154] In some embodiments, if the complexity of each audio frame is represented by a value between 0 and 1. Then, in the case that the complexity of an audio frame is greater than a threshold, the audio frame is represented as a first type of audio frame. In the case that the complexity of an audio frame is less than or equal to the threshold, the audio frame is represented as a second type of audio frame.

[0155] For example, the implementation in which the encoding device divides the plurality of audio frames into the first type of audio frame and the second type of audio frame according to the complexity of each audio frame includes: taking an audio frame with a complexity greater than a threshold as the first type of audio frame, and taking an audio frame with a complexity less than or equal to the threshold as the second type of audio frame, the complexity of the audio frame being a value between 0 and 1.

[0156] Of course, in some embodiments, an audio frame with a value greater than a threshold can also be taken as a second type of audio frame, and an audio frame with a value less than or equal to the threshold can be taken as a first type of audio frame.

[0157] In some embodiments, the threshold can be indicated by the network device, or the threshold can be pre-set, or the threshold can be adaptively determined according to the scene, which is not limited by the present disclosure.

[0158] Step S2104, the encoding device 101 determines a total bit budget according to the complexity of each audio frame.

[0159] In some embodiments, the encoding device can allocate a total bit budget for the plurality of audio frames by performing a VBR strategy. For example, the encoding device determines a total bit budget for the plurality of audio frames based on a VBR strategy according to the complexity of each audio frame in the plurality of audio frames.

[0160] Wherein, the total bit budget is a maximum value for constraining the total encoding length (the length of the first encoding data) of the plurality of audio frames.

[0161] For example, the encoding device can dynamically adjust the bit rate according to the complexity of each audio frame in the plurality of audio frames based on the VBR principle. Although the bit rate is variable, a target average bit rate (ABR) can be set. This target average bit rate is a rough guide value for controlling the overall encoding quality. The encoding device can estimate a total bit budget according to the target average bit rate and the total duration of the content. This estimated total bit budget is used to guide the entire encoding process to ensure that the final output file size (i.e. the first encoding data) does not exceed the total bit budget. That is, the encoding device needs to ensure that the total bits used by all audio frames (i.e. the plurality of audio frames) do not exceed the total bit budget, while intelligently allocating bits according to the complexity of each audio frame to optimize the encoding quality.

[0162] At step S2105, the encoding device 101 performs full-amount encoding on the first type of audio frame to obtain second encoding data.

[0163] In some embodiments, the encoding device comprises a full-amount encoding module configured to perform full-amount encoding on the first type of audio frame with high complexity to obtain second encoding data.

[0164] At step S2106, the encoding device 101 performs non-full-amount encoding on the second type of audio frame to obtain third encoding data.

[0165] In some embodiments, the encoding device comprises a non-full-amount encoding module configured to perform non-full-amount encoding on the second type of audio frame with low complexity to obtain third encoding data. The term “non-full-amount encoding” can be replaced by the terms “core encoding”, “lossy encoding”, “degrading encoding”, “lightweight encoding”, and the like.

[0166] In some embodiments, the encoding bitrates of the audio frames of different categories are different. For example, the encoding bitrate of the full-amount encoding on the first type of audio frame is greater than the encoding bitrate of the non-full-amount encoding on the second type of audio frame.

[0167] In some embodiments, the encoding device can determine the bit budget of each audio frame according to the total bit budget and the complexity of each audio frame. Optionally, the higher the complexity of an audio frame, the more bit budget is allocated to the audio frame. That is, the complexity of an audio frame is positively correlated with the corresponding allocated bit budget. The bit budget of an audio frame is the maximum value used to constrain the length of the encoding data of the audio frame.

[0168] In some embodiments, the encoding device can comprise a bit allocator. The bit allocator is an algorithm module rather than a learning module. The bit allocator is configured to determine the bit budget of each audio frame according to the complexity of each audio frame and the total bit budget.

[0169] In some embodiments, if the complexity of each audio frame is represented by a value between 0 and 1, the bit allocator is configured to calculate the bit budget of each audio frame according to the following formula:

[0170]

[0171] wherein b i represents the bit budget of the i-th audio frame, w i represents the complexity of the i-th audio frame, sum(w) represents the sum of the complexities of all audio frames, and b total represents the total bit budget.

[0172] During the encoding process where the encoding device performs full encoding of the first type of audio frames to obtain the second type of encoded data, and performs partial encoding of the second type of audio frames to obtain the third type of encoded data, the encoding device needs to ensure that the number of bits used in the i-th audio frame does not exceed b. i Furthermore, it is necessary to ensure that the total number of bits used by all audio frames does not exceed the total bit budget.

[0173] In some embodiments, the bit allocator is a learnable module. If the complexity of an audio frame is a first value or a second value, then the bit allocator is used to ensure that, during the encoding process of fully encoding a first type of audio frame to obtain second encoded data and partially encoding a second type of audio frame to obtain third encoded data, the bits allocated to the first type of audio frame are as many as possible as the bits allocated to the second type of audio frame.

[0174] It should be noted that the bit rate of full encoding is greater than that of non-full encoding. Full encoding may encode the complete audio spectrum information to ensure high-fidelity reconstruction quality by subsequent decoding equipment. Non-full encoding may use an extremely low bit rate and may only encode very basic information (e.g., very low-dimensional representations or only fundamental frequency information).

[0175] In some embodiments, the encoding device 101 determines the first encoded data based on the second encoded data and the third encoded data.

[0176] For example, the second and third encoded data are concatenated or merged to obtain the first encoded data.

[0177] In step S2107, the encoding device 101 sends the first encoded data to the decoding device 102.

[0178] In some embodiments, the decoding device receives first encoded data. For example, encoding device 101 sends the first encoded data to decoding device 102, and decoding device 102 receives the first encoded data sent by encoding device 101. The first encoded data includes second encoded data and third encoded data.

[0179] In some embodiments, the encoding device may encapsulate the second encoded data and the third encoded data into a multi-frame data packet and send the data packet to the decoding device. The decoding device receives the data packet and obtains the first encoded data.

[0180] In step S2108, the decoding device 102 performs full decoding of the second encoded data to obtain the first type of decoded audio frame.

[0181] In some embodiments, the decoding device includes a full decoding module for performing full decoding on the second encoded data to obtain a first type of decoded audio frame.

[0182] Step S2109, the decoding device 102 performs non-full-amount decoding on the third encoded data, and predicts non-full-amount encoding loss information according to the non-full-amount decoding result, and obtains the second type of decoded audio frame according to the non-full-amount decoding result and the non-full-amount encoding loss information.

[0183] In some embodiments, the decoding device comprises a non-full-amount decoding module, which comprises a core decoding module and a bandwidth extension (BWE) module. The core decoding module is used to decode low-frequency basic information from the third encoded data, and then input the low-frequency basic information into the bandwidth extension module to obtain the second type of decoded audio frame.

[0184] The bandwidth extension module can be a neural network, which intelligently predicts and reconstructs high-frequency information from low-frequency information, thereby restoring full-bandwidth audio with very little transmission data.

[0185] Step S2110, the decoding device 102 determines a plurality of decoded audio frames according to the first type of decoded audio frame and the second type of decoded audio frame.

[0186] In some embodiments, the decoding device determines a plurality of decoded audio frames according to the first type of decoded audio frame and the second type of decoded audio frame, and the plurality of decoded audio frames are reconstruction results of the plurality of audio frames.

[0187] In some embodiments, the encoding device and the decoding device are obtained through end-to-end joint training. The training process can refer to the implementation of model training described below, which will not be repeated here.

[0188] By using the above method, the encoding device classifies the audio frames based on the complexity of each audio frame, and encodes the audio frames of different types at different bit rates. This realizes adaptive variable bit rate encoding based on the content of the audio frames. This differentiated encoding method for different types of audio frames can optimize bit allocation to the extreme, and almost no bandwidth is used for audio frames that are not sensitive to hearing or simple. The saved bandwidth is used for key frames, so that the best overall sound quality is achieved in subjective perception at a very low average bit rate. The decoding device uses a hierarchical decoding strategy of "full-amount decoding + non-full-amount decoding". Full-amount decoding ensures perfect reconstruction of important frames (i.e. audio frames with high complexity), and non-full-amount decoding optimizes intelligent recovery of high-frequency information and psychoacoustic details of simple frames (i.e. audio frames with low complexity). The two work together to improve the reconstruction fidelity of all types of audio (speech, music, environmental sound).

[0189] In some embodiments, the names of data and the like are not limited to the names described in the embodiments, and the terms of "data", "information", "signal", "audio", "voice", "sound wave", and the like can be replaced with each other.

[0190] In some embodiments, "acquire", "obtain", "get", "receive", "transmit", "bidirectional transmission", "send and / or receive" can be replaced with each other.

[0191] In some embodiments, the terms of "send", "transmit", "report", "issue", "transmit", "bidirectional transmission", "send and / or receive", and the like can be replaced with each other.

[0192] The method related to the embodiments of the present disclosure can include at least one of the steps S2101-S2110 described above. For example, step S2103 can be implemented as an independent embodiment, steps S2103 and S2105 are implemented as independent embodiments, steps S2103, S2106 are implemented as independent embodiments, steps S2108 and S2109 are implemented as independent embodiments, but not limited thereto.

[0193] In some embodiments, any of steps S2101-S2110 is optional, and one or more of these steps can be omitted or replaced in different embodiments.

[0194] In some embodiments, reference can be made to the steps and optional implementation manners thereof in other embodiments described before or after the description of the present embodiment, and other related parts in the description, which will not be repeated here.

[0195] Figure 2B is an interaction diagram of a coding method according to an embodiment of the present disclosure. The method can be performed by the coding system 100 described above. As shown in Figure 2B The method includes at least one of the following steps:

[0196] Step S2201, the encoding device 101 obtains at least one category of audio frames according to the complexity of each audio frame in a plurality of audio frames.

[0197] In some embodiments, the at least one category of audio frames is the first category of audio frames.

[0198] In some embodiments, the at least one category of audio frames includes the first category of audio frames and the second category of audio frames.

[0199] For example, the encoding device divides the plurality of audio frames into first type audio frames and second type audio frames according to the complexity of each audio frame. The complexity of the first type audio frames is different from the complexity of the second type audio frames. For example, the complexity of the first type audio frames is higher than the complexity of the second type audio frames.

[0200] In some embodiments, the at least one type of audio frames includes the first type audio frames, the second type audio frames, and the third type audio frames.

[0201] For example, the encoding device divides the plurality of audio frames into first type audio frames, second type audio frames, and third type audio frames according to the complexity of each audio frame.

[0202] The complexity of the first type audio frames, the complexity of the second type audio frames, and the complexity of the third type audio frames are all different. For example, the complexity of the first type audio frames is higher than the complexity of the second type audio frames, and the complexity of the second type audio frames is higher than the complexity of the third type audio frames.

[0203] In some embodiments, the at least one type of audio frames includes the first type audio frames, the second type audio frames, the third type audio frames, and the fourth type audio frames.

[0204] For example, the encoding device divides the plurality of audio frames into first type audio frames, second type audio frames, third type audio frames, and fourth type audio frames according to the complexity of each audio frame.

[0205] The complexity of the first type audio frames, the complexity of the second type audio frames, the complexity of the third type audio frames, and the complexity of the fourth type audio frames are all different. For example, the complexity of the first type audio frames is higher than the complexity of the second type audio frames, the complexity of the second type audio frames is higher than the complexity of the third type audio frames, and the complexity of the third type audio frames is higher than the complexity of the fourth type audio frames.

[0206] It should be noted that the encoding device can divide the plurality of audio frames into more types according to the complexity of each audio frame, which is not limited in the present disclosure.

[0207] In step S2202, the encoding device 101 encodes each type of audio frames respectively, and determines the first encoding data according to the encoding result of each type of audio frames.

[0208] For example, the encoding device encodes the first type audio frames.

[0209] For example, the encoding device encodes the first type audio frames and the second type audio frames respectively. The encoding bit rate of different types of audio frames is different. For example, the encoding bit rate of the first type audio frames is higher than the encoding bit rate of the second type audio frames.

[0210] For example, the encoding device encodes the first type of audio frame, the second type of audio frame, and the third type of audio frame respectively. The code rate (i.e., the encoding bit rate) of the first type of audio frame, the code rate of the second type of audio frame, and the code rate of the third type of audio frame are all different. For example, the code rate of the first type of audio frame is higher than the code rate of the second type of audio frame, and the code rate of the second type of audio frame is higher than the code rate of the third type of audio frame.

[0211] For example, the encoding device encodes the first type of audio frame, the second type of audio frame, the third type of audio frame, and the fourth type of audio frame respectively. The code rate of the first type of audio frame, the code rate of the second type of audio frame, the code rate of the third type of audio frame, and the code rate of the fourth type of audio frame are all different. For example, the code rate of the first type of audio frame is higher than the code rate of the second type of audio frame, the code rate of the second type of audio frame is higher than the code rate of the third type of audio frame, and the code rate of the third type of audio frame is higher than the code rate of the fourth type of audio frame.

[0212] It should be noted that the above examples do not exhaust all embodiments, and the encoding device can also encode more types of audio frames respectively, and the encoding bit rates corresponding to different types of audio frames are different.

[0213] In some embodiments, in the encoding process, a VBR strategy can be performed to determine a total bit budget to constrain the length of the first encoding data, and a bit budget of each type of audio frame can be determined to constrain the encoding length of the corresponding type of audio frame.

[0214] In step S2203, the encoding device 101 sends the first encoding data to the decoding device 102.

[0215] In step S2204, the decoding device 102 decodes the encoding data of each type of audio frame respectively to obtain reconstructed multiple audio frames.

[0216] In some embodiments, the decoding process is the inverse process of the encoding process. If the encoding device performs full-amount encoding on a certain type of audio frame, the decoding device performs full-amount decoding on the encoding result of the type of audio frame. If the encoding device performs non-full-amount encoding on a certain type of audio frame, the decoding device performs non-full-amount decoding on the encoding result of the type of audio frame. The full-amount decoding and the non-full-amount decoding can be referred to the embodiments of steps S2108 and S2109 of the encoding device 101, which will not be described herein. Figure 2A

[0217] In some embodiments, the steps and optional implementation manners thereof in other embodiments described before or after the description of the present embodiment, and other related parts of the description can be referred to, which will not be described herein.

[0218] Figure 2C ​is a flowchart of a model training method according to an embodiment of the present disclosure. The method can be performed by any device or apparatus with model training capability or a cluster composed of multiple devices / apparatuses to train a coding-decoding model (or called a coder-decoder, a coder-decoder model) for the purpose of Figure 2C The method can include at least one of the following steps:

[0219] In step S2301, the encoding device obtains at least one category of sample audio frames according to the complexity of each sample audio frame in the plurality of sample audio frames.

[0220] For example, the encoding device classifies the plurality of sample audio frames according to the complexity of each sample audio frame in the plurality of sample audio frames to obtain at least one category of sample audio frames.

[0221] In some embodiments, the training sample set includes a plurality of groups of training data, and each group of training data includes a plurality of sample audio frames.

[0222] In step S2302, the encoding device encodes each category of sample audio frames respectively, and determines sample first encoding data according to the encoding result of each category of sample audio frames.

[0223] In some embodiments, the encoding bit rate of sample audio frames of different categories is different.

[0224] In step S2303, the decoding device decodes the sample first encoding data to obtain a plurality of sample decoded audio frames.

[0225] In some embodiments, the decoding device decodes the encoding result of each category of sample audio frames in the first encoding data respectively to obtain a plurality of sample decoded audio frames.

[0226] In step S2304, a loss value is determined according to the difference between the plurality of sample audio frames and the plurality of sample decoded audio frames, and the number of bits of the sample first encoding data.

[0227] In step S2305, the trainable parameters in the model are updated according to the loss value.

[0228] In some embodiments, the model includes an encoding device and a decoding device. The encoding device and the decoding device are jointly trained in an end-to-end manner. The loss function not only includes audio reconstruction quality (such as multi-scale spectral loss MS-SSL, adversarial loss, etc.), but also adds a code rate constraint. The loss function is approximately: Total_Loss=Distortion(original audio, reconstructed audio)+λ*Rate(total number of bits actually generated).

[0229] Distortion represents the difference between the reconstructed audio and the original audio to ensure the audio quality.

[0230] Rate indicates an incentive for the model to conserve bits and control the output bit rate.

[0231] λ is a hyperparameter used to control the trade-off between audio quality and bitrate. The larger λ is, the more the model tends to favor low bitrate, and vice versa.

[0232] In some embodiments, by performing multiple epochs of steps S2301 to S2305, a trained encoding device and a decoding device can be obtained.

[0233] It should be noted that the training method in this embodiment is a single-stage end-to-end joint training, which performs global optimization with the perceptual quality of the final output audio as the target. This enables the modules to work collaboratively, and the frame classification strategy directly serves the optimal sound quality target, avoiding the suboptimal problems caused by submodule optimization.

[0234] The methods involved in the embodiments of this disclosure may include at least one of the steps S2301 to S2306 described above. For example, step S2301 may be implemented as a separate embodiment, and step S2302 may be implemented as a separate embodiment, but are not limited thereto.

[0235] In some embodiments, steps S2301 to S2305 are optional, and one or more of these steps may be omitted or substituted in different embodiments.

[0236] In some embodiments, the steps and their optional implementations in other embodiments described before or after this embodiment, as well as other related parts in the specification, can be referred to, and will not be repeated here.

[0237] Figure 3A This is a flowchart illustrating an encoding method according to an embodiment of the present disclosure. Figure 3A As shown, the embodiments of this disclosure relate to an encoding method, which is executed by an encoding device. The method includes at least one of the following steps:

[0238] Step S3101: Based on the complexity of each audio frame in the multiple audio frames, obtain at least one category of audio frames.

[0239] Optionally, the multiple audio frames are classified according to the complexity of each audio frame to obtain audio frames of at least one category.

[0240] Optionally, the at least one category of audio frames includes a first category of audio frames and a second category of audio frames, wherein the complexity of the first category of audio frames is higher than that of the second category of audio frames.

[0241] Optionally, the audio frame belongs to the first type of audio frame when the complexity of the audio frame is a first value, and belongs to the second type of audio frame when the complexity of the audio frame is a second value.

[0242] Optionally, the audio frame is indicated as the first type of audio frame when the complexity of the audio frame is greater than a threshold value, and is indicated as the second type of audio frame when the complexity of the audio frame is less than or equal to the threshold value, wherein the complexity of the audio frame is a value between 0 and 1.

[0243] Optionally, the classification of the plurality of audio frames according to the complexity of each audio frame in the plurality of audio frames comprises: dividing the plurality of audio frames into first type of audio frames and second type of audio frames, wherein the complexity of the first type of audio frames is higher than the complexity of the second type of audio frames.

[0244] Optionally, the complexity of an audio frame is a first value or a second value, wherein the first value indicates that the audio frame is the first type of audio frame, and the second value indicates that the audio frame is the second type of audio frame.

[0245] Optionally, the complexity of an audio frame is a value between 0 and 1.

[0246] When the value is greater than a threshold value, the audio frame is indicated as the first type of audio frame.

[0247] When the value is less than or equal to the threshold value, the audio frame is indicated as the second type of audio frame.

[0248] Step S3102, respectively encode each type of audio frame, and obtain first encoding data according to the encoding result of each type of audio frame.

[0249] Optionally, the encoding bit rates of different types of audio frames are different.

[0250] Optionally, the encoding of each type of audio frame to obtain the first encoding data comprises: full-amount encoding of the first type of audio frame to obtain second encoding data; non-full-amount encoding of the second type of audio frame to obtain third encoding data, wherein the encoding bit rate of the full-amount encoding is higher than the encoding bit rate of the non-full-amount encoding; and determining the first encoding data according to the second encoding data and the third encoding data.

[0251] Optionally, before encoding each category of audio frames respectively to obtain the first encoded data, the method further comprises: determining a total bit budget of the plurality of audio frames according to the complexity of each audio frame in the plurality of audio frames, the total bit budget being a maximum value for constraining the length of the first encoded data.

[0252] Optionally, the method further comprises: determining a bit budget of each audio frame according to the total bit budget and the complexity of each audio frame, wherein the complexity of an audio frame is positively correlated with the corresponding allocated bit budget, and the bit budget of an audio frame is a maximum value for constraining the length of the encoded data of the audio frame.

[0253] Optionally, before classifying the plurality of audio frames according to the complexity of each audio frame in the plurality of audio frames, the method further comprises: determining an audio feature of the plurality of audio frames; and determining the complexity of each audio frame based on the audio feature.

[0254] Optionally, the method further comprises: sending the first encoded data to a decoding device.

[0255] In some embodiments, refer to other embodiments described before or after the description of the present embodiment, the steps and optional implementation manners thereof, and other related parts of the description, which will not be repeated here.

[0256] Figure 3B is a flowchart of a decoding method according to an embodiment of the present disclosure. As shown in Figure 3B The present disclosure relates to a decoding method, which is performed by a decoding device, and the method comprises at least one of the following steps:

[0257] Step S3201: receiving first encoded data sent by an encoding device.

[0258] Optionally, the first encoded data is obtained by the encoding device according to the complexity of each audio frame in the plurality of audio frames to determine at least one category of audio frames, and encoding each category of audio frames respectively, wherein the encoding bit rates of different categories of audio frames are different.

[0259] Step S3202: decoding the first encoded data to obtain a plurality of decoded audio frames.

[0260] Optionally, the first encoded data comprises second encoded data and third encoded data, wherein the second encoded data is obtained by full-amount encoding of the first type of audio frames by the encoding device, and the third encoded data is obtained by non-full-amount encoding of the second type of audio frames by the encoding device, the complexity of the first type of audio frames being higher than that of the second type of audio frames; the decoding of the first encoded data to obtain a plurality of decoded audio frames comprises: full-amount decoding of the second encoded data to obtain the first type of decoded audio frames; and non-full-amount decoding of the third encoded data to obtain the second type of decoded audio frames based on a non-full-amount decoding result.

[0261] Optionally, the obtaining of the second type of decoded audio frames based on the non-full-amount decoding result comprises: predicting non-full-amount encoding loss information according to the non-full-amount decoding result, and obtaining the second type of decoded audio frames according to the non-full-amount decoding result and the non-full-amount encoding loss information.

[0262] In some embodiments, see other embodiments described before or after the description of the present embodiment, the steps and optional implementations thereof, and other related parts of the description.

[0263] It should be noted that the variable bit rate neural audio codec method proposed in the embodiments of the present disclosure is suitable for various audio transmission and storage scenarios with strict requirements on bandwidth, sound quality and power consumption, and can effectively maintain high perceptual sound quality at extremely low average code rate.

[0264] Specific application scenarios include but are not limited to the following:

[0265] 1. Real-time voice communication and conference: In real-time interactive scenarios such as mobile voice calls, video conferences, and in-game voice chats, network bandwidth fluctuates dramatically, and user device power consumption is sensitive. The embodiments of the present disclosure can dynamically adjust the code rate according to the voice content (such as significantly reducing the code rate during the silent period), significantly reduce the overall bandwidth occupancy and mobile terminal encoding energy consumption while ensuring the high definition of key voice frames, thereby providing a smoother and more durable high-quality real-time communication experience.

[0266] 2. Audio and video streaming services: For music streaming services (such as NetEase Music) and video sharing platforms (such as Bilibili), the storage and bandwidth costs are huge. The embodiments of the present disclosure can intelligently encode audio streams, preserving rich details in complex musical passages and efficiently compressing simple passages, thereby reducing the average code rate to the minimum under the same perceptual sound quality, saving a large amount of cost for the platform, and reducing data traffic consumption for users.

[0267] 3. Smart Internet of Things (IoT) Devices and Voice Assistants: In IoT devices such as smart home speakers, wearable devices, and in-vehicle voice assistants, computing resources, storage space, and network bandwidth are extremely limited. The embodiments of this disclosure employ a frame classification mechanism to perform high-fidelity encoding of key frames such as wake words and voice commands, and to perform extreme compression of environmental noise and inactive audio. This can reduce the computational complexity of device-side encoding, storage pressure, and cloud uplink transmission bandwidth requirements, thereby extending device battery life.

[0268] 4. Large-scale voice acquisition and archiving: In scenarios requiring long-term storage of massive amounts of voice data, such as call center recordings, conference system recordings, judicial evidence collection, and medical records, storage cost is a core concern. The embodiments disclosed herein enable content-adaptive variable bitrate encoding, generating audio archives that are significantly smaller than traditional fixed bitrate encoding while ensuring speech intelligibility and clear speaker characteristics, thus substantially reducing long-term storage costs for customers.

[0269] 5. Immersive Interactive Media and Cloud Gaming: In emerging applications such as VR / AR and cloud gaming, audio and video need to be transmitted synchronously in real time, posing a significant challenge to network bandwidth. The embodiments of this disclosure can dynamically allocate valuable bandwidth resources to the most critical moments in visual and auditory experiences (such as explosions and conversations), maximizing the overall experience quality of immersive media under strict overall bitrate constraints.

[0270] The encoding and decoding method proposed in this disclosure is a variable bit rate neural audio encoding and decoding method based on content-aware frame classification and hierarchical coding. It can intelligently identify the importance of audio content and dynamically allocate bit resources to the most needed key frames, thereby significantly improving the overall perceived sound quality at an extremely low average bit rate, while greatly reducing the encoding computation complexity. It is particularly good at handling music and complex environmental sounds.

[0271] In some embodiments, the audio codec of this disclosure is implemented based on a neural network model. Variable bit rate (VBR) is achieved by classifying frames during multi-frame packet encapsulation. A bandwidth extension module is introduced to adapt to communication under weak network conditions and further conserve encoding bits for non-critical frames. The encoding and decoding method of this disclosure optimizes bit allocation to the extreme, consuming almost no bandwidth for frames that are not auditoryly sensitive or simple, and using all the saved bandwidth for critical frames, thereby achieving the best perceived overall sound quality at an extremely low average bit rate.

[0272] In some embodiments, the flowchart of the encoding / decoding method is as follows: Figure 4 As shown.

[0273] In some embodiments, the encoding end includes:

[0274] Multi-frame packing and analysis: the encoder receives multiple consecutive audio frames at once.

[0275] Frame classification and VBR: analyze the perceptual importance or audio content complexity of each frame in the multi-frame, and according to the analysis results: classify the frame as "important frame" (complex audio such as music climax, percussion) and "unimportant frame" (simple audio such as smooth paragraph, silence). And implement the VBR strategy: allocate a total bit budget for the entire data packet, and allocate more bits to "important frames" and fewer bits to "unimportant frames".

[0276] Hierarchical coding: encode "important frames" using a full-amount coding model. This model encodes the complete audio spectrum information, providing high-fidelity reconstruction. Encode "unimportant frames" using very low bit rate, possibly only encoding very basic information (e.g. very low-dimensional representation or only fundamental frequency information). The decoding end will rely on "bandwidth extension" (BWE) technology to restore high-frequency details.

[0277] In some embodiments, the decoding end includes:

[0278] Receive and unpack.

[0279] For "important frames": directly use the full-amount decoding model for decoding to obtain high-quality audio.

[0280] For "unimportant frames": first decode the low-frequency basic information using a core decoder, and then input it into a bandwidth extension (BWE) model. The BWE model is a neural network that intelligently predicts and reconstructs high-frequency information based on low-frequency information, thereby restoring full-bandwidth audio with very little transmission data.

[0281] In some embodiments, the key steps include frame classification and VBR control, full-amount codec and core codec model, bandwidth extension model.

[0282] In some embodiments, for frame classification and VBR control, the implementation method can adopt a scheme based on neural network implicit analysis and joint optimization. Generally, it is integrated as a lightweight side branch network in the front end of the encoder. The core idea is not to train a separate classifier, but to let the entire encoder automatically learn how to judge the importance of the frame during the training process, and jointly optimize the VBR bit allocation strategy. The network building process is as follows:

[0283] 1. Feature extraction backbone:

[0284] Input: an audio time window (contains multiple frames of data including the current frame and the context before and after).

[0285] Backbone network: several layers of 1D-Conv or Gated Recurrent Unit (GRU). Conv layers can quickly extract local spectral features, while GRUs can model the dependencies well on time series.

[0286] Output: a high-dimensional feature vector, representing the complexity and perceptual importance of the input audio.

[0287] 2. Importance evaluation head:

[0288] Structure: a fully connected layer (FC) + Softmax or Sigmoid activation function after the backbone network.

[0289] Output:

[0290] Scheme A (directly output weights): output N importance weights [w1, w2,..., wN] between 0 and 1 for N frames in a packet. The higher the weight, the more important the frame, and the more bits are allocated.

[0291] Scheme B (output mode selection): directly output an N-dimensional binary decision [0, 1,..., 0], where 1 represents using "full-amount encoding" for the frame, and 0 represents using "BWE encoding". This usually requires a Gumbel-Softmax trick to simulate discrete decisions during training.

[0292] 3. Bit allocator:

[0293] This is an algorithmic module, not a learning module. It calculates the actual number of bits b_i allocated to each frame based on the weights or decisions output by the above network and a preset total bit budget B_total.

[0294] For scheme A: b_i = (w_i / sum(w))*B_total.

[0295] For scheme B, frames with decision 1 share the "high-bit" budget, and frames with decision 0 share the "very low-bit" budget.

[0296] In some embodiments, the entire system is trained end-to-end jointly. The loss function not only includes audio reconstruction quality (such as multi-scale spectral loss MS-SSL, adversarial loss), but also must add a bit rate constraint. The loss function is roughly as follows:

[0297] Total_Loss = Distortion (original audio, reconstructed audio) + λ * Rate (actual total number of bits generated).

[0298] Distortion: measures the difference between the reconstructed audio and the original audio, ensuring audio quality.

[0299] Rate: Encourages the model to conserve bits and controls the output bit rate.

[0300] λ: A hyperparameter that controls the trade-off between audio quality and bitrate. The larger λ is, the more the model tends to favor low bitrate, and vice versa.

[0301] The beneficial effects of the embodiments of this disclosure include at least one of the following:

[0302] 1. Content-aware bit allocation for optimal perceived sound quality: By intelligently identifying the perceived importance of each frame through a frame classification network, high bits are dynamically allocated to complex frames, while simple frames are encoded with extremely low bits. This non-uniform, content-adaptive allocation strategy uses limited bit resources wisely, significantly improving subjective listening quality at the same average bit rate.

[0303] 2. Dynamic computational complexity significantly improves coding efficiency: The encoding and decoding system only activates the computationally complex full encoder for the classified "important frames," while calling the extremely lightweight core encoder for "non-important frames." This dynamic computation mechanism significantly reduces average coding latency and power consumption, making real-time applications of high-performance neural codecs possible on edge devices such as mobile phones and IoT devices.

[0304] 3. Layered Decoding Architecture for Enhanced Reconstruction Fidelity: Employing a layered decoding strategy combining a "full-length decoding model" and a "bandwidth extension (BWE) model," the full-length model ensures perfect reconstruction of details in important frames; the BWE model is specifically optimized for intelligent recovery of high-frequency and psychoacoustic details in simple frames. Working together, they enhance the reconstruction fidelity for all types of audio (speech, music, ambient sound).

[0305] 4. Packet-level optimization and multi-frame encapsulation significantly reduce transmission overhead: The introduction of multi-frame encapsulation technology combines multiple frames into a single packet for transmission, greatly reducing the proportion of metadata. Simultaneously, global bit allocation optimization within the packet further improves the effective payload ratio and overall compression efficiency.

[0306] 5. End-to-end joint optimization to avoid error accumulation: The frame classification network, encoder, decoder, and BWE model are jointly trained end-to-end, and global optimization is performed with the perceptual quality of the final output audio as the goal. This enables the modules to work together, and the frame classification strategy directly serves the optimal sound quality goal, avoiding suboptimal problems caused by submodule optimization.

[0307] In some embodiments of this disclosure, an encoding / decoding system is provided, which may include an encoding device and a decoding device, wherein the encoding device may execute the encoding method performed by the encoding device in the foregoing embodiments of this disclosure; and the decoding device may execute the decoding method performed by the decoding device in the foregoing embodiments of this disclosure.

[0308] This disclosure also provides embodiments of an apparatus for implementing any of the above methods. For example, an apparatus is provided that includes units or modules for implementing the steps performed by the encoding device in any of the above methods. Furthermore, another apparatus is provided that includes units or modules for implementing the steps performed by the decoding device in any of the above methods.

[0309] It should be understood that the division of units or modules in the above device is only a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, the units or modules in the device can be implemented by a processor calling software: for example, the device includes a processor connected to a memory containing instructions. The processor calls the instructions stored in the memory to implement any of the above methods or to implement the functions of the units or modules in the above device. The processor can be, for example, a general-purpose processor, such as a Central Processing Unit (CPU) or a microprocessor, and the memory can be internal or external to the device. Alternatively, the units or modules in the device can be implemented in the form of hardware circuits. The functionality of some or all of the units or modules can be achieved through the design of these hardware circuits, which can be understood as one or more processors. For example, in one implementation, the hardware circuit is an Application-Specific Integrated Circuit (ASIC), and the functionality of some or all of the units or modules is achieved through the design of the logical relationships between the components within the circuit. In another implementation, the hardware circuit can be implemented using a Programmable Logic Device (PLD), such as a Field Programmable Gate Array (FPGA), which can include a large number of logic gates. The connection relationships between the logic gates are configured through configuration files, thereby achieving the functionality of some or all of the units or modules. All units or modules of the above device can be implemented entirely through processor-called software, entirely through hardware circuits, or partially through processor-called software with the remaining parts implemented through hardware circuits.

[0310] In this embodiment, the processor is a circuit with signal processing capabilities. In one implementation, the processor can be a circuit with instruction read and execute capabilities, such as a Central Processing Unit (CPU), a microprocessor, a Graphics Processing Unit (GPU) (which can be understood as a microprocessor), or a Digital Signal Processor (DSP). In another implementation, the processor can implement certain functions through the logical relationships of hardware circuits. The logical relationships of the aforementioned hardware circuits are fixed or reconfigurable. For example, the processor is a hardware circuit implemented using an Application-Specific Integrated Circuit (ASIC) or a Programmable Logic Device (PLD), such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document and configuring the hardware circuit can be understood as the process of the processor loading instructions to implement the functions of some or all of the above units or modules. In addition, it can also be hardware circuits designed for artificial intelligence, which can be understood as ASICs, such as Neural Network Processing Units (NPUs), Tensor Processing Units (TPUs), and Deep Learning Processing Units (DPUs).

[0311] Figure 5A This is a schematic diagram of the structure of an encoding device according to an embodiment of this disclosure. Figure 5A As shown, the encoding device 5100 may include at least one of a processing module 5101, a transceiver module 5102, etc. In some embodiments, the processing module 5101 is used to obtain at least one category of audio frames based on the complexity of each audio frame in a plurality of audio frames; and to encode each category of audio frames to obtain first encoded data, wherein the encoding bit rates of different categories of audio frames are different. Optionally, the transceiver module 5101 may be used to perform at least one of the communication steps such as sending and / or receiving performed by the encoding device 101 in any of the above methods (e.g., step S2107, but not limited thereto), which will not be described in detail here. Optionally, the processing module 5102 may be used to execute at least one of the other steps (e.g., steps S2101, S2102, S2103, S2104, S2105, S2106, S2108, S2109, S2110, but not limited thereto) executed by the encoding device 101 in any of the above methods, which will not be elaborated here.

[0312] Figure 5B This is a schematic diagram of the structure of a decoding device according to an embodiment of this disclosure. Figure 5B As shown, the decoding device 5200 may include at least one of a processing module 5201 and a transceiver module 5202. In some embodiments, the transceiver module 5202 is used to receive first encoded data sent by the encoding device, wherein the first encoded data is obtained by the encoding device determining at least one category of audio frames based on the complexity of each audio frame in a plurality of audio frames, and encoding each category of audio frames respectively, wherein the encoding bit rates of different categories of audio frames are different. The processing module 5201 is used to decode the first encoded data to obtain a plurality of decoded audio frames. Optionally, the transceiver module 5201 may be used to perform at least one of the communication steps such as sending and / or receiving performed by the decoding device 102 in any of the above methods (e.g., step S2107, but not limited thereto), which will not be described in detail here. Optionally, the processing module 5202 may be used to execute at least one of the other steps executed by the decoding device 102 in any of the above methods (e.g., steps S2101, S2102, S2103, S2104, S2105, S2106, S2108, S2109, S2110, but not limited thereto), which will not be elaborated here.

[0313] In some embodiments, the transceiver module may include a transmitting module and / or a receiving module, which may be separate or integrated. Optionally, the transceiver module may be interchangeable with a transceiver.

[0314] In some embodiments, the processing module may be a single module or may include multiple sub-modules. Optionally, the multiple sub-modules may each perform all or part of the steps required by the processing module. Optionally, the processing module may be interchangeable with a processor.

[0315] Figure 6A This is a schematic diagram of the structure of a communication device 6100 according to an embodiment of this disclosure. The communication device 6100 can be an encoding device, a decoding device, or a chip, chip system, or processor that supports the encoding device in implementing any of the above methods; it can also be a chip, chip system, or processor that supports the decoding device in implementing any of the above methods. The communication device 6100 can be used to implement the methods described in the above method embodiments, and specific details can be found in the descriptions in the above method embodiments.

[0316] like Figure 6AAs shown, the communication device 6100 is used to execute any of the above methods. In some embodiments, the communication device 6100 includes one or more processors 6101. The processor 6101 may be a general-purpose processor or a special-purpose processor, such as a baseband processor or a central processing unit. The baseband processor may be used to process communication protocols and communication data, and the central processing unit may be used to control communication devices (e.g., base stations, baseband chips, terminal devices, terminal device chips, DUs or CUs, etc.), execute programs, and process program data. Optionally, the communication device 6100 is used to execute any of the above methods. Optionally, one or more processors 6101 are used to invoke instructions to cause the communication device 6100 to execute any of the above methods.

[0317] In some embodiments, the communication device 6100 further includes one or more transceivers 6102. When the communication device 6100 includes one or more transceivers 6102, the transceiver 6102 performs at least one of the communication steps such as sending and / or receiving in the above method (e.g., step S2107, but not limited thereto), and the processor 6101 performs at least one of other steps (e.g., steps S2101, S2102, S2103, S2104, S2105, S2106, S2108, S2109, S2110, but not limited thereto). In optional embodiments, the transceiver may include a receiver and / or a transmitter, which may be separate or integrated together. Optionally, the terms transceiver, transceiver unit, transceiver, transceiver circuit, interface circuit, interface, etc., can be used interchangeably; the terms transmitter, transmitting unit, transmitter, transmitting circuit, etc., can be used interchangeably; the terms receiver, receiving unit, receiver, receiving circuit, etc., can be used interchangeably.

[0318] In some embodiments, the communication device 6100 further includes one or more memories 6103 for storing data and / or instructions. Optionally, one or more processors 6101 are used to invoke instructions stored in the memory 6103 to cause the communication device 6100 to perform any of the above methods. Optionally, all or part of the memory 6103 may also be located outside the communication device 6100. In an optional embodiment, the communication device 6100 may include one or more interface circuits 6104. Optionally, the interface circuit 6104 is connected to the memory 6103 and can be used to receive data and / or instructions from the memory 6103 or other devices, and can be used to send data and / or instructions to the memory 6103 or other devices. For example, the interface circuit 6104 can read data and / or instructions stored in the memory 6103 and send the data and / or instructions to the processor 6101.

[0319] The communication device 6100 described in the above embodiments may be a network device or a terminal, but the scope of the communication device 6100 described in this disclosure is not limited thereto, and the structure of the communication device 6100 may vary. Figure 6A The limitations. The communication device may be a standalone device or part of a larger device. For example, the communication device may be: (1) a standalone integrated circuit IC, or chip, or chip system or subsystem; (2) a collection of one or more ICs, optionally including storage components for storing data, programs and / or instructions; (3) an ASIC, such as a modem; (4) a module that can be embedded in other devices; (5) a receiver, terminal device, smart terminal device, cellular phone, wireless device, handheld device, mobile unit, vehicle device, network device, cloud device, artificial intelligence device, etc.; (6) others, etc.

[0320] Figure 6B This is a schematic diagram of the structure of chip 6200 according to an embodiment of this disclosure. For cases where the communication device 6100 can be a chip or a chip system, please refer to... Figure 6B The diagram shown is a schematic representation of the structure of chip 6200, but it is not limited to this.

[0321] Chip 6200 includes one or more processors 6201. Chip 6200 is used to perform any of the methods described above.

[0322] In some embodiments, chip 6200 further includes one or more interface circuits 6202. Optionally, terms such as interface circuit, interface, and transceiver pin can be used interchangeably. In some embodiments, chip 6200 further includes one or more memories 6203 for storing data and / or instructions. Optionally, all or part of the memories 6203 may be located outside of chip 6200. Optionally, interface circuit 6202 is connected to memory 6203, and interface circuit 6202 can be used to receive data and / or instructions from memory 6203 or other devices, and interface circuit 6202 can be used to send data and / or instructions to memory 6203 or other devices. For example, interface circuit 6202 can read data and / or instructions stored in memory 6203 and send the data and / or instructions to processor 6201.

[0323] In some embodiments, the interface circuit 6202 performs at least one of the communication steps such as sending and / or receiving in the above-described method (e.g., step S2107, but not limited thereto). The interface circuit 6202 performing the communication steps such as sending and / or receiving in the above-described method refers, for example, to the interface circuit 6202 performing data and / or instruction interaction between the processor 6201, the chip 6200, the memory 6203, or the transceiver device. In some embodiments, the processor 6201 performs at least one of other steps (e.g., steps S2101, S2102, S2103, S2104, S2105, S2106, S2108, S2109, S2110, but not limited thereto).

[0324] The modules and / or devices described in the various embodiments, such as virtual devices, physical devices, and chips, can be combined or separated arbitrarily as needed. Optionally, some or all steps can also be performed collaboratively by multiple modules and / or devices, which is not limited here.

[0325] This disclosure also proposes a storage medium storing instructions that, when executed on a communication device, cause the communication device to perform any of the above methods. Optionally, the storage medium is an electronic storage medium. Optionally, the storage medium is a computer-readable storage medium, but not limited thereto; it may also be a storage medium readable by other devices. Optionally, the storage medium may be a non-transitory storage medium, but not limited thereto; it may also be a temporary storage medium.

[0326] This disclosure also proposes a program product, including a program and / or instructions, which, when executed by a communication device, cause the communication device to perform any of the above methods. Optionally, the program product is a computer program product. Optionally, the program product is stored on the storage medium.

[0327] This disclosure also proposes a computer program that, when run on a computer, causes the computer to perform any of the above methods.

Claims

1. An encoding method characterized by comprising: The method is performed by an encoding device, and comprises: obtaining at least one category of audio frames according to complexity of each audio frame in a plurality of audio frames; encoding each category of audio frames respectively to obtain first encoding data, wherein encoding bit rates of different categories of audio frames are different.

2. The method of claim 1, wherein, The at least one category of audio frames comprises first category audio frames and second category audio frames, and complexity of the first category audio frames is higher than complexity of the second category audio frames.

3. The method of claim 2, wherein, in a case where the complexity of the audio frame is a first value, the audio frame belongs to the first category audio frames; in a case where the complexity of the audio frame is a second value, the audio frame belongs to the second category audio frames.

4. The method of claim 2, wherein, in a case where the complexity of the audio frame is greater than a threshold value, the audio frame is indicated as the first category audio frames; in a case where the complexity of the audio frame is less than or equal to the threshold value, the audio frame is indicated as the second category audio frames; wherein the complexity of the audio frame is a value between 0 and 1.

5. The method according to any one of claims 2-4, characterized in that, The encoding each category of audio frames respectively to obtain first encoding data comprises: full-amount encoding the first category audio frames to obtain second encoding data; non-full-amount encoding the second category audio frames to obtain third encoding data, wherein an encoding bit rate of the full-amount encoding is higher than an encoding bit rate of the non-full-amount encoding; determining the first encoding data according to the second encoding data and the third encoding data.

6. The method according to any one of claims 1-5, characterized in that, The method further comprises: determining a total bit budget of the plurality of audio frames according to the complexity of each audio frame in the plurality of audio frames, the total bit budget being used to constrain a maximum value of a length of the first encoding data.

7. The method of claim 6, wherein, The method further comprises: determining a bit budget of each audio frame according to the total bit budget and the complexity of each audio frame, wherein the complexity of an audio frame is positively correlated with a corresponding allocated bit budget, and the bit budget of the audio frame is used to constrain a maximum value of a length of encoding data of the audio frame.

8. The method according to any one of claims 1-7, characterized in that, The method further comprises: determining audio features of the plurality of audio frames; determining the complexity of each audio frame based on the audio features.

9. The method according to any one of claims 1-8, characterized in that, The method further comprises: sending the first encoding data to a decoding device.

10. A decoding method, comprising: The method is performed by a decoding device, and comprises: receiving first encoding data sent by an encoding device, wherein the first encoding data is determined by the encoding device according to complexity of each audio frame in a plurality of audio frames, at least one category of audio frames is obtained, and each category of audio frames is encoded respectively to obtain the first encoding data, wherein encoding bit rates of different categories of audio frames are different; decoding the first encoding data to obtain a plurality of decoded audio frames.

11. The method of claim 10, wherein, The first encoding data comprises second encoding data and third encoding data, wherein the second encoding data is obtained by full-amount encoding first category audio frames by the encoding device, the third encoding data is obtained by non-full-amount encoding second category audio frames by the encoding device, and complexity of the first category audio frames is higher than complexity of the second category audio frames. The decoding of the first encoded data obtains a plurality of decoded audio frames, comprising: full-quantity decoding the second encoded data to obtain first-type decoded audio frames; non-full-quantity decoding the third encoded data to obtain second-type decoded audio frames based on a non-full-quantity decoding result.

12. The method of claim 11, wherein, The second-type decoded audio frames obtained based on the non-full-quantity decoding result, comprising: predicting non-full-quantity encoding loss information according to the non-full-quantity decoding result, and obtaining the second-type decoded audio frames according to the non-full-quantity decoding result and the non-full-quantity encoding loss information.

13. A model training method, comprising: The model comprises an encoding device and a decoding device, and the method comprises: The encoding device obtains at least one type of sample audio frames according to the complexity of each sample audio frame in a plurality of sample audio frames; The encoding device respectively encodes each type of sample audio frames to obtain sample first encoded data, wherein the encoding bit rates of sample audio frames of different types are different; The decoding device decodes the sample first encoded data to obtain a plurality of sample decoded audio frames; determining a loss value according to the difference between the plurality of sample audio frames and the plurality of sample decoded audio frames and the bit number of the sample first encoded data; updating the trainable parameters in the model according to the loss value.

14. An encoding device, comprising: The encoding device is configured to implement the encoding method of any one of claims 1-9.

15. A decoding device, comprising: The decoding device is configured to implement the decoding method of any one of claims 10-12.

16. A coding system characterized by The model comprises an encoding device and a decoding device, wherein the encoding device is configured to implement the encoding method of any one of claims 1-9, and the decoding device is configured to implement the decoding method of any one of claims 10-12.

17. A storage medium, the storage medium storing instructions, wherein, When the instructions run on the communication device, the communication device is caused to perform the method of any one of claims 1-13.

18. A program product comprising at least one of a program, instructions, characterized in that The program and the instructions are configured to implement the method of any one of claims 1-13 when executed by the communication device. The program and the instructions are configured to implement the method of any one of claims 1-13 when executed by the communication device.

Images