Audio coding method, apparatus and device

Abstract

Provided are an audio coding method, device and equipment, relating to the field of audio and video coding. The audio decoding method comprises: parsing a to-be-decoded code stream to obtain a quantization result; performing inverse quantization on the quantization result to obtain a reconstructed coding vector; and obtaining a combined reconstructed coding vector according to the reconstructed coding vector and padding data, wherein the length of the padding data is determined according to the effective input length corresponding to a decoding network, and the padding data comprises the reconstructed coding vector of the decoded code stream; inputting the reconstructed coding vector corresponding to the effective input length in the combined reconstructed coding vector into the decoding network; performing a decoding operation on the input reconstructed coding vector by using the decoding network to output a reconstructed signal; the reconstructed signal is related to the reconstructed coding vector of the to-be-decoded code stream and is not related to the reconstructed coding vector of the decoded code stream, and the decoding network does not perform non-real data padding. The embodiments of the present application can improve the coding efficiency and quality of the audio coding architecture based on deep learning.

Description

Technical Field

[0001] This application relates to the field of multimedia technology, and in particular to an audio encoding and decoding method, apparatus, and device. Background Technology

[0002] With the rapid development of deep learning technology, it has been widely applied to the processing of signals in different dimensions, such as audio, images, and video. Taking audio signals as an example, in an end-to-end audio codec scheme based on deep learning, the encoder maps the audio signal into a coding vector through an encoding network, and further generates a corresponding binary bitstream file through quantization technology. The decoder reads the binary bitstream file to obtain the quantization result, and then performs inverse quantization on the quantization result to obtain the reconstructed coding vector. This reconstructed coding vector is then used as the input to the decoding network to decode the final reconstructed audio signal. How to further improve the coding efficiency and quality of deep learning-based audio codecs remains a pressing issue. Summary of the Invention

[0003] This application provides an audio encoding / decoding method, apparatus, and device that can improve the encoding / decoding efficiency and quality of deep learning-based audio encoding / decoding architectures.

[0004] In a first aspect, embodiments of this application provide an audio decoding method, including:

[0005] Parse the bitstream to be decoded to obtain the quantization result corresponding to the bitstream to be decoded;

[0006] The quantization result is dequantized to obtain the reconstructed encoding vector of the bitstream to be decoded;

[0007] Based on the reconstructed coding vector and the padding data, a combined reconstructed coding vector is obtained; wherein, the length of the padding data is determined according to the effective input length corresponding to the decoding network, and the padding data includes the reconstructed coding vector of the decoded bitstream;

[0008] The reconstructed encoding vector corresponding to the effective input length in the combined reconstructed encoding vector is input into the decoding network, and the decoding network is used to perform a decoding operation on the input reconstructed encoding vector to output a reconstructed signal; wherein, the reconstructed signal is related to the reconstructed encoding vector of the bitstream to be decoded in the input reconstructed encoding vector and is not related to the reconstructed encoding vector of the decoded bitstream in the input reconstructed encoding vector, and the decoding network does not fill the input reconstructed encoding vector with non-real data.

[0009] Secondly, embodiments of this application provide an audio encoding method, including:

[0010] Obtain the first audio data to be encoded;

[0011] Based on the first audio data and the padding data, combined audio data is obtained; wherein, the length of the padding data is determined according to the effective input length corresponding to the encoding network, and the padding data includes the encoded audio data;

[0012] The second audio data corresponding to the effective input length in the combined audio data is input into the encoding network, and the encoding network is used to encode the second audio data to output an encoding vector; wherein, the encoding vector is related to the audio data to be encoded in the second audio data but is not related to the encoded audio data in the second audio data, and the encoding network does not fill the second audio data with non-real data;

[0013] The encoded vector is quantized to obtain a quantization result, and the bitstream of the first audio data is obtained based on the quantization result.

[0014] Thirdly, embodiments of this application provide an audio decoding apparatus, including:

[0015] The parsing unit is used to parse the bitstream to be decoded and obtain the quantization result corresponding to the bitstream to be decoded.

[0016] The dequantization unit is used to dequantize the quantization result to obtain the reconstructed encoding vector of the bitstream to be decoded;

[0017] A padding unit is used to obtain a combined reconstructed coding vector based on the reconstructed coding vector and padding data; wherein the length of the padding data is determined according to the effective input length corresponding to the decoding network, and the padding data includes the reconstructed coding vector of the decoded bitstream;

[0018] The decoding unit is used to input the reconstructed coding vector corresponding to the effective input length in the combined reconstructed coding vector into the decoding network, and use the decoding network to perform a decoding operation on the input reconstructed coding vector to output a reconstructed signal; wherein, the reconstructed signal is related to the reconstructed coding vector of the bitstream to be decoded in the input reconstructed coding vector and is not related to the reconstructed coding vector of the decoded bitstream in the input reconstructed coding vector, and the decoding network does not fill the input reconstructed coding vector with non-real data.

[0019] Fourthly, embodiments of this application provide an audio encoding apparatus, including:

[0020] The acquisition unit is used to acquire the first audio data to be encoded.

[0021] A padding unit is used to obtain combined audio data based on the first audio data and the padding data; wherein the length of the padding data is determined according to the effective input length corresponding to the encoding network, and the padding data includes the encoded audio data;

[0022] An encoding unit is used to input the second audio data corresponding to the effective input length in the combined audio data into the encoding network, use the encoding network to encode the second audio data, and output an encoding vector; wherein, the encoding vector is related to the audio data to be encoded in the second audio data but is not related to the encoded audio data in the second audio data, and the encoding network does not fill the second audio data with non-real data;

[0023] A quantization unit is used to quantize the encoding vector to obtain a quantization result, and to obtain the bitstream of the first audio data based on the quantization result.

[0024] Fifthly, embodiments of this application provide an electronic device, including: a processor and a memory, the memory being used to store a computer program, and the processor being used to call and run the computer program stored in the memory to perform the methods as described in the first or second aspect.

[0025] In a sixth aspect, embodiments of this application provide a computer-readable storage medium including instructions that, when executed on a computer, cause the computer to perform the methods as described in the first or second aspect.

[0026] In a seventh aspect, embodiments of this application provide a computer program product including computer program instructions that cause a computer to perform the methods described in the first or second aspect.

[0027] Eighthly, embodiments of this application provide a computer program that causes a computer to perform the methods described in the first or second aspect.

[0028] The above technical solution, by determining the length of the padding data based on the effective input length of the encoding / decoding network, achieves precise control over the amount of encoded data padded during encoding and decoding. This ensures that the output of the encoding / decoding network in a single inference depends only on the current data to be encoded or decoded, helping to avoid introducing additional computations, reducing waste of computing resources, and thus improving encoding and decoding efficiency. Simultaneously, precise control over the amount of encoded data padded during encoding and decoding prevents the encoding / decoding network from padding the input data with non-real data, avoiding the influence of non-real data padding in the encoding network on the encoding and decoding process, thereby improving encoding quality. Attached Figure Description

[0029] Figure 1This is a schematic block diagram of an audio encoding and decoding system according to an embodiment of this application;

[0030] Figure 2 This is a schematic diagram of an end-to-end audio codec system based on deep learning, as described in an embodiment of this application.

[0031] Figure 3 This is a network structure block diagram of the codec built based on a deep learning network involved in the embodiments of this application;

[0032] Figure 4 This is a schematic flowchart of an audio encoding method provided in an embodiment of this application;

[0033] Figures 5A to 5C Here are some schematic diagrams illustrating the processing of convolutional networks;

[0034] Figures 6A to 6C A diagram illustrating three scenarios for filling in data using pre-coded data;

[0035] Figure 7 This is a schematic diagram of another audio encoding method provided in an embodiment of this application;

[0036] Figures 8A to 8D Here are several schematic diagrams illustrating the encoding process involved in the embodiments of this application;

[0037] Figure 9 This is a schematic flowchart of an audio decoding method provided in an embodiment of this application;

[0038] Figures 10A to 10C A diagram illustrating three scenarios where data is filled using decoded data;

[0039] Figure 11 This is a schematic diagram of another audio decoding method provided in an embodiment of this application;

[0040] Figures 12A to 12C Here are several schematic diagrams illustrating the decoding process involved in the embodiments of this application;

[0041] Figure 13 This is a schematic block diagram of an audio decoding apparatus according to an embodiment of this application;

[0042] Figure 14 This is a schematic block diagram of an audio encoding apparatus according to an embodiment of this application;

[0043] Figure 15 This is a schematic block diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0044] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0045] It should be understood that in the embodiments of this application, "B corresponding to A" means that B is associated with A. In one implementation, B can be determined based on A. However, it should also be understood that determining B based on A does not mean determining B solely based on A; B can also be determined based on A and / or other information.

[0046] In the description of this application, unless otherwise stated, "at least one" means one or more, and "multiple" means two or more. Additionally, "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0047] It should also be understood that the descriptions of "first", "second", etc. appearing in the embodiments of this application are only for illustration and to distinguish the objects being described, and there is no order to them. They do not indicate any special limitation on the number of devices in the embodiments of this application, and cannot constitute any limitation on the embodiments of this application.

[0048] It should also be understood that specific features, structures, or characteristics relating to embodiments in the specification are included in at least one embodiment of this application. Furthermore, these specific features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0049] Furthermore, the terms “comprising” and “having”, and any variations thereof, are intended to cover non-exclusive inclusion, such that a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such processes, methods, products, or devices.

[0050] The embodiments of this application are applied to the field of artificial intelligence technology.

[0051] The relevant concepts involved in the embodiments of this application are introduced below.

[0052] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0053] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0054] Machine learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning.

[0055] With the research and advancement of artificial intelligence (AI) technology, AI is being studied and applied in various fields, such as smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, autonomous driving, drones, robots, smart healthcare, and smart customer service. It is believed that with the development of technology, AI will be applied in more fields and play an increasingly important role.

[0056] This application mainly introduces the application of artificial intelligence technology in audio encoding and decoding technology.

[0057] Audio encoding and decoding: The audio encoding process compresses audio into smaller data sets, while the decoding process restores the audio from those smaller data sets. The smaller encoded data is then used for network transmission, consuming less bandwidth.

[0058] Audio sampling rate: The audio sampling rate describes the number of data points contained within a unit of time (1 second). For example, an 8kHz sampling rate contains 8000 sample points, each corresponding to a short integer.

[0059] Audio sampling duration: refers to the duration of audio data, usually expressed in seconds (s). Sampling duration equals the number of sampling points divided by the sampling rate. The number of sampling points refers to the total number of samples in the audio data.

[0060] Codebook: A collection of multiple vectors, with the encoder and decoder storing identical codebooks.

[0061] Quantization: Find the nearest vector in the codebook to the input vector, return it as the replacement of the input vector, and return the corresponding codebook index position.

[0062] Quantizer: The quantizer is responsible for quantization and updating the vectors within the codebook.

[0063] Audio frame: Represents the minimum duration of a single audio transmission in a network.

[0064] Short-Time Fourier Transform (STFT) divides a long signal into several shorter, equally long signals, and then calculates the Fourier transform of each shorter segment. It is typically used to depict changes in the frequency and time domains and is an important tool in time-frequency analysis.

[0065] The audio encoding and decoding method provided in this application can be applied to the fields of audio encoding and decoding, hardware audio encoding and decoding, dedicated circuit video encoding and decoding, and real-time audio encoding and decoding. For example, the solution of this application can be combined with audio video coding standards (AVS), such as the H.264 / Audio Video Coding (AVC) standard. Alternatively, the solution of this application can be combined with other proprietary or industry standards. It should be understood that the technology of this application is not limited to any specific encoding and decoding standard or technology.

[0066] The audio encoding and decoding method provided in this application can be applied to any end-to-end audio encoding and decoding scheme based on deep learning.

[0067] To facilitate understanding, let's first combine... Figure 1 The audio encoding and decoding system involved in the embodiments of this application will be described.

[0068] Figure 1 This is a schematic block diagram of an audio encoding / decoding system according to an embodiment of this application. It should be noted that... Figure 1This is merely an example; the audio encoding and decoding system in this application includes, but is not limited to, [other systems]. Figure 1 As shown. Figure 1 As shown, the audio encoding / decoding system 100 includes an encoding device 110 and a decoding device 120. The encoding device encodes (can be understood as compressing) audio data to generate a bitstream and transmits the bitstream to the decoding device. The decoding device decodes the bitstream generated by the encoding device to obtain the decoded audio data.

[0069] The encoding device 110 in this application embodiment can be understood as a device with audio encoding function, and the decoding device 120 can be understood as a device with audio decoding function. That is, the encoding device 110 and the decoding device 120 in this application embodiment include a wider range of devices, such as smartphones, desktop computers, mobile computing devices, laptops (e.g., laptops), tablet computers, set-top boxes, televisions, cameras, playback devices, digital media players, audio game consoles, in-vehicle computers, etc.

[0070] In some embodiments, encoding device 110 may transmit encoded audio data (such as a bitstream) to decoding device 120 via channel 130. Channel 130 may include one or more media and / or means capable of transmitting encoded audio data from encoding device 110 to decoding device 120.

[0071] In one example, channel 130 includes one or more communication media that enable encoding device 110 to transmit encoded audio data directly to decoding device 120 in real time. In this example, encoding device 110 can modulate the encoded audio data according to a communication standard and transmit the modulated audio data to decoding device 120. The communication media includes wireless communication media, such as radio frequency spectrum; optionally, the communication media may also include wired communication media, such as one or more physical transmission lines.

[0072] In another example, channel 130 includes a storage medium that can store audio data encoded by encoding device 110. The storage medium includes various local access data storage media, such as optical discs, DVDs, flash memory, etc. In this example, decoding device 120 can retrieve the encoded audio data from the storage medium.

[0073] In another example, channel 130 may include a storage server that stores the audio data encoded by encoding device 110. In this example, decoding device 120 can download the stored encoded audio data from the storage server. Optionally, the storage server can store and transmit the encoded audio data to decoding device 120, such as a web server (e.g., for a website), a file transfer protocol (FTP) server, etc.

[0074] In some embodiments, the encoding device 110 includes an audio encoder 112 and an output interface 113. The output interface 113 may include a modulator / demodulator (modem) and / or a transmitter.

[0075] In some embodiments, the encoding device 110 may include an audio source 111 in addition to the audio encoder 112 and the input interface 113.

[0076] Audio source 111 may include at least one of an audio acquisition device (e.g., a microphone), an audio archive, an audio input interface, and a computer speech system, wherein the audio input interface is used to receive audio data from an audio content provider, and the computer speech system is used to generate the audio data.

[0077] Audio encoder 112 encodes audio data from audio source 111 to generate a bitstream. The bitstream contains the encoding information of the audio data in the form of a bitstream. The encoding information may include encoded audio data and associated data. The associated data may include quantization parameters and other syntax structures. The syntax structure refers to the set of zero or more syntax elements arranged in a specified order in the bitstream.

[0078] The audio encoder 112 transmits the encoded audio data directly to the decoding device 120 via the output interface 113. The encoded audio data can also be stored on a storage medium or a storage server for subsequent retrieval by the decoding device 120.

[0079] In some embodiments, the decoding device 120 includes an input interface 121 and an audio decoder 122.

[0080] In some embodiments, in addition to the input interface 121 and the audio decoder 122, the decoding device 120 may also include a playback device 123.

[0081] The input interface 121 includes a receiver and / or a modem. The input interface 121 can receive encoded audio data through channel 130.

[0082] The audio decoder 122 is used to decode the encoded audio data to obtain the decoded audio data, and transmit the decoded audio data to the playback device 123.

[0083] Playback device 123 plays the decoded audio data. Playback device 123 may be integrated with decoding device 120 or external to decoding device 120. Playback device 123 may include various playback devices.

[0084] also, Figure 1 This is merely an example; the technical solutions in the embodiments of this application are not limited to... Figure 1 For example, the technology of this application can also be applied to one-sided audio encoding or one-sided audio decoding.

[0085] Figure 2 This is a schematic diagram of an end-to-end audio codec system based on deep learning, as described in an embodiment of this application. Figure 2 As shown, the audio encoding and decoding system of this application embodiment includes: encoding network 210, quantization module 211, inverse quantization module 212 and decoding network 213.

[0086] During encoding, the encoding end (also called the transmitting end) first inputs the input audio data into the encoding network 210 for nonlinear transformation, obtaining the encoded vector of the input audio data (also called the embedded sequence or latent variables, etc.). Next, the encoded vector of the audio data is quantized by the quantization module 211 to obtain the quantization result of the encoded vector. For example, a residual-based vector quantizer is used, and the corresponding quantization parameters are selected according to the target bit rate. Finally, the quantized encoded vector is encoded into a binary bitstream.

[0087] During decoding, the decoding end (also known as the receiving end) first recovers the quantization result of the encoded vector from the bitstream, and then further recovers the encoded vector through the dequantization module 212, and inputs it into the decoding network 213 for nonlinear transformation to obtain the reconstructed audio data.

[0088] Figure 3 This is a block diagram of the network structure of an encoder-decoder built based on a deep learning network in one embodiment of this application.

[0089] like Figure 3 As shown, the network structure of the codec includes an encoding network 310 and a decoding network 320, wherein the encoding network 310 can be implemented in software as follows: Figure 1 The audio / video encoding device 110 and decoding network 320 shown can be implemented in software, such as... Figure 1 The audio / video decoding device 120 shown. In some embodiments, the encoding network 310 is also referred to as encoder 310, and the decoding network 320 is also referred to as decoder 320.

[0090] At the data transmitting end, audio data can be encoded and compressed using an encoding network 310. In one embodiment of this application, the encoding network 310 may include an input layer 311, one or more encoding modules 312, and an output layer 313.

[0091] For example, the input layer 311 and the output layer 313 can be convolutional layers constructed based on one-dimensional convolutional kernels, with multiple (e.g., four) encoder blocks 312 connected sequentially between the input layer 311 and the output layer 313. Each encoder block 312 includes multiple residual unit modules, and each residual unit module contains multiple convolutional layers.

[0092] For example, in the input stage of the encoder, the raw audio data to be encoded is sampled to obtain a vector with c channels and w dimensions. This vector is then input to the input layer 311, and after convolution processing, a feature vector with 32c channels and w dimensions is obtained. In some optional implementations, to improve encoding efficiency, the encoding network 310 can encode a batch of audio vectors simultaneously.

[0093] During the downsampling stage of the encoder, the first encoding module reduces the vector dimension to 1 / 2 and doubles the number of channels, resulting in a feature vector with 64 channels and a dimension of 1 / 2w; the second encoding module reduces the vector dimension to 1 / 4 and doubles the number of channels, resulting in a feature vector with 128 channels and a dimension of 1 / 8w; the third encoding module reduces the vector dimension to 1 / 5 and doubles the number of channels, resulting in a feature vector with 256 channels and a dimension of 1 / 40w; and the fourth encoding module reduces the vector dimension to 1 / 8 and doubles the number of channels, resulting in a feature vector with 512 channels and a dimension of 1 / 320w.

[0094] In the output stage of the encoder, the output layer 313 performs convolution processing on the feature vector with 512c channels and 1 / 320w dimension to obtain an encoded vector with 1 channel and K dimension.

[0095] The encoded vector is input to the quantizer 330, and the codebook index corresponding to the encoded vector can be found in the codebook. The codebook index is then encoded to obtain a binary code stream, which is then sent to the data receiving end.

[0096] The data receiver decodes the received binary bitstream to obtain the codebook index, and performs inverse quantization based on the codebook index to obtain the reconstructed encoding vector. Finally, the reconstructed encoding vector is decoded by the decoding network 320 to obtain the restored audio data.

[0097] In one embodiment of this application, the decoding network 320 may include an input layer 321, one or more decoding modules 322, and an output layer 323. Each decoding module 322 includes multiple residual unit modules, and each residual unit module contains multiple convolutional layers.

[0098] After the data receiver decodes the bitstream to obtain the codebook index, it can first query the codebook vector corresponding to the codebook index in the codebook using the quantizer 320, and then obtain the encoded vector for audio data reconstruction based on the codebook vector. For example, the reconstructed encoded vector can be a vector with 1 channel and K dimensions. In some optional implementations, to improve decoding efficiency, the data receiver can decode a batch of codebook vectors simultaneously.

[0099] In the input stage of the decoder, the reconstructed encoded vector is input to the input layer 321. After convolution processing, a feature vector with 512c channels and a dimension of 1 / 320w can be obtained.

[0100] In the decoding stage of the decoder, the first decoding module increases the vector dimension by 8 times and reduces the number of channels by 2 times, resulting in a feature vector with 256 channels and a dimension of 1 / 40w; the second decoding module increases the vector dimension by 5 times and reduces the number of channels by 2 times, resulting in a feature vector with 128 channels and a dimension of 1 / 8w; the third decoding module increases the vector dimension by 4 times and reduces the number of channels by 2 times, resulting in a feature vector with 64 channels and a dimension of 1 / 2w; the fourth decoding module increases the vector dimension by 2 times and reduces the number of channels by 2 times, resulting in a feature vector with 32 channels and a dimension of w.

[0101] In the output stage of the decoder, the output layer 323 performs convolution processing on the feature vector with 32 channels and w dimensions to restore the reconstructed audio data with 1 channel and w dimensions.

[0102] In some embodiments, to improve audio coding performance, a residual-based vector quantizer is used when quantizing the encoded vector of the audio data, i.e., the one described above. Figure 3 The quantizer 330 in the code is a residual-based vector quantizer.

[0103] In related technologies, deep learning-based audio codecs support variable-length input data. If the data for a single inference is larger than the receptive field N of the neural network model... rf If the data used for a single inference is less than the receptive field N of the neural network model, then this inference will utilize all the data, leading to additional computation and a waste of computing resources. rfTherefore, the features corresponding to the data to be encoded will be affected by the non-real data padding of the convolutional network, thus affecting the encoding quality and the quality of the reconstructed signal. Thus, how to further improve the encoding and decoding efficiency and quality of deep learning-based audio encoding and decoding urgently needs to be addressed.

[0104] In view of this, embodiments of this application provide an audio encoding and decoding method, apparatus, and device, which improve the encoding and decoding efficiency and quality of a deep learning-based audio encoding and decoding architecture by controlling the amount of encoded data filled during encoding at the encoding end and controlling the amount of decoded data filled during decoding at the decoding end.

[0105] Specifically, at the encoding end, the first audio data to be encoded is acquired; combined audio data is obtained based on the first audio data and padding data; wherein, the length of the padding data is determined according to the effective input length corresponding to the encoding network, and the padding data includes the already encoded audio data; the second audio data corresponding to the effective input length in the combined audio data is input into the encoding network, and the encoding network is used to encode the second audio data to output an encoding vector; wherein, the encoding vector is related to the audio data to be encoded in the second audio data but not related to the already encoded audio data in the second audio data, and the encoding network does not pad the second audio data with non-real data; the encoding vector is quantized to obtain the quantization result, and the bitstream of the first audio data is obtained based on the quantization result.

[0106] Therefore, this embodiment of the application determines the length of the padding data based on the effective input length corresponding to the encoding network, thereby achieving precise control over the amount of encoded data padded during encoding. This ensures that the output encoding vector of the encoding network in one inference depends only on the current data to be encoded, which helps avoid introducing additional computations, reduces waste of computing resources, and thus improves encoding efficiency. Simultaneously, precise control over the amount of encoded data padded during encoding prevents the encoding network from padding the input audio data with non-real data, avoiding the encoding vector being affected by non-real data padding in the encoding network, thereby improving encoding quality.

[0107] At the decoding end, the code stream to be decoded is parsed to obtain the quantization result corresponding to the code stream to be decoded; the quantization result is dequantized to obtain the reconstructed encoding vector of the code stream to be decoded; based on the reconstructed encoding vector and padding data, a combined reconstructed encoding vector is obtained; wherein, the length of the padding data is determined according to the effective input length corresponding to the decoding network, and the padding data includes the reconstructed encoding vector of the decoded code stream; the encoding vector corresponding to the effective input length in the combined reconstructed encoding vector is input to the decoding network, and the decoding network performs a decoding operation on the input reconstructed encoding vector to output a reconstructed signal; wherein, the reconstructed signal is related to the reconstructed encoding vector of the code stream to be decoded in the input reconstructed encoding vector and is not related to the reconstructed encoding vector of the decoded code stream in the input reconstructed encoding vector, and the decoding network does not pad the input reconstructed encoding vector with non-real data.

[0108] Therefore, this embodiment of the application determines the length of the padding data based on the effective input length corresponding to the decoding network, thereby achieving precise control over the reconstructed encoding vector of the decoded bitstream during decoding. This ensures that the output reconstructed signal of the decoding network in one inference depends only on the current reconstructed encoding vector, which helps avoid introducing additional computations, reduces waste of computing resources, and thus improves decoding efficiency. Simultaneously, precise control over the reconstructed encoding vector of the decoded bitstream during decoding prevents the decoding network from padding the input reconstructed encoding vector with non-real data, avoiding the influence of non-real data padding in the decoding network on the reconstructed signal, thereby improving the quality of the reconstructed signal.

[0109] In some embodiments, the length of the audio data to be encoded in the second audio data input to the encoding network is less than or equal to the data size of one frame. Here, the second audio data is the audio data input to the encoding network for one inference iteration, and the length of the second audio data is the aforementioned effective input length. Thus, the output encoding vector of the encoding network is a frame-level encoding vector, thereby achieving the effect of separable frame-by-frame bitstreams.

[0110] In some embodiments, the length of the reconstructed encoding vector to be decoded in the input reconstructed encoding vector of the decoding network is equal to the length of the reconstructed encoding vector corresponding to one frame of data. Here, the input reconstructed encoding vector is the encoding vector input to the decoding network for one inference, and the length of the input reconstructed encoding vector is the aforementioned effective input length. In this way, the output reconstructed signal of the decoding network is a frame-level reconstructed signal, thereby achieving the effect of separable frame-by-frame reconstructed signals.

[0111] The technical solutions of the embodiments of this application will be described in detail below through some examples. The following embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0112] First, taking the encoding end as an example, the audio encoding method provided in the embodiments of this application will be introduced below.

[0113] Figure 4 This is a schematic flowchart of an audio encoding method provided in an embodiment of this application. The execution entity of this embodiment can be a device with a specific audio encoding function, such as an audio encoding apparatus. In some embodiments, the audio encoding apparatus can be... Figure 1 The encoding device in the process. For ease of description, the embodiments of this application are illustrated using the encoding device as the execution subject.

[0114] like Figure 4 As shown, the audio encoding method of this application embodiment includes the following steps 410 to 440:

[0115] 410, Obtain the first audio data to be encoded.

[0116] The first audio data to be encoded can be a segment of audio data of any length, and this application embodiment does not limit this.

[0117] In some embodiments, the first audio data to be encoded includes M frames of data and data corresponding to an incomplete frame length; M is a positive integer greater than or equal to 1. The M frames of data are one or more audio frames of complete frame length, and the data corresponding to the incomplete frame length is an incomplete audio frame, such as a 1 / 2 audio frame, a 1 / 4 audio frame, or an audio frame of other lengths, without limitation.

[0118] In the embodiments of this application, an audio frame can be understood as a data segment with a specified time length obtained after performing frame-segmentation and windowing processing on the original audio data.

[0119] For example, the raw audio data can be audio data with a certain sampling duration obtained according to a preset sampling rate. In some examples, the raw audio data can be speech collected by the terminal. In some examples, the raw audio data can be sound signals collected in network voice calls or video calls. In some examples, the raw audio data can be sound signals collected in live streaming scenarios, online singing scenarios, or voice broadcasting scenarios. In some examples, the raw audio data can be audio data obtained from storage resources. For example, the raw audio data can be stored speech, music, video, etc. This application embodiment does not limit the specific method of obtaining the raw audio data.

[0120] In one possible implementation, when dividing the original audio data into audio frames, this application embodiment can set a preset duration for the division. For example, the original audio data can be divided into one audio frame every 10ms, or the original audio data can be divided into one audio frame every 20ms. This application does not limit this. The preset duration is the frame length.

[0121] In order to enable audio data to be stored and transmitted over long distances, the acquired raw audio data needs to be encoded to reduce the size of the audio data, thereby reducing the storage space required for audio data or reducing the bandwidth consumed for long-distance transmission.

[0122] 420. Based on the first audio data and the padding data, combined audio data is obtained; wherein, the length of the padding data is determined according to the effective input length corresponding to the encoding network, and the padding data includes the encoded audio data.

[0123] Specifically, in deep learning-based end-to-end audio codec systems, the encoding network uses convolutional networks for data processing. Each convolutional layer in a convolutional network uses a convolutional kernel to encode the input data, and the output data length of each convolutional layer is shorter than its input data length. To ensure the required length of the convolutional network output, the input audio data needs to be padded.

[0124] As an example, taking a regular convolutional network as an example, the processing procedure for one element in a batch (each element in the batch undergoes the same process) is as follows: Figure 5A As you can see, to compute an element of the output, we need to examine a series of consecutive elements of the input with a length equal to the kernel size (kernel_size). In this example, kernel_size is 3. To obtain the output, a subsequence of the input is dot-producted with a kernel vector of learned weights of the same length. To obtain the next element of the output, the same process is applied, but the window of the input sequence, which is kernel_size in size, is shifted one element to the right (for this model, the stride is set to 1). Note that the weights of the same set of kernel vectors are used here to compute each output of a convolutional layer. To ensure that the output sequence has the same length as the input sequence, the input sequence can be zero-paddinged, such as by adding extra zero values ​​to the beginning or end of the input vector, to ensure that the output has the length required by the network.

[0125] As an example, taking a causal convolutional network as an example, for a causal convolutional layer, for the i-th element in the input sequence {0, ..., input_length-1}, the i-th element in the output sequence depends only on the elements of the input sequence with indices {0, ..., i}. In other words, an element of the output sequence can only depend on the elements preceding that element in the input sequence. To ensure that the output vector has the same length as the input vector, zero padding is needed. For example, zero padding can be applied to the left side of the input vector. Since there is no padding on the right side of the input sequence, its last dependency is the last element of the input. Compared to the last output element, the kernel window of the second-to-last output element of the output sequence has shifted one position to the left, meaning that its last dependency in the input sequence is the second-to-last element of the input sequence. By induction, it can be concluded that for each element in the output sequence, its latest dependency in the input sequence has the same index as itself.

[0126] Figure 5B The example shown has an input length of 4 and a kernel_size of 3. It demonstrates that by adding left-hand zero padding to two entries, the same output length can be achieved while adhering to the causal relationship. The number of zero-padding entries required to maintain the input length is always equal to kernel_size - 1. Similarly, as... Figure 5C As shown, when a convolutional network uses causal convolution, the data to be processed is placed on the far right of the input data. The network's effective input length requirement is achieved by padding the left side of the input data with encoded data or zeros.

[0127] Based on this, the audio data is padded with a certain length of padding data in this step to ensure that the encoding network outputs the required encoding vector length.

[0128] In this embodiment, by determining the length of the padding data based on the effective input length corresponding to the encoding network, the length of the encoded data padding the input audio data during encoding can be precisely controlled, thereby ensuring that the output encoding vector of the encoding network in one inference depends only on the current data to be encoded.

[0129] The encoding process for different lengths of encoded data is described in detail below with reference to the accompanying drawings.

[0130] The process is described using the encoding of one-dimensional data in an end-to-end manner. It is assumed that the end-to-end encoder-decoder network uses a causal convolutional network, and the effective input length required for each inference operation at the encoder end is N. opFor ease of discussion, the input data N corresponding to one encoding inference can be further represented as [N2, N1, N0], where N0 represents the current data to be encoded with a length of N0, N1 represents the encoded data with a length of N1, and N2 represents other padding data (such as zero padding) with a length of N2.

[0131] Optionally, N0 can be a value representing the frame length of the codec system (i.e., the amount of data corresponding to one frame). For a given end-to-end codec system, N0 can be assumed to be a fixed value.

[0132] If the input data of the encoding network contains only the current data to be encoded, N0, then the encoding network will pad the left side of the input data with (N... op -N0) non-real data (such as zero) are used to ensure that the network can perform operations normally. However, this will introduce a lot of zeros into the calculation process, resulting in poor quality of the reconstructed signal after decoding.

[0133] Therefore, in this embodiment of the application, by configuring the padding data as encoded data, a certain amount of encoded data N1 can be filled to the left of the current data to be encoded N0, so as to use real data as much as possible in the calculation and ensure the quality of the reconstructed signal after decoding.

[0134] Assuming that encoded data of length N1 is used for padding, there are three possible scenarios for the padding data.

[0135] The first case (corresponding to) Figure 6A If N1 + N0 < N op Then, the convolutional network will continue to pad the input data with zeros (adding a certain length of other padding data, as shown by the arrow in Case 1) to ensure the effective input length required for encoding inference. This situation is equivalent to the input data being [N2, N1, N0]. Thus, the output length of the encoding network includes the position from the rightmost point to the position corresponding to the arrow in Case 1, including the bitstream corresponding to N0, N1, and N2. In other words, some features corresponding to the data to be encoded will be affected by the zero-padding operation of the convolutional network itself, thereby affecting the quality of the reconstructed signal.

[0136] The second scenario (corresponding to) Figure 6B If N1 + N0 > N opIn this case, the convolutional network will use the encoded data as padding, and the inference will use all the data (N1+N0) for inference. The input data is then [N1, N0]. Thus, the output length of the encoding network includes the position from the rightmost point to the position corresponding to the arrow in case 2, including the bitstream corresponding to N0 and the bitstream corresponding to N1. Therefore, part of the generated bitstream corresponds to the data to be encoded N0, and part corresponds to the encoded data N1. However, this introduces additional computation to calculate (N1+N0-N... op This resulted in a waste of computing resources.

[0137] The third scenario (corresponding to) Figure 6C If N1 + N0 = N op In this case, the convolutional network will use the encoded data as padding data, and the inference will still use all the data (N1+N0) for inference. The input data is now [N1, N0]. However, since N1+N0 = N... op Therefore, the output length of the encoding network includes only the bitstream corresponding to N0, from the rightmost position to the position indicated by the arrow in Case 3. Thus, the generated bitstream entirely corresponds to the data N0 to be encoded, without introducing additional computation to calculate the encoded data, achieving the theoretically minimum computational cost required for encoding. Furthermore, in this case, no additional zero-padding is needed on the input data, thus avoiding a series of processing steps caused by zero-padding and contributing to better encoding quality.

[0138] Therefore, in this embodiment, the length of the input data after padding is made equal to the effective input length N required by the encoding network for inference at the encoding end by padding the data to be encoded. op It can precisely control the length of the encoded data that fills the input audio data during encoding, thereby ensuring that the output encoding vector of the encoding network in one inference depends only on the current data to be encoded, and at the same time ensuring that no non-real data is introduced into the calculation process.

[0139] In some embodiments, for the third case described above, when N0 is the length of the encoding / decoding system frame (i.e., the amount of data corresponding to one frame of data), the bitstream generated in case 3 corresponds to the bitstream of the system frame length (i.e., one frame of data), so that the bitstream of the current frame generated by the encoding network depends only on the current data to be encoded, thus achieving the effect of frame-level bitstream separability.

[0140] It should be noted that, for a given encoding network, the corresponding effective input length N can be calculated based on the encoding network architecture. opIn some embodiments, for an encoding network (the network architecture may be known or unknown), the effective input length of the encoding network can be determined by testing. For example, different sizes of audio data can be input into the encoding network, and when the input audio data length is less than the effective data length N, the effective input length can be determined. op The system reported an error when the length of the input audio data was greater than or equal to the valid data length N. op The system does not report errors, thus determining the effective input length of the encoding network.

[0141] In some embodiments, the length of the padding data is determined by the effective input length corresponding to the encoding network and the data margin that is not an integer multiple of the frame length of the first audio data.

[0142] Specifically, the data reserve of the first audio data that is not a multiple of the frame length refers to the audio data remaining after removing audio frames with a frame length that is a multiple of the frame length from the first audio data. For example, the data reserve of the first audio data that is not a multiple of the frame length can be expressed by the following formula:

[0143]

[0144] Where, N r This represents a non-integer multiple of the frame length of the audio data, where N0 represents the first audio data, and floor() is the round-down integer operation. f The length of one frame of audio data (i.e., one audio frame).

[0145] In some embodiments, it is possible to record This indicates the number of complete audio frames contained in the first audio data.

[0146] For example, when the first audio data includes M frames of data, the length of the first audio data is an integer multiple of the frame length, and the corresponding data margin of the first audio data that is not an integer multiple of the frame length is zero.

[0147] For example, when the audio data includes M frames of data and data corresponding to an incomplete frame length, the length of the first audio data is a non-integer multiple of the frame length, and the data margin of the corresponding non-integer multiple frame length of the first audio data is the amount of data corresponding to the incomplete frame length.

[0148] For example, when the audio data includes data corresponding to an incomplete frame length, the data margin of the first audio data that is not an integer multiple of the frame length is the amount of data corresponding to the incomplete frame length.

[0149] In some embodiments, if the data margin of the first audio data is not an integer multiple of the frame length and is not zero, then the length of the padding data is the difference between the effective input length of the encoding network and the data margin.

[0150] For example, when the first audio data includes M frames of data and data of an incomplete frame length, or when the first audio data includes only data of an incomplete frame length, the length of the padding data is the difference between the effective input length of the encoding network and the amount of data corresponding to the incomplete frame length.

[0151] For example, the length of the padding data can be expressed by the following formula:

[0152] N1 = N op -N r

[0153] Where N1 is the length of the padding data, N op This represents the effective input length of the encoding network.

[0154] In some embodiments, if the data margin of the first audio data that is not an integer multiple of the frame length is zero, then the length of the padding data is the difference between the effective input length of the encoding network and the amount of data corresponding to one frame.

[0155] For example, when the first audio data consists of only M frames, the length of the padding data is the difference between the effective input length of the encoding network and the amount of data corresponding to one frame. Thus, the sum of the lengths of one frame and the padding data in the first audio data is the effective input length of the encoding network.

[0156] For example, the length of the padding data can be expressed by the following formula:

[0157] N1 = N op -N f

[0158] Where N1 is the length of the padding data, N op This represents the effective input length of the encoding network.

[0159] After determining the length of the padding data, the padding data can be combined with the first audio data to obtain combined audio data. For example, the combined audio data can be represented as N, where N = [N1, N0].

[0160] In some embodiments, when the encoding network includes a causal convolutional network, padding data can be padded before the first audio data to be encoded to obtain combined audio data, such that the padding data is combined with the first data to be encoded in the first audio data as the input to the encoding network. For example, the positions of the padding data and the audio data can be referenced... Figures 6A to 6C The positional relationship between the pre-encoded data padding (N1) and the data to be encoded (N0) is determined, and the padding data is then placed to the left of the first audio data to be encoded.

[0161] 430. Input the second audio data corresponding to the effective input length in the combined audio data into the encoding network, use the encoding network to encode the second audio data, and output the encoding vector; wherein, the encoding vector is related to the audio data to be encoded in the second audio data but is not related to the encoded audio data in the second audio data, and the encoding network does not fill the second audio data with non-real data.

[0162] Specifically, the encoding network encodes the combined audio data and outputs an encoded vector. The length of the second audio data input to the encoding network each time is the effective input length. This ensures that the encoded vector output by the encoding network at each inference step is related to the audio data to be encoded in the second audio data input for the current inference, but not to the decoded audio data in the second audio data input for the current inference step. Furthermore, the encoding network does not pad the input second audio data with non-real data during each inference step. This helps ensure that the output encoded vector of the encoding network at each inference step depends only on the current audio data to be encoded, and that no non-real data is introduced into the computation during each inference process. For example, the non-real data is zero.

[0163] In some embodiments, the audio data to be encoded in the second audio data of the encoding network is less than or equal to the data size of one frame. Here, the second audio data is the audio data input to the encoding network for one inference iteration, and the length of the second audio data is the aforementioned effective input length.

[0164] Specifically, traditional audio codecs typically process input data of a specific length (e.g., frame length), such as a 20ms frame. If the input data length is less than 20ms, it needs to be padded with zeros until the data length reaches the preset frame length. Conversely, if the input data length is greater than 20ms, it is sliced ​​so that each slice, except the last one, has the preset frame length, and the last frame is padded with zeros as needed. Afterward, each frame of data is used as a basic encoding unit for encoding and decoding, enabling frame-by-frame bitstream separability.

[0165] Deep learning-based audio codecs support variable-length input data. If the data for a single inference iteration is larger than the receptive field N of the neural network model... rf In this case, the inference process would use all the data, resulting in a bitstream that cannot be directly mapped to the input data. However, in real-time communication scenarios, the data to be encoded is newly received data, assuming its length is N. sr If the exact amount of data to be filled cannot be calculated, N srThe first part is used for the current encoding, while the remaining part is used for the next encoding. This loses the function of separating the frame-by-frame data bitstream and requires additional post-processing to align the output data.

[0166] In this embodiment, by configuring the amount of audio data to be encoded in the input audio data of the encoding network to be less than or equal to the amount of data in one frame, the encoding network can output a frame-level encoding vector during each inference. For example, when the amount of audio data to be encoded in the input audio data of the encoding network is less than the amount of data corresponding to one frame, the audio data to be encoded corresponds to a data length of an incomplete frame, and the output encoding vector of the current frame includes the encoding vector corresponding to the incomplete frame length. As another example, when the amount of audio data to be encoded in the input audio data of the encoding network is equal to the amount of data in one frame, the audio data to be encoded corresponds to a complete audio frame, and the output encoding vector of the current frame is the encoding vector corresponding to the complete audio frame. Therefore, this embodiment can achieve the effect of frame-by-frame bitstream separability.

[0167] In some embodiments, see Figure 7 The output encoding vector of the encoding network can be obtained according to the following step 431.

[0168] 431. Input the audio data corresponding to the first valid input length in the combined audio data into the encoding network, use the encoding network to encode the input audio data, and output the first encoding vector corresponding to the first frame data.

[0169] In this context, the length of the combined audio data is greater than or equal to the effective input length. As an example, when the first audio data to be encoded is less than or equal to the data size of one frame, the length of the combined audio data equals the effective input length. In this case, the audio data corresponding to the first effective input length in the combined audio data is the combined audio data itself. As another example, when the first audio data to be encoded is greater than the data size of one frame, the length of the combined audio data is greater than the effective input length. In this case, the audio data corresponding to the first effective input length in the combined audio data is the padding data and the data margin (not an integer multiple of the frame length) in the combined audio data, where this data margin is the earliest temporally significant data in the first audio data.

[0170] The audio data corresponding to the first valid input length of the combined audio data, that is, the audio data of the first batch input to the encoding network in the combined audio data. The length of each batch of data input to the encoding network is the valid input length.

[0171] In the first possible scenario, when the length of the combined audio data equals the effective input length, the entire combined audio data is used as the audio data corresponding to the first effective input length and input into the encoding network. For example... Figure 8A As shown, pre-encoded data N1 (i.e., padding data) can be combined with the data to be encoded N0, and concatenated to the left of the data to be encoded N0 before being input into the encoding network. The length of the combination of pre-encoded data N1 and data to be encoded N0 is equal to the effective input length. Correspondingly, the encoding network performs encoding operations on the input audio data and outputs an encoded vector for one frame of data. This encoded vector includes the encoded vector corresponding to the data to be encoded N0.

[0172] Optionally, if the data N0 to be encoded is less than the data size of a frame, the encoding vector of that frame also includes the encoding vector corresponding to N1 filled with some already encoded data. Optionally, if the data N0 to be encoded is equal to the data size of a frame, the encoding vector corresponding to that frame only includes the encoding vector corresponding to the data N0 to be encoded.

[0173] The second possible scenario is that when the length of the combined audio data exceeds the effective input length, the audio data corresponding to the first effective input length in the combined audio data is input into the encoding network. For example... Figure 8B As shown, if the data to be encoded, N0, is larger than the data size of one frame, then the already encoded data padding N1 (i.e., padding data) can be combined with the remaining data (Nr) in the data to be encoded, and concatenated to the left of the remaining data Nr before being input into the encoding network. The length of the combination of the already encoded data padding N1 and the remaining data Nr is equal to the effective input length. Correspondingly, the encoding network performs encoding operations on the input audio data (the audio data corresponding to the first effective input length) and outputs the encoded vector of the first frame of data. This encoded vector includes the encoded vector corresponding to the remaining data Nr and the encoded vector corresponding to part of the already encoded data padding N1.

[0174] Optionally, in other embodiments, when the data margin Nr is zero, the encoded data padding N1 can be combined with the first audio frame in the data to be encoded N0, and concatenated to the left side of the audio frame before being input into the encoding network. The length of the combined encoded data padding N1 and the first audio frame is equal to the effective input length. Correspondingly, the encoding vector of the first frame of data output by the encoding network includes the encoding vector of the first audio frame in the data to be encoded N0.

[0175] Optionally, for the second case, after obtaining the encoding vector of the first frame of data, the following step 432 can be performed to continue encoding other audio data to be encoded.

[0176] 432. Using the data size corresponding to one frame of data as the step length, the audio data corresponding to the i-th valid input length in the combined audio data is input into the encoding network. The encoding network is used to encode the input audio data and output the second encoding vector corresponding to the i-th frame of data, where i is a positive integer greater than 1.

[0177] For example, in the second possible case in step 431 above, the remaining audio data in the data to be encoded N0 can be encoded according to step 432.

[0178] Figure 8C This illustrates a schematic diagram of encoding audio frame 1. (For example...) Figure 8C As shown, after encoding the data surplus Nr to obtain the encoding vector of the first frame of data, the effective input length window is shifted to the right with the data size corresponding to one frame as the step size to obtain the audio data of the second effective input length in the combined audio data, which includes audio frame 1, data surplus Nr, and partially encoded padding data N1. At this time, the data surplus Nr in the second effective input length is the encoded audio data, and audio frame 1 is the audio data to be encoded. The audio data of the second effective input length is input into the encoding network, and the encoding network performs an encoding operation on the input audio data (the audio data corresponding to the second effective input length). In this convolution operation, the data surplus Nr and partially encoded padding data N1 are used as padding data for audio frame 1, and the encoding vector of the second frame of data is output. This encoding vector includes the encoding vector corresponding to audio frame 1.

[0179] Figure 8D A schematic diagram of encoding audio frame 2 is shown. For example... Figure 8D As shown, after encoding audio frame 1 to obtain the encoding vector of the second frame, the effective input length window is shifted to the right by a step size equal to the amount of data corresponding to one frame, resulting in the audio data of the third effective input length in the combined audio data. This includes audio frame 2, audio frame 1, data margin Nr, and partially encoded padding data N1. At this point, in the third effective input length, audio frame 1 and data margin Nr are encoded audio data, and audio frame 2 is the audio data to be encoded. The audio data of the third effective input length is input into the encoding network, which performs encoding operations on the input audio data (the audio data corresponding to the third effective input length). In this convolution operation, audio frame 1, data margin Nr, and partially encoded padding data N1 are used as padding data for audio frame 2, outputting the encoding vector of the third frame. This encoding vector includes the encoding vector corresponding to audio frame 2.

[0180] It is understood that by including data margin in the first valid input length, the embodiments of this application can first encode the data margin that is not an integer multiple of the frame length in the data to be encoded, and then encode the audio frames with integer multiples of the frame length in the data to be encoded in turn, so that the frames after the first frame of the data to be encoded correspond to the encoding vector of the complete audio frame.

[0181] It should be understood that Figures 8B to 8D The encoding process of the data N0 to be encoded is illustrated illustratively, and the embodiments of this application are not limited thereto. For example, in some embodiments, the audio data corresponding to the third valid input length may not include the encoded data padding N1. Furthermore, in some embodiments, the data N0 to be encoded may also include more audio frames, such as audio frame 3, audio frame 4, etc. The encoding process for audio frames after audio frame 2 is similar to that for audio frame 1 or audio frame 2, and can be referred to the relevant description above, which will not be repeated here.

[0182] In other embodiments, when the data margin Nr is zero, the encoding process for the audio data of the i-th valid input length is similar to that when the data margin Nr is non-zero, and can be referred to... Figure 8C and Figure 8D The relevant descriptions will not be repeated here.

[0183] In some embodiments, when the first audio data to be encoded includes M frames of data and data corresponding to an incomplete frame length, the audio data to be encoded can be processed as described above. Figures 8B to 8D The data is input into the encoding network in (M+1) sequential steps, resulting in encoding vectors for (M+1) frames of data. The encoding vectors of the last M frames (i.e., frames 2 to (M+1)) output by the encoding network correspond one-to-one with the last M input frames of data. The encoding vector of the last (M+1)th frame (i.e., frame 1) is included in the encoding vector of the last (M+1)th frame (i.e., frame 1) output by the encoding network. Optionally, the encoding vector of the last (M+2)th frame and all preceding encoding vectors are unrelated to the currently encoded data N0. Therefore, this embodiment enables the separation of the bitstream frame by frame during encoding.

[0184] In other embodiments, the length of the audio data to be encoded in the second audio data input to the encoding network can be greater than the data volume corresponding to one frame, for example, including two or more audio frames. Here, the input second audio data is the audio data input to the encoding network for one inference iteration, and the length of the second audio data is the aforementioned effective input length. Correspondingly, the encoding vector output by the encoding network for one inference iteration is greater than the encoding vector corresponding to the frame length, for example, the encoding vector of two or more audio frames; this embodiment does not limit this.

[0185] After obtaining the encoded vector of the audio data based on the above steps, the encoding end performs the following step 440.

[0186] 440. Quantize the encoded vector to obtain the quantization result, and obtain the bitstream of the first audio data based on the quantization result.

[0187] For example, the encoded vector can be input to a quantizer, the codebook index corresponding to the encoded vector can be retrieved from the codebook, and the codebook index can be encoded to obtain a binary bitstream. This application embodiment does not limit the specific method by which the encoding end quantizes the encoded vector of the first audio data.

[0188] Therefore, this embodiment of the application determines the length of the padding data based on the effective input length corresponding to the encoding network, thereby achieving precise control over the amount of encoded data padded during encoding. This ensures that the output encoding vector of the encoding network in one inference depends only on the current data to be encoded, which helps avoid introducing additional computations, reduces waste of computing resources, and thus improves encoding efficiency. Simultaneously, precise control over the amount of encoded data padded during encoding prevents the encoding network from padding the input audio data with non-real data, avoiding the encoding vector being affected by non-real data padding in the encoding network, thereby improving encoding quality.

[0189] In some embodiments, the bitstream may further include first indication information for indicating whether the frame-by-frame bitstream corresponds to audio data of a complete frame length.

[0190] For example, the first indication information may include at least one first bit, each first bit corresponding to the bitstream of an audio frame, used to indicate whether the corresponding audio frame is audio data of a complete frame length. As an example, the first bit may include 1 bit, where a value of 1 indicates that the corresponding audio frame is audio data of a complete frame length; and a value of 0 indicates that the corresponding audio frame is audio data of an incomplete frame length.

[0191] As a concrete example, when the first audio data to be encoded includes M frames of data and data with an incomplete frame length, the bitstream of the audio data obtained by quantizing the encoded vector output by the encoding network has M first bits corresponding to the last M frames (i.e., frames 2 to (M+1)th) of the data, each with a bit value of 1, indicating that the bitstream of these M frames corresponds to audio data with a complete frame length. The bitstream of the last (M+1)th frame (i.e., frame 1) corresponds to 1 first bit with a bit value of 0, indicating that the bitstream of this frame corresponds to audio data with an incomplete frame length.

[0192] Therefore, by carrying first indication information in the bitstream, the embodiments of this application can indicate whether each frame of the bitstream corresponds to audio data of a complete frame length. Based on this, the decoding end can determine whether each frame of the bitstream corresponds to audio data of a complete frame length according to the first indication information, which is beneficial for the decoding end to achieve accurate decoding.

[0193] In some embodiments, the bitstream may also include second indication information for indicating the amount of audio data corresponding to the incomplete frame length.

[0194] Specifically, when the first indication in the bitstream is used to indicate audio data of an incomplete frame length corresponding to the bitstream, the second indication information can further specify the amount of audio data corresponding to that incomplete frame length. For example, the second indication information may include at least one second bit used to indicate the data length of the corresponding audio frame. For example, the data type of the second bit is integer.

[0195] Therefore, by carrying second indication information in the bitstream, the embodiments of this application can further indicate the amount of audio data of incomplete frame length. Based on this, the decoding end can determine the amount of audio data of incomplete frame length according to the second indication information, which is beneficial for the decoding end to achieve accurate decoding.

[0196] The audio encoding method involved in the embodiments of this application has been described above. The audio decoding method provided in the embodiments of this application will now be described using the decoding end as an example.

[0197] Figure 9 This is a schematic flowchart of an audio decoding method provided in an embodiment of this application. The execution entity of this embodiment can be a device with a specific audio decoding function, such as an audio decoding device. In some embodiments, the audio decoding device can be... Figure 1 The decoding device in the process. For ease of description, this application's embodiments use a decoding device as an example for illustration.

[0198] like Figure 9 As shown, the audio decoding method of this application embodiment includes the following steps 910 to 940:

[0199] 910. Parse the bitstream to be decoded to obtain the quantization result corresponding to the bitstream to be decoded.

[0200] Specifically, to enable the storage and long-distance transmission of audio data, the acquired raw audio data needs to be encoded to reduce its size, thereby reducing storage space or bandwidth consumption for long-distance transmission. After encoding the audio data to be encoded into a bitstream, the encoding end sends the bitstream to the decoding end. The corresponding decoding end receives the bitstream, parses it, and obtains the quantization result of the audio data.

[0201] The quantization result is obtained by quantizing the encoded vector of the audio data. The encoded vector is obtained by encoding the audio data to be encoded using an encoding network. The audio data to be encoded can be referenced from... Figure 4 The description in step 410 will not be repeated here.

[0202] In some embodiments, the bitstream to be decoded further includes first indication information, used to indicate whether the frame-by-frame bitstream corresponds to audio data of a complete frame length. The decoding end can also obtain the first indication information by parsing the bitstream.

[0203] For example, the first indication information may include at least one first bit, each first bit corresponding to a bitstream, used to indicate whether the corresponding bitstream is audio data of the corresponding complete frame length. For example, if the audio data to be encoded has been encoded (as described above)... Figure 4 If the encoding process shown yields a frame of bitstream, then that frame of bitstream is said to correspond to the audio data to be encoded. As an example, the amount of audio data in this incomplete frame length can be the data margin mentioned above.

[0204] As an example, the first bit may include 1 bit. When the value of the first bit is 1, it can indicate that the bitstream corresponds to audio data of a complete frame length; when the value of the first bit is 0, it can indicate that the bitstream corresponds to audio data of an incomplete frame length.

[0205] As a concrete example, when the first audio data to be encoded includes M frames of data and data with an incomplete frame length, the bitstream of the audio data obtained by quantizing the encoded vector output by the encoding network has M first bits corresponding to the last M frames (i.e., frames 2 to (M+1)th) of the data, each with a bit value of 1, indicating that the bitstream of these M frames corresponds to audio data with a complete frame length. The bitstream of the last (M+1)th frame (i.e., frame 1) corresponds to 1 first bit with a bit value of 0, indicating that the bitstream of this frame corresponds to audio data with an incomplete frame length.

[0206] Therefore, by carrying first indication information in the bitstream, the embodiments of this application can indicate whether each frame of the bitstream corresponds to audio data of a complete frame length. Based on this, the decoding end can determine whether each frame of the bitstream corresponds to audio data of a complete frame length according to the first indication information, which is beneficial for the decoding end to achieve accurate decoding.

[0207] In some embodiments, the bitstream may further include second indication information, used to indicate the amount of audio data to be encoded corresponding to incomplete frame length audio data to be encoded. The decoding end can also obtain the second indication information by parsing the bitstream.

[0208] Specifically, when the first indication in the bitstream indicates that the corresponding audio frame is an incomplete audio frame, the decoding end can further obtain the second indication information to determine the amount of audio data corresponding to the length of the incomplete frame. For example, the second indication information may include at least one second bit used to indicate the data length of the corresponding audio frame. For example, the data type of the second bit is integer.

[0209] 920, dequantize the quantization result to obtain the reconstructed encoding vector of the bitstream to be decoded.

[0210] Specifically, the decoding end parses the acquired bitstream, obtains the quantization result corresponding to the bitstream to be decoded, and then performs dequantization on the quantization result to obtain the reconstructed encoding vector of the bitstream to be decoded.

[0211] 930. Based on the reconstructed coding vector and the padding data, a combined reconstructed coding vector is obtained; wherein, the length of the padding data is determined according to the effective input length corresponding to the decoding network, and the padding data includes the reconstructed coding vector of the decoded bitstream.

[0212] Specifically, in a deep learning-based end-to-end audio codec system, the decoding network uses convolutional networks for data processing. Each convolutional layer in the network uses a convolutional kernel to decode the input data, and the output data length of each convolutional layer is shorter than its input data length. To ensure the required length of the convolutional network output, the input audio data needs to be padded. Therefore, in this step, the reconstructed encoding vector is padded with a certain length of padding data to ensure the required reconstructed signal length for the decoding network output.

[0213] For details on the processing steps of convolutional networks, please refer to the above text. Figures 5A to 5C The relevant descriptions in the text will not be repeated here.

[0214] In this embodiment, by determining the length of the padding data according to the effective input length corresponding to the decoding network, the length of the reconstructed encoding vector of the decoded bitstream filled by the input reconstructed encoding vector during decoding can be precisely controlled, thereby ensuring that the reconstructed signal output by the decoding network in one inference depends only on the reconstructed encoding vector of the current bitstream to be decoded.

[0215] The following describes in detail, with reference to the accompanying drawings, the decoding process corresponding to the reconstructed encoding vector length of decoded bitstreams of different lengths.

[0216] The decoding process of one-dimensional data from end to end is described. Assume the end-to-end encoding / decoding network uses a causal convolutional network, and the effective input bitstream length required for each inference at the decoding end is B. opFor ease of discussion, the reconstructed encoding vector of the input bitstream B corresponding to a single decoding inference can be further represented as [B2, B1, B0], where B0 represents the reconstructed encoding vector of the current bitstream to be decoded with length B0, B1 represents the reconstructed encoding vector of the decoded bitstream with length B1, and B2 represents other padding data (such as zero padding) with length B2.

[0217] Optionally, B0 can be the reconstructed coding vector length corresponding to the frame length (i.e., one frame of data) of the encoding / decoding system. For a given end-to-end encoding / decoding system, B0 can be assumed to be a fixed value.

[0218] If the input data to the decoding network contains only the reconstructed encoding vector of the current bitstream B0 to be decoded, then the decoding network will pad the left side of the input reconstructed encoding vector with (B op -B0) zeros (or other non-real data) are used to ensure that the network can perform operations normally. However, this will introduce a lot of zeros into the calculation process, which will lead to a deterioration in the quality of the reconstructed signal after decoding.

[0219] In this embodiment, a certain amount of reconstructed encoding vector from the decoded bitstream B1 can be padded to the left of the reconstructed encoding vector of the current bitstream B0 to be decoded, using real data as much as possible in the calculation to ensure the quality of the reconstructed signal after decoding. Assume that a reconstructed encoding vector of length B1 from the decoded bitstream is used for padding. In this case, the padding data corresponds to three scenarios.

[0220] The first case (corresponding to) Figure 10A If B1 + B0 < B op Then, the convolutional network will continue to pad the input reconstructed encoding vector with zeros (adding a certain length of padding data, as shown by the arrow in Case 1) to ensure the effective input length required for decoding inference. This is equivalent to the input reconstructed encoding vector being [B2, B1, B0]. Thus, the output length of the decoding network includes the position from the rightmost point to the position corresponding to the arrow in Case 1, including the reconstructed signals corresponding to B0, B1, and B2. In other words, a portion of the features corresponding to the bitstream to be decoded will be affected by the zero-padding operation of the convolutional network on the input reconstructed encoding vector, thereby affecting the quality of the reconstructed signal.

[0221] The second scenario (corresponding to) Figure 10B If Bi + B0 > B opThe convolutional network will then use the reconstructed encoding vector of the decoded bitstream as padding data, and this inference will use the reconstructed encoding vectors of all bitstreams (B1+B0) for inference. The input reconstructed encoding vector is then [B1, B0]. Thus, the output length of the decoding network includes the position from the rightmost point to the position corresponding to the arrow in case 2, including the reconstructed signal corresponding to B0 and the reconstructed signal corresponding to B1. Therefore, part of the reconstructed signal corresponds to the bitstream to be decoded B0, and part corresponds to the decoded bitstream B1. However, this introduces additional computation to calculate (B1+B0-B0). op This resulted in a waste of computing resources.

[0222] The third scenario (corresponding to) Figure 10C If B1 + B0 = B op The convolutional network will then use the reconstructed encoding vector of the decoded bitstream as padding data, while the inference will still use the reconstructed encoding vectors of all bitstreams (B1+B0) for inference. The input reconstructed encoding vector is then [B1, B0]. However, since B1+B0 = B... op Therefore, the output length of the decoding network includes only the reconstructed signal corresponding to B0, from the rightmost position to the position indicated by the arrow in Case 3. Thus, the generated reconstructed signal corresponds entirely to the bitstream B0 to be decoded, without introducing additional computation, and no additional zero-padding is required on the input reconstructed encoding vector.

[0223] Therefore, in this embodiment, the reconstructed encoding vector corresponding to the encoded bitstream is filled with the reconstructed encoding vector of the bitstream to be decoded, so that the length of the input data after filling is the effective input length B required by the decoding network for inference at the decoding end. op It can precisely control the length of the reconstructed encoding vector of the decoded bitstream that fills the input reconstructed encoding vector during decoding, thereby ensuring that the output reconstructed signal of the decoding network in one inference depends only on the reconstructed encoding vector of the current bitstream to be decoded, while ensuring that no non-real data is introduced into the calculation process.

[0224] In some embodiments, for the third case described above, when B0 is the length of the reconstructed coding vector corresponding to the frame length of the encoding and decoding system, the reconstructed signal generated in case 3 corresponds to a complete frame of audio data, so that the reconstructed signal of the current frame generated by the decoding network depends only on the reconstructed coding vector of the current bitstream to be decoded, thus achieving the effect of separable frame-level reconstructed signals.

[0225] It should be noted that, for a given decoding network, the corresponding effective input length B can be calculated based on the decoding network architecture. opIn some embodiments, for a decoding network (the network architecture may be known or unknown), the effective input length of the decoding network can be determined by testing. For example, reconstructed encoding vectors of different sizes can be input into the decoding network. When the length of the input reconstructed encoding vector is less than the effective data length B, the effective input length can be determined. op The system reported an error when the length of the input reconstructed encoding vector was greater than or equal to the length of the effective data B. op The system does not report an error, thus determining the effective input length of the decoding network.

[0226] In some embodiments, the length of the padding data is determined based on the effective input length of the decoding network and the length of the reconstructed coding vector corresponding to a frame of data.

[0227] In some embodiments, it is possible to record This indicates the number of audio frames contained in the bitstream. Where B0 represents the reconstructed encoding vector of the bitstream to be decoded, B... f This represents the length of the reconstructed coding vector corresponding to a frame of the bitstream.

[0228] In some embodiments, the length of the padding data is the difference between the effective input length and the length of the reconstructed coding vector corresponding to one frame of data. For example, the length of the padding data can be expressed by the following formula:

[0229] B1 = B op -B f

[0230] Where B1 is the length of the padding data, B op This represents the effective input length of the decoding network.

[0231] After determining the length of the padding data, the padding data of that length can be combined with the reconstructed coding vector to obtain the combined reconstructed coding vector. For example, the combined reconstructed coding vector can be represented as B, B = [B1, B0].

[0232] In some embodiments, when the decoding network includes a causal convolutional network, padding data can be padded before the reconstructed coding vector of the bitstream to be decoded to obtain a combined reconstructed coding vector, such that the padding data and the first reconstructed coding vector in the reconstructed coding vector of the bitstream to be decoded are combined as the input to the decoding network. For example, the positions of the padding data and the reconstructed coding vector can be referenced... Figures 10A to 10C The positional relationship between the reconstructed coding vector padding (B1) of the decoded bitstream and the reconstructed coding vector (B0) of the bitstream to be decoded is determined, and the padding data is filled to the left of the reconstructed coding vector of the bitstream to be decoded.

[0233] 940. The reconstructed coding vector corresponding to the effective input length in the combined reconstructed coding vector is input into the decoding network. The decoding network performs a decoding operation on the input reconstructed coding vector and outputs a reconstructed signal. The reconstructed signal is related to the reconstructed coding vector of the bitstream to be decoded in the input reconstructed coding vector and is not related to the reconstructed coding vector of the decoded bitstream in the input reconstructed coding vector. The decoding network does not fill the input reconstructed coding vector with non-real data.

[0234] Specifically, the decoding network performs decoding operations on the combined reconstructed encoding vector, outputting the reconstructed signal of the bitstream. The length of the input reconstructed encoding vector to the decoding network each time is the effective input length. This ensures that the reconstructed signal output by the decoding network at each inference step is related to the reconstructed encoding vector of the bitstream to be decoded in the input reconstructed encoding vector of the current inference, but not related to the reconstructed encoding vector of the already decoded bitstream in the input audio data of the current inference. Furthermore, the decoding network does not pad the input reconstructed encoding vector with non-real data during each inference step. This helps ensure that the output reconstructed signal of the decoding network at each inference step depends only on the reconstructed encoding vector of the current bitstream to be decoded, and that no non-real data is introduced into the computation during each inference process. For example, the non-real data is zero.

[0235] In some embodiments, the length of the reconstructed coding vector of the bitstream to be encoded in the input reconstructed coding vector of the decoding network is equal to the length of the reconstructed coding vector corresponding to one frame of data. Here, the input reconstructed coding vector is the feature vector of the decoding network's input for one inference, and the length of the input reconstructed coding vector is the aforementioned effective input length.

[0236] Specifically, traditional audio codecs typically process input data of a specific length (e.g., frame length), such as a 20ms frame. If the input data length is less than 20ms, it needs to be padded with zeros until the data length reaches the preset frame length. Conversely, if the input data length is greater than 20ms, it is sliced ​​so that each slice, except the last one, has the preset frame length, and the last frame is padded with zeros as needed. Each frame then serves as a basic encoding / decoding unit, enabling frame-by-frame bitstream separability.

[0237] Deep learning-based audio decoders support variable-length input data. If the data for a single inference is larger than the receptive field of the neural network model, then that inference will use all the data, resulting in a reconstructed signal that cannot be directly mapped to the input data. However, in real-time communication scenarios, the bitstream to be decoded is newly received data. If the amount of padding data that needs to be accurately calculated cannot be determined, the first part of the bitstream to be decoded will be used for the current decoding, while the remaining part will be used for the next decoding, thus losing the ability to reconstruct and separate the signal frame by frame.

[0238] In this embodiment, by configuring the length of the reconstructed encoding vector of the input bitstream to the decoding network to be equal to the length of the reconstructed encoding vector corresponding to one frame of data, the decoding network can output a frame-level reconstructed signal each time it infers. Therefore, this embodiment can achieve the effect of frame-by-frame bitstream separability.

[0239] In some embodiments, see Figure 11 The output reconstructed signal of the decoding network can be obtained according to the following step 941.

[0240] 941. Input the reconstructed coding vector corresponding to the first valid input length in the combined reconstructed coding vector into the decoding network, and use the decoding network to perform decoding operation on the input reconstructed coding vector to output the first reconstructed signal of the first frame bitstream.

[0241] In this context, the length of the combined reconstructed coding vector is greater than or equal to the effective input length. As an example, when the bitstream to be decoded includes a bitstream corresponding to one frame of data (i.e., one frame of bitstream), the length of the combined reconstructed coding vector is equal to the effective input length. In this case, the coding vector corresponding to the first effective input length in the combined reconstructed coding vector is the combined reconstructed coding vector itself. As another example, when the bitstream to be decoded includes a bitstream corresponding to multiple frames of data (i.e., multiple frame of bitstream), the length of the combined reconstructed coding vector is greater than the effective input length. In this case, the coding vector corresponding to the first effective input length in the combined reconstructed coding vector is the reconstructed coding vector of the padding data and the first frame of bitstream in the bitstream to be decoded.

[0242] The encoding vector corresponding to the first valid input length of the combined reconstructed encoding vector is the encoding vector of the first batch of inputs to the decoding network in the combined reconstructed encoding vector. The data length of each batch input to the decoding network is the valid input length.

[0243] In the first possible scenario, when the length of the combined reconstructed coding vector equals the effective input length, the entire combined reconstructed coding vector is used as the coding vector corresponding to the first effective input length and input into the decoding network. For example... Figure 12AAs shown, the reconstructed encoding vector B1 (i.e., padding data) of the decoded bitstream can be combined with the reconstructed encoding vector B0 of the bitstream to be decoded, and then concatenated to the left of the reconstructed encoding vector B0 of the bitstream to be decoded before being input into the decoding network. The length of the combined reconstructed encoding vector B1 and the reconstructed encoding vector B0 of the bitstream to be decoded is equal to the effective input length. Correspondingly, the decoding network performs a decoding operation on the input encoding vector and outputs a reconstructed signal of one frame of the bitstream. This reconstructed signal includes the reconstructed signal corresponding to the bitstream to be decoded.

[0244] The second possible scenario is that when the length of the combined reconstructed coding vector is greater than the effective input length, the coding vector corresponding to the first effective input length in the combined reconstructed coding vector is input into the decoding network. For example... Figure 12B As shown, the bitstream to be decoded includes multiple frames. In this case, the reconstructed encoding vector of the already decoded bitstream, padded with B1 (i.e., padding data), can be combined with the reconstructed encoding vector of the first frame of the bitstream to be decoded. This combination is then appended to the left side of the reconstructed encoding vector of the first frame and input into the decoding network. The length of the combination of the reconstructed encoding vector of the already decoded bitstream, padded with B1, and the reconstructed encoding vector of the first frame is equal to the effective input length. Correspondingly, the decoding network performs a decoding operation on the input encoding vector (the reconstructed encoding vector corresponding to the first effective input length) and outputs the reconstructed signal of the first frame. This reconstructed signal includes the reconstructed signal of the first frame.

[0245] Optionally, for the second case, after obtaining the reconstructed signal of the first frame bitstream, the following step 432 can be performed to continue decoding other bitstreams to be decoded.

[0246] 942. Using the length of the reconstructed coding vector corresponding to a frame of data as the step length, the reconstructed coding vector corresponding to the i-th effective input length in the combined reconstructed coding vector is input into the decoding network. The decoding network is used to perform decoding operations on the input reconstructed coding vector and output the second reconstructed signal corresponding to the i-th frame bitstream, where i is a positive integer greater than 1.

[0247] For example, in the second possible case in step 941 above, the remaining bitstream in the bitstream to be decoded can be encoded according to step 942.

[0248] Figure 12C This diagram illustrates the encoding of the second frame bitstream. Figure 12CAs shown, after decoding the first frame bitstream to obtain its reconstructed signal, the effective input length window is shifted to the right with a step size of the reconstructed coding vector length corresponding to one frame of data to obtain the second effective input length coding vector in the combined reconstructed coding vector. This includes the reconstructed coding vector of the second frame bitstream, the reconstructed coding vector of the second frame bitstream, and the reconstructed coding vector of a partially decoded bitstream filling B1. At this time, the first frame bitstream in the second effective input length is the decoded bitstream, and audio frame 1 is the bitstream to be encoded. The reconstructed coding vector of the second effective input length is input into the decoding network, which performs a decoding operation on the input reconstructed coding vector (the reconstructed coding vector corresponding to the second effective input length). In this convolution operation, the reconstructed coding vector of the first frame bitstream and the reconstructed coding vector of a partially decoded bitstream are used as padding data for the reconstructed coding vector of the second frame bitstream, and the reconstructed signal of the second frame bitstream is output. This reconstructed signal includes the reconstructed signal corresponding to the second frame bitstream.

[0249] The encoding process for the third frame is similar to that for the first or second frame, as described above, and will not be repeated here. Optionally, the decoding process for the subsequent frames is similar to that for the first or second frame.

[0250] In some embodiments, when the bitstream to be decoded includes (M+1) frame bitstreams, after obtaining the reconstructed coding vector of the bitstream to be decoded and combining it with padding data to obtain the combined reconstructed coding vector, it can be processed according to the above. Figures 12A to 12B The method involves dividing the combined reconstructed coding vector into (M+1) portions and inputting them sequentially into the decoding network, resulting in reconstructed signals corresponding to (M+1) frame bitstreams. The reconstructed signals of the last (M+1) frame bitstreams output by the decoding network correspond one-to-one with the input (M+1) frame bitstreams. Therefore, the embodiments of this application enable the separation of frame-by-frame reconstructed signals during decoding.

[0251] Optionally, when the first bit corresponding to the (M+1)th frame data in the bitstream is 0, indicating that the frame data corresponds to an audio frame with an incomplete frame length, the decoding end can further obtain the second bit in the bitstream and determine the amount of audio data for the incomplete frame length based on the second bit, thereby determining the effective data length in the reconstructed signal. For example, this effective data length corresponds to the audio data to be encoded and is unrelated to the encoded audio data padded during encoding. Therefore, by carrying second indication information in the bitstream, this embodiment of the application can further indicate the amount of audio data for the incomplete frame length. Based on this, the decoding end can determine the amount of audio data for the incomplete frame length according to the second indication information, thereby facilitating accurate decoding.

[0252] Therefore, this embodiment of the application determines the length of the padding data based on the effective input length corresponding to the decoding network, thereby achieving precise control over the reconstructed encoding vector of the decoded bitstream during decoding. This ensures that the output reconstructed signal of the decoding network in one inference depends only on the current reconstructed encoding vector, which helps avoid introducing additional computations, reduces waste of computing resources, and thus improves decoding efficiency. Simultaneously, precise control over the reconstructed encoding vector of the decoded bitstream during decoding prevents the decoding network from padding the input reconstructed encoding vector with non-real data, avoiding the influence of non-real data padding in the decoding network on the reconstructed signal, thereby improving the quality of the reconstructed signal.

[0253] The specific embodiments of this application have been described in detail above with reference to the accompanying drawings. However, this application is not limited to the specific details of the above embodiments. Within the scope of the technical concept of this application, various simple modifications can be made to the technical solutions of this application, and these simple modifications all fall within the protection scope of this application. For example, the various specific technical features described in the above embodiments can be combined in any suitable manner without contradiction. To avoid unnecessary repetition, this application will not describe the various possible combinations separately. Furthermore, various different embodiments of this application can also be arbitrarily combined, as long as they do not violate the spirit of this application, they should also be considered as the content disclosed in this application.

[0254] It should also be understood that, in the various method embodiments of this application, the sequence numbers of the above processes do not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. It should be understood that these sequence numbers can be interchanged where appropriate so that the embodiments of this application described can be implemented in a sequence other than those illustrated or described.

[0255] The method embodiments of this application have been described in detail above. The following description, in conjunction with... Figures 13 to 15 The following describes in detail the device embodiments of this application.

[0256] Figure 13 This is a schematic block diagram of the audio decoding device 10 according to an embodiment of this application. Figure 13 As shown, the audio decoding device 10 may include a parsing unit 11, an inverse quantization unit 12, a filling unit 13, and a decoding unit 14.

[0257] Parsing unit 11 is used to parse the code stream to be decoded and obtain the quantization result corresponding to the code stream to be decoded;

[0258] The dequantization unit 12 is used to dequantize the quantization result to obtain the reconstructed encoding vector of the code stream to be decoded;

[0259] The padding unit 13 is used to obtain a combined reconstructed coding vector based on the reconstructed coding vector and the padding data; wherein the length of the padding data is determined according to the effective input length corresponding to the decoding network, and the padding data includes the reconstructed coding vector of the decoded bitstream;

[0260] Decoding unit 14 is used to input the reconstructed encoding vector corresponding to the effective input length in the combined reconstructed encoding vector into the decoding network, and use the decoding network to perform a decoding operation on the input reconstructed encoding vector to output a reconstructed signal; wherein, the reconstructed signal is related to the reconstructed encoding vector of the bitstream to be decoded in the input reconstructed encoding vector and is not related to the reconstructed encoding vector of the decoded bitstream in the input reconstructed encoding vector, and the decoding network does not fill the input reconstructed encoding vector with non-real data.

[0261] In some embodiments, the length of the reconstructed coding vector of the bitstream to be decoded in the input reconstructed coding vector is equal to the length of the reconstructed coding vector corresponding to one frame of data.

[0262] In some embodiments, the length of the padding data is determined based on the effective input length and the length of the reconstructed coding vector corresponding to a frame of data.

[0263] In some embodiments, the decoding unit 14 is specifically used for:

[0264] The reconstruction coding vector corresponding to the first valid input length in the combined reconstruction coding vector is input into the decoding network, and the decoding network is used to perform a decoding operation on the input reconstruction coding vector to output the first reconstruction signal of the first frame bitstream.

[0265] In some embodiments, the decoding unit 14 is further configured to:

[0266] Using the length of the reconstructed coding vector corresponding to a frame of data as the step length, the reconstructed coding vector corresponding to the i-th effective input length in the combined reconstructed coding vector is input into the decoding network. The decoding network is used to perform a decoding operation on the input reconstructed coding vector, and the second reconstructed signal of the i-th frame bitstream is output, where i is a positive integer greater than 1.

[0267] In some embodiments, the bitstream to be decoded further includes first indication information for indicating whether the frame-by-frame bitstream corresponds to audio data of a complete frame length.

[0268] In some embodiments, the bitstream to be decoded further includes second indication information, used to indicate the amount of audio data corresponding to the incomplete frame length in the bitstream.

[0269] In some embodiments, the decoding network includes a causal convolutional network.

[0270] It should be understood that the device embodiments and method embodiments can correspond to each other, and similar descriptions can be referred to the method embodiments. To avoid repetition, further details will not be provided here. Specifically, Figure 13 The audio decoding device 10 shown can execute the above method embodiments, and the aforementioned and other operations and / or functions of each module in the audio decoding device 10 are respectively for implementing the corresponding process in the above method 400. For the sake of brevity, they will not be described in detail here.

[0271] Figure 14 This is a schematic block diagram of an audio encoding device 20 according to an embodiment of this application. Figure 14 As shown, the audio encoding device 20 may include an acquisition unit 21, a filling unit 22, an encoding unit 23, and a quantization unit 24.

[0272] Acquisition unit 21 is used to acquire the first audio data to be encoded;

[0273] The padding unit 22 is used to obtain combined audio data based on the first audio data and the padding data; wherein the length of the padding data is determined according to the effective input length corresponding to the encoding network, and the padding data includes the encoded audio data;

[0274] Encoding unit 23 is used to input the second audio data corresponding to the effective input length in the combined audio data into the encoding network, use the encoding network to encode the second audio data, and output an encoding vector; wherein, the encoding vector is related to the audio data to be encoded in the second audio data and is not related to the encoded audio data in the second audio data, and the encoding network does not fill the second audio data with non-real data;

[0275] The quantization unit 24 is used to quantize the encoding vector to obtain a quantization result, and to obtain the bitstream of the first audio data based on the quantization result.

[0276] In some embodiments, the audio data to be encoded in the second audio data is less than or equal to the amount of data in one frame.

[0277] In some embodiments, the length of the padding data is determined based on the effective input length and the data margin that is not an integer multiple of the frame length of the first audio data.

[0278] In some embodiments, if the data margin is non-zero, the length of the padding data is the difference between the effective input length and the data margin.

[0279] In some embodiments, if the data margin is zero, the length of the padding data is the difference between the effective input length and the amount of data corresponding to one frame of data.

[0280] In some embodiments, the encoding unit 23 is specifically used for:

[0281] The audio data corresponding to the first valid input length in the combined audio data is input into the encoding network, and the encoding network is used to encode the input audio data to output the first encoding vector corresponding to the first frame data.

[0282] In some embodiments, the encoding unit 23 is further configured to:

[0283] Using the data size corresponding to one frame as the step length, the audio data corresponding to the i-th valid input length in the combined audio data is input into the encoding network. The encoding network is used to encode the input audio data and output the second encoding vector corresponding to the i-th frame, where i is a positive integer greater than 1.

[0284] In some embodiments, the bitstream further includes first indication information for indicating whether the frame-by-frame bitstream corresponds to audio data of a complete frame length.

[0285] In some embodiments, the bitstream further includes second indication information for indicating the amount of audio data corresponding to incomplete frame lengths in the bitstream.

[0286] In some embodiments, the first audio data includes M frames of data and data corresponding to an incomplete frame length; M is a positive integer greater than or equal to 1.

[0287] In some embodiments, the encoding network includes a causal convolutional network.

[0288] It should be understood that the device embodiments and method embodiments can correspond to each other, and similar descriptions can be referred to the method embodiments. To avoid repetition, further details will not be provided here. Specifically, Figure 14 The audio encoding device 20 shown can execute the above method embodiments, and the aforementioned and other operations and / or functions of each module in the audio encoding device 20 are respectively to implement the corresponding process in the above method 900. For the sake of brevity, they will not be described in detail here.

[0289] The apparatus of this application embodiment has been described above from the perspective of functional modules in conjunction with the accompanying drawings. It should be understood that this functional module can be implemented in hardware, in software instructions, or in a combination of hardware and software modules. Specifically, the steps of the method embodiments in this application can be completed by integrated logic circuits in the processor's hardware and / or by software instructions. The steps of the method disclosed in this application embodiment can be directly embodied as being executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. Optionally, the software module can reside in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps in the above method embodiments.

[0290] Figure 15 This is a schematic block diagram of the electronic device provided in the embodiments of this application.

[0291] like Figure 15 As shown, the electronic device 30 may include:

[0292] The system includes a memory 33 for storing a computer program 34 and a processor 32 for transferring the program code 34 to the processor 32. In other words, the processor 32 can retrieve and run the computer program 34 from the memory 33 to implement the methods described in the embodiments of this application.

[0293] For example, the processor 32 can be used to execute the steps in the methods 400 or 900 described above according to the instructions in the computer program 34.

[0294] In some embodiments of this application, the processor 32 may include, but is not limited to:

[0295] General-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0296] In some embodiments of this application, the memory 33 includes, but is not limited to:

[0297] Volatile memory and / or non-volatile memory. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0298] In some embodiments of this application, the computer program 34 may be divided into one or more units, which are stored in the memory 33 and executed by the processor 32 to perform the method provided in this application. The one or more units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program 34 in the electronic device 30.

[0299] Optional, such as Figure 15 As shown, the electronic device 30 may further include:

[0300] Transceiver 33, which can be connected to processor 32 or memory 33.

[0301] The processor 32 can control the transceiver 33 to communicate with other devices; specifically, it can send information or data to other devices or receive information or data sent by other devices. The transceiver 33 may include a transmitter and a receiver. The transceiver 33 may further include antennas, and the number of antennas may be one or more. It should be understood that the various components in this electronic device are connected through a bus system, which includes, in addition to a data bus, a power bus, a control bus, and a status signal bus.

[0302] This application also provides a computer storage medium storing a computer program thereon, which, when executed by a computer, enables the computer to perform the methods of the above-described method embodiments. Alternatively, embodiments of this application also provide a computer program product containing instructions that, when executed by a computer, cause the computer to perform the methods of the above-described method embodiments.

[0303] When implemented using software, it can be implemented entirely or partially as a computer program product. This computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital video disc (DVD)), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0304] It is understood that in the specific implementation of this application, when the above embodiments of this application are applied to specific products or technologies and involve user information and other related data, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0305] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0306] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0307] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. For example, the functional modules in the various embodiments of this application may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0308] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An audio decoding method, characterized in that, include: Parse the bitstream to be decoded to obtain the quantization result corresponding to the bitstream to be decoded; The quantization result is dequantized to obtain the reconstructed encoding vector of the bitstream to be decoded; Based on the reconstructed coding vector and the padding data, a combined reconstructed coding vector is obtained; wherein, the length of the padding data is determined according to the effective input length corresponding to the decoding network, and the padding data includes the reconstructed coding vector of the decoded bitstream; The reconstructed encoding vector corresponding to the effective input length in the combined reconstructed encoding vector is input into the decoding network, and the decoding network is used to perform a decoding operation on the input reconstructed encoding vector to output a reconstructed signal; wherein, the reconstructed signal is related to the reconstructed encoding vector of the bitstream to be decoded in the input reconstructed encoding vector and is not related to the reconstructed encoding vector of the decoded bitstream in the input reconstructed encoding vector, and the decoding network does not fill the input reconstructed encoding vector with non-real data.

2. The method according to claim 1, characterized in that, The length of the reconstructed coding vector of the bitstream to be decoded in the input reconstructed coding vector is equal to the length of the reconstructed coding vector corresponding to one frame of data.

3. The method according to claim 2, characterized in that, The length of the padding data is determined based on the effective input length and the length of the reconstructed coding vector corresponding to one frame of data.

4. The method according to claim 2, characterized in that, The step of inputting the encoding vector corresponding to the effective input length in the combined reconstructed encoding vector into the decoding network, and using the decoding network to perform a decoding operation on the input reconstructed encoding vector to output a reconstructed signal includes: The reconstruction coding vector corresponding to the first valid input length in the combined reconstruction coding vector is input into the decoding network, and the decoding network is used to perform a decoding operation on the input reconstruction coding vector to output the first reconstruction signal of the first frame bitstream.

5. The method according to claim 4, characterized in that, Also includes: Using the length of the reconstructed coding vector corresponding to a frame of data as the step length, the reconstructed coding vector corresponding to the i-th effective input length in the combined reconstructed coding vector is input into the decoding network. The decoding network is used to perform a decoding operation on the input reconstructed coding vector, and the second reconstructed signal of the i-th frame bitstream is output, where i is a positive integer greater than 1.

6. The method according to claim 4, characterized in that, The bitstream to be decoded includes first indication information, used to indicate whether the frame-by-frame bitstream corresponds to audio data of a complete frame length.

7. The method according to claim 6, characterized in that, The bitstream to be decoded also includes second indication information, used to indicate the amount of audio data corresponding to the incomplete frame length in the bitstream.

8. The method according to any one of claims 1-7, characterized in that, The decoding network includes a causal convolutional network.

9. An audio encoding method, characterized in that, include: Obtain the first audio data to be encoded; Based on the first audio data and the padding data, combined audio data is obtained; wherein, the length of the padding data is determined according to the effective input length corresponding to the encoding network, and the padding data includes the encoded audio data; The second audio data corresponding to the effective input length in the combined audio data is input into the encoding network, and the encoding network is used to encode the second audio data to output an encoding vector; wherein, the encoding vector is related to the audio data to be encoded in the second audio data but is not related to the encoded audio data in the second audio data, and the encoding network does not fill the second audio data with non-real data; The encoded vector is quantized to obtain a quantization result, and the bitstream of the first audio data is obtained based on the quantization result.

10. The method according to claim 9, characterized in that, The audio data to be encoded in the second audio data is less than or equal to the data volume of one frame.

11. The method according to claim 10, characterized in that, The length of the padding data is determined based on the effective input length and the data margin that is not an integer multiple of the frame length of the first audio data. If the data margin is non-zero, the length of the padding data is the difference between the effective input length and the data margin; if the data margin is zero, the length of the padding data is the difference between the effective input length and the data volume corresponding to one frame of data.

12. The method according to claim 10, characterized in that, The step of inputting the audio data corresponding to the effective input length in the combined audio data into the encoding network, and using the encoding network to encode the input audio data to output an encoding vector includes: The audio data corresponding to the first valid input length in the combined audio data is input into the encoding network, and the encoding network is used to encode the input audio data to output the first encoding vector corresponding to the first frame data.

13. The method according to claim 12, characterized in that, Also includes: Using the data size corresponding to one frame as the step length, the audio data corresponding to the i-th valid input length in the combined audio data is input into the encoding network. The encoding network is used to encode the input audio data and output the second encoding vector corresponding to the i-th frame, where i is a positive integer greater than 1.

14. The method according to claim 12, characterized in that, The bitstream includes first indication information, used to indicate whether the frame-by-frame bitstream corresponds to audio data of a complete frame length.

15. The method according to claim 14, characterized in that, The bitstream also includes second indication information, used to indicate the amount of audio data corresponding to incomplete frame lengths in the bitstream.

16. An audio decoding device, characterized in that, include: The parsing unit is used to parse the bitstream to be decoded and obtain the quantization result corresponding to the bitstream to be decoded. The dequantization unit is used to dequantize the quantization result to obtain the reconstructed encoding vector of the bitstream to be decoded; A padding unit is used to obtain a combined reconstructed coding vector based on the reconstructed coding vector and padding data; wherein the length of the padding data is determined according to the effective input length corresponding to the decoding network, and the padding data includes the reconstructed coding vector of the decoded bitstream; The decoding unit is used to input the reconstructed coding vector corresponding to the effective input length in the combined reconstructed coding vector into the decoding network, and use the decoding network to perform a decoding operation on the input reconstructed coding vector to output a reconstructed signal; wherein, the reconstructed signal is related to the reconstructed coding vector of the bitstream to be decoded in the input reconstructed coding vector and is not related to the reconstructed coding vector of the decoded bitstream in the input reconstructed coding vector, and the decoding network does not fill the input reconstructed coding vector with non-real data.

17. An audio encoding device, characterized in that, include: The acquisition unit is used to acquire the first audio data to be encoded. A padding unit is used to obtain combined audio data based on the first audio data and the padding data; wherein the length of the padding data is determined according to the effective input length corresponding to the encoding network, and the padding data includes the encoded audio data; An encoding unit is used to input the second audio data corresponding to the effective input length in the combined audio data into the encoding network, use the encoding network to encode the second audio data, and output an encoding vector; wherein, the encoding vector is related to the audio data to be encoded in the second audio data but is not related to the encoded audio data in the second audio data, and the encoding network does not fill the second audio data with non-real data; A quantization unit is used to quantize the encoding vector to obtain a quantization result, and to obtain the bitstream of the first audio data based on the quantization result.

18. An electronic device, characterized in that, The method includes a processor and a memory, wherein the memory stores instructions, and when the processor executes the instructions, it causes the processor to perform the method according to any one of claims 1-15.

19. A computer storage medium, characterized in that, Used for storing computer programs, said computer programs including methods for performing any one of claims 1-15.

20. A computer program product, characterized in that, It includes computer program code that, when executed by an electronic device, causes the electronic device to perform the method of any one of claims 1-15.

Images