Coding method, decoding method, device and equipment

Abstract

The embodiment of the invention provides a coding method, a decoding method, a device and equipment, and the coding method comprises the steps: carrying out the processing of a first signal through one or more neural networks, obtaining a feature quantity corresponding to each neural network, carrying out the coding of at least part of the feature quantity, and obtaining a feature code stream; and determining a difference characteristic quantity between the characteristic quantities corresponding to different neural networks, and encoding the difference characteristic quantity to obtain a difference characteristic code stream.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of communication technology, and specifically to an encoding method, a decoding method, an apparatus and a device. Background Art

[0002] In the current audio encoding and decoding field, many systems rely on fixed-architecture neural networks, which are usually designed and trained for specific tasks or application scenarios. With the continuous advancement of technology, new codec solutions and neural network frameworks continue to emerge. However, traditional codec solutions often lack flexibility and cannot effectively adapt to these changes. For example, the new version of the codec file may not be correctly parsed in the old decoding environment, thus affecting the overall codec efficiency. Summary of the invention

[0003] The embodiments of the present application provide an encoding method, a decoding method, an apparatus and a device to solve the problem of how to improve the encoding and decoding efficiency.

[0004] In a first aspect, a coding method is provided, which is applied to an encoder, wherein the encoder includes one or more neural networks, including:

[0005] Processing the first signal through the one or more neural networks to obtain a feature quantity corresponding to each neural network, and encoding at least part of the feature quantity to obtain a feature code stream;

[0006] The difference feature quantities between the feature quantities corresponding to different neural networks are determined, and the difference feature quantities are encoded to obtain a difference feature code stream.

[0007] Optionally, each of the one or more neural networks included in the encoder corresponds to at least one version of the encoder.

[0008] Optionally, determining the difference feature quantities between feature quantities corresponding to different neural networks includes:

[0009] Determine the difference between the feature quantity corresponding to the neural network matching the current version of the encoder and the feature quantity corresponding to other neural networks.

[0010] Optionally, the method further includes:

[0011] The feature code stream and / or the difference feature code stream is sent to a decoder, wherein the difference feature code stream is used to match the feature code stream to a feature quantity corresponding to a neural network of the decoder, and the neural network of the decoder is used to output the first signal.

[0012] Optionally, sending the feature code stream and / or the difference feature code stream to a decoder includes:

[0013] The first code stream and / or the second code stream is sent to the decoder, the first code stream includes first information and second information, the first information is used to indicate that the first code stream is the feature code stream, the second information is used to indicate the neural network corresponding to the feature code stream, the second code stream includes third information and fourth information, the third information is used to indicate that the second code stream is the difference feature code stream, and the fourth information is used to indicate the neural network corresponding to the difference feature code stream.

[0014] In a second aspect, a decoding method is provided, which is applied to a decoder, comprising:

[0015] Receiving a feature code stream sent by an encoder, and decoding to obtain a feature quantity corresponding to a neural network of the encoder, wherein the feature quantity is obtained by processing a first signal based on one or more neural networks of the encoder;

[0016] According to the neural network of the decoder and the feature code stream, a target difference feature code stream is obtained, and a target difference feature quantity is obtained by decoding, wherein the target difference feature quantity is used to represent the difference between the feature quantity corresponding to the neural network of the encoder and the feature quantity corresponding to the neural network of the decoder;

[0017] Determining the feature quantity corresponding to the neural network of the decoder according to the feature quantity corresponding to the neural network of the encoder and the target difference feature quantity;

[0018] The first signal is generated by the feature quantity corresponding to the neural network of the decoder.

[0019] Optionally, receive the feature code stream sent by the encoder, including:

[0020] Receive a first code stream sent by the encoder, the first code stream includes first information and second information, the first information is used to indicate that the first code stream is the feature code stream, and the second information is used to indicate the neural network of the encoder corresponding to the feature code stream.

[0021] Optionally, obtaining a target difference feature code stream according to the neural network of the decoder and the feature code stream includes:

[0022] Receive a second code stream sent by the encoder, the second code stream includes third information and fourth information, the third information is used to indicate that the second code stream is a difference feature code stream corresponding to difference feature quantities of different neural networks of the encoder, and the fourth information is used to indicate the neural network of the encoder corresponding to the difference feature code stream;

[0023] According to the neural network of the decoder and the feature code stream sent by the encoder, the target difference feature code stream is selected from the difference feature code stream sent by the encoder.

[0024] Optionally, obtaining a target difference feature code stream according to the neural network of the decoder and the feature code stream includes:

[0025] Sending the feature code stream and fifth information to a cloud transcoder, wherein the fifth information is used to indicate a neural network of the decoder;

[0026] The target difference feature code stream is received, which is sent by the cloud transcoder according to the feature code stream and the fifth information.

[0027] In a third aspect, there is provided an encoding device, applied to an encoder, the encoder comprising one or more neural networks, the device comprising: a first receiving unit and a first processing unit;

[0028] The first processing unit is used to process the first signal through one or more neural networks to obtain a feature quantity corresponding to each neural network, and encode at least part of the feature quantity to obtain a feature code stream;

[0029] The first processing unit is further used to determine the difference feature quantities between the feature quantities corresponding to different neural networks, and encode the difference feature quantities to obtain a difference feature code stream.

[0030] In a fourth aspect, a decoding device is provided, which is applied to a decoder, and the device includes a second receiving unit and a second processing unit;

[0031] The second receiving unit is used to receive a feature code stream sent by an encoder, and decode to obtain a feature quantity corresponding to a neural network of the encoder, wherein the feature quantity is obtained by processing the first signal based on one or more neural networks of the encoder;

[0032] The second processing unit is used to obtain a target difference feature code stream according to the neural network of the decoder and the feature code stream, and decode to obtain a target difference feature quantity, wherein the target difference feature quantity is used to represent the difference between the feature quantity corresponding to the neural network of the encoder and the feature quantity corresponding to the neural network of the decoder;

[0033] The second processing unit is further used to determine the feature quantity corresponding to the neural network of the decoder according to the feature quantity corresponding to the neural network of the encoder and the target difference feature quantity;

[0034] The second processing unit is further used to generate the first signal through the feature quantity corresponding to the neural network of the decoder.

[0035] In a fifth aspect, a communication device is provided, comprising a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the method described in the first aspect or the second aspect.

[0036] In a sixth aspect, a readable storage medium is provided, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method described in the first aspect or the second aspect are implemented.

[0037] In a seventh aspect, a computer program product is provided, comprising computer instructions, which, when executed by a processor, implement the steps of the method described in the first aspect or the second aspect.

[0038] In this application, in view of the situation that the neural network architecture and neural network parameters of the encoder and decoder are updated quickly, a method combining feature code stream and network difference feature code stream is adopted. Even if the neural network structure and parameters of the decoder and encoder are different, this scheme can still complete the analysis of the voice or audio code stream and reconstruct the complete sound signal, ensuring that after the neural network is updated, the old codec or encoded file can still be compatible with the new system. At the same time, this encoding and decoding method combining feature code stream and difference feature significantly improves the efficiency of encoding and decoding because it can adapt to a variety of neural network architectures and changes in neural network parameters. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0040] Figure 1 is a flowchart of an encoding method provided by an embodiment of the present application;

[0041] Figure 2 is a flowchart of a decoding method provided by an embodiment of the present application;

[0042] Figure 3 This is a flowchart of a codec provided by an embodiment of the present application;

[0043] Figure 4 is a schematic diagram of an encoding device provided in an embodiment of the present application;

[0044] Figure 5 is a schematic diagram of a decoding device provided in an embodiment of the present application;

[0045] Figure 6 It is a schematic diagram of a communication processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0047] The term "comprise" and any variation thereof in the specification and claims of the present application are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units need not be limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices. In addition, the use of "and / or" in the specification and claims to indicate at least one of the connected objects, such as A and / or B, means that A alone, B alone, and A and B exist in three cases.

[0048] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0049] Voice audio coding and decoding are essential basic links in communication. With the development of communication technology and the popularization of digital devices, the demand for voice audio transmission and storage is growing. As a key technology for achieving efficient transmission and high-quality reproduction, voice audio coding and decoding technology has become the basis of many fields such as communication, broadcasting, and storage. Traditional analog signal transmission methods have problems such as large bandwidth consumption, susceptibility to interference, and difficulty in long-distance transmission. In order to solve these problems, voice audio coding technology came into being. It converts the original analog voice signal into a digital signal through a compression algorithm and effectively compresses it to reduce the bandwidth and storage space required for transmission.

[0050] During the encoding process, the speech audio encoder analyzes the characteristics of the speech signal, such as frequency distribution, amplitude change, etc., and uses the corresponding algorithm to encode the signal. Common encoding algorithms include but are not limited to Linear Predictive Coding (LPC), transform coding (such as Moving Pictures Experts Group (MPEG)-Advanced Audio Coding (AAC)), Subband Coding (SBC), etc. These algorithms achieve effective data compression by removing redundant information in the signal.

[0051] The decoding process is the reverse process of encoding. After receiving the encoded voice data, the decoder restores the voice signal close to the original sound quality according to the encoding algorithm. This process is crucial to ensure the clarity and intelligibility of voice communication.

[0052] See also Figure 1 An embodiment of the present application provides a coding method applied to an encoder (or artificial intelligence (AI) encoder), wherein the encoder includes one or more neural networks, and the specific steps include: step 11 and step 12.

[0053] Step 11: Processing the first signal through the one or more neural networks to obtain a feature quantity corresponding to each neural network, and encoding at least part of the feature quantity to obtain a feature code stream;

[0054] In step 11, the first signal is processed by one or more neural networks to obtain corresponding feature quantities. First, the first signal is passed to a pre-trained neural network. These networks can be convolutional neural networks (CNN), recurrent neural networks (RNN) or other forms of deep learning models. After each neural network is processed by multiple layers, a feature quantity corresponding to the neural network is obtained, which reflects certain attributes or information in the first signal. The feature quantity is used by the decoder to reconstruct the first signal. Next, the extracted feature quantities are encoded to create a feature code stream. This part of the operation involves compressing high-dimensional features into smaller representations for easy storage and transmission.

[0055] Optionally, each of the one or more neural networks included in the encoder corresponds to at least one version of the encoder.

[0056] Optionally, encoding at least part of the feature quantity to obtain a feature code stream includes: encoding the feature quantity corresponding to the neural network matching the current version of the encoder to obtain the feature code stream. For example, the encoder includes a neural network Y and a neural network X, wherein the neural network Y corresponds to the current version of the encoder, and the neural network X corresponds to the previous version of the encoder, that is, the version of the neural network Y is higher than the version of the neural network X. In step 11, the encoder can quantize and encode the features corresponding to the neural network Y matching the current version of the encoder to obtain a feature code stream and send it to the decoder.

[0057] In this application, the specific version of the neural network used by the encoder (such as its architecture, parameters, and training method) is closely related to the software or hardware version of the encoder. This means that the neural network used by the encoder when processing the signal should be consistent with its own implementation version to ensure the best performance and compatibility.

[0058] Optionally, the first signal may be a preprocessed speech or audio signal, where the preprocessing includes but is not limited to at least one of sub-band separation, bandpass filtering, and other filtering processes.

[0059] Optionally, the feature quantities corresponding to the neural network of the encoder include but are not limited to at least one of the following: input features, which refer to the feature representation of the audio signal used for training or inference; hidden layer features, which are intermediate features generated by each hidden layer of the neural network, which can capture the complex patterns and relationships in the speech or audio signals; network structure, including the hierarchical structure of the network (such as convolutional layers, fully connected layers, etc.), the number of nodes in each layer, and the selection of activation functions; weight parameters for each connection in the neural network; normalization and standardization; loss function, which is a function used to measure the gap between the model output and the actual audio features; hyperparameters, which are parameters specified during the training process, such as learning rate, batch size, and regularization parameters.

[0060] Step 12: determining the difference feature quantities between the feature quantities corresponding to different neural networks, and encoding the difference feature quantities to obtain a difference feature code stream;

[0061] Optionally, determining the difference feature quantities of different neural networks includes: determining the difference feature quantities between the feature quantities corresponding to the neural network matching the current version of the encoder and the feature quantities corresponding to other neural networks.

[0062] It is understandable that if the feature code stream corresponding to the neural network Y in the encoder is obtained in step 11, the difference feature quantity between the neural network Y and the neural network X is determined in step 12. The number of neural networks matching the current version of the encoder can be one or more.

[0063] Optionally, the difference characteristics of different neural networks can include the following aspects: 1. Structural features, for example: a) Number of layers: Different models may contain different numbers of layers, such as input layer, multiple hidden layers and output layer. b) Number of neurons in each layer: The number of neurons (or units) in each layer will affect the expressiveness of the model; 2. Type and architecture, a) Network type: such as Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), Long Short-Term Memory (LSTM), etc. Various architectures are suitable for different tasks. b) Connection method: such as full connection, residual connection, skip connection, etc. 3. Activation function: Different networks may use different activation functions, which will affect the hidden layer output and model performance. 4. Training method: a) Optimization algorithm: such as Stochastic Gradient Descent (SGD), Adaptive Moment Estimation (Adam), Root Mean Square Propagation (RMSprop), etc. Different optimization algorithms have different learning strategies and effects. b) Loss function: The loss function selected for the task type (regression, classification, etc.) may be different, which will naturally lead to differences in feature quantities. 5. Regularization technology: a) The degree and type of application of various regularization methods (such as L1 / L2 regularization, dropout, batch normalization, etc.) can affect the generalization ability of the model. 6. How to process input data: Whether data preprocessing uses normalization, standardization, data augmentation, etc. will affect the input feature performance of the model. 7. Hyperparameter settings: Different settings such as learning rate, batch size, number of iterations, network initialization method, etc. will lead to differences in model performance during training. 8. Capacity and complexity: The capacity (expressive power) and complexity (computational amount and memory usage) of the model directly affect its performance on specific tasks and training efficiency.

[0064] In the process of determining the difference feature quantities of different neural networks and encoding to generate the difference feature code stream, first, the corresponding feature quantities are extracted when each neural network processes the first signal. These feature quantities can be represented in the form of high-dimensional vectors, reflecting the key characteristics of the first signal. Then, feature comparison techniques (for example, Euclidean distance or cosine similarity) can be used to analyze the similarity between the extracted feature quantities, thereby identifying features with significant differences. These are defined as "difference feature quantities". After determining the difference feature quantities of different neural networks, they can be encoded. For example, quantization techniques can be used to map the difference feature quantities to discrete values ​​to achieve compression. At the same time, model-based encoding methods, such as Huffman coding or arithmetic coding, are applied to give these features efficient bit representations. In addition, deep learning models such as autoencoders can be used to optimize the representation of difference feature quantities through training. Finally, the encoded difference feature quantities are connected in series into a data stream, which is a difference feature code stream. This process ensures that the information remains intact during transmission while effectively reducing the amount of data.

[0065] It can be understood that step 11 can be performed before step 12, or step 11 can be performed after step 12, or step 11 and step 12 can be performed simultaneously.

[0066] In one embodiment, after step 12, the method further includes:

[0067] The feature code stream and / or the difference feature code stream is sent to a decoder, and the difference feature code stream is used to match the feature code stream to a feature quantity corresponding to the neural network of the decoder, and the neural network of the decoder is used to output the first signal.

[0068] Optionally, the version of the decoder's neural network can be higher or lower than the version of the encoder's neural network.

[0069] In one implementation of the present application, sending the feature code stream and / or the difference feature code stream to a decoder includes:

[0070] The first code stream and / or the second code stream is sent to the decoder, the first code stream includes first information and second information, the first information is used to indicate that the first code stream is the feature code stream, the second information is used to indicate the neural network corresponding to the feature code stream, the second code stream includes third information and fourth information, the third information is used to indicate that the second code stream is the difference feature code stream, and the fourth information is used to indicate the neural network corresponding to the difference feature code stream.

[0071] See also Figure 3, the encoder includes neural network A, neural network B, neural network C, ..., neural network n, wherein neural network A matches the current version of the encoder, and neural networks B to neural network C correspond to previous versions of the encoder. Input speech or audio into the encoder, pre-process the speech or audio inside the encoder to obtain a first signal, input the first signal into neural network A, neural network B, neural network C, ..., neural network n respectively, process the first signal by neural network A to obtain feature quantity A corresponding to neural network A, quantize and encode feature quantity A to obtain feature code stream corresponding to neural network A. Calculate feature difference between feature quantity corresponding to neural network A and feature quantity corresponding to neural network B to obtain difference feature quantity AB between neural network A and neural network B, and encode the difference feature quantity AB to obtain difference feature code stream AB, and similarly obtain difference feature code stream AC, ..., difference feature code stream An.

[0072] In this embodiment, in view of the fact that the architecture of the neural network and the parameters of the neural network are updated quickly, a method combining a feature code stream and a network difference feature code stream is adopted. Even when the neural network architecture and neural network parameters of the decoder and the encoder are different, the speech or audio code stream can be parsed and a complete reconstructed sound signal can be output. The decoder does not need to update or frequently upgrade the neural network, thus achieving neural network compatibility of stock devices or stock coded files and improving the efficiency of speech or audio coding.

[0073] See also Figure 2 An embodiment of the present application provides a decoding method, which is applied to a decoder (or called an AI decoder), and the specific steps include: step 21, step 22, step 23, and step 24.

[0074] Step 21: receiving a feature code stream sent by an encoder, and decoding to obtain a feature quantity corresponding to a neural network of the encoder, wherein the feature quantity is obtained by processing the first signal based on one or more neural networks of the encoder;

[0075] Optionally, each of the one or more neural networks included in the encoder corresponds to at least one version of the encoder.

[0076] Optionally, each of the one or more neural networks included in the decoder corresponds to at least one version of the decoder.

[0077] Optionally, if the neural network of the encoder and the neural network of the decoder corresponding to the feature code stream are the same pair of neural networks, steps 22 to 23 may be omitted and step 24 may be directly executed to transmit the feature quantity corresponding to the feature code stream to the neural network of the decoder for signal reconstruction to obtain a first signal.

[0078] Step 22: According to the neural network of the decoder and the feature code stream, a target difference feature code stream is obtained, and a target difference feature quantity is decoded to indicate the difference between the feature quantity corresponding to the neural network of the encoder and the feature quantity corresponding to the neural network of the decoder;

[0079] Optionally, if the decoder receives multiple difference feature code streams from the encoder, the decoder may select a difference feature code stream associated with the neural network of the encoder from the multiple difference feature code streams based on the neural network included in the decoder.

[0080] Step 23: determining the feature quantity corresponding to the neural network of the decoder according to the feature quantity corresponding to the neural network of the encoder and the target difference feature quantity;

[0081] In an embodiment, the decoder processes the received feature code stream and the difference feature code stream to determine the feature quantity corresponding to the neural network of the decoder. First, the decoder extracts basic feature quantities from the feature code stream, which are generated by encoding the first signal through one or more neural networks. This process usually involves back propagation and feature reconstruction, and the decoder uses a predefined model structure to parse these feature data. Next, the decoder uses the difference feature code stream to identify and adjust the extracted feature quantities. The information in the difference feature code stream comes from the difference feature quantities between different neural networks. The decoder will analyze these difference features, evaluate their impact on the basic feature quantities, and then correct or supplement them. Specifically, the decoder uses a corresponding mapping function or layer, such as a fully connected layer, to combine the basic feature quantities obtained by decoding with the difference feature quantities to determine the feature quantity corresponding to the neural network of the decoder.

[0082] Step 24: Generate the first signal through the feature quantity corresponding to the neural network of the decoder.

[0083] In an embodiment, the decoder performs reverse parsing on the feature code stream to extract the encoded basic feature quantities, which are obtained by processing the first signal through one or more neural networks. The feature extraction process adopts a decoding scheme corresponding to the encoding to ensure accurate recovery of the information. Then, the decoder uses the difference feature code stream to correct and supplement the basic feature quantities. The difference feature quantities generated can help the decoder identify the difference feature quantities between different neural networks, thereby optimizing the output results. The decoder combines the feature code stream with the difference feature code stream, and uses a series of linear and nonlinear transformations to enhance the features using appropriate structures such as fully connected layers and convolutional layers. Finally, the decoder's neural network generates the final first signal through the adjusted feature quantities.

[0084] See also Figure 3The decoder includes a neural network k, which receives a feature code stream A, a difference feature code stream AB, a difference feature code stream AC, ..., a difference feature code stream An corresponding to the neural network A sent by the encoder. The decoder selects a difference feature code stream Ak from the difference feature code stream AB, the difference feature code stream AC, ..., the difference feature code stream An according to the neural network k, quantizes and decodes the feature code stream A corresponding to the neural network A to obtain a feature quantity A, quantizes and decodes the difference feature code stream Ak to obtain a difference feature quantity Ak, determines a feature quantity k of the neural network k of the decoder according to the feature quantity A and the difference feature quantity Ak, and the feature quantity k can reflect the key characteristics of the first signal, and then generates the first signal through the neural network k of the decoder and the feature quantity k.

[0085] In one implementation of the present application, receiving a feature code stream sent by an encoder includes:

[0086] Receive a first code stream sent by the encoder, the first code stream includes first information and second information, the first information is used to indicate that the first code stream is the feature code stream, and the second information is used to indicate the neural network of the encoder corresponding to the feature code stream.

[0087] In one embodiment of the present application, obtaining a target difference feature code stream according to the neural network of the decoder and the feature code stream includes:

[0088] Receive a second code stream sent by the encoder, the second code stream includes third information and fourth information, the third information is used to indicate that the second code stream is a difference feature code stream corresponding to difference feature quantities of different neural networks of the encoder, and the fourth information is used to indicate the neural network of the encoder corresponding to the difference feature code stream;

[0089] According to the neural network of the decoder and the feature code stream sent by the encoder, the target difference feature code stream is selected from the difference feature code stream sent by the encoder.

[0090] In one embodiment of the present application, obtaining a target difference feature code stream according to the neural network of the decoder and the feature code stream includes:

[0091] Sending the feature code stream and fifth information to a cloud transcoder, wherein the fifth information is used to indicate a neural network of the decoder;

[0092] The target difference feature code stream is received, which is sent by the cloud transcoder according to the feature code stream and the fifth information.

[0093] Optionally, the cloud transcoder can be an application running in a cloud computing environment, whose main function is to encode and decode voice or audio signals. For example, the cloud transcoder has multiple neural networks. After receiving the decoder request, the cloud transcoder quantizes and decodes the feature code stream, and reconstructs the first signal; then the first signal is sent to the neural network corresponding to the feature code stream, and the corresponding feature quantity and the difference feature quantity between the neural network corresponding to the feature code stream and the neural network of the decoder are calculated, and then the difference feature code stream between the neural network corresponding to the feature code stream and the neural network of the decoder is obtained by quantization encoding, and sent to the decoding end.

[0094] In this embodiment, when the neural networks of the encoder and decoder are different, the decoder can also complete the parsing of the speech or audio code stream and output a complete reconstructed sound signal.

[0095] The implementation methods of the present application are introduced below in conjunction with Examples 1 to 4.

[0096] Embodiment 1: A speech and audio encoding method.

[0097] Step 1: Receive the voice or audio signal S at the current time t t , the received voice or audio signal S t Perform preprocessing to obtain the preprocessed signal SP t (ie the first signal).

[0098] Optionally, in step 1, S t The duration of the signal may be 10 milliseconds (ms), 20 ms, or other durations.

[0099] Optionally, in step 1, S t The signal sampling can be 8K, 16K, 32K, or other sampling.

[0100] Optionally, in step 1, S t The bit depth of the signal can be 8 bit, 16 bit, or other bit depths.

[0101] Optionally, in step 1, S t The number of channels of a signal can be 1, 2, or any other number.

[0102] The preprocessing in step 1 includes but is not limited to at least one of sub-band separation, band-pass filtering, and other filtering processes.

[0103] Step 2: The preprocessed signal SP t Send it to the neural network for encoding to get the feature value F t_Ni . Where Ni represents the i-th neural network.

[0104] In step 2, multiple neural networks can be fed simultaneously, each corresponding to a version of the encoder.

[0105] In step 2, only one neural network may be input, that is, when the computing and storage performance of the terminal is limited, only the feature quantity corresponding to the current neural network version of the encoder is calculated.

[0106] Step 3: Calculate the difference in features of different neural networks, where the difference feature quantity corresponding to the Ni and Nj neural networks is F t_Ni_Nj .

[0107] Step 4: Encode the feature code stream S corresponding to the neural network Ni t_Ni .

[0108] In step 4, the feature quantity is first quantized. The quantization can be implemented based on a quantization table or other quantization methods.

[0109] In step 4, the quantized feature quantity is further entropy encoded to form a feature code stream.

[0110] Step 5: Encode the difference feature quantities corresponding to the Ni neural network and the Nj neural network, and output the difference feature code stream D t_Ni_Nj .

[0111] In step 5, the feature quantities of the neural network can be encoded simultaneously, thus achieving the representation of multiple neural network versions.

[0112] In step 5, the difference feature quantity may not be encoded, so that the encoder only implements the encoding of a single neural network version.

[0113] Step 6: Send the feature code stream through the network or store it. If there is a difference feature code stream, it is also sent through the network or stored.

[0114] The above steps are executed in parallel during the encoding process.

[0115] In the encoded output bitstream, the bitstream can set fields to describe its corresponding neural network, including version, whether it is a feature bitstream, whether it is a difference feature bitstream, etc.

[0116] Embodiment 2: A speech and audio decoding method.

[0117] Step 1: The received feature code stream S t_Ni Quantize and decode to get the feature value F t_Ni .

[0118] In step 1, the quantization and decoding process corresponds to the encoding process, using the same inverse quantization table and entropy coding method.

[0119] Step 2: Determine whether the neural network version Ni corresponding to the feature code stream is the same as the neural network version Nk of the current decoder.

[0120] Step 3: If the neural network version is the same, that is, the encoder and decoder are the same pair of neural networks. Then the feature quantity F t_Ni Send it to the neural network for signal reconstruction and get R t .

[0121] Step 4: If the neural network version is different, the decoder requests the difference feature code stream D corresponding to the network t_Ni_Nk .

[0122] Step 5: D t_Ni_Nk Perform inverse quantization and decoding to obtain the difference feature F t_Ni_Nk .

[0123] Step 6: Feature adaptation module, the feature quantity F t_Ni , difference feature quantity F t_Ni_Nk . Calculate the feature value F of the decoder corresponding to the neural network t_Nk .

[0124] Step 7: The feature quantity F t_Nk Send it to the neural network for signal reconstruction and get R t .

[0125] Step 8: R t Do post-processing to get the output sound signal SE t .

[0126] The above steps are executed in parallel during the decoding process.

[0127] In this embodiment, in view of the fact that the architecture and neuron parameters of the neural network are updated quickly, a combination of feature code stream and network difference feature code stream is adopted to achieve neural network compatibility of existing devices or existing coded files, further improving the efficiency of speech and audio coding. Even when the neural network architecture and parameters of the decoding end and the encoding end are different, the parsing of the speech or audio code stream can be completed and a complete reconstructed sound signal can be output.

[0128] Example 3: The encoder supports multiple neural networks, and the decoder has only one neural network

[0129] The encoder neural network version is Y, while supporting the older neural network version X. The decoder neural network version is X.

[0130] During encoding, the encoder preprocesses the input signal and sends it to neural network Y and neural network X for feature extraction. The feature corresponding to neural network Y is quantized and encoded and sent to the decoder. The encoder calculates the difference feature between neural network Y and neural network X, and sends the difference feature code stream to the decoder after quantization encoding.

[0131] The decoder quantizes and decodes the received feature code stream and difference feature code stream to obtain the feature quantity of neural network Y and the difference feature quantity between neural network Y and neural network X. The decoder calculates the feature quantity of neural network X based on the feature quantity of neural network Y and the difference feature quantity between neural network Y and neural network X, and sends the feature quantity to the reconstruction neural network X to further synthesize the sound waveform.

[0132] Example 4: The encoder supports 1 neural network and the decoder has only 1 neural network.

[0133] The neural network version of the encoder is Y. The neural network version of the decoder is X.

[0134] During encoding, the encoder preprocesses the input signal and sends it to the neural network Y for feature extraction. The feature quantity corresponding to the neural network Y is quantized and encoded to obtain the feature code stream corresponding to the neural network Y and sent to the decoder.

[0135] The decoder quantizes and decodes the received feature code stream to obtain the feature quantity of neural network Y. Since the decoder only has neural network X, it cannot reconstruct the feature quantity of neural network Y. The decoder sends the feature code stream corresponding to neural network Y to the cloud transcoder and requests the difference feature code stream between neural network Y and neural network X.

[0136] The cloud transcoder has multiple neural network versions. After receiving the decoder request, the feature code stream corresponding to neural network Y is quantized and decoded, and the speech signal is reconstructed; the speech signal is then sent to neural network X, the feature quantity corresponding to neural network X and the difference feature quantity between neural network X and neural network Y are calculated, and then the difference feature code stream between neural network Y and neural network X is quantized and encoded, and sent to the decoding end.

[0137] The decoder quantizes and decodes the received difference feature code stream to obtain the difference feature quantity between neural network Y and neural network X, and then calculates the feature quantity of neural network X based on the feature quantity of neural network Y, and sends this feature quantity to the reconstruction neural network X to further synthesize the sound waveform.

[0138] See also Figure 4An embodiment of the present application provides a coding device, applied to an encoder, wherein the encoder includes one or more neural networks, and the device 400 includes: a first receiving unit 401 and a first processing unit 402.

[0139] A first processing unit 402 is configured to process the first signal through the one or more neural networks to obtain a feature quantity corresponding to each neural network, and encode at least part of the feature quantity to obtain a feature code stream;

[0140] The first processing unit 402 is further used to determine the difference feature quantities between the feature quantities corresponding to different neural networks, and encode the difference feature quantities to obtain a difference feature code stream.

[0141] In one embodiment of the present application, each of the one or more neural networks included in the encoder corresponds to at least one version of the encoder.

[0142] In one embodiment of the present application, the first processing unit 402 is further used to determine the difference feature quantity between the feature quantity corresponding to the neural network matching the current version of the encoder and the feature quantity corresponding to other neural networks.

[0143] In one embodiment of the present application, the first receiving unit 401 is used to send the feature code stream and / or the difference feature code stream to a decoder, and the difference feature code stream is used to match the feature code stream to a feature value corresponding to the neural network of the decoder, and the neural network of the decoder is used to output the first signal.

[0144] In one embodiment of the present application, the first processing unit 401 is further used to send a first code stream and / or a second code stream to the decoder, the first code stream includes first information and second information, the first information is used to indicate that the first code stream is the feature code stream, the second information is used to indicate the neural network corresponding to the feature code stream, the second code stream includes third information and fourth information, the third information is used to indicate that the second code stream is the difference feature code stream, and the fourth information is used to indicate the neural network corresponding to the difference feature code stream.

[0145] The device provided in the embodiment of the present application can achieve Figure 1 The various processes implemented by the method embodiment shown achieve the same technical effect and will not be described again here to avoid repetition.

[0146] See also Figure 5 An embodiment of the present application provides a decoding device, which is applied to a decoder. The device 500 includes a second receiving unit 501 and a second processing unit 502.

[0147] The second receiving unit 501 is used to receive a feature code stream sent by an encoder, and decode to obtain a feature quantity corresponding to a neural network of the encoder, where the feature quantity is obtained by processing the first signal based on one or more neural networks of the encoder;

[0148] A second processing unit 502 is used to obtain a target difference feature code stream according to the neural network of the decoder and the feature code stream, and decode to obtain a target difference feature quantity, wherein the target difference feature quantity is used to represent the difference between the feature quantity corresponding to the neural network of the encoder and the feature quantity corresponding to the neural network of the decoder;

[0149] The second processing unit 502 is further used to determine the feature quantity corresponding to the neural network of the decoder according to the feature quantity corresponding to the neural network of the encoder and the target difference feature quantity;

[0150] The second processing unit 502 is further used to generate the first signal through the feature quantity corresponding to the neural network of the decoder.

[0151] In one embodiment of the present application, the second receiving unit 501 is further used to: receive a first code stream sent by the encoder, the first code stream includes first information and second information, the first information is used to indicate that the first code stream is the feature code stream, and the second information is used to indicate the neural network of the encoder corresponding to the feature code stream.

[0152] In one embodiment of the present application, the second receiving unit 501 is further used to: receive a second code stream sent by the encoder, the second code stream includes third information and fourth information, the third information is used to indicate that the second code stream is a difference feature code stream corresponding to the difference feature quantities of different neural networks of the encoder, and the fourth information is used to indicate the neural network of the encoder corresponding to the difference feature code stream;

[0153] The second processing unit 502 is further configured to select the target difference feature code stream from the difference feature code stream sent by the encoder according to the neural network of the decoder and the feature code stream sent by the encoder.

[0154] In one embodiment of the present application, the second receiving unit 501 is also used to: send the feature code stream and the fifth information to the cloud transcoder, the fifth information is used to indicate the neural network of the decoder, and receive the target difference feature code stream sent by the cloud transcoder based on the feature code stream and the fifth information.

[0155] The device provided in the embodiment of the present application can achieve Figure 2 The various processes implemented by the method embodiment shown achieve the same technical effect and will not be described again here to avoid repetition.

[0156] like Figure 6 As shown, the embodiment of the present application also provides a communication device 600, including a processor 601, a memory 602, a program or instruction stored in the memory 602 and executable on the processor 601, and the program or instruction is executed by the processor 601 to implement the above Figure 1 or Figure 2 The various processes of the method embodiment can achieve the same technical effect. To avoid repetition, it will not be repeated here. The communication device can be a terminal or a network side device, that is, the terminal can include the above encoder and / or decoder, and the network side device can also include the above encoder and / or decoder.

[0157] The embodiment of the present application also provides a readable storage medium on which a program or instruction is stored. When the program or instruction is executed by a processor, the above Figure 1 or Figure 2 The various processes of the method embodiment shown can achieve the same technical effect, and will not be described again here to avoid repetition.

[0158] The processor is the processor in the terminal described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0159] The steps of the method or algorithm described in conjunction with the contents disclosed in this application can be implemented in hardware or by executing software instructions in a processor. The software instructions can be composed of corresponding software modules, and the software modules can be stored in RAM, flash memory, ROM, EPROM, EEPROM, registers, hard disks, mobile hard disks, read-only optical disks, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be carried in an ASIC. In addition, the ASIC can be carried in a core network interface device. Of course, the processor and the storage medium can also exist in the core network interface device as discrete components.

[0160] Those skilled in the art should be aware that in one or more of the above examples, the functions described in this application can be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any media that facilitates the transmission of a computer program from one place to another. The storage medium can be any available medium that a general or special-purpose computer can access.

[0161] The specific implementation methods described above further illustrate the purpose, technical solutions and beneficial effects of the present application in detail. It should be understood that the above description is only the specific implementation methods of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solutions of the present application should be included in the scope of protection of the present application.

[0162] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the embodiments of the present application may adopt the form of complete hardware embodiments, complete software embodiments, or embodiments in combination with software and hardware. Moreover, the embodiments of the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0163] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0164] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1A function specified in one or more boxes.

[0165] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0166] Obviously, those skilled in the art can make various changes and modifications to the embodiments of the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the embodiments of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A coding method, applied to an encoder, the encoder comprising one or more neural networks, characterized in that: include: Processing the first signal through the one or more neural networks to obtain a feature quantity corresponding to each neural network, and encoding at least part of the feature quantity to obtain a feature code stream; The difference feature quantities between the feature quantities corresponding to different neural networks are determined, and the difference feature quantities are encoded to obtain a difference feature code stream.

2. The method according to claim 1, characterized in that Each of the one or more neural networks included in the encoder corresponds to at least one version of the encoder.

3. The method according to claim 2, characterized in that Determine the difference between the feature quantities corresponding to different neural networks, including: Determine the difference between the feature quantity corresponding to the neural network matching the current version of the encoder and the feature quantity corresponding to other neural networks.

4. The method according to claim 1, characterized in that: The method further comprises: The feature code stream and / or the difference feature code stream is sent to a decoder, wherein the difference feature code stream is used to match the feature code stream to a feature quantity corresponding to a neural network of the decoder, and the neural network of the decoder is used to output the first signal.

5. The method according to claim 4, characterized in that Sending the feature code stream and / or the difference feature code stream to a decoder includes: The first code stream and / or the second code stream is sent to the decoder, the first code stream includes first information and second information, the first information is used to indicate that the first code stream is the feature code stream, the second information is used to indicate the neural network corresponding to the feature code stream, the second code stream includes third information and fourth information, the third information is used to indicate that the second code stream is the difference feature code stream, and the fourth information is used to indicate the neural network corresponding to the difference feature code stream.

6. A decoding method, applied to a decoder, characterized in that: include: Receiving a feature code stream sent by an encoder, and decoding to obtain a feature quantity corresponding to a neural network of the encoder, wherein the feature quantity is obtained by processing a first signal based on one or more neural networks of the encoder; According to the neural network of the decoder and the feature code stream, a target difference feature code stream is obtained, and a target difference feature quantity is obtained by decoding, wherein the target difference feature quantity is used to represent the difference between the feature quantity corresponding to the neural network of the encoder and the feature quantity corresponding to the neural network of the decoder; Determining the feature quantity corresponding to the neural network of the decoder according to the feature quantity corresponding to the neural network of the encoder and the target difference feature quantity; The first signal is generated by the feature quantity corresponding to the neural network of the decoder.

7. The method according to claim 6, characterized in that Receive the feature code stream sent by the encoder, including: Receive a first code stream sent by the encoder, the first code stream includes first information and second information, the first information is used to indicate that the first code stream is the feature code stream, and the second information is used to indicate the neural network of the encoder corresponding to the feature code stream.

8. The method according to claim 6, characterized in that According to the neural network of the decoder and the feature code stream, a target difference feature code stream is obtained, including: Receive a second code stream sent by the encoder, the second code stream includes third information and fourth information, the third information is used to indicate that the second code stream is a difference feature code stream corresponding to difference feature quantities of different neural networks of the encoder, and the fourth information is used to indicate the neural network of the encoder corresponding to the difference feature code stream; According to the neural network of the decoder and the feature code stream sent by the encoder, the target difference feature code stream is selected from the difference feature code stream sent by the encoder.

9. The method according to claim 6, characterized in that According to the neural network of the decoder and the feature code stream, a target difference feature code stream is obtained, including: Sending the feature code stream and fifth information to a cloud transcoder, wherein the fifth information is used to indicate a neural network of the decoder; The target difference feature code stream is received, which is sent by the cloud transcoder according to the feature code stream and the fifth information.

10. A coding device, applied to an encoder, the encoder comprising one or more neural networks, characterized in that: The device comprises: a first receiving unit and a first processing unit; The first processing unit is used to process the first signal through one or more neural networks to obtain a feature quantity corresponding to each neural network, and encode at least part of the feature quantity to obtain a feature code stream; The first processing unit is further used to determine the difference feature quantities between the feature quantities corresponding to different neural networks, and encode the difference feature quantities to obtain a difference feature code stream.

11. A decoding device, applied to a decoder, characterized in that: The device comprises a second receiving unit and a second processing unit; The second receiving unit is used to receive a feature code stream sent by an encoder, and decode to obtain a feature quantity corresponding to a neural network of the encoder, wherein the feature quantity is obtained by processing the first signal based on one or more neural networks of the encoder; The second processing unit is used to obtain a target difference feature code stream according to the neural network of the decoder and the feature code stream, and decode to obtain a target difference feature quantity, wherein the target difference feature quantity is used to represent the difference between the feature quantity corresponding to the neural network of the encoder and the feature quantity corresponding to the neural network of the decoder; The second processing unit is further used to determine the feature quantity corresponding to the neural network of the decoder according to the feature quantity corresponding to the neural network of the encoder and the target difference feature quantity; The second processing unit is further used to generate the first signal through the feature quantity corresponding to the neural network of the decoder.

12. A communication device, characterized in that: The method comprises a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the method according to any one of claims 1 to 9.

13. A readable storage medium, characterized in that: The readable storage medium stores a program or an instruction, and when the program or the instruction is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

14. A computer program product, characterized in that The method comprises computer instructions, which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 9.

Images