Training method and device of coding model, equipment and storage medium

Abstract

The application discloses a training method and device of a coding and decoding model, equipment and a storage medium, and relates to the technical field of computers. The method comprises the following steps: obtaining a pre-training coding and decoding model; wherein the pre-training coding and decoding model comprises a pre-training encoder and a pre-training decoder; performing pruning processing on the pre-training decoder to obtain a pruned decoder; performing encoding processing on second sample multimedia data through the pre-training encoder to obtain encoded feature data, and performing decoding processing on the encoded feature data through the pruned decoder to obtain reconstructed multimedia data; performing parameter optimization on the pruned decoder based on the difference between the reconstructed multimedia data and the second sample multimedia data to obtain a target decoder; and obtaining a target coding and decoding model according to the pre-training decoder and the target decoder. Through the stage-by-stage training mode, the application realizes a large-encoder+small-decoder coding and decoding model, and improves the performance of the coding and decoding model.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a training method, apparatus, device, and storage medium for an encoding / decoding model. Background Technology

[0002] Encoding and decoding have always been an important research branch in the field of multimedia data. Compression of multimedia data based on signal processing and psychoacoustic models can reduce the recording and transmission of redundant data.

[0003] In related technologies, the encoding and decoding are usually implemented using the DAC (Digital-to-Analog Converter) model. The DAC model includes four modules: encoder, quantizer, decoder, and discriminator. The encoder performs convolutional encoding on the input multimedia data, the quantizer quantizes the encoded multimedia data into a smaller integer sequence, the decoder decodes and recovers the quantized multimedia data, and the discriminator makes the decoded multimedia data more closely approximate the original multimedia data.

[0004] However, the DAC model in the above method is trained by directly using a large encoder + small decoder, which can easily affect the feature extraction capability of the encoder, causing the new model to produce illusions and affecting the encoding and decoding effect. Summary of the Invention

[0005] This application provides a method, apparatus, device, and storage medium for training an encoding / decoding model. The technical solution provided by this application is as follows: According to one aspect of the embodiments of this application, a method for training an encoding / decoding model is provided, the method comprising: Obtain a pre-trained encoder-decoder model; wherein the pre-trained encoder-decoder model includes a pre-trained encoder and a pre-trained decoder; the pre-trained encoder-decoder model is obtained by training an initial encoder-decoder model based on first sample multimedia data; The pre-trained decoder is pruned to obtain the pruned decoder; The second sample multimedia data is encoded by the pre-trained encoder to obtain encoded feature data, and the encoded feature data is decoded by the pruned decoder to obtain reconstructed multimedia data. Based on the difference between the reconstructed multimedia data and the second sample multimedia data, the parameters of the pruned decoder are optimized to obtain the target decoder.

[0006] According to one aspect of the embodiments of this application, a training apparatus for an encoding / decoding model is provided, the apparatus comprising: A model pre-training module is used to obtain a pre-trained encoder-decoder model; wherein, the pre-trained encoder-decoder model includes a pre-trained encoder and a pre-trained decoder; the pre-trained encoder-decoder model is obtained by training an initial encoder-decoder model based on first sample multimedia data; The decoder pruning module is used to prune the pre-trained decoder to obtain the pruned decoder; The data reconstruction module is used to encode the second sample multimedia data through the pre-trained encoder to obtain encoded feature data, and to decode the encoded feature data through the pruned decoder to obtain reconstructed multimedia data. The decoder optimization module is used to optimize the parameters of the pruned decoder based on the difference between the reconstructed multimedia data and the second sample multimedia data to obtain the target decoder. The model determination module is used to obtain the target encoding / decoding model based on the pre-trained decoder and the target decoder.

[0007] According to one aspect of the embodiments of this application, a computer device is provided, the computer device including a processor and a memory, the memory storing a computer program, the computer program being loaded and executed by the processor to implement the above-described training method for the encoding and decoding model.

[0008] According to one aspect of the embodiments of this application, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, the computer program being loaded and executed by a processor to implement the above-described training method for the encoding and decoding model.

[0009] According to one aspect of the embodiments of this application, a computer program product is provided, the computer program product including a computer program, the computer program being loaded and executed by a processor to implement the above-described training method for the encoding and decoding model.

[0010] The technical solution provided in this application can bring the following beneficial effects: By pruning the pre-trained decoder in the pre-trained codec model, removing some neurons to obtain a pruned decoder, and then freezing the parameters of other components in the pre-trained codec model, the parameters of the pruned decoder are optimized to obtain the target decoder. Compared with the method of directly training the codec model containing a large encoder and a small decoder in related technologies, the technical solution provided in this application adopts a staged training approach. First, the initial codec model is trained to obtain a pre-trained codec model. Under the premise that the feature extraction capability of the large encoder has been guaranteed in the pre-training, ensuring that detailed information in high bitrate scenarios can be extracted, the size of the pre-trained decoder is reduced by pruning the pre-trained decoder to obtain a small decoder. The small decoder is then fine-tuned, so that the small decoder has a strong ability to restore details while minimizing the performance loss caused by the reduction in the size of the pre-trained decoder. This improves the performance of the codec model in high bitrate scenarios, avoids the illusion problem in related technologies, and improves the model's encoding and decoding effect. Attached Figure Description

[0011] Figure 1 This is a schematic diagram of a computer system provided in one embodiment of this application; Figure 2 This is a flowchart of a training method for an encoding / decoding model provided in one embodiment of this application; Figure 3 This is a schematic diagram of the architecture of an encoding / decoding model provided in one embodiment of this application; Figure 4 This is a schematic diagram of the training process of an encoding / decoding model provided in one embodiment of this application; Figure 5 This is a flowchart of a training method for an audio codec model provided in one embodiment of this application; Figure 6 This is a flowchart of a training method for a codec model applied to the video field, provided in one embodiment of this application; Figure 7 This is a block diagram of a training apparatus for an encoding / decoding model provided in one embodiment of this application; Figure 8 This is a structural block diagram of a computer device provided in one embodiment of this application. Detailed Implementation

[0012] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0013] Please refer to Figure 1 The diagram illustrates a computer system provided in one embodiment of this application. The computer system may include a model training device 10 and a model usage device 20.

[0014] The model training device 10 can be an electronic device such as a mobile phone, tablet computer, laptop computer, desktop computer, smart TV, multimedia playback device, vehicle terminal, server, intelligent robot, or other electronic devices with strong computing power. The model training device 10 is used to train the encoding and decoding model.

[0015] In this embodiment, the encoding / decoding model is a machine learning model trained using a training method based on the encoding / decoding model, used for encoding and decoding multimedia data. The model training device 10 can train the encoding / decoding model using machine learning to enable it to encode and decode multimedia data. Specific model training methods can be found in the following embodiments.

[0016] The trained encoding / decoding model can be deployed on the model-using device 20. The model-using device 20 can be an electronic device such as a mobile phone, tablet, laptop, desktop computer, smart TV, multimedia playback device, in-vehicle terminal, server, intelligent robot, or other electronic devices with strong computing capabilities. When multimedia data needs to be encoded and decoded, the model-using device 20 can achieve the above functions by completing the trained encoding / decoding model.

[0017] The model training device 10 and the model usage device 20 can be two separate devices or the same device. If the model training device 10 and the model usage device 20 are the same device, then the model training device 10 can be deployed in the model usage device 20.

[0018] In this embodiment of the application, the entity performing each step can be a computer device, which can be, for example, a computer device... Figure 1 The model training device 10 can also be the model usage device 20. The server can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server providing cloud computing services.

[0019] Please refer to Figure 2 This document illustrates a flowchart of a training method for an encoding / decoding model provided in one embodiment of this application. The execution entity for each step of this method can be a computer device. The method may include at least one of the following steps 210-250: Step 210: Obtain the pre-trained codec model; wherein, the pre-trained codec model includes a pre-trained encoder and a pre-trained decoder; the pre-trained codec model is obtained by training the initial codec model based on the first sample multimedia data.

[0020] The pre-training process of the pre-trained codec model is as follows: Obtain a first training dataset, which includes at least one first sample multimedia data. Train the initial codec model using at least one first sample multimedia data to obtain the pre-trained codec model, wherein the initial codec model includes an initial encoder and an initial decoder.

[0021] Multimedia data is composite data formed by various forms of data, such as text, graphics, images, sound, animation, and moving images, which are interconnected in content. The presentation forms of multimedia data include, but are not limited to, images, audio, and video. For example, if multimedia data is presented as an image, it can be composite data formed by various forms of data, such as text, graphics, images, and moving images, which are interconnected in content; if multimedia data is presented as audio, it is audio data in the form of sound; if multimedia data is presented as video, it is composite data formed by various forms of data, such as text, images, sound, and animation, which are interconnected in content. If the sample multimedia data is used to indicate sample audio, the training method of the encoding and decoding model provided in this application is a training method for an audio encoding and decoding model; if the sample multimedia data is used to indicate sample video, the training method of the encoding and decoding model provided in this application is a training method for a video encoding and decoding model; if the sample multimedia data is used to indicate sample image, the training method of the encoding and decoding model provided in this application is a training method for an image encoding and decoding model.

[0022] In some embodiments, the initial encoding / decoding model includes only an initial encoder and an initial decoder. The initial encoder encodes the first sample multimedia data to obtain encoded feature data corresponding to the first sample multimedia data, and the initial decoder decodes the encoded feature data corresponding to the first sample multimedia data to obtain reconstructed multimedia data corresponding to the first sample multimedia data.

[0023] Based on the difference between the reconstructed multimedia data corresponding to the first sample multimedia data and the first sample multimedia data, the data loss of the first sample multimedia data is obtained. This data loss measures the similarity between the reconstructed multimedia data and the first sample multimedia data. Based on the data loss of the first sample multimedia data, the parameters of the initial encoder-decoder model are optimized to obtain a pre-trained encoder-decoder model. The pre-trained encoder-decoder model includes a pre-trained encoder and a pre-trained decoder.

[0024] In some embodiments, the initial encoding / decoding model includes an initial encoder, an initial quantizer, and an initial decoder. The initial encoder encodes the first sample multimedia data to obtain encoded feature data corresponding to the first sample multimedia data. The initial quantizer quantizes the encoded feature data corresponding to the first sample multimedia data to obtain quantized feature data corresponding to the first sample multimedia data. The initial decoder decodes the quantized feature data corresponding to the first sample multimedia data to obtain reconstructed multimedia data corresponding to the first sample multimedia data.

[0025] Based on the difference between the reconstructed multimedia data corresponding to the first sample multimedia data and the first sample multimedia data, the data loss of the first sample multimedia data is obtained. This data loss measures the similarity between the reconstructed multimedia data and the first sample multimedia data. Based on the generation process of the reconstructed multimedia data corresponding to the first sample multimedia data, the quantizer loss of the initial quantizer is obtained. This quantizer loss measures the information loss during the quantization process. Based on the data loss of the first sample multimedia data and the quantizer loss of the initial quantizer, the comprehensive loss corresponding to the first sample multimedia data is obtained. Based on the comprehensive loss corresponding to the first sample multimedia data, the parameters of the initial encoder-decoder model are adjusted to obtain the pre-trained encoder-decoder model. The pre-trained encoder-decoder model includes a pre-trained encoder, a pre-trained quantizer, and a pre-trained decoder.

[0026] The initial encoder can include multiple one-dimensional convolutional blocks. These convolutional layers sequentially encode the first sample multimedia data, yielding the encoded feature data. By setting the stride of the initial encoder, downsampling can be implemented during the encoding process, reducing the amount of data while preserving the main features of the first sample multimedia data and compressing its dimensionality. The encoding process transforms the first sample multimedia data into a more compact representation; the encoded feature data is a low-dimensional latent representation, facilitating subsequent processing.

[0027] The initial quantizer employs Residual Vector Quantization (RVQ), progressively quantizing the encoded feature data through multiple stages. Each stage uses a codebook to map the latent representation to a discrete sequence of integers, quantizing the encoded feature data corresponding to the first sample multimedia data into a smaller-dimensional integer sequence, thus obtaining the quantized feature data corresponding to the first sample multimedia data. This approach allows for further data compression while preserving, to some extent, the important information of the first sample multimedia data. This process may involve multiple quantization steps or levels to progressively improve quantization accuracy or compression ratio.

[0028] The initial decoder may include multiple one-dimensional deconvolutional blocks. Multiple deconvolutional layers sequentially decode the quantized feature data corresponding to the first sample multimedia data, gradually restoring the data dimension of the first sample multimedia data to obtain the reconstructed multimedia data. Deconvolution is the inverse operation of convolution; it can convert encoded and quantized compressed integer sequences back to near-original data. This process requires progressively increasing the dimensionality and complexity of the data to reconstruct the features of the first sample multimedia data.

[0029] In some embodiments, the initial codec model may also include an initial discriminator, and the initial encoder, initial quantizer and initial decoder may also be collectively referred to as an initial generator, that is, the initial codec model includes an initial generator and an initial discriminator.

[0030] An initial discriminator compares the reconstructed multimedia data corresponding to the first sample multimedia data with the original first sample multimedia data. The initial discriminator may include multiple two-dimensional convolutional blocks, employing multi-scale short-time Fourier transforms to distinguish between the reconstructed multimedia data and the original first sample multimedia data. Then, adversarial training prompts the initial generator to adjust its parameters, making the data generated by the initial decoder more closely approximate the original data, thus improving decoding quality. For example, the discriminator might determine whether the reconstructed first sample multimedia data resembles the original first sample multimedia data; if so, positive feedback is given; otherwise, negative feedback is given, thereby continuously optimizing the entire model during training.

[0031] The data loss for the first sample multimedia data can include the generator loss and the discriminator loss. The generator loss measures the error level of the reconstructed multimedia data corresponding to the first sample multimedia data generated by the initial generator, which is then recognized by the initial discriminator. Minimizing the generator loss encourages the initial generator to produce more realistic prediction data to fool the initial discriminator, thus improving the quality of the initial generator's data generation. The discriminator loss measures the error level of the initial discriminator in the classification task. Minimizing the discriminator loss encourages the initial discriminator to more accurately identify the reconstructed multimedia data and the first sample multimedia data, thus improving the initial discriminator's discrimination ability.

[0032] The quantizer loss of the initial quantizer can be divided into the quantization loss and the entropy encoding loss. The quantization loss measures the similarity between the codebook of the initial quantizer and the data before quantization, reflecting the degree of information loss caused by converting continuous data into a discrete integer sequence during quantization. Minimizing the quantization loss ensures that more information is retained during the quantization process. The entropy encoding loss measures the similarity of the encoded feature data after secondary compression. Further compression operations such as entropy encoding may be performed after quantization; the entropy encoding loss assesses the impact of this additional compression on the data. Minimizing the entropy encoding loss optimizes the compression efficiency during the quantization process.

[0033] Optionally, the data loss of the first sample multimedia data and the quantizer loss of the initial quantizer can be added together to obtain the comprehensive loss corresponding to the first sample multimedia data. Alternatively, the data loss of the first sample multimedia data and the quantizer loss of the initial quantizer can be weighted to obtain the comprehensive loss corresponding to the first sample multimedia data. With minimizing the comprehensive loss corresponding to the first sample multimedia data as the training objective, the parameters of the initial encoder-decoder model are adjusted to obtain the pre-trained encoder-decoder model. Adjusting the parameters of the initial encoder-decoder model refers to adjusting the parameters of one or more components in the initial encoder, initial quantizer, initial decoder, and initial discriminator.

[0034] In some embodiments, when the sample multimedia data is sample audio, the comprehensive loss corresponding to the first sample multimedia data further includes the spectral loss and temporal loss of the first sample multimedia data; wherein, the spectral loss is used to measure the spectral similarity between the reconstructed multimedia data corresponding to the first sample multimedia data and the first sample multimedia data, and the temporal loss is used to measure the temporal similarity between the reconstructed multimedia data corresponding to the first sample multimedia data and the first sample multimedia data.

[0035] Based on the spectrograms of the reconstructed multimedia data and the first sample multimedia data corresponding to the first sample multimedia data, the energy distribution information at different frequencies of the reconstructed multimedia data and the first sample multimedia data is analyzed, and the spectral loss of the first sample multimedia data is calculated. The spectral loss of the first sample multimedia data is used to indicate the spectral similarity between the reconstructed multimedia data and the first sample multimedia data. By calculating the spectral loss, it can be ensured that the spectral characteristics of the first sample multimedia data are not excessively altered during encoding and decoding. Based on the waveform diagrams of the reconstructed multimedia data and the first sample multimedia data, the waveform shape, amplitude changes, and other characteristic information of the audio signal in the time domain of the reconstructed multimedia data and the first sample multimedia data are analyzed, and the temporal loss of the first sample multimedia data is calculated. The temporal loss of the first sample multimedia data is used to indicate the temporal similarity between the reconstructed multimedia data and the first sample multimedia data. By calculating the temporal loss, it can be ensured that the waveform characteristics of the first sample multimedia data are not excessively altered during encoding and decoding.

[0036] It's important to note that for high bitrate scenarios, the aforementioned pre-trained encoder-decoder model needs stronger feature extraction and reconstruction capabilities. Therefore, a large encoder and decoder are used in the pre-trained model. There are many ways to increase the encoder and decoder size, such as widening the network to increase the number of neurons in each layer, or deepening the network to increase the number of convolutional residual units. The training of the pre-trained encoder-decoder model aims to ensure that the pre-trained encoder has strong feature extraction capabilities and the pre-trained discriminator has strong ability to distinguish spectral details.

[0037] Step 220: Prune the pre-trained decoder to obtain the pruned decoder.

[0038] Pruning is used to remove some neurons from the pre-trained decoder, reducing the number of neurons in the pre-trained decoder and thus reducing its size. This ensures that the size of the pruned decoder meets the pruning criteria, meaning the pre-trained decoder is a "large decoder" and the pruned decoder is a "small decoder." The specific steps of pruning are detailed in the following examples and will not be described in detail here.

[0039] Step 230: The second sample multimedia data is encoded by a pre-trained encoder to obtain encoded feature data, and the encoded feature data is decoded by a pruned decoder to obtain reconstructed multimedia data.

[0040] In some embodiments, if the pre-trained encoder-decoder model includes only a pre-trained encoder and a pre-trained decoder, then the pruned encoder-decoder model includes a pre-trained encoder and a pruned decoder. The pre-trained encoder encodes the second sample multimedia data to obtain encoded feature data, and the pruned decoder decodes the encoded feature data to obtain reconstructed multimedia data.

[0041] In some embodiments, the pre-trained encoding / decoding model includes a pre-trained encoder, a pre-trained quantizer, and a pre-trained decoder; the pruned encoding / decoding model includes a pre-trained encoder, a pre-trained quantizer, and a pruned decoder. The second sample multimedia data is encoded using the pre-trained encoder to obtain encoded feature data; the encoded feature data is quantized using the pre-trained quantizer to obtain quantized feature data; and the quantized feature data is decoded using the pruned decoder to obtain reconstructed multimedia data.

[0042] In some embodiments, the pre-trained codec model may further include a pre-trained discriminator. The pre-trained encoder, pre-trained quantizer, and post-pruned decoder can also be collectively referred to as a post-pruned generator, meaning that the pruned codec model includes a post-pruned generator and a pre-trained discriminator. The pre-trained discriminator is used to compare the reconstructed multimedia data corresponding to the second sample multimedia data and the second sample multimedia data to optimize the parameters of the post-pruned decoder and obtain the target decoder.

[0043] The specific encoding and decoding process of the pre-trained encoding and decoding model can be referred to the encoding and decoding process of the initial encoding and decoding model mentioned above, and will not be repeated here.

[0044] Step 240: Based on the difference between the reconstructed multimedia data and the second sample multimedia data, the parameters of the pruned decoder are optimized to obtain the target decoder.

[0045] In some embodiments, the pruned encoder-decoder model includes only a pre-trained encoder and a pruned decoder. Then, based on the difference between the reconstructed multimedia data corresponding to the second sample multimedia data and the second sample multimedia data, the data loss of the second sample multimedia data is obtained. Based on the data loss of the second sample multimedia data, the parameters of the pruned decoder are optimized to obtain the target decoder.

[0046] In some embodiments, the data loss of the second sample multimedia data is obtained based on the difference between the reconstructed multimedia data corresponding to the second sample multimedia data and the second sample multimedia data. This data loss measures the similarity between the reconstructed multimedia data and the second sample multimedia data. The quantizer loss of the pre-trained quantizer is obtained based on the generation process of the reconstructed multimedia data corresponding to the second sample multimedia data. This quantizer loss measures the information loss of the pre-trained quantizer during the quantization process. The comprehensive loss corresponding to the second sample multimedia data is obtained based on the data loss of the second sample multimedia data and the quantizer loss of the pre-trained quantizer. Based on the comprehensive loss corresponding to the second sample multimedia data, the parameters of the pruned decoder are optimized to obtain the target decoder.

[0047] The data loss for the second sample multimedia data can include the generator loss and the discriminator loss. The generator loss measures the error level of the reconstructed multimedia data corresponding to the second sample multimedia data generated by the generator after horizontal pruning, which is then recognized by the pre-trained discriminator. The discriminator loss measures the error level of the pre-trained discriminator in the classification task. The quantizer loss for the pre-trained quantizer can include the quantization loss and the entropy encoding loss. The quantization loss measures the similarity between the codebook of the pre-trained quantizer and the data before quantization, while the entropy encoding loss measures the similarity of the encoded feature data after secondary compression.

[0048] Optionally, the data loss of the second sample multimedia data and the quantizer loss of the pre-trained quantizer can be added together to obtain the comprehensive loss corresponding to the second sample multimedia data. Alternatively, the data loss of the second sample multimedia data and the quantizer loss of the pre-trained quantizer can be weighted to obtain the comprehensive loss corresponding to the second sample multimedia data. With minimizing the comprehensive loss corresponding to the second sample multimedia data as the training objective, the parameters of the pruned decoder are optimized to obtain the target decoder. During pre-training, this parameter optimization requires freezing the parameters of the pre-trained encoder, pre-trained quantizer, and pre-trained discriminator in the pruned encoder-decoder model, and only adjusting the parameters of the pruned decoder in the pruned encoder-decoder model.

[0049] In some embodiments, when the sample multimedia data is sample audio, the comprehensive loss corresponding to the second sample multimedia data also includes the spectral loss and temporal loss of the second sample multimedia data. Based on the spectrograms of the reconstructed multimedia data and the second sample multimedia data, the energy distribution information of the reconstructed multimedia data and the second sample multimedia data at different frequencies is analyzed to calculate the spectral loss of the second sample multimedia data. The spectral loss of the second sample multimedia data is used to indicate the spectral similarity between the reconstructed multimedia data and the second sample multimedia data. By calculating the spectral loss, it can be ensured that the spectral characteristics of the second sample multimedia data are not excessively altered during encoding and decoding. Based on the waveform diagrams of the reconstructed multimedia data and the second sample multimedia data, the waveform shape, amplitude changes, and other characteristic information of the audio signal in the time domain are analyzed to calculate the temporal loss of the second sample multimedia data. The temporal loss of the second sample multimedia data is used to indicate the temporal similarity between the reconstructed multimedia data and the second sample multimedia data. By calculating the temporal loss, it can be ensured that the waveform characteristics of the second sample multimedia data are not excessively altered during encoding and decoding.

[0050] Figure 3 The diagram illustrates the architecture of the encoding / decoding model, which includes an encoder, quantizer, decoder, and discriminator. Based on the quantization process, the quantizer loss can be obtained, which includes quantization loss and entropy coding loss. The reconstructed multimedia data output from the decoder and the sample multimedia data are input into the discriminator to obtain the data loss of the sample multimedia data. When the sample multimedia data is sample audio, the spectral and temporal losses of the sample multimedia data can be obtained based on the sample multimedia data and the reconstructed multimedia data output from the decoder.

[0051] Step 250: Obtain the target encoding / decoding model based on the pre-trained decoder and the target decoder.

[0052] If the pre-trained encoder-decoder model includes only a pre-trained encoder and a pre-trained decoder, then the target encoder-decoder model includes a pre-trained decoder and a target decoder. If the pre-trained encoder-decoder model includes a pre-trained encoder, a pre-trained quantizer, and a pre-trained decoder, then the target encoder-decoder model includes a pre-trained decoder, a pre-trained quantizer, and a target decoder. The size of the target decoder is smaller than the size of the pre-trained decoder.

[0053] The technical solution provided in this application prunes the pre-trained decoder in the pre-trained codec model, removing some neurons to obtain a pruned decoder. Then, the parameters of other components in the pre-trained codec model are frozen, and the parameters of the pruned decoder are optimized to obtain the target decoder. Compared to related technologies that directly train a codec model containing a large encoder and a small decoder, the technical solution provided in this application uses a staged training approach. First, an initial codec model is trained to obtain a pre-trained codec model. Since the pre-training already ensures the feature extraction capability of the large encoder, guaranteeing the extraction of detailed information in high-bitrate scenarios, the size of the pre-trained decoder is reduced by pruning, resulting in a small decoder. This small decoder is then fine-tuned, giving it strong detail restoration capabilities while minimizing the performance loss caused by the reduction in the size of the pre-trained decoder. This improves the performance of the codec model in high-bitrate scenarios, avoids the illusion problem in related technologies, and enhances the model's encoding and decoding effect.

[0054] In some embodiments, step 230 includes at least one of sub-steps 231 to 233.

[0055] Sub-step 231: Obtain the weight parameters corresponding to at least two neurons in the pre-trained decoder. The weight parameters are used to measure the role of the neurons in the decoding process.

[0056] Neurons are the basic computational units for building a pre-trained decoder, used to reconstruct information during the decoding process. Before the first stage of training, the weight parameters of at least two neurons are equal. During the first stage of training, the weight parameters of each neuron are continuously adjusted through optimization algorithms to improve the decoding performance of the pre-trained decoder. The weight parameters of a neuron indicate the activation influence of the current neuron on downstream neurons, measuring the decoding intensity from the current neuron to the downstream neuron and reflecting the role of the current neuron in the decoding process. Therefore, the higher the weight parameter of a neuron, the stronger its activation influence on downstream neurons, and the greater its role in the decoding process; conversely, the lower the weight parameter, the weaker its activation influence on downstream neurons, and the smaller its role in the decoding process.

[0057] Sub-step 232: Based on the weight parameters corresponding to at least two neurons, obtain the proportion parameters corresponding to at least two neurons. The proportion parameters are used to indicate the importance of the weight parameters of the neurons in at least two neurons.

[0058] The proportion parameter of a neuron directly measures the importance of the neuron's weight parameter among at least two neurons, and indirectly reflects the role of the neuron in the decoding process.

[0059] In some embodiments, the sum of squared weights of at least two neurons is calculated based on the weight parameters corresponding to at least two neurons respectively; for each neuron, the proportion parameter of the neuron is obtained based on the neuron's weight parameters and the sum of squared weights of at least two neurons.

[0060] For example, the sum of squared weights of at least two neurons can be expressed as:

[0061] The proportion parameter of the i-th neuron can be expressed as:

[0062] in, Let N represent the i-th neuron, and N represent the number of neurons in the pre-trained decoder. This represents the sum of squares of the weights of at least two neurons. This represents the percentage of the i-th neuron.

[0063] In some embodiments, the sum of absolute values ​​of at least two neurons is calculated based on the weight parameters corresponding to at least two neurons respectively; for each neuron, the proportion parameter of the neuron is obtained based on the weight parameters of the neuron and the sum of absolute values ​​of at least two neurons.

[0064] For example, the sum of squared weights of at least two neurons can be expressed as:

[0065] The proportion parameter of the i-th neuron can be expressed as:

[0066] in, This represents the sum of the absolute values ​​of at least two neurons. This represents the percentage of the i-th neuron.

[0067] By using the two methods described above for calculating the proportion parameters of neurons, the importance of the weight parameters of neurons in at least two neurons can be measured relatively accurately based on the proportion parameters. Therefore, pruning based on the proportion parameters can improve the accuracy of pruning and ensure the decoding capability of the pruned decoder while reducing the size of the pre-trained decoder.

[0068] Sub-step 233: Based on the proportion parameters corresponding to at least two neurons, prune the pre-trained decoder to obtain the pruned decoder.

[0069] By performing pruning based on the proportion parameter, the neurons that have the least impact on the decoding effect are removed from the pre-trained decoder. This minimizes the performance loss caused by shrinking the pre-trained decoder and ensures the decoding capability of the pruned decoder and the performance of the pruned encoding / decoding model.

[0070] Figure 4 The diagram illustrates the training process of the codec model. After pre-training the initial codec model, which includes a large encoder, quantizer, large decoder, and discriminator, the proportion parameters of each neuron in the large decoder are calculated. Based on these proportion parameters, the large decoder is pruned to obtain a small decoder. The pruned codec model then includes a large encoder, quantizer, small decoder, and discriminator. During training of the pruned codec model, the parameters of the large encoder, quantizer, and discriminator are frozen. The parameters of the small decoder in the pruned model are then optimized to obtain the target codec model.

[0071] In some embodiments, sub-step 233 includes at least one of sub-steps 2331 to 2332.

[0072] Sub-step 2331: Based on the number of neurons in the pre-trained decoder and the target number of neurons in the pruned decoder, obtain the pruning rate corresponding to the pruning process.

[0073] For example, if the number of neurons in the pre-trained decoder is S1 and the target number of neurons in the pre-trained decoder is S2, then the pruning rate corresponding to the pruning process is 1-(S1 / S2).

[0074] Sub-step 2332: Based on the proportion parameters corresponding to at least two neurons, remove neurons in the pre-trained decoder whose proportion parameters are less than the pruning rate to obtain the pruned decoder.

[0075] If, after removing neurons whose percentage parameter is less than the pruning rate, the number of neurons in the pre-trained decoder is less than or equal to the target number, then the pre-trained decoder at this point is determined to be the pruned decoder. If, after removing neurons whose percentage parameter is less than the pruning rate, the number of neurons in the pre-trained decoder is greater than the target number, then neurons at the top of the remaining neurons are gradually removed according to their percentage parameters in ascending order, until the number of neurons in the pre-trained decoder equals the target number, then the pre-trained decoder at this point is determined to be the pruned decoder.

[0076] The pre-trained encoder, pre-trained quantizer, and pre-trained discriminator in the pre-trained codec model are retained, and the pre-trained decoder in the pre-trained codec model is replaced with a pruned decoder to obtain the pruned codec model. That is to say, only the pruned decoder is changed in the pruned codec model, and other components remain unchanged.

[0077] By removing neurons whose percentage parameter is less than the pruning rate according to the pruning rate corresponding to the pre-trained decoder, the neurons with the least impact on the decoding effect can be directly eliminated, ensuring the procedural nature of the pruning process and helping to minimize the performance loss caused by shrinking the pre-trained decoder.

[0078] In some embodiments, sub-step 233 includes sub-step 2333. The pruning schemes corresponding to sub-steps 2331-2332 and the pruning scheme corresponding to sub-step 2333 are parallel schemes.

[0079] Sub-step 2333: Based on the proportion parameters corresponding to at least two neurons and the target number of neurons in the pruned decoder, remove some neurons from the pre-trained decoder to obtain the pruned decoder. The number of neurons in the pruned decoder is less than or equal to the target number.

[0080] In addition to removing some neurons according to the pruning rate, you can also remove the neurons that have the least impact on the decoding effect directly according to the proportion parameter of neurons, so that the number of neurons in the decoder after pruning is less than or equal to the target number.

[0081] In some embodiments, the neurons with the highest target number in the pre-trained decoder are retained according to the sorting order of the proportion parameters corresponding to at least two neurons from largest to smallest, to obtain the pruned decoder.

[0082] Alternatively, neurons can be gradually removed from the top of the list according to the order of the percentage parameters corresponding to at least two neurons from smallest to largest, until the number of neurons in the pre-trained decoder equals the target number. Then, the pre-trained decoder at this point is determined as the pruned decoder.

[0083] In some embodiments, the pre-trained decoder includes at least two decoding layers, and each decoding layer includes at least two neurons. Based on the proportion parameters corresponding to the at least two neurons, the sum of the proportions corresponding to the at least two decoding layers is calculated; based on the sum of the proportions corresponding to the at least two decoding layers, some neurons in the pre-trained decoder are removed to obtain the pruned decoder.

[0084] The percentage of the decoding layer is the sum of the percentage parameters corresponding to the at least two neurons included in the decoding layer. The percentage of the decoding layer is used to indicate the role of the decoding layer in the decoding process and can be used to measure the decoding effect of the decoding layer in at least two decoding layers.

[0085] Optionally, according to the proportions of at least two decoding layers and their sorting order from largest to smallest, remove at least one decoding layer that is ranked last in the pre-trained decoder to obtain the pruned decoder.

[0086] For example, the last decoding layer in the sorting can be removed first. If the number of neurons in the pre-trained decoder after removing the last decoding layer is less than or equal to the target number, then the pre-trained decoder at this point is determined as the pruned decoder. If the number of neurons in the pre-trained decoder after removing the last decoding layer is greater than the target number, then the last decoding layer in the sorting can be removed, until the number of neurons in the pre-trained decoder is less than or equal to the target number, then the pre-trained decoder at this point is determined as the pruned decoder.

[0087] For example, at least one decoding layer in the pre-trained decoder can be removed first, according to the proportions of at least two decoding layers and the sorting order from largest to smallest. At this time, the number of neurons in the pre-trained decoder is greater than the target number. However, if another decoding layer is removed, the number of neurons in the pre-trained decoder will be less than the target number. Then, neurons in the later-ranked decoding layer can be removed step by step, according to the proportion parameters of at least two neurons in the later-ranked decoding layer from largest to smallest, until the number of neurons in the pre-trained decoder is equal to the target number. The pre-trained decoder at this time is then determined as the pruned decoder.

[0088] Optionally, based on the proportions corresponding to at least two decoding layers, at least one neuron in each decoding layer is removed to obtain the pruned decoder.

[0089] The number of neurons removed from the decoding layer is negatively correlated with the percentage of the corresponding decoding layer. The smaller the percentage of the corresponding decoding layer, the more neurons are removed from the decoding layer, and vice versa.

[0090] For example, neurons can be removed sequentially from the later-ranked decoding layers according to their respective proportions and sorted in descending order. If, during the removal process, the number of neurons in the pre-trained decoder equals the target number, then the pre-trained decoder at this point is identified as the pruned decoder. If, after the first round of removal, the number of neurons in the pre-trained decoder is still greater than or equal to the target number, a second round of removal is performed until the number of neurons in the pre-trained decoder equals the target number, at which point the pre-trained decoder is identified as the pruned decoder.

[0091] The above methods provide two pruning approaches: removing neurons from the entire layer and removing a portion of neurons within each layer. This offers more possibilities for pruning and also allows for greater flexibility in the decoding performance of the pruned decoder, enabling the selection of the pruning method that best enhances the decoding effect.

[0092] Application scenario examples: The training method for the encoding and decoding model provided in this application embodiment can be applied to the audio field, video field, image field, etc. The training method for the encoding and decoding model applied to the audio field and video field will be introduced below.

[0093] Applications in the audio field: In one application scenario, the training method for the codec model can be applied to the audio field. Figure 5 A flowchart illustrating a training method for an audio codec model according to an embodiment of this application is shown. The execution entity for each step of this method can be a computer device. The method may include at least one of the following steps 510-550.

[0094] Step 510: Obtain the pre-trained audio codec model; wherein, the pre-trained audio codec model includes a pre-trained encoder and a pre-trained decoder; the pre-trained audio codec model is obtained by training the initial audio codec model based on the first sample audio.

[0095] Step 520: Prune the pre-trained decoder to obtain the pruned decoder.

[0096] Step 530: The second sample audio is encoded using a pre-trained encoder to obtain encoded feature data, and the encoded feature data is then decoded using a pruned decoder to obtain the reconstructed audio.

[0097] Step 540: Based on the difference between the reconstructed audio and the second sample audio, optimize the parameters of the pruned decoder to obtain the target decoder.

[0098] Step 550: Obtain the target audio codec model based on the pre-trained decoder and the target decoder.

[0099] In some embodiments, the weight parameters corresponding to at least two neurons in the pre-trained decoder are obtained, and the weight parameters are used to measure the role of the neurons in the decoding process; based on the weight parameters corresponding to at least two neurons, the proportion parameters corresponding to at least two neurons are obtained, and the proportion parameters are used to indicate the importance of the weight parameters of the neurons in the at least two neurons; based on the proportion parameters corresponding to at least two neurons, the pre-trained decoder is pruned to obtain the pruned decoder.

[0100] In some embodiments, the sum of squared weights of at least two neurons is calculated based on the weight parameters corresponding to each of the at least two neurons; for each neuron, the proportion parameter of the neuron is obtained based on the neuron's weight parameters and the sum of squared weights of the at least two neurons. Alternatively, the sum of absolute values ​​of at least two neurons is calculated based on the weight parameters corresponding to each of the at least two neurons; for each neuron, the proportion parameter of the neuron is obtained based on the neuron's weight parameters and the sum of absolute values ​​of the at least two neurons.

[0101] In some embodiments, the pruning rate corresponding to the pruning process is obtained based on the number of neurons in the pre-trained decoder and the target number of neurons in the pruned decoder; based on the proportion parameters corresponding to at least two neurons, neurons in the pre-trained decoder whose proportion parameters are less than the pruning rate are removed to obtain the pruned decoder.

[0102] In some embodiments, based on the proportion parameters corresponding to at least two neurons and the target number of neurons in the pruned decoder, some neurons in the pre-trained decoder are removed to obtain the pruned decoder, wherein the number of neurons in the pruned decoder is less than or equal to the target number.

[0103] In some embodiments, the pre-trained decoder includes at least two decoding layers, and each decoding layer includes at least two neurons. Based on the descending order of the proportion parameters corresponding to the at least two neurons, the neurons with the highest number of target neurons in the pre-trained decoder are retained to obtain the pruned decoder. Alternatively, based on the proportion parameters corresponding to the at least two neurons, the sum of the proportions corresponding to the at least two decoding layers is calculated; based on the sum of the proportions corresponding to the at least two decoding layers, some neurons in the pre-trained decoder are removed to obtain the pruned decoder.

[0104] In some embodiments, at least one decoding layer in the pre-trained decoder is removed according to the descending order of the proportions of the at least two decoding layers to obtain the pruned decoder. Alternatively, at least one neuron in each decoding layer is removed according to the sum of the proportions of the at least two decoding layers to obtain the pruned decoder.

[0105] In some embodiments, the pre-trained audio codec model further includes a pre-trained quantizer, which is used to quantize the encoded feature data to obtain quantized feature data. The pruned decoder is used to decode the quantized feature data to obtain reconstructed audio. Based on the difference between the reconstructed audio and the second sample audio, the data loss of the second sample audio is obtained, which measures the similarity between the reconstructed audio and the second sample audio. Based on the generation process of the reconstructed audio, the quantizer loss of the pre-trained quantizer is obtained, which measures the information loss of the pre-trained quantizer during the quantization process. Based on the reconstructed audio and the second sample audio, the spectral loss and temporal loss of the second sample audio are obtained; wherein the spectral loss measures the spectral similarity between the reconstructed audio and the second sample audio, and the temporal loss measures the temporal similarity between the reconstructed audio and the second sample audio. Based on the data loss, the quantizer loss of the pre-trained quantizer, the spectral loss, and the temporal loss of the second sample audio, the comprehensive loss corresponding to the second sample audio is obtained. Based on the comprehensive loss corresponding to the second sample audio, the parameters of the pruned decoder are optimized to obtain the target decoder.

[0106] Applications in the video field: In one application scenario, the training method for encoding and decoding models can be applied to the video field. Figure 6 A flowchart illustrating a training method for a video encoding / decoding model according to an embodiment of this application is shown. The execution entity for each step of this method can be a computer device. The method may include at least one of steps 610-650 below.

[0107] Step 610: Obtain a pre-trained video codec model; wherein, the pre-trained video codec model includes a pre-trained encoder and a pre-trained decoder; the pre-trained video codec model is obtained by training the initial video codec model based on the first sample video.

[0108] Step 620: Prune the pre-trained decoder to obtain the pruned decoder.

[0109] Step 630: The second sample video is encoded using a pre-trained encoder to obtain encoded feature data, and the encoded feature data is decoded using a pruned decoder to obtain the reconstructed video.

[0110] Step 640: Based on the difference between the reconstructed video and the second sample video, optimize the parameters of the pruned decoder to obtain the target decoder.

[0111] Step 650: Obtain the target video encoding / decoding model based on the pre-trained decoder and the target decoder.

[0112] In some embodiments, the weight parameters corresponding to at least two neurons in the pre-trained decoder are obtained, and the weight parameters are used to measure the role of the neurons in the decoding process; based on the weight parameters corresponding to at least two neurons, the proportion parameters corresponding to at least two neurons are obtained, and the proportion parameters are used to indicate the importance of the weight parameters of the neurons in the at least two neurons; based on the proportion parameters corresponding to at least two neurons, the pre-trained decoder is pruned to obtain the pruned decoder.

[0113] In some embodiments, the sum of squared weights of at least two neurons is calculated based on the weight parameters corresponding to each of the at least two neurons; for each neuron, the proportion parameter of the neuron is obtained based on the neuron's weight parameters and the sum of squared weights of the at least two neurons. Alternatively, the sum of absolute values ​​of at least two neurons is calculated based on the weight parameters corresponding to each of the at least two neurons; for each neuron, the proportion parameter of the neuron is obtained based on the neuron's weight parameters and the sum of absolute values ​​of the at least two neurons.

[0114] In some embodiments, the pruning rate corresponding to the pruning process is obtained based on the number of neurons in the pre-trained decoder and the target number of neurons in the pruned decoder; based on the proportion parameters corresponding to at least two neurons, neurons in the pre-trained decoder whose proportion parameters are less than the pruning rate are removed to obtain the pruned decoder.

[0115] In some embodiments, based on the proportion parameters corresponding to at least two neurons and the target number of neurons in the pruned decoder, some neurons in the pre-trained decoder are removed to obtain the pruned decoder, wherein the number of neurons in the pruned decoder is less than or equal to the target number.

[0116] In some embodiments, the pre-trained decoder includes at least two decoding layers, and each decoding layer includes at least two neurons. Based on the descending order of the proportion parameters corresponding to the at least two neurons, the neurons with the highest number of target neurons in the pre-trained decoder are retained to obtain the pruned decoder. Alternatively, based on the proportion parameters corresponding to the at least two neurons, the sum of the proportions corresponding to the at least two decoding layers is calculated; based on the sum of the proportions corresponding to the at least two decoding layers, some neurons in the pre-trained decoder are removed to obtain the pruned decoder.

[0117] In some embodiments, at least one decoding layer in the pre-trained decoder is removed according to the descending order of the proportions of the at least two decoding layers to obtain the pruned decoder. Alternatively, at least one neuron in each decoding layer is removed according to the sum of the proportions of the at least two decoding layers to obtain the pruned decoder.

[0118] In some embodiments, the pre-trained video encoding / decoding model further includes a pre-trained quantizer, which is used to quantize the encoded feature data to obtain quantized feature data. The pruned decoder is used to decode the quantized feature data to obtain the reconstructed video. Based on the difference between the reconstructed video and the second sample video, the data loss of the second sample video is obtained, which measures the similarity between the reconstructed video and the second sample video. Based on the generation process of the reconstructed video, the quantizer loss of the pre-trained quantizer is obtained, which measures the information loss of the pre-trained quantizer during the quantization process. Based on the data loss of the second sample video and the quantizer loss of the pre-trained quantizer, the comprehensive loss corresponding to the second sample video is obtained. Based on the comprehensive loss corresponding to the second sample video, the parameters of the pruned decoder are optimized to obtain the target decoder.

[0119] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0120] Please refer to Figure 7 This diagram illustrates a block diagram of a training apparatus for an encoding / decoding model according to an embodiment of this application. The apparatus has the function of implementing the training method for the aforementioned encoding / decoding model; this function can be implemented in hardware or by hardware executing corresponding software. The apparatus can be the terminal device described above, or it can be installed within a terminal device. Figure 7 As shown, the device 700 may include: a model pre-training module 710, a decoder pruning module 720, a data reconstruction module 730, a decoder optimization module 740, and a model determination module 750.

[0121] The model pre-training module 710 is used to obtain a pre-trained encoder-decoder model; wherein, the pre-trained encoder-decoder model includes a pre-trained encoder and a pre-trained decoder; the pre-trained encoder-decoder model is obtained by training an initial encoder-decoder model based on first sample multimedia data.

[0122] The decoder pruning module 720 is used to prune the pre-trained decoder to obtain the pruned decoder.

[0123] The data reconstruction module 730 is used to encode the second sample multimedia data through the pre-trained encoder to obtain encoded feature data, and to decode the encoded feature data through the pruned decoder to obtain reconstructed multimedia data.

[0124] The decoder optimization module 740 is used to optimize the parameters of the pruned decoder based on the difference between the reconstructed multimedia data and the second sample multimedia data to obtain the target decoder.

[0125] The model determination module 750 is used to obtain the target encoding / decoding model based on the pre-trained decoder and the target decoder.

[0126] In some embodiments, the decoder pruning module 720 is configured to: Obtain the weight parameters corresponding to at least two neurons in the pre-trained decoder, wherein the weight parameters are used to measure the role of the neurons in the decoding process; Based on the weight parameters corresponding to the at least two neurons, the proportion parameters corresponding to the at least two neurons are obtained, and the proportion parameters are used to indicate the importance of the weight parameters of the neurons in the at least two neurons; Based on the proportion parameters corresponding to the at least two neurons, the pre-trained decoder is pruned to obtain the pruned decoder.

[0127] In some embodiments, the decoder pruning module 720 is configured to: Based on the weight parameters corresponding to the at least two neurons respectively, calculate the sum of squared weights of the at least two neurons; for each neuron, obtain the proportion parameter of the neuron based on the weight parameters of the neuron and the sum of squared weights of the at least two neurons. or, The sum of absolute values ​​of the at least two neurons is calculated based on their respective weight parameters; for each neuron, the percentage parameter of the neuron is obtained based on its weight parameters and the sum of absolute values ​​of the at least two neurons.

[0128] In some embodiments, the decoder pruning module 720 is configured to: The pruning rate corresponding to the pruning process is obtained based on the number of neurons in the pre-trained decoder and the target number of neurons in the pruned decoder. Based on the proportion parameters corresponding to the at least two neurons, neurons in the pre-trained decoder whose proportion parameters are less than the pruning rate are removed to obtain the pruned decoder.

[0129] In some embodiments, the decoder pruning module 720 is configured to: Based on the proportion parameters corresponding to the at least two neurons and the target number of neurons in the pruned decoder, some neurons in the pre-trained decoder are removed to obtain the pruned decoder, wherein the number of neurons in the pruned decoder is less than or equal to the target number.

[0130] In some embodiments, the pre-trained decoder includes at least two decoding layers, each decoding layer including at least two neurons; the decoder pruning module 720 is used for: According to the sorting order of the proportion parameters corresponding to the at least two neurons from largest to smallest, retain the target number of neurons in the pre-trained decoder that are ranked first, and obtain the pruned decoder. or, Based on the proportion parameters corresponding to the at least two neurons, calculate the sum of proportions corresponding to the at least two decoding layers; based on the sum of proportions corresponding to the at least two decoding layers, remove some neurons from the pre-trained decoder to obtain the pruned decoder.

[0131] In some embodiments, the decoder pruning module 720 is configured to: According to the proportions of the at least two decoding layers and their sorting order from largest to smallest, remove at least one decoding layer that is ranked last in the pre-trained decoder to obtain the pruned decoder. or, Based on the respective proportions of the at least two decoding layers, at least one neuron in each decoding layer is removed to obtain the pruned decoder.

[0132] In some embodiments, the pre-trained encoding / decoding model further includes a pre-trained quantizer, which is used to quantize the encoded feature data to obtain quantized feature data, and the pruned decoder is used to decode the quantized feature data to obtain the reconstructed multimedia data; the decoder optimization module 740 is used to: Based on the difference between the reconstructed multimedia data and the second sample multimedia data, the data loss of the second sample multimedia data is obtained, and the data loss is used to measure the similarity between the reconstructed multimedia data and the second sample multimedia data; Based on the process of generating the reconstructed multimedia data, the quantizer loss of the pre-trained quantizer is obtained, and the quantizer loss is used to measure the information loss of the pre-trained quantizer during the quantization process. Based on the data loss of the second sample multimedia data and the quantizer loss of the pre-trained quantizer, the comprehensive loss corresponding to the second sample multimedia data is obtained; Based on the comprehensive loss corresponding to the second sample multimedia data, the parameters of the pruned decoder are optimized to obtain the target decoder.

[0133] In some embodiments, when the second sample multimedia data is sample audio, the comprehensive loss corresponding to the second sample multimedia data further includes the spectral loss and temporal loss of the second sample multimedia data; wherein, the spectral loss is used to measure the spectral similarity between the reconstructed multimedia data and the second sample multimedia data, and the temporal loss is used to measure the temporal similarity between the reconstructed multimedia data and the second sample multimedia data.

[0134] It should be noted that the apparatus provided in the above embodiments is only illustrated by the division of the above functional modules when implementing its functions. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the content structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0135] Please refer to Figure 8 This diagram illustrates a structural block diagram of a computer device 800 provided in one embodiment of this application. The computer device 800 can be any electronic device capable of data computation, processing, and storage. The computer device 800 can be used to implement the training method for the encoding / decoding model provided in the above embodiments.

[0136] Typically, computer device 800 includes a processor 801 and a memory 802.

[0137] Processor 801 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 801 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field Programmable Gate Array), and PLA (Programmable Logic Array). Processor 801 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 801 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 801 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0138] The memory 802 may include one or more computer-readable storage media, which may be non-transitory. The memory 802 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 802 are used to store a computer program configured to be executed by one or more processors to implement the above-described encoding / decoding model training method.

[0139] Those skilled in the art will understand that Figure 8 The structure shown does not constitute a limitation on the computer device 800, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0140] In an illustrative embodiment, a computer-readable storage medium is also provided, wherein a computer program is stored in the storage medium, and the computer program, when executed by a processor of a computer device, implements the above-described training method for the encoding and decoding model. Optionally, the above-described computer-readable storage medium may be ROM (Read-Only Memory), RAM (Random Access Memory), CD-ROM (Compact Disc Read-Only Memory), magnetic tape, floppy disk, and optical data storage device, etc.

[0141] In an exemplary embodiment, a computer program product is also provided, comprising a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform the training method for the encoding / decoding model described above.

[0142] It should be noted that this application may display prompt interfaces, pop-ups, or output voice prompts before and during the collection of user data. These prompt interfaces, pop-ups, or voice prompts are used to inform users that their data is being collected. This ensures that the application only begins the steps for collecting user data after receiving confirmation from the user regarding the prompt interface or pop-up; otherwise (i.e., without user confirmation), the steps for collecting user data end, meaning no user data is collected. In other words, all user data collected by this application is processed strictly in accordance with the requirements of relevant national laws and regulations. The informed consent or separate consent of the data subject is obtained only with the user's consent and authorization. Subsequent data use and processing are conducted within the scope of laws, regulations, and the data subject's authorization, and the collection, use, and processing of relevant user data must comply with the relevant laws, regulations, and standards of the relevant countries and regions.

[0143] It should be understood that "multiple" as used herein refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. Furthermore, the step numbers described herein are merely illustrative of one possible execution order. In some other embodiments, the steps may not be executed in numerical order, such as two steps with different numbers being executed simultaneously, or two steps with different numbers being executed in the reverse order of the illustration. This application does not limit this.

[0144] The above description is merely an exemplary embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A training method for an encoding / decoding model, characterized in that, The method includes: Obtain a pre-trained encoder-decoder model; wherein the pre-trained encoder-decoder model includes a pre-trained encoder and a pre-trained decoder; the pre-trained encoder-decoder model is obtained by training an initial encoder-decoder model based on first sample multimedia data; The pre-trained decoder is pruned to obtain the pruned decoder; The second sample multimedia data is encoded by the pre-trained encoder to obtain encoded feature data, and the encoded feature data is decoded by the pruned decoder to obtain reconstructed multimedia data. Based on the difference between the reconstructed multimedia data and the second sample multimedia data, the parameters of the pruned decoder are optimized to obtain the target decoder; The target encoding / decoding model is obtained based on the pre-trained decoder and the target decoder.

2. The method according to claim 1, characterized in that, The step of pruning the pre-trained decoder to obtain the pruned decoder includes: Obtain the weight parameters corresponding to at least two neurons in the pre-trained decoder, wherein the weight parameters are used to measure the role of the neurons in the decoding process; Based on the weight parameters corresponding to the at least two neurons, the proportion parameters corresponding to the at least two neurons are obtained, and the proportion parameters are used to indicate the importance of the weight parameters of the neurons in the at least two neurons; Based on the proportion parameters corresponding to the at least two neurons, the pre-trained decoder is pruned to obtain the pruned decoder.

3. The method according to claim 2, characterized in that, The step of obtaining the proportion parameters corresponding to the at least two neurons based on the weight parameters corresponding to the at least two neurons includes: Based on the weight parameters corresponding to the at least two neurons respectively, calculate the sum of squared weights of the at least two neurons; for each neuron, obtain the proportion parameter of the neuron based on the weight parameters of the neuron and the sum of squared weights of the at least two neurons. or, The sum of absolute values ​​of the at least two neurons is calculated based on their respective weight parameters; for each neuron, the percentage parameter of the neuron is obtained based on its weight parameters and the sum of absolute values ​​of the at least two neurons.

4. The method according to claim 2 or 3, characterized in that, The step of pruning the pre-trained decoder based on the proportion parameters corresponding to the at least two neurons to obtain the pruned decoder includes: The pruning rate corresponding to the pruning process is obtained based on the number of neurons in the pre-trained decoder and the target number of neurons in the pruned decoder. Based on the proportion parameters corresponding to the at least two neurons, neurons in the pre-trained decoder whose proportion parameters are less than the pruning rate are removed to obtain the pruned decoder.

5. The method according to claim 2 or 3, characterized in that, The step of pruning the pre-trained decoder based on the proportion parameters corresponding to the at least two neurons to obtain the pruned decoder includes: Based on the proportion parameters corresponding to the at least two neurons and the target number of neurons in the pruned decoder, some neurons in the pre-trained decoder are removed to obtain the pruned decoder, wherein the number of neurons in the pruned decoder is less than or equal to the target number.

6. The method according to claim 5, characterized in that, The pre-trained decoder includes at least two decoding layers, and each decoding layer includes at least two neurons; The step of removing some neurons from the pre-trained decoder based on the proportion parameters corresponding to the at least two neurons and the target number of neurons in the pruned decoder to obtain the pruned decoder includes: According to the sorting order of the proportion parameters corresponding to the at least two neurons from largest to smallest, retain the target number of neurons in the pre-trained decoder that are ranked first, and obtain the pruned decoder. or, Based on the proportion parameters corresponding to the at least two neurons, calculate the sum of proportions corresponding to the at least two decoding layers; based on the sum of proportions corresponding to the at least two decoding layers, remove some neurons from the pre-trained decoder to obtain the pruned decoder.

7. The method according to claim 6, characterized in that, The step of removing a portion of neurons from the pre-trained decoder based on the sum of the proportions corresponding to the at least two decoding layers to obtain the pruned decoder includes: According to the proportions of the at least two decoding layers and their sorting order from largest to smallest, remove at least one decoding layer that is ranked last in the pre-trained decoder to obtain the pruned decoder. or, Based on the respective proportions of the at least two decoding layers, at least one neuron in each decoding layer is removed to obtain the pruned decoder.

8. The method according to any one of claims 1 to 7, characterized in that, The pre-trained encoding / decoding model further includes a pre-trained quantizer, which is used to quantize the encoded feature data to obtain quantized feature data. The pruned decoder is used to decode the quantized feature data to obtain the reconstructed multimedia data. The step of optimizing the parameters of the pruned decoder based on the difference between the reconstructed multimedia data and the second sample multimedia data to obtain the target decoder includes: Based on the difference between the reconstructed multimedia data and the second sample multimedia data, the data loss of the second sample multimedia data is obtained, and the data loss is used to measure the similarity between the reconstructed multimedia data and the second sample multimedia data; Based on the process of generating the reconstructed multimedia data, the quantizer loss of the pre-trained quantizer is obtained, and the quantizer loss is used to measure the information loss of the pre-trained quantizer during the quantization process. Based on the data loss of the second sample multimedia data and the quantizer loss of the pre-trained quantizer, the comprehensive loss corresponding to the second sample multimedia data is obtained; Based on the comprehensive loss corresponding to the second sample multimedia data, the parameters of the pruned decoder are optimized to obtain the target decoder.

9. The method according to claim 8, characterized in that, When the second sample multimedia data is sample audio, the comprehensive loss corresponding to the second sample multimedia data also includes the spectral loss and temporal loss of the second sample multimedia data; wherein, the spectral loss is used to measure the spectral similarity between the reconstructed multimedia data and the second sample multimedia data, and the temporal loss is used to measure the temporal similarity between the reconstructed multimedia data and the second sample multimedia data.

10. A training device for an encoding / decoding model, characterized in that, The device includes: A model pre-training module is used to obtain a pre-trained encoder-decoder model; wherein, the pre-trained encoder-decoder model includes a pre-trained encoder and a pre-trained decoder; the pre-trained encoder-decoder model is obtained by training an initial encoder-decoder model based on first sample multimedia data; The decoder pruning module is used to prune the pre-trained decoder to obtain the pruned decoder; The data reconstruction module is used to encode the second sample multimedia data through the pre-trained encoder to obtain encoded feature data, and to decode the encoded feature data through the pruned decoder to obtain reconstructed multimedia data. The decoder optimization module is used to optimize the parameters of the pruned decoder based on the difference between the reconstructed multimedia data and the second sample multimedia data to obtain the target decoder. The model determination module is used to obtain the target encoding / decoding model based on the pre-trained decoder and the target decoder.

11. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing a computer program that is loaded and executed by the processor to implement the training method for the encoding / decoding model as described in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which is loaded and executed by a processor to implement the training method of the encoding / decoding model as described in any one of claims 1 to 9.

13. A computer program product, characterized in that, The computer program product includes a computer program that is loaded and executed by a processor to implement the training method of the encoding / decoding model as described in any one of claims 1 to 9.

Images