End-to-end video coding and decoding method and training method, device and equipment thereof

Abstract

The invention provides an end-to-end video coding and decoding method, and a training method, device and equipment thereof, which can be applied to the technical field of video coding and decoding. The method comprises the following steps: inputting first discrimination results respectively corresponding to an rth predetermined code rate and an (r + 1) th predetermined code rate into an adversarial loss function to obtain an adversarial loss value corresponding to the rth predetermined code rate; and obtaining a target loss value according to the confrontation loss value corresponding to the rth predetermined code rate, the first original video sample and the first reconstructed video corresponding to the rth predetermined code rate, and obtaining a target model under the condition that the target loss value or the training round meets a first predetermined condition. And under the condition that both the target loss value and the training round do not meet a first predetermined condition, updating network parameters of the initial end-to-end video coding and decoding model by using a gradient descent method according to the adversarial loss function, and taking the updated end-to-end video coding and decoding model as the initial end-to-end video coding and decoding model of the next training.

Description

Technical Field

[0001] This disclosure relates to the field of video encoding and decoding technology, and more specifically, to an end-to-end video encoding and decoding method, training method, apparatus and equipment thereof. Background Technology

[0002] In related technologies, a variable bitrate video codec model is trained using perceptual loss or adversarial loss to obtain a target video codec model. Then, the target video codec model is used to reconstruct a predetermined video, resulting in a reconstructed video.

[0003] In the process of realizing the present invention, it was found that when video reconstruction is performed using a target video encoding and decoding model based on perceptual loss or adversarial loss, the quality and detail of the reconstructed video are insufficient, which affects the visual experience. Summary of the Invention

[0004] In view of this, this disclosure provides an end-to-end video encoding and decoding method, training method, apparatus and equipment thereof.

[0005] One aspect of this disclosure provides a training method for an end-to-end video codec model, comprising: inputting a first original video sample into an initial end-to-end video codec model at a predetermined bitrate of r to obtain a first reconstructed video corresponding to the predetermined bitrate r; inputting first discrimination results corresponding to the predetermined bitrate r and the (r+1)th predetermined bitrate respectively into an adversarial loss function to obtain an adversarial loss value corresponding to the predetermined bitrate r, wherein the first discrimination result corresponding to each predetermined bitrate is a discrimination result output by a trained discriminator based on the first reconstructed video corresponding to each predetermined bitrate, the (r+1)th predetermined bitrate and the predetermined bitrate r are adjacent predetermined bitrates among a plurality of predetermined bitrates, and the adversarial loss value is calculated based on the magnitude relationship between the first discrimination results corresponding to the predetermined bitrate r and the (r+1)th predetermined bitrate respectively and the first discrimination result corresponding to the predetermined bitrate r. Based on the adversarial loss value corresponding to the r-th predetermined bitrate, the first original video sample, and the first reconstructed video corresponding to the r-th predetermined bitrate, a target loss value is obtained. If the target loss value or the number of training epochs satisfies a first predetermined condition, a target model is obtained. If neither the target loss value nor the number of training epochs satisfies the first predetermined condition, gradient descent is used to update the network parameters of the initial end-to-end video codec model according to the adversarial loss function. The updated end-to-end video codec model is then used as the initial end-to-end video codec model for the next training iteration.

[0006] According to an embodiment of this disclosure, the (r+1)th predetermined code rate is greater than the rth predetermined code rate. The step of inputting the first discrimination results corresponding to the rth and (r+1)th predetermined code rates into the adversarial loss function to obtain the adversarial loss value corresponding to the rth predetermined code rate includes: if the first discrimination result corresponding to the (r+1)th predetermined code rate is greater than the first discrimination result corresponding to the rth predetermined code rate, multiplying a first value by the first discrimination result corresponding to the rth predetermined code rate to obtain the adversarial loss value corresponding to the rth predetermined code rate. If the first discrimination result corresponding to the (r+1)th predetermined code rate is less than or equal to the first discrimination result corresponding to the rth predetermined code rate, multiplying a second value by the first discrimination result corresponding to the rth predetermined code rate to obtain the adversarial loss value corresponding to the rth predetermined code rate.

[0007] According to embodiments of this disclosure, the training method further includes: when the bitrate is a predetermined bitrate of r, inputting a first original video sample into an initial end-to-end video codec model to obtain a bitstream corresponding to the predetermined bitrate of r. Obtaining the target loss value based on the adversarial loss value corresponding to the predetermined bitrate of r, the first original video sample, and the first reconstructed video corresponding to the predetermined bitrate of r includes: inputting both the first original video sample and the first reconstructed video corresponding to the predetermined bitrate of r into a reconstruction loss function and a perceptual loss function to obtain reconstruction loss values ​​and perceptual loss values ​​corresponding to the predetermined bitrate of r, and inputting the bitstream corresponding to the predetermined bitrate of r into a bitrate loss function to obtain a bitrate loss value corresponding to the predetermined bitrate of r. The target loss value is obtained based on the adversarial loss value, reconstruction loss value, perceptual loss value, and bitrate loss value corresponding to the predetermined bitrate of r.

[0008] According to an embodiment of this disclosure, the training method further includes: in each model iteration training round, using the first original video samples to iteratively train the initial end-to-end video codec model a first predetermined number of times, wherein the bitrates corresponding to the first predetermined number of model iteration training processes are different from each other and the bitrates change from large to small, wherein the first predetermined number is equal to the total number of multiple predetermined bitrates.

[0009] According to embodiments of this disclosure, the training method further includes: when the bitrate is the maximum predetermined bitrate among multiple predetermined bitrates, inputting the first discrimination results corresponding to the maximum predetermined bitrate and the first original video sample respectively into an adversarial loss function to obtain an adversarial loss value corresponding to the maximum predetermined bitrate. Wherein, the first discrimination result corresponding to the first original video sample is the discrimination result output by the trained discriminator based on the first original video sample, and the adversarial loss value corresponding to the maximum predetermined bitrate is calculated based on the magnitude relationship between the first discrimination results corresponding to the maximum predetermined bitrate and the first original video sample, and the first discrimination result corresponding to the maximum predetermined bitrate.

[0010] According to an embodiment of this disclosure, when neither the target loss value nor the training epochs meet the first predetermined condition, updating the network parameters of the initial end-to-end video codec model using gradient descent and the adversarial loss function includes: updating the network parameters of the initial end-to-end video codec model using gradient descent and the adversarial loss function, reconstruction loss function, perceptual loss function, and bitrate loss function.

[0011] According to embodiments of this disclosure, the trained discriminator is obtained by iteratively training the discriminator for a second predetermined number of rounds when the discriminator loss value does not meet the second predetermined condition, according to the following operation: In each discriminator iterative training round, the discriminator is iteratively trained for a first predetermined number of times according to the following operation: When the bitrate is a predetermined bitrate of r, a second original video sample is input into the initial end-to-end video codec model to obtain a second reconstructed video corresponding to the predetermined bitrate of r, wherein the bitrate changes from large to small in each discriminator iterative training round, and the second original video sample used in different discriminator iterative training rounds is different. The second reconstructed video corresponding to the predetermined bitrate of r is input into the discriminator to obtain a second discrimination result corresponding to the predetermined bitrate of r. The second discrimination results corresponding to the r-th and (r+1)-th predetermined bitrates are input into a predetermined loss function to obtain the discriminator loss value. The second discrimination result corresponding to the (r+1)-th predetermined bitrate is the discriminator's output based on the second reconstructed video corresponding to the (r+1)-th predetermined bitrate. The discriminator loss value is calculated based on the difference between the second discrimination results corresponding to the r-th and (r+1)-th predetermined bitrates. Using gradient descent, the network parameters of the discriminator are updated according to the predetermined loss function, and the updated discriminator is used as the discriminator for the next training iteration.

[0012] Another aspect of this disclosure provides an end-to-end video encoding and decoding method, comprising: encoding and decoding an original video using a target model obtained by training an end-to-end video encoding and decoding model according to any one of the preceding claims, to obtain a reconstructed video.

[0013] Another aspect of this disclosure provides a training apparatus for an end-to-end video codec model, comprising: a first obtaining module, configured to input a first original video sample into an initial end-to-end video codec model at a predetermined bitrate of r, to obtain a first reconstructed video corresponding to the predetermined bitrate r; and a second obtaining module, configured to input first discrimination results corresponding to the predetermined bitrate r and the (r+1)th predetermined bitrate, respectively, into an adversarial loss function to obtain an adversarial loss value corresponding to the predetermined bitrate r, wherein the first discrimination result corresponding to each predetermined bitrate is a discrimination result output by a trained discriminator based on the first reconstructed video corresponding to each predetermined bitrate, the (r+1)th predetermined bitrate and the predetermined bitrate r are adjacent predetermined bitrates among a plurality of predetermined bitrates, and the adversarial loss value is calculated based on the magnitude relationship between the first discrimination results corresponding to the predetermined bitrate r and the (r+1)th predetermined bitrate, respectively, and the first discrimination result corresponding to the predetermined bitrate r. The third obtaining module is used to obtain a target loss value based on the adversarial loss value corresponding to the r-th predetermined bitrate, the first original video sample, and the first reconstructed video corresponding to the r-th predetermined bitrate, and to obtain a target model if the target loss value or the number of training epochs meets a first predetermined condition. The updating module is used to update the network parameters of the initial end-to-end video codec model using gradient descent based on the adversarial loss function if neither the target loss value nor the number of training epochs meets the first predetermined condition, and to use the updated end-to-end video codec model as the initial end-to-end video codec model for the next training iteration.

[0014] Another aspect of this disclosure provides an electronic device comprising:

[0015] One or more processors;

[0016] Memory, used to store one or more programs.

[0017] When the one or more programs are executed by the one or more processors, the one or more processors implement the method described above.

[0018] According to embodiments of this disclosure, by inputting the first discrimination results corresponding to the r-th predetermined bitrate and the (r+1)-th predetermined bitrate into the adversarial loss function, an adversarial loss value corresponding to the r-th predetermined bitrate is obtained. Based on the adversarial loss value corresponding to the r-th predetermined bitrate, the first original video sample, and the first reconstructed video corresponding to the r-th predetermined bitrate, a target loss value is obtained. A target model is obtained when the target loss value or the number of training epochs satisfies a first predetermined condition. This achieves a subjective quality ranking relationship between reconstructed videos at different bitrates by introducing a loss function based on chain-order constraints (i.e., the adversarial loss function). This requires the reconstruction quality to progressively improve with increasing bitrate, and ensures that the variable bitrate model only receives valid gradients when the ranking condition is met, thereby avoiding invalid or misleading updates. When neither the target loss value nor the number of training epochs satisfies the first predetermined condition, gradient descent is used to update the network parameters of the initial end-to-end video codec model according to the adversarial loss function. This design enables the variable bitrate model to obtain more fine-grained and accurate perceptual feedback, improving the quality of reconstructed videos in multi-bitrate scenarios. Attached Figure Description

[0019] The above and other objects, features and advantages of this disclosure will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0020] Figure 1 The illustration shows a comparison of images reconstructed using a variable bitrate model at different bitrates;

[0021] Figure 2 A flowchart illustrating a training method for an end-to-end video codec model according to an embodiment of the present disclosure is shown schematically.

[0022] Figure 3 This illustration schematically shows a comparison of reconstructed images from an end-to-end video encoding / decoding method according to embodiments of the present disclosure and related technologies;

[0023] Figure 4 This illustration schematically shows a comparison of artifact suppression effects of an end-to-end video encoding / decoding method according to embodiments of the present disclosure and related technologies on reconstructed images;

[0024] Figure 5 A block diagram illustrating a training apparatus for an end-to-end video codec model according to embodiments of the present disclosure is shown; and

[0025] Figure 6 The diagram illustrates a block diagram of an electronic device suitable for implementing the training method for an end-to-end video codec model described above, according to embodiments of the present disclosure. Detailed Implementation

[0026] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0027] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0028] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0029] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0030] The goal of video encoding and decoding is to reduce the amount of data in video storage and transmission, thereby reducing storage space requirements and saving transmission bandwidth. Traditional video encoding and decoding models can be composed of modules such as motion estimation, motion information encoding / decoding, motion compensation, and residual encoding / decoding. At the encoding end of the video encoding and decoding model, motion estimation is first performed using a reference frame and the current frame to obtain motion vectors, which are then encoded into a motion bitstream. Subsequently, the motion vectors are motion-compensated to assist the current frame in generating a residual signal, which is then converted into a residual bitstream by a residual encoder. The final bitstream is composed of the motion bitstream and the residual bitstream. At the decoding end of the video encoding and decoding model, the motion bitstream is first decoded to obtain reconstructed motion vectors, which are then used by motion compensation to participate in the decoding of the residual bitstream, thereby recovering the reconstructed frame.

[0031] In contrast, end-to-end video codec models leverage the feature modeling capabilities and end-to-end optimization characteristics of deep neural networks to replace individual modules in the traditional video codec process with learnable neural network modules, and perform overall optimization based on rate-distortion loss. Among related technologies, end-to-end video codec models can be divided into two categories according to the method of eliminating inter-frame redundancy: one is a residual modeling-based framework, which manually calculates the reference frame and predicts the difference between the reference frame and the current frame to obtain the residual signal and encodes it, thereby removing temporal redundancy; the other is a conditional modeling-based model, which adaptively learns the temporal correlation between the reference frame and the current frame through the network, directly modeling inter-frame redundancy under conditional input. Compared with residual coding, conditional modeling-based neural video compression models can learn the temporal context more flexibly, and their theoretical entropy lower bound is usually lower, thus achieving higher video compression efficiency.

[0032] End-to-end video encoding / decoding models can include fixed-bitrate models and variable-bitrate models. When using a fixed-bitrate model for video encoding / decoding, only one bitstream can be obtained, and the bitrate of this stream is fixed. When using a variable-bitrate model for video encoding / decoding, different bitstreams and bitrates can be obtained by controlling the modulation model's parameters (e.g., quality parameters, quality factor, quantization parameters, or rate-distortion factor), thereby enhancing the controllability of the video encoding / decoding model. Furthermore, by adjusting the video encoding bitrate, it can be adapted to different transmission conditions in communication systems.

[0033] Mainstream neural video encoding and decoding methods generally employ variable bitrate (VBR) models. Using VBR models allows for the reconstruction of videos at various bitrates using a single video encoding / decoding model, eliminating the need to train multiple models for different bitrates, thus reducing model training costs and storage overhead. During the inference phase, VBR models can flexibly adjust the bitrate according to the application scenario, allowing the video bitrate to balance transmission bandwidth and video quality requirements. Furthermore, by introducing adaptive quantization strategies, VBR models can dynamically adjust the quantization step size at different levels, such as global, channel, and spatial, to achieve fine-grained bit allocation. This allows important regions to receive more encoding resources, while less important regions are processed with a higher compression rate, resulting in smooth bitrate control and superior rate-distortion capabilities. For example, methods such as Deep Contextual Video Compression with Hybrid Entropy Modeling (DCVC-HEM), Deep Contextual Video Compression with Diverse Contexts (DCVC-DC), Deep Contextual Video Compression using Feature Modulation (DCVC-FM), and Deep Contextual Video Compression with Real-Time Capability (DCVC-RT) all utilize the design concept of variable bitrate models. These methods improve upon motion estimation, entropy modeling, and contextual condition modeling. The use of variable bitrate mechanisms enables these methods to achieve adaptive compression of video across multiple bitrate ranges within a single model.

[0034] Figure 1 The illustration shows a comparison of images reconstructed using a variable bitrate model at different bitrates.

[0035] like Figure 1 As shown, Figure 1 The images in columns (a), (b), (c), and (d) are reconstruction models obtained by using the Variable Bit Rate Model (DCVC-DC) to reconstruct the original images at different bit rates. Figure 1In the diagram, (a), (b), (c), and (d) correspond to rate index 0, rate index 1, rate index 2, and rate index 3, respectively, with the rate increasing progressively. The variable bitrate model has the ability to generate a series of reconstruction results at different bitrates, and the quality of these results improves progressively with the bitrate. For the same input, the variable bitrate model can generate a series of reconstruction results at different bitrates, and as the bitrate increases, both the objective and subjective quality of the reconstruction results should gradually improve, forming a natural hierarchical structure where "the higher the bitrate, the better the quality."

[0036] Video encoding and decoding methods can measure the degree of image quality distortion after video compression using objective metrics such as Peak Signal-to-Noise Ratio (PSNR). In some schemes, traditional end-to-end video encoding and decoding methods exhibit a smooth transition in reconstruction results at low bitrates, making it difficult to meet the quality requirements perceived by the human eye. Therefore, deep neural network-based video encoding and decoding methods introduce perceptual optimization approaches, including optimization based on perceptual loss and discriminator design combined with generative adversarial networks. In related technologies, different types of discriminators can be used to improve the subjective quality of the reconstructed video. During training, the discriminator provides perceptual feedback to the encoder (i.e., the variable bitrate video encoding and decoding model), enabling the encoder to not only optimize the objective metrics of the reconstructed video but also generate results that better match human perception. For example, discriminators can be directly introduced into the encoder structure to improve overall perception by comparing the distribution differences between reconstructed and real images; combining them with recurrent neural networks can enable the discriminator to have temporal modeling capabilities, thereby maintaining perceptual temporal consistency in video sequences; Transformer discriminators can be used to enhance feature modeling capabilities to better capture complex global dependencies; discriminators can be designed in the frequency domain or 3D space to strengthen spatiotemporal consistency and reduce artifacts; and discriminators can be configured separately in keyframes (such as I-frames) to improve the perceptual quality of keyframes and drive the overall subjective performance of subsequent frames. However, existing perceptual optimization methods have not fully utilized this characteristic of variable bitrate models, and therefore have failed to realize their potential advantages in multi-bitrate scenarios.

[0037] However, the above solutions have shortcomings in the application of variable bitrate scenarios. They do not fully utilize the characteristic that higher bitrate leads to better quality in the variable bitrate framework, and there is still room for improvement in the quality of reconstructed video under the variable bitrate framework.

[0038] Therefore, variable bitrate video encoding / decoding models can generate a series of progressively reconstructed videos at different bitrates, with quality gradually improving as the bitrate increases. This "increasing bitrate, improving quality" principle itself provides additional constraints for perceptual optimization. While related technologies apply perceptual loss or adversarial loss to variable bitrate models, this characteristic is not systematically utilized. There is still room for improvement in the quality of reconstructed videos under multi-bitrate frameworks, resulting in insufficient refinement of reconstructed video quality in variable bitrate scenarios, impacting the visual experience.

[0039] In view of this, in order to further utilize the subjective quality increment relationship brought about by the increase in bit rate, thereby realizing refined perceptual learning across bit rates, the embodiments of this disclosure provide an end-to-end video encoding and decoding method and its training method, apparatus and device, which can be applied to the field of video encoding and decoding technology.

[0040] Figure 2 A flowchart illustrating a training method for an end-to-end video codec model according to an embodiment of the present disclosure is shown.

[0041] like Figure 2 As shown, the training method for the end-to-end video codec model includes operations S210~S240.

[0042] In operation S210, with a bit rate of the r-th predetermined bit rate, the first original video sample is input into the initial end-to-end video codec model to obtain the first reconstructed video corresponding to the r-th predetermined bit rate.

[0043] The first raw video sample is the original video data used to train the initial end-to-end video codec model; it is video data that has not undergone encoding or decoding processing. Bitrate is the amount of video data transmitted or stored per unit of time, and it can be used to measure the speed of video transmission.

[0044] The initial end-to-end video codec model is an untrained codec model capable of reconstructing videos at multiple bitrates. For example, the initial end-to-end video codec model can support n bits of the first original video. rate Bitrate mode, n rate Let X be the first raw video sample of the initial end-to-end video codec model, and let r be the bitrate of the r-th predetermined bitrate. , where 0, 1, ..., n rate -1 corresponds to different bitrates, and a larger r indicates a larger bitrate. Given a predetermined bitrate of r, In this case, the first reconstructed video obtained can be .

[0045] In operation S220, the first discrimination result corresponding to the r-th predetermined code rate and the (r+1)-th predetermined code rate is input into the adversarial loss function to obtain the adversarial loss value corresponding to the r-th predetermined code rate.

[0046] The first discrimination result corresponding to each predetermined bitrate is the discrimination result output by the trained discriminator based on the first reconstructed video corresponding to each predetermined bitrate. The (r+1)th predetermined bitrate and the rth predetermined bitrate are adjacent predetermined bitrates among multiple predetermined bitrates. The adversarial loss value is calculated based on the magnitude relationship between the first discrimination results corresponding to the rth predetermined bitrate and the (r+1)th predetermined bitrate, respectively, and the first discrimination result corresponding to the rth predetermined bitrate.

[0047] The discriminator can output a quantized score, i.e., a first discrimination result, based on the first original video and the first reconstructed video generated at a predetermined bitrate r. The first discrimination result measures the subjective quality of the first reconstructed video; a higher score indicates better quality of the first reconstructed video.

[0048] The adversarial loss function is constructed based on the first discrimination result corresponding to the r-th predetermined code rate and the first discrimination result corresponding to the (r+1)-th predetermined code rate. The adversarial loss function is used to decompose the global ranking objective into multiple local ranking constraints, and the complete chain-like ranking optimization is gradually implemented during training.

[0049] The trained discriminator can be used to quantitatively evaluate the first reconstructed video and obtain the discrimination result. For example, the discriminator's processing procedure can be denoted as... Then the first reconstructed video After inputting into the discriminator, the first discrimination result can be obtained. The first discrimination result can measure the subjective quality of the first reconstructed video.

[0050] According to embodiments of this disclosure, the trained discriminator has the ability to score perceived quality. Based on this, this disclosure describes the structural consistency of the discriminator output as it increases progressively with the bit rate using a chain ordering constraint, that is, the discriminator output can explicitly reflect the positive correlation between bit rate and subjective quality. The chain ordering constraint can be expressed by formula (1).

[0051] (1)

[0052] In operation S230, the target loss value is obtained based on the adversarial loss value corresponding to the r-th predetermined bitrate, the first original video sample, and the first reconstructed video corresponding to the r-th predetermined bitrate. The target model is obtained when the target loss value or the number of training rounds meets the first predetermined condition.

[0053] The target loss value calculation function can measure the training effect of the end-to-end video codec model. By integrating multi-dimensional loss terms, it can evaluate the model's coding efficiency, reconstruction accuracy, and subjective perception quality, and provide optimization directions for model parameter updates.

[0054] The first predetermined condition can be used to determine whether the initial end-to-end video codec model has been sufficiently trained to the point where iteration can stop and the target model can be output. The first predetermined condition can be the value of the target loss or the number of training epochs.

[0055] In operation S240, if neither the target loss value nor the number of training epochs meets the first predetermined condition, the gradient descent method is used to update the network parameters of the initial end-to-end video codec model according to the adversarial loss function, and the updated end-to-end video codec model is used as the initial end-to-end video codec model for the next training.

[0056] Updating the network parameters of the initial end-to-end video codec model using gradient descent allows the network parameters to be gradually adjusted along the gradient direction of the adversarial loss function, thereby improving the performance of the end-to-end video codec model.

[0057] According to embodiments of this disclosure, by inputting the first discrimination results corresponding to the r-th predetermined bitrate and the (r+1)-th predetermined bitrate into the adversarial loss function, an adversarial loss value corresponding to the r-th predetermined bitrate is obtained. Based on the adversarial loss value corresponding to the r-th predetermined bitrate, the first original video sample, and the first reconstructed video corresponding to the r-th predetermined bitrate, a target loss value is obtained. A target model is obtained when the target loss value or the number of training epochs satisfies a first predetermined condition. This achieves a subjective quality ranking relationship between reconstructed videos at different bitrates by introducing a loss function based on chain-order constraints (i.e., the adversarial loss function). This requires the reconstruction quality to progressively improve with increasing bitrate, and ensures that the variable bitrate model only receives valid gradients when the ranking condition is met, thereby avoiding invalid or misleading updates. When neither the target loss value nor the number of training epochs satisfies the first predetermined condition, gradient descent is used to update the network parameters of the initial end-to-end video codec model according to the adversarial loss function. This design enables the variable bitrate model to obtain more fine-grained and accurate perceptual feedback, improving the quality of reconstructed videos in multi-bitrate scenarios.

[0058] The following explains how to obtain the adversarial loss value corresponding to the r-th predetermined bitrate. According to an embodiment of this disclosure, for the (r+1)-th predetermined bitrate, And the (r+1)th predetermined code rate is greater than the rth predetermined code rate.

[0059] The adversarial loss function can be expressed by formula (2).

[0060] (2)

[0061] in, It is a counter-loss value. It is an indicator function that can be used to control the efficient propagation of gradients. It is the first discrimination result corresponding to the (r+1)th predetermined bit rate. It is the first discrimination result corresponding to the r-th predetermined code rate.

[0062] According to embodiments of this disclosure, for Figure 2 The operation S220 shown, which inputs the first discrimination results corresponding to the r-th predetermined code rate and the (r+1)-th predetermined code rate into the adversarial loss function to obtain the adversarial loss value corresponding to the r-th predetermined code rate, may include the following operations: if the first discrimination result corresponding to the (r+1)-th predetermined code rate is greater than the first discrimination result corresponding to the r-th predetermined code rate, multiply the first value by the first discrimination result corresponding to the r-th predetermined code rate to obtain the adversarial loss value corresponding to the r-th predetermined code rate; if the first discrimination result corresponding to the (r+1)-th predetermined code rate is less than or equal to the first discrimination result corresponding to the r-th predetermined code rate, multiply the second value by the first discrimination result corresponding to the r-th predetermined code rate to obtain the adversarial loss value corresponding to the r-th predetermined code rate.

[0063] If the first discrimination result corresponding to the (r+1)th predetermined bitrate is greater than the first discrimination result corresponding to the rth predetermined bitrate, the indicator function is assigned a value of 1, that is, the first value is 1. In this case, the adversarial loss function can be expressed as: The first discrimination result corresponding to the r-th predetermined bit rate is the adversarial loss value.

[0064] If the first discrimination result corresponding to the (r+1)th predetermined bitrate is less than or equal to the first discrimination result corresponding to the rth predetermined bitrate, the indicator function is assigned a value of 0, i.e., the second value is 0. In this case, the adversarial loss function can be expressed as: The resistance loss value is 0.

[0065] According to embodiments of this disclosure, an indicator function is assigned a value based on a first discrimination result corresponding to the (r+1)th predetermined bitrate and a second discrimination result corresponding to the rth predetermined bitrate to obtain a first value or a second value. The first value or the second value is multiplied by the first discrimination result corresponding to the rth predetermined bitrate to obtain an adversarial loss value corresponding to the rth predetermined bitrate. This can control the effective propagation of gradients. The calculation mechanism of the adversarial loss value ensures the consistency of perceptual quality of video encoding and decoding in multi-bitrate scenarios and effectively improves the stability of adversarial training.

[0066] According to embodiments of this disclosure, Figure 2The training method for the end-to-end video codec model shown further includes: when the bit rate is a predetermined bit rate of r, inputting the first original video sample into the initial end-to-end video codec model to obtain the bit stream corresponding to the predetermined bit rate of r.

[0067] A bitstream is compressed data generated after the first original video sample has been processed by an initial end-to-end video codec model; it is a data stream that can carry video information.

[0068] The r-th predetermined bitrate is one of several predetermined bitrates. When the first original video sample is input into the initial end-to-end video codec model, the model encodes the first original video sample to obtain the bitstream corresponding to the r-th predetermined bitrate.

[0069] According to embodiments of this disclosure, for example, Figure 2 Operation S230, as shown, obtains the target loss value based on the adversarial loss value corresponding to the r-th predetermined bitrate, the first original video sample, and the first reconstructed video corresponding to the r-th predetermined bitrate. This operation may include the following steps:

[0070] The first original video sample and the first reconstructed video corresponding to the r-th predetermined bitrate are both input into the reconstruction loss function and the perceptual loss function to obtain the reconstruction loss value and the perceptual loss value corresponding to the r-th predetermined bitrate. The bitstream corresponding to the r-th predetermined bitrate is then input into the bitrate loss function to obtain the bitrate loss value corresponding to the r-th predetermined bitrate. Based on the adversarial loss value, reconstruction loss value, perceptual loss value, and bitrate loss value corresponding to the r-th predetermined bitrate, the target loss value is obtained.

[0071] The reconstruction loss function measures the difference between the pixel values ​​in the first original video sample and the first reconstructed video by calculating the squared difference between the corresponding pixels in the first original video sample and the first reconstructed video, thus obtaining the reconstruction loss value.

[0072] The perceptual loss function measures the similarity between the first original video sample and the first reconstructed video. It can use a pre-trained deep network to extract features from images in the video and calculate the distance between the features to evaluate the similarity of the videos and obtain the perceptual loss value.

[0073] The bitrate loss function can measure the compression efficiency of video encoding and quantify the data size of the bitstream, enabling end-to-end video encoding and decoding models to minimize data redundancy after encoding while ensuring the quality of reconstructed video, thereby reducing storage and transmission costs.

[0074] The calculation function for the target loss value can be expressed by formula (3).

[0075] (3)

[0076] in, R is the target loss value, and R is the bitrate loss value. It is the reconstruction loss value. It is a counter-loss value. It is a countermeasure against loss value. , , , These are the weights. For example, in In the case of four bitrate points They are 85, 170, 340 and 850 respectively; , , .

[0077] According to embodiments of this disclosure, the target loss value is calculated by using adversarial loss value, reconstruction loss value, perceptual loss value, and bitrate loss value, enabling the end-to-end video codec model to balance video compression efficiency, reconstruction accuracy, and subjective perceptual quality during video encoding and decoding, thereby making the model training process more efficient and stable.

[0078] When implementing chained sort constraints, if we use a method that maximizes the difference between all sorted chains at once, for example:

[0079] (4)

[0080] The video encoding / decoding model is prone to degenerate into a coarse-grained binary classification that only focuses on the original sample and the lowest bitrate reconstruction, ignoring the fine-grained ranking structure of the intermediate layers.

[0081] To address the aforementioned issues, this disclosure proposes a phased training strategy that goes beyond coarse-grained distinction between true and false results, further leveraging the subjective quality increment resulting from increased bitrate. This strategy equivalently decomposes the global chain-order constraint into... Local constraints between adjacent reconstruction results:

[0082] (5)

[0083] The global ranking is optimized gradually through phased training.

[0084] According to embodiments of this disclosure, Figure 2 The training method for the end-to-end video codec model further includes: in each model iteration training round, using the first original video samples to iteratively train the initial end-to-end video codec model a first predetermined number of times. The bitrates corresponding to each of the first predetermined number of model iteration training processes are different and change from large to small, wherein the first predetermined number is equal to the total number of multiple predetermined bitrates.

[0085] In each iteration of model training, the model is divided into n parts according to a first predetermined number of times. rate The training process consists of multiple phases, each maintaining only one pair of adjacent bitrate reconstruction results in a ranking relationship. The training process begins at the highest bitrate and gradually progresses to lower bitrates.

[0086] As one implementation, each of the above training phases can be performed in the encoder, which can support n... rate A variable bitrate encoder (i.e., a variable bitrate video codec model) with various bitrate modes. In each training phase, the model can cache the high bitrate reconstruction results from the previous training phase for use in the ranking constraints of the next training phase. rate The bitrate modes can be denoted as 0, 1, ..., n respectively. rate -1. For the r-th predetermined bit rate, the encoder generates the reconstruction result at the r-th predetermined bit rate. This data, along with the cached previous high-bitrate reconstruction result, is input into the trained discriminator to obtain the corresponding discrimination result. and If and only if When the model determines that the current discrimination result satisfies the ranking constraint, the encoder receives the effective gradient signal provided by the discriminator, and then utilizes the corresponding discrimination result. and Update the parameters.

[0087] For example, in the first training phase, the first The first raw video sample X at a predetermined bitrate is input to the encoder, and the encoder generates the second... First reconstructed video at the predetermined bitrate Since the first original video sample X is readily available, it can be denoted as... As the first reconstruction result under extremely high bitrate, and will and Input the discriminator to obtain the first discrimination result. and Calculate the adversarial loss value according to the adversarial loss function formula. And according to the formula for calculating the target loss value. Calculate the target loss value of the encoder, and based on the calculation results... Perform gradient backpropagation to update the encoder, while... Cached for use in the second phase.

[0088] In the second training phase, extract the cached data. At the same time, the encoder is used to generate the first First reconstructed video at the predetermined bitrate The two are then fed into a discriminator to obtain their respective discrimination scores. and Calculate the adversarial loss value according to the adversarial loss function formula. And according to the formula for calculating the target loss value. Calculate the target loss value of the encoder. And based on the calculation results... Perform gradient backpropagation to update the encoder, while... Cached for use in the third stage.

[0089] This process continues until the lowest bitrate stage, where the cached data is retrieved. Simultaneously, the encoder is used to generate the first reconstructed video at a predetermined bitrate of 0. The two are then fed into a discriminator to obtain their respective discrimination scores. and Calculate the adversarial loss value according to the adversarial loss function formula. And according to the formula for calculating the target loss value. Calculate the target loss value of the encoder. And based on the calculation results... Perform gradient backpropagation to update the encoder.

[0090] According to embodiments of this disclosure, in each iteration of model training, the model sequentially adapts to multiple predetermined bitrates, thus eliminating the need to train a model separately for a single bitrate. Furthermore, the training results from the high bitrate training phase can provide a data foundation for adjacent low bitrate training phases, enabling the model training process to follow a hierarchical pattern where higher bitrates result in better quality, thereby enhancing the model's perceptual consistency across multi-bitrate scenarios.

[0091] The initial end-to-end video codec model iterative training process starts from the highest bitrate and gradually progresses to lower bitrates. The following describes the training process when the bitrate is the maximum predetermined bitrate.

[0092] When the bitrate is the maximum predetermined bitrate among multiple predetermined bitrates, the first discrimination result corresponding to the maximum predetermined bitrate and the first original video sample is input into the adversarial loss function to obtain the adversarial loss value corresponding to the maximum predetermined bitrate.

[0093] The first discrimination result corresponding to the first original video sample is the discrimination result output by the trained discriminator based on the first original video sample. The adversarial loss value corresponding to the maximum predetermined bitrate is calculated based on the size relationship between the first discrimination results corresponding to the maximum predetermined bitrate and the first original video sample, and the first discrimination result corresponding to the maximum predetermined bitrate.

[0094] Given the maximum predetermined bitrate, the bitrate of the first original video sample can be expressed as the bitrate of the nth sample. rate Predetermined bitrate. At this point, the reconstructed video obtained from the first original video sample after passing through the encoder (i.e., the variable bitrate video codec model) can be represented as: Since there are no reconstruction results with higher bitrates for the maximum pre-determined bitrate as a reference, the first original video can be used as a reference, represented as follows: .

[0095] The first discrimination result output by the trained discriminator corresponding to the first original video sample can be expressed as: The first discrimination result corresponding to the maximum pre-defined bit rate can be expressed as: .

[0096] The adversarial loss function measures the perceptual difference between the reconstructed video at the maximum predetermined bitrate and the first original video sample, with the goal of making the reconstructed video at the maximum predetermined bitrate as close as possible to the quality of the original video.

[0097] According to embodiments of this disclosure, when the bitrate is the maximum predetermined bitrate among multiple predetermined bitrates, since there is no reconstructed video with a higher bitrate, the first original video sample can be used as the reconstructed video with a higher bitrate. This avoids the situation where the reconstructed video obtained under the maximum predetermined bitrate has no comparison, which would lead to the inability to calculate the adversarial loss value.

[0098] The initial end-to-end video codec model iterative training process, starting from the highest bitrate and gradually progressing to lower bitrates, can be divided into n... rate Each sub-stage maintains only one pair of adjacent bitrate reconstruction results in a sorting relationship. In each sub-stage, the system caches the high bitrate reconstruction results from the previous stage and uses them together with the reconstruction results at the current bitrate to participate in the sorting constraints.

[0099] For example, in the first stage, the encoder (i.e., the variable bitrate video codec model) generates... and with Forming constraints At the same time cache.

[0100] In the second stage, the encoder is generated. and cached Establish constraints At the same time cache.

[0101] This process continues until the lowest bitrate stage, at which point the encoder generates... and cached Establish constraints .

[0102] Through this step-by-step approach, the model can learn the correct ranking relationship among all adjacent bitrates, thus maintaining a complete chain structure overall. The core feature of this phased strategy is that each phase involves only one pair of adjacent reconstruction results, avoiding the problem of the global ranking objective being too complex and difficult to converge; at the same time, a caching mechanism ensures that there is always a high bitrate reference, thereby maintaining the continuity and stability of the ranking constraints.

[0103] If neither the target loss value nor the number of training epochs meets the first predetermined condition, the network parameters of the initial end-to-end video codec model are updated using gradient descent and according to the adversarial loss function, including:

[0104] Using gradient descent, the network parameters of the initial end-to-end video codec model are updated based on the adversarial loss function, reconstruction loss function, perceptual loss function, and bitrate loss function.

[0105] The loss values ​​output by the adversarial loss function, reconstruction loss function, perceptual loss function, and bitrate loss function are used as the basis for joint updates. The four loss values ​​are weighted and fused, and the gradient descent method is used to update the network parameters of the initial end-to-end video codec model to optimize the video codec performance of the model.

[0106] According to embodiments of this disclosure, when neither the target loss value nor the number of training epochs meets the first predetermined condition, gradient descent is used to update the network parameters of the initial end-to-end video codec model, providing an adjustment direction for updating the model's network parameters.

[0107] The training process of the discriminator is explained below.

[0108] According to embodiments of this disclosure, the trained discriminator is obtained by iteratively training the discriminator for a second predetermined number of rounds according to the following operation when the discriminator loss value does not meet the second predetermined condition:

[0109] In each discriminator iteration training round, the discriminator is iteratively trained a first predetermined number of times according to the following operations: Given a predetermined bitrate of r, a second original video sample is input into the initial end-to-end video codec model to obtain a second reconstructed video corresponding to the predetermined bitrate of r. In each discriminator iteration training round, the bitrate decreases, and the second original video sample used in different discriminator iteration training rounds is different. The second reconstructed video corresponding to the predetermined bitrate of r is input into the discriminator to obtain a second discrimination result corresponding to the predetermined bitrate of r. The second discrimination results corresponding to the predetermined bitrate of r and the (r+1)th predetermined bitrate are input into a predetermined loss function to obtain the discriminator loss value. The second discrimination result corresponding to the (r+1)th predetermined bitrate is the discrimination result output by the discriminator based on the second reconstructed video corresponding to the (r+1)th predetermined bitrate, and the discriminator loss value is calculated based on the difference between the second discrimination results corresponding to the predetermined bitrate of r and the (r+1)th predetermined bitrate.

[0110] Using gradient descent, the network parameters of the discriminator are updated according to a predetermined loss function, and the updated discriminator is used as the discriminator for the next training iteration.

[0111] The second predetermined condition can be used to determine whether the discriminator has been trained sufficiently. If the discriminator's loss value meets the second predetermined condition, the discriminator can be considered trained successfully, resulting in a trained discriminator. If the discriminator's loss value does not meet the second predetermined condition, the discriminator needs to be trained for a second predetermined number of epochs to fully train it, thus obtaining a trained discriminator. As one implementation, if the discriminator's loss value first meets the requirements (e.g., drops to a threshold, tends to converge), training can be terminated early, without needing to complete the second predetermined number of epochs.

[0112] According to embodiments of this disclosure, the first original video sample and the second original video sample are both original video samples, which may be the same or different. They are used to distinguish whether the original video sample is used to train the encoder (i.e., a variable bitrate video codec model) or the discriminator, and have no substantial difference. Therefore, in this disclosure, for simplicity, the same symbols are used to represent the first original video sample and the second original video sample when explaining the processes of training the encoder and training the discriminator. Similarly, for simplicity, the same symbols are used to represent the first reconstructed video and the second reconstructed video, and the same symbols are used to represent the first discrimination result and the second discrimination result.

[0113] The second discrimination result corresponding to the r-th predetermined code rate can be expressed as: The second discrimination result corresponding to the (r+1)th predetermined code rate can be expressed as: .

[0114] The predetermined loss function can be expressed by formula (6).

[0115] (6)

[0116] in, This represents the discriminator loss value. and These represent the reconstruction results for adjacent high and low bit rates, respectively.

[0117] During the discriminator training phase, the original sample X is used as the ideal lossless reconstruction result, i.e., as the highest bitrate reconstruction. The process starts from the highest bitrate point and gradually moves towards lower bitrate points. For a predetermined bitrate of r, the encoder (i.e., the variable bitrate video codec model) first generates the reconstruction result at the corresponding bitrate. and the previous high bitrate result in the cache. and Both are input into the discriminator for discrimination, and the discrimination results are obtained. and The goal of the discriminator optimization is to maximize the score difference between adjacent high and low bitrate discrimination results, ensuring that the high bitrate discrimination result consistently scores higher than the low bitrate result. The discriminator's parameters are updated based on its loss value, thus completing the discriminator optimization. Each time the discriminator training is completed, the system updates the current reconstruction results. Cache for the next (i.e., bitrate index) Continue to use it.

[0118] For example, in the first training round, there are naturally original samples. Meanwhile, the encoder generates The two are then input into the discriminator to obtain their respective discrimination results. and According to the discriminator loss function The discriminator loss value is calculated, and the loss gradient is backpropagated based on the discriminator loss value to update the discriminator. Simultaneously, the discriminator is... The cache is used for the second training iteration of the first round.

[0119] During the second training session of the first training session, the cached data was extracted. Meanwhile, the encoder generates The two are then input into the discriminator to obtain their respective discrimination results. and According to the discriminator loss function The discriminator loss value is calculated, and the loss gradient is backpropagated based on the discriminator loss value to update the discriminator. Simultaneously, the discriminator is... The cache is used for the third training iteration of the first round.

[0120] This process continues until the first round of training with the lowest bitrate, retrieving data from the cache. Meanwhile, the encoder generates The results will be fed into the discriminator to obtain their respective discrimination results. and According to the discriminator loss function The discriminator loss value is calculated, and the loss gradient is backpropagated based on the discriminator loss value to update the discriminator.

[0121] According to embodiments of this disclosure, an end-to-end video codec model can gradually learn the subjective quality degradation pattern from high bitrate to low bitrate by using a multi-round training mechanism. Using round-based training improves model stability while achieving perceptual ranking consistency and refined optimization across bitrates, thereby enhancing the model's subjective visual quality at low bitrates.

[0122] Under the multi-round training strategy proposed in this disclosure, each training round can be regarded as a local binary perceptual ranking task. That is, the discriminator needs to compare the two reconstruction results of the same input at high bitrate and low bitrate, and ensure that the high bitrate result has a higher discrimination score. Through this "high and low bitrate pairing" method, the ranking signal of perceptual quality can be gradually transmitted during the training process.

[0123] Based on the above strategy, this disclosure further proposes a Rank Chain GAN (Rc-GAN) structure. This structure does not rely solely on the binary constraint between the original sample and a single generated sample, but rather establishes an order relationship between adjacent reconstruction results under multi-bitrate conditions. The discriminator maintains only one pair of adjacent bitrates in each stage. To effectively maintain the order relationship, this disclosure uses a staged training strategy, decomposing the global chain constraint into multiple binary sub-constraints to progressively optimize the global ranking. In this process, the discriminator provides perceptual feedback by learning the ranking relationship across bitrates. By explicitly introducing chain constraints, the discriminator is guided to learn the subjective quality-increasing relationship across bitrates, thereby providing the encoder (i.e., the variable bitrate video codec model) with finer-grained and more accurate perceptual feedback. The encoder only receives valid gradients when the ranking conditions are met, thus achieving a chain transfer from high bitrate to low bitrate, making the overall training process both stable and consistent.

[0124] The following describes the training method for an end-to-end video codec model according to embodiments of this disclosure. The training method for the end-to-end video codec model is constructed based on an adversarial training process with phased ranking constraints. This training method, by alternately optimizing the discriminator and encoder, enables the model to progressively learn the hierarchical ranking relationship of subjective quality under multi-bitrate conditions.

[0125] During training, the discriminator and encoder in the end-to-end video codec model are updated alternately, with each stage including a discriminator update stage and an encoder update stage. Because the variable bitrate coding model supports... The bitrate mode is further divided into different training phases within each training stage. There are *r* training epochs, each corresponding to a predetermined bitrate at the *r*th training epoch. The input to the end-to-end video codec model is defined as the original video sample X, and the reconstructed video at the r-th predetermined bitrate is... .

[0126] Initialize the model parameters, setting the discriminator learning rate to α1, the encoder (i.e., the variable bitrate video codec model) learning rate to α2, the cropping parameter to c, the number of samples to m, and the number of bitrate points to n. rate Set the number of discriminator iterations to n before each encoder parameter update. critic .

[0127] An encoder can be represented as Set the loss function , .

[0128] Before training, the encoder's update parameters u and the discriminator's update parameters w are first initialized. Then, the rate-distortion loss is minimized. Pre-trained encoder.

[0129] Before the encoder's updated parameters u converge, the discriminator is iteratively trained n times. critic Second-rate.

[0130] Because the encoder supports n rate The discriminator training process is further divided into n steps within each training iteration, using different bitrate patterns. rate Each sub-stage corresponds to a different bitrate index. .

[0131] In each training iteration of the discriminator, the process of updating the discriminator parameters w is as follows: Select video data from dataset D. Each original video sample X from the video data is input into the encoder, and the encoder is used to generate a reconstructed image at a predetermined bitrate r. The discriminator loss value corresponding to the r-th predetermined bit rate is calculated using a preset loss function. And calculate the discriminator loss value. gradient a w Based on a corresponding to each original video sample in the video data wGiven the discriminator learning rate α1, the discriminator parameters w are updated using the RMSProp optimizer. The process of updating the discriminator parameters w using the RMSProp optimizer can be represented as follows: The discriminator parameter w is updated by pruning w.

[0132] Each original video sample X from the video data is input into the encoder, which generates a reconstructed image at a predetermined bitrate of r-1. The discriminator loss value and gradient corresponding to the (r-1)th predetermined bitrate are calculated using a preset loss function. Based on the gradient corresponding to each original video sample in the video data and the discriminator learning rate α1, the discriminator parameters w are updated using the RMSProp optimizer. This process is repeated until the reconstructed images at the 0th predetermined bitrate corresponding to each original video sample in the video data are input into the discriminator. Then, the discriminator parameters w are updated based on the discrimination results and the corresponding preset loss function.

[0133] During the training of the encoder, video data is selected from dataset D. Each original video sample X from the video data is input into the encoder, and the encoder is used to generate a reconstructed image at a predetermined bitrate r. The adversarial loss value of the encoder is calculated using the adversarial loss function. The target loss value is obtained based on the adversarial loss value corresponding to the r-th predetermined bitrate, the reconstruction loss value, the perceptual loss value, and the bitrate loss value. The gradient a is then backpropagated based on the target loss value. u Based on a corresponding to each original video sample in the video data. u Given the encoder learning rate α2, the encoder parameters are updated using the RMSProp optimizer. The process of updating the encoder parameters u using the RMSProp optimizer can be represented as follows: .

[0134] Each original video sample X from the video data is input into the encoder, which generates a reconstructed image at a predetermined bitrate of r-1. The target loss value and backpropagation gradient corresponding to the (r-1)th predetermined bitrate are calculated using the adversarial loss function. Based on the backpropagation gradient corresponding to each original video sample in the video data and the encoder learning rate α2, the encoder parameters u are updated using the RMSProp optimizer. This process is repeated until each original video sample X in the video data is input into the encoder, which generates the reconstructed image at the 0th predetermined bitrate. The target loss value and backpropagation gradient corresponding to the 0th predetermined bitrate are calculated by using the adversarial loss function. Based on the backpropagation gradient corresponding to each original video sample in the video data and the encoder learning rate α2, the encoder parameters u are updated using the RMSProp optimizer, thus completing the update of the encoder parameters u.

[0135] During training, the discriminator and encoder are updated alternately. Each training round includes a discriminator update phase and an encoder update phase. This phased training strategy enables fine-grained perceptual optimization.

[0136] In each training phase, the discriminator compares the discrimination results of only two adjacent reconstruction results and uses the output as a sorting signal to guide the encoder optimization. The encoder only receives valid gradients based on the discrimination results corresponding to adjacent bitrates when local sorting conditions are met, thereby avoiding misleading updates. The training method for end-to-end video codec models provided in this disclosure achieves cross-stage perceptual consistency and global chained sorting preservation through stage-by-stage optimization, significantly improving the subjective visual quality of video reconstruction.

[0137] This embodiment provides a training method for end-to-end video codec models that maintains the consistency of ranking in high-quality reconstruction results. During training, the global chained ranking constraint is decomposed into multiple local ranking sub-constraints, and each training stage focuses only on the ranking relationship between two adjacent reconstruction results. A key feature is that in each stage, the system caches the reconstruction result of the previous high bitrate point, which is used to form a binary discriminant constraint with the current bitrate reconstruction result, thereby gradually maintaining the global ranking structure. In this way, the training process is more stable, and the discriminator can effectively learn the local ranking relationships across stages.

[0138] This disclosure also provides an end-to-end video encoding and decoding method, comprising: encoding and decoding the original video using a target model obtained by the training method described above for the end-to-end video encoding and decoding model, to obtain a reconstructed video.

[0139] According to embodiments of this disclosure, this disclosure also provides an end-to-end video encoding and decoding apparatus, which may include: an encoding and decoding module, used to encode and decode the original video using a target model obtained by any of the training methods for end-to-end video encoding and decoding models described above, to obtain a reconstructed video.

[0140] Table 1 schematically illustrates a comparison of the video encoding and decoding performance of the end-to-end video encoding and decoding method provided in the embodiments of this disclosure and the video decoding method in related technologies on multiple public datasets.

[0141] As shown in Table 1, the end-to-end video encoding / decoding method provided in this disclosure and the video encoding / decoding methods in related technologies were tested using multiple publicly available datasets. The related technologies used the Universal Video Coding Test Model (VTM), the DCVC-DC method, and the DCVC-FM method, respectively. The multiple publicly available datasets are HEVC_B, HEVC_C, HEVC_D, HEVC_E, MCL-JCV, USTC-TD, and UVG datasets.

[0142] LPIPS, DISTS, KID, FID, and PSNR were tested respectively. YUV420 Five categories of subjective quality metrics are used. Among them, LPIPS quantifies the objective quality difference between the reconstructed video and the original video after encoding and decoding; DISTS evaluates the perceptual consistency of image structure and texture; KID measures the distributional similarity between the generated image and the real image; FID verifies the semantic and quality consistency between the reconstructed video and the original video; and PSNR... YUV420 It represents the overall peak signal-to-noise ratio of the entire YUV420 format image, reflecting the overall image quality.

[0143] In the test, no fixed I-frame interval was set, and the length of the test video sequence was 96 frames. Based on the test results of the VTM method, the percentage reduction in required bitrate of the video encoding and decoding method disclosed in this invention compared with related video encoding and decoding methods under the same image quality was compared for each dataset and each index. Negative values ​​indicate savings, and positive values ​​indicate increases.

[0144] Table 1 Comparison of Video Encoding and Decoding Effects

[0145]

[0146] The end-to-end video encoding and decoding method provided in this disclosure achieves significant performance improvements on standard datasets such as HEVC, UVG, and MCL-JCV. Using BD-rate as the evaluation metric, compared with the traditional optimal standard H.266 / VVC reference VTM model and the existing representative end-to-end DCVC-FM method, the end-to-end video encoding and decoding method provided in this disclosure achieves the best results in subjective quality metrics (LPIPS, DISTS, KID, FID), significantly reducing bitrate overhead. Simultaneously, in terms of the objective metric PSNR... YUV420 The model also maintained a high level, indicating that it can improve perception quality while maintaining perception reconstruction accuracy, demonstrating superior comprehensive coding performance.

[0147] Figure 3 The illustration shows a comparison of reconstructed images from an end-to-end video encoding / decoding method according to embodiments of the present disclosure and related technologies.

[0148] like Figure 3 As shown, at the same bitrate, the BQMall sequence and RaceHorses sequence images in the 720P resolution HEVC_C dataset are reconstructed. The end-to-end video encoding and decoding method provided in this disclosure can generate visual reconstruction effects with clearer structure and richer details under low bitrate conditions.

[0149] When reconstructing images from the BQMall sequence, the stripes of the trousers and clothing are clearly discernible. The bit-per-pixel (bpp) of the image reconstructed using VTM is 0.0195, the bpp of the image reconstructed using the DCVC-DC method is 0.0297, the bpp of the image reconstructed using the DCVC-FM method is 0.0257, and the bpp of the image reconstructed using the end-to-end video codec method provided in this embodiment is 0.0194. When reconstructing images from the RaceHorses sequence, details such as the horse's eyes and tether are more realistically restored. The bpp of the image reconstructed using VTM is 0.0332, the bpp of the image reconstructed using the DCVC-DC method is 0.0534, the bpp of the image reconstructed using the DCVC-FM method is 0.0481, and the bpp of the image reconstructed using the end-to-end video codec method provided in this embodiment is 0.0327. Overall, the end-to-end video encoding and decoding method of this disclosure is significantly superior to existing methods at the same bitrate, with a more natural subjective viewing experience and more refined detail.

[0150] Figure 4 The illustration shows a comparison of artifact suppression effects of an end-to-end video encoding / decoding method according to embodiments of the present disclosure and related technologies on reconstructed images.

[0151] like Figure 4 As shown, using the model The reconstructed image has a bpp of 0.0039, using the model. The reconstructed image has a bpp of 0.0045. While the reconstruction enhances texture details (such as hair and clothing edges), it also introduces noticeable artifacts. Using the model... The reconstructed image has a bpp of 0.0041, using the model. The reconstructed image has a bpp of 0.0037. The model provided in this disclosure... It effectively enhances texture details while significantly suppressing artifacts, and does not suffer from cumulative artifact aggravation in long sequence reconstruction, thus achieving the best subjective visual performance.

[0152] Table 2 schematically illustrates a comparison of the compression efficiency of the end-to-end video encoding / decoding method provided in the embodiments of this disclosure with other models.

[0153] As shown in Table 2, the baseline objective model Purely objective optimization lacks specificity for perception. Introducing LPIPS loss yields a model... .Model Approximately 49.8% BD-rate savings were achieved on the LPIPS metric. In the model... Based on this, a model is obtained by combining WGAN adversarial loss. .Model There is a slight improvement in LPIPS, and artifacts in early frames are somewhat mitigated. However, as the frame sequence increases, artifacts gradually worsen, affecting the overall perception quality. In contrast, the model provided in this disclosure... This further improves the BD-rate savings to approximately 79.6% on LPIPS.

[0154] Table 2 Comparison of Compression Efficiency of Video Encoding and Decoding Methods

[0155]

[0156] Figure 5 A block diagram of a training apparatus for an end-to-end video codec model according to an embodiment of the present disclosure is shown schematically.

[0157] like Figure 5 As shown, the training device 500 for the end-to-end video codec model includes a first obtaining module 510, a second obtaining module 520, a third obtaining module 530, and an updating module 540.

[0158] The first obtaining module 510 is used to input the first original video sample into the initial end-to-end video codec model when the bit rate is a predetermined bit rate of r, so as to obtain the first reconstructed video corresponding to the predetermined bit rate of r.

[0159] The second obtaining module 520 is used to input the first discrimination result corresponding to the r-th predetermined code rate and the (r+1)-th predetermined code rate into the adversarial loss function to obtain the adversarial loss value corresponding to the r-th predetermined code rate.

[0160] The first discrimination result corresponding to each predetermined bitrate is the discrimination result output by the trained discriminator based on the first reconstructed video corresponding to each predetermined bitrate. The (r+1)th predetermined bitrate and the rth predetermined bitrate are adjacent predetermined bitrates among multiple predetermined bitrates. The adversarial loss value is calculated based on the size relationship between the first discrimination results corresponding to the rth predetermined bitrate and the (r+1)th predetermined bitrate, respectively, and the first discrimination result corresponding to the rth predetermined bitrate.

[0161] The third module 530 is used to obtain a target loss value based on the adversarial loss value corresponding to the r-th predetermined bitrate, the first original video sample, and the first reconstructed video corresponding to the r-th predetermined bitrate, and to obtain a target model if the target loss value or the number of training rounds meets the first predetermined condition.

[0162] The update module 540 is used to update the network parameters of the initial end-to-end video codec model using gradient descent and adversarial loss function when the target loss value and training epochs do not meet the first predetermined condition. The updated end-to-end video codec model is then used as the initial end-to-end video codec model for the next training.

[0163] It should be noted that the training device part for the end-to-end video codec model in the embodiments of this disclosure corresponds to the training method part for the end-to-end video codec model in the embodiments of this disclosure. For a detailed description of the training device part for the end-to-end video codec model, please refer to the training method part for the end-to-end video codec model, which will not be repeated here.

[0164] Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure, or at least part of the functions of any one or more of them, can be implemented in one module. Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be implemented by dividing them into multiple modules. Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be at least partially implemented as hardware circuitry, such as a Field-Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a System-on-Chip, a System-on-a-Substrate, a System-on-Package, an Application-Specific Integrated Circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be at least partially implemented as computer program modules, which, when run, can perform corresponding functions.

[0165] For example, any plurality of the first obtaining module 510, the second obtaining module 520, the third obtaining module 530, and the updating module 540 can be combined into one module / unit / subunit, or any one of these modules / units / subunits can be split into multiple modules / units / subunits. Alternatively, at least part of the functionality of one or more of these modules / units / subunits can be combined with at least part of the functionality of other modules / units / subunits and implemented in one module / unit / subunit. According to embodiments of this disclosure, at least one of the first obtaining module 510, the second obtaining module 520, the third obtaining module 530, and the updating module 540 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging the circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the first obtaining module 510, the second obtaining module 520, the third obtaining module 530, and the updating module 540 may be implemented at least partially as a computer program module, which can perform corresponding functions when the computer program module is run.

[0166] Figure 6 A block diagram of an electronic device suitable for implementing the training method for end-to-end video encoding and decoding described above, according to embodiments of the present disclosure, is illustrated. Figure 6 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0167] like Figure 6 As shown, an electronic device 600 according to an embodiment of this disclosure includes a processor 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage portion 608 into a random access memory (RAM) 603. The processor 601 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 601 may also include onboard memory for caching purposes. The processor 601 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this disclosure.

[0168] RAM 603 stores various programs and data required for the operation of electronic device 600. Processor 601, ROM 602, and RAM 603 are interconnected via bus 604. Processor 601 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 602 and / or RAM 603. It should be noted that the programs may also be stored in one or more memories other than ROM 602 and RAM 603. Processor 601 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in said one or more memories.

[0169] According to embodiments of this disclosure, the electronic device 600 may further include an input / output (I / O) interface 605, which is also connected to a bus 604. The electronic device 600 may also include one or more of the following components connected to the input / output (I / O) interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the input / output (I / O) interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 610 as needed so that computer programs read from it can be installed into the storage section 608 as needed.

[0170] According to embodiments of this disclosure, the method flow according to embodiments of this disclosure can be implemented as a computer software program. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the methods shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network via communication section 609, and / or installed from removable medium 611. When the computer program is executed by processor 601, it performs the functions defined in the system of embodiments of this disclosure. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0171] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0172] According to embodiments of this disclosure, the computer-readable storage medium can be a non-volatile computer-readable storage medium. Examples include, but are not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0173] For example, according to embodiments of this disclosure, a computer-readable storage medium may include the ROM 602 and / or RAM 603 described above and / or one or more memories other than ROM 602 and RAM 603.

[0174] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods provided in the embodiments of this disclosure. When the computer program product is run on an electronic device, the program code is used to enable the electronic device to implement the methods provided in the embodiments of this disclosure.

[0175] When the computer program is executed by the processor 601, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0176] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and downloaded and installed via the communication section 609, and / or installed from the removable medium 611. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0177] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on a user's computing device, partially on a user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0178] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions. Those skilled in the art will understand that the features recited in the various embodiments and / or claims of this disclosure can be combined and / or combined in various ways, even if such combinations or combinations are not expressly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure may be combined and / or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0179] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A training method for an end-to-end video codec model, comprising: With a bit rate of r-th predetermined bit rate, the first original video sample is input into the initial end-to-end video codec model to obtain the first reconstructed video corresponding to the r-th predetermined bit rate; The first discrimination results corresponding to the r-th predetermined bitrate and the (r+1)-th predetermined bitrate are respectively input into the adversarial loss function to obtain the adversarial loss value corresponding to the r-th predetermined bitrate. The first discrimination result corresponding to each predetermined bitrate is the discrimination result output by the trained discriminator based on the first reconstructed video corresponding to each predetermined bitrate. The (r+1)-th predetermined bitrate and the r-th predetermined bitrate are adjacent predetermined bitrates among a plurality of predetermined bitrates. The adversarial loss value is calculated based on the size relationship between the first discrimination results corresponding to the r-th predetermined bitrate and the (r+1)-th predetermined bitrate, and the first discrimination result corresponding to the r-th predetermined bitrate. Based on the adversarial loss value corresponding to the r-th predetermined bitrate, the first original video sample, and the first reconstructed video corresponding to the r-th predetermined bitrate, the target loss value is obtained, and the target model is obtained when the target loss value or the number of training rounds meets the first predetermined condition. If neither the target loss value nor the training rounds meet the first predetermined condition, the network parameters of the initial end-to-end video codec model are updated using gradient descent and the adversarial loss function, and the updated end-to-end video codec model is used as the initial end-to-end video codec model for the next training.

2. The training method according to claim 1, wherein, The (r+1)th predetermined code rate is greater than the rth predetermined code rate; The step of inputting the first discrimination result corresponding to the r-th predetermined code rate and the (r+1)-th predetermined code rate into the adversarial loss function to obtain the adversarial loss value corresponding to the r-th predetermined code rate includes: If the first discrimination result corresponding to the (r+1)th predetermined code rate is greater than the first discrimination result corresponding to the rth predetermined code rate, the first value is multiplied by the first discrimination result corresponding to the rth predetermined code rate to obtain the adversarial loss value corresponding to the rth predetermined code rate. If the first discrimination result corresponding to the (r+1)th predetermined code rate is less than or equal to the first discrimination result corresponding to the rth predetermined code rate, the second value is multiplied by the first discrimination result corresponding to the rth predetermined code rate to obtain the adversarial loss value corresponding to the rth predetermined code rate.

3. The training method according to claim 1 or 2, wherein, The training method also includes: With a bit rate of r-th predetermined bit rate, the first original video sample is input into the initial end-to-end video codec model to obtain the bit stream corresponding to the r-th predetermined bit rate; The step of obtaining the target loss value based on the adversarial loss value corresponding to the r-th predetermined bitrate, the first original video sample, and the first reconstructed video corresponding to the r-th predetermined bitrate includes: The first original video sample and the first reconstructed video corresponding to the r-th predetermined bit rate are both input into the reconstruction loss function and the perceptual loss function to obtain the reconstruction loss value and the perceptual loss value corresponding to the r-th predetermined bit rate. The bit stream corresponding to the r-th predetermined bit rate is input into the bit rate loss function to obtain the bit rate loss value corresponding to the r-th predetermined bit rate. The target loss value is obtained based on the adversarial loss value, reconstruction loss value, perception loss value, and bitrate loss value corresponding to the r-th predetermined bitrate.

4. The training method according to claim 3, wherein, The training method further includes: in each model iteration training round, using the first original video samples to iteratively train the initial end-to-end video codec model a first predetermined number of times, wherein the bitrates corresponding to the first predetermined number of model iteration training processes are different from each other and the bitrates change from large to small, wherein the first predetermined number is equal to the total number of multiple predetermined bitrates.

5. The training method according to claim 4, wherein, The training method also includes: When the bitrate is the maximum predetermined bitrate among multiple predetermined bitrates, the first discrimination result corresponding to the maximum predetermined bitrate and the first original video sample are respectively input into the adversarial loss function to obtain the adversarial loss value corresponding to the maximum predetermined bitrate. The first discrimination result corresponding to the first original video sample is the discrimination result output by the trained discriminator based on the first original video sample. The adversarial loss value corresponding to the maximum predetermined bitrate is calculated based on the size relationship between the first discrimination results corresponding to the maximum predetermined bitrate and the first original video sample, and the first discrimination result corresponding to the maximum predetermined bitrate.

6. The training method according to claim 3, wherein, The step of updating the network parameters of the initial end-to-end video codec model using gradient descent and according to the adversarial loss function when neither the target loss value nor the training epochs meet the first predetermined condition includes: Using gradient descent, the network parameters of the initial end-to-end video codec model are updated based on the adversarial loss function, reconstruction loss function, perceptual loss function, and bitrate loss function.

7. The training method according to claim 4, wherein, The trained discriminator is obtained by iteratively training the discriminator for a second predetermined number of rounds according to the following operation when the discriminator loss value does not meet the second predetermined condition: In each discriminator iteration training round, the discriminator is iteratively trained a first predetermined number of times according to the following operation: Given a predetermined bitrate of r, the second original video sample is input into the initial end-to-end video codec model to obtain the second reconstructed video corresponding to the predetermined bitrate of r. In each discriminator iteration training round, the bitrate changes from large to small, and the second original video sample used in different discriminator iteration training rounds is different. The second reconstructed video corresponding to the r-th predetermined bit rate is input into the discriminator to obtain the second discrimination result corresponding to the r-th predetermined bit rate; The second discrimination results corresponding to the r-th predetermined bitrate and the (r+1)-th predetermined bitrate are respectively input into the predetermined loss function to obtain the discriminator loss value. The second discrimination result corresponding to the (r+1)-th predetermined bitrate is the discrimination result output by the discriminator based on the second reconstructed video corresponding to the (r+1)-th predetermined bitrate. The discriminator loss value is calculated based on the difference between the second discrimination results corresponding to the r-th predetermined bitrate and the (r+1)-th predetermined bitrate. Using gradient descent, the network parameters of the discriminator are updated according to the predetermined loss function, and the updated discriminator is used as the discriminator for the next training iteration.

8. An end-to-end video encoding / decoding method, comprising: Using the target model obtained by the training method for end-to-end video encoding and decoding models according to any one of claims 1 to 7, the original video is encoded and decoded to obtain the reconstructed video.

9. A training apparatus for an end-to-end video codec model, comprising: The first obtaining module is used to input the first original video sample into the initial end-to-end video codec model when the bit rate is a predetermined bit rate of r, and obtain the first reconstructed video corresponding to the predetermined bit rate of r. The second obtaining module is used to input the first discrimination results corresponding to the r-th predetermined bitrate and the (r+1)-th predetermined bitrate respectively into the adversarial loss function to obtain the adversarial loss value corresponding to the r-th predetermined bitrate. The first discrimination result corresponding to each predetermined bitrate is the discrimination result output by the trained discriminator based on the first reconstructed video corresponding to each predetermined bitrate. The (r+1)-th predetermined bitrate and the r-th predetermined bitrate are adjacent predetermined bitrates among a plurality of predetermined bitrates. The adversarial loss value is calculated based on the size relationship between the first discrimination results corresponding to the r-th predetermined bitrate and the (r+1)-th predetermined bitrate respectively and the first discrimination result corresponding to the r-th predetermined bitrate. The third obtaining module is used to obtain the target loss value based on the adversarial loss value corresponding to the r-th predetermined bitrate, the first original video sample and the first reconstructed video corresponding to the r-th predetermined bitrate, and to obtain the target model when the target loss value or the training rounds meet the first predetermined condition. The update module is used to update the network parameters of the initial end-to-end video codec model using gradient descent and according to the adversarial loss function when neither the target loss value nor the training rounds meet the first predetermined condition, and to use the updated end-to-end video codec model as the initial end-to-end video codec model for the next training.

10. An electronic device, comprising: One or more processors; Memory, used to store one or more programs. Wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1 to 8.

Images