End-to-end video coding model training and video coding method, device and equipment

Abstract

The invention discloses an end-to-end video coding model training and video coding method, device and equipment, and the method comprises the steps: obtaining an end-to-end video coding model which is obtained through the training of a video data set of a source domain; a plurality of training stages are adopted, a video data set of a target domain is utilized to train an end-to-end video coding model, in each training stage, a non-key module is firstly frozen, parameter updating of a key module is carried out based on a preset local training loss function corresponding to the training stage, and when a local training ending condition is met, the key module is updated. If yes, unfreezing the non-key module, and updating parameters based on a preset global training loss function; the preset local training loss function of the non-first training stage comprises a preset local training loss function of the previous training stage and a newly added local training loss function of the current training stage. Therefore, the stability of the training process can be guaranteed, the model performance is improved, and the video coding quality is improved.

Description

Technical Field

[0001] This application relates to the field of video processing technology, and in particular to end-to-end video coding model training and video coding methods, apparatus, and equipment. Background Technology

[0002] The performance of end-to-end video coding models is highly dependent on the distribution characteristics of the training data. When the model is transferred to a target domain that differs significantly from the original training domain, the reconstruction quality and bitrate control performance will significantly degrade due to the domain shift in feature distribution and spatiotemporal structure. Currently, to adapt to the target domain, a fine-tuning strategy is typically adopted: that is, based on the pre-trained model, some module parameters are retrained using a specific loss function based on the characteristics of the target domain. This can lead to the introduction of multiple losses at once, causing conflicts in optimization objectives, instability in the training process, and localized training, resulting in an inability to coordinate the improvement of the overall network performance, thus affecting the quality of video coding.

[0003] Therefore, ensuring the stability of the end-to-end video coding model training process and improving the performance of the end-to-end video coding model to improve video coding quality are technical problems that need to be solved by those skilled in the art. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide an end-to-end video coding model training and video coding method, apparatus, and equipment that can ensure the stability of the end-to-end video coding model training process and improve the performance of the end-to-end video coding model, thereby improving the video coding quality. The specific solution is as follows:

[0005] Firstly, this application provides an end-to-end video coding model training method, including:

[0006] Obtain an end-to-end video coding model, wherein the end-to-end video coding model is trained from a video dataset in the source domain;

[0007] The end-to-end video coding model is trained using multiple training stages and a video dataset from the target domain. In each training stage, non-critical modules in the end-to-end video coding model are frozen, and the parameters of critical modules are updated based on a preset local training loss function corresponding to that training stage. When the local training termination condition is met, the non-critical modules are unfrozen, and the parameters of both critical and non-critical modules in the end-to-end video coding model are updated based on a preset global training loss function. When the multiple training stages are completed, the current end-to-end video coding model is determined as the target end-to-end video coding model.

[0008] Among the multiple training stages, the preset local training loss function for non-first training stages includes the preset local training loss function of the previous training stage and the newly added local training loss function in the current training stage.

[0009] Optionally, when the key module is a motion estimation module, the multiple training stages include a first training stage, a second training stage, and a third training stage.

[0010] In the first training phase, non-critical modules in the end-to-end video coding model are frozen, and the parameters of the motion estimation module are updated based on the pixel-level reconstruction loss function.

[0011] In the second training phase, the non-critical modules in the end-to-end video coding model are first frozen, and the parameters of the motion estimation module are updated based on the pixel-level reconstruction loss function and the structural similarity loss function.

[0012] In the third training phase, non-critical modules in the end-to-end video coding model are first frozen, and the parameters of the motion estimation module are updated based on the pixel-level reconstruction loss function, structural similarity loss function, and perceptual loss function.

[0013] Optionally, the parameters of the motion estimation module are updated based on the pixel-level reconstruction loss function, the structural similarity loss function, and the perceptual loss function, including:

[0014] The comprehensive loss is calculated based on the pixel-level reconstruction loss function and the first weight coefficient corresponding to the pixel-level reconstruction loss function, the structural similarity loss function and the second weight system corresponding to the structural similarity loss function, the perceptual loss function and the third weight coefficient corresponding to the perceptual loss function;

[0015] The parameters of the motion estimation module are updated based on the comprehensive loss.

[0016] Optionally, updating the parameters of the key modules and non-key modules in the end-to-end video coding model based on a preset global training loss function includes:

[0017] The parameters of the critical and non-critical modules in the end-to-end video coding model are updated based on the rate-distortion loss function.

[0018] Optionally, the parameters of key modules are updated based on the preset local training loss function corresponding to this training phase, including:

[0019] The parameters of key modules are updated based on the first learning rate and the preset local training loss function corresponding to this training phase.

[0020] Accordingly, the parameters of the key modules and non-key modules in the end-to-end video coding model are updated based on a preset global training loss function, including:

[0021] The parameters of the key modules and non-key modules in the end-to-end video coding model are updated based on the second learning rate and the preset global training loss function.

[0022] The initial value of the first learning rate is higher than the initial value of the second learning rate.

[0023] Secondly, this application provides a video coding method, including:

[0024] Obtain the video to be encoded in the target domain;

[0025] The video to be encoded is input into the target end-to-end video coding model to obtain the encoding result output by the target end-to-end video coding model, wherein the target end-to-end video coding model is trained by the aforementioned end-to-end video coding model training method.

[0026] Thirdly, this application provides an end-to-end video coding model training device, comprising:

[0027] An acquisition module is used to acquire an end-to-end video coding model, wherein the end-to-end video coding model is trained from a video dataset in the source domain;

[0028] The training module is used to train the end-to-end video coding model using multiple training stages and a video dataset of the target domain. In each training stage, the non-critical modules in the end-to-end video coding model are frozen first, and the parameters of the critical modules are updated based on the preset local training loss function corresponding to the training stage. When the local training termination condition is met, the non-critical modules are unfrozen, and the parameters of the critical modules and non-critical modules in the end-to-end video coding model are updated based on the preset global training loss function. When the multiple training stages are completed, the current end-to-end video coding model is determined as the target end-to-end video coding model.

[0029] Among the multiple training stages, the preset local training loss function for non-first training stages includes the preset local training loss function of the previous training stage and the newly added local training loss function in the current training stage.

[0030] Fourthly, this application provides an electronic device, including a memory and a processor, wherein:

[0031] The memory is used to store computer programs;

[0032] The processor is configured to execute the computer program to implement the aforementioned end-to-end video coding model training method and / or the aforementioned video coding method.

[0033] Fifthly, this application provides a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the aforementioned end-to-end video coding model training method and / or the aforementioned video coding method.

[0034] Sixthly, this application provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the aforementioned end-to-end video coding model training method and / or the aforementioned video coding method.

[0035] As can be seen from the above scheme, this application provides an end-to-end video coding model training method, including: obtaining an end-to-end video coding model, wherein the end-to-end video coding model is trained from a video dataset in the source domain; employing multiple training stages and training the end-to-end video coding model using a video dataset in the target domain; in each training stage, first freezing non-critical modules in the end-to-end video coding model, and updating the parameters of critical modules based on a preset local training loss function corresponding to the training stage; when the local training termination condition is met, unfreezing the non-critical modules, and updating the parameters of the critical modules and non-critical modules in the end-to-end video coding model based on a preset global training loss function; when the multiple training stages are completed, the current end-to-end video coding model is determined as the target end-to-end video coding model; wherein the preset local training loss function of the multiple training stages (excluding the first training stage) includes the preset local training loss function of the previous training stage and the newly added local training loss function in the current training stage.

[0036] As can be seen, the beneficial effects of this application are as follows: In the process of migrating the end-to-end video coding model from the source domain to the target domain, this application adopts multiple training stages and gradually introduces multiple local training loss functions. In each training stage, the non-critical modules in the end-to-end video coding model are frozen first, and the parameters of the critical modules are updated. When the local training termination condition is met, the non-critical modules are unfrozen, and the end-to-end video coding model is trained globally based on the preset global training loss function. In this way, the loss is introduced in stages, enabling the model to gradually learn the corresponding capabilities. Furthermore, by first training the critical modules locally in each stage and then adjusting globally, the stability of the end-to-end video coding model training process can be guaranteed, and the performance of the end-to-end video coding model can be improved, thereby improving the video coding quality.

[0037] Correspondingly, the end-to-end video coding model training device, equipment, readable storage medium and product, and video coding method provided in this application also have the above-mentioned technical effects. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0039] Figure 1 A flowchart of an end-to-end video coding model training device method provided in this application embodiment;

[0040] Figure 2 This application provides a schematic diagram of scene transfer training for an end-to-end video coding model.

[0041] Figure 3 A flowchart of a video encoding method provided in this application embodiment;

[0042] Figure 4 A schematic diagram of an end-to-end video coding model training device provided in an embodiment of this application;

[0043] Figure 5 This is a structural diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0044] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0045] In recent years, EVC (End-to-End Video Compression) has gradually become a new direction replacing traditional hybrid coding frameworks such as H.265 / HEVC (High Efficiency Video Coding) and VVC (Versatile Video Coding). EVC directly models motion estimation, motion compensation, entropy coding, and reconstruction processes through deep neural networks, achieving significant compression gains and perceptual quality improvements on typical training sets (such as natural scene videos). However, the superior performance of end-to-end models is highly dependent on the distribution characteristics of the training data. When the model is transferred to a target domain that differs significantly from the original training domain (e.g., using a natural video model for game videos, surveillance videos, or virtual scene videos), the reconstruction quality and bitrate control performance will significantly decrease due to domain shift in feature distribution and spatiotemporal structure. To adapt to the target domain, a fine-tuning strategy is usually adopted: that is, based on the pre-trained model, some module parameters are retrained using a specialized loss function based on the characteristics of the target domain. When key modules in an end-to-end compression system (such as motion estimation networks, reconstruction networks, and residual prediction networks) rely on multi-level, multi-dimensional loss functions for joint optimization, several problems arise: Introducing multiple losses at once for training can easily lead to conflicting optimization objectives and unstable training; strong coupling exists between modules and the overall network, preventing new loss information from being passed to other modules during local training, resulting in uncoordinated performance improvements across the entire network; and the lack of sequential logic and hierarchy in multi-loss joint training can easily lead to gradient competition, making it difficult for the model to converge to a satisfactory equilibrium point. To address these issues, this application provides an end-to-end video coding model training scheme that ensures the stability of the training process and improves the performance of the end-to-end video coding model, thereby enhancing video coding quality.

[0046] See Figure 1 As shown in the figure, this application discloses an end-to-end video coding model training method, which includes the following steps.

[0047] Step S11: Obtain the end-to-end video coding model, wherein the end-to-end video coding model is trained from the video dataset of the source domain.

[0048] The source domain can be a natural domain, a game domain, a surveillance domain, a virtual scene domain, etc., and the corresponding video datasets are video datasets composed of natural videos, game videos, surveillance videos, and virtual scene videos, respectively. Natural videos are videos captured by cameras. For example, in the embodiments of this application, an end-to-end video coding model can be trained using the natural domain.

[0049] Step S12: Multiple training stages are employed, and the end-to-end video coding model is trained using a video dataset from the target domain. In each training stage, non-critical modules in the end-to-end video coding model are first frozen, and the parameters of critical modules are updated based on the preset local training loss function corresponding to that training stage. When the local training termination condition is met, the non-critical modules are unfrozen, and the parameters of the critical modules and non-critical modules in the end-to-end video coding model are updated based on the preset global training loss function. When the multiple training stages are completed, the current end-to-end video coding model is determined as the target end-to-end video coding model.

[0050] Among the multiple training stages, the preset local training loss function for non-first training stages includes the preset local training loss function of the previous training stage and the newly added local training loss function in the current training stage.

[0051] In this embodiment, the target domain can be a natural domain, a game domain, a surveillance domain, a virtual scene domain, etc., and the target domain is a different domain from the natural domain. Freezing can be understood as not updating parameters during local training; that is, non-critical modules in the end-to-end video coding model do not have their parameters updated, while critical modules are updated based on a preset local training loss function corresponding to this training stage. When the local training termination condition is met, the non-critical modules are unfrozen, and the parameters of both critical and non-critical modules are updated based on a preset global training loss function. The local training termination condition can be that the number of model iterations reaches a preset local iteration threshold or the loss converges. The parameters of both critical and non-critical modules in the end-to-end video coding model are updated based on the preset global training loss function. When the global training termination condition is met, the next stage of training begins. The global training termination condition can be that the number of model iterations reaches a preset global iteration threshold or the loss converges. When the last stage meets the global training termination condition, all training stages are complete.

[0052] This embodiment can identify key modules based on the types of the source and target domains; that is, the modules that cause the main performance degradation during transfer. Furthermore, multiple corresponding training loss functions can be determined as local training loss functions based on these key modules, and these functions can be sorted according to semantic hierarchy, importance, or dependency to determine the local training loss function introduced at each training stage. The semantic hierarchy can range from low to high levels, and the dependency relationship can be understood as training on a stable foundation built upon the earlier introduced loss.

[0053] In an optional implementation, when the key module is a motion estimation module, the multiple training stages include a first training stage, a second training stage, and a third training stage. In the first training stage, non-key modules in the end-to-end video coding model are first frozen, and the parameters of the motion estimation module are updated based on a pixel-level reconstruction loss function. In the second training stage, non-key modules in the end-to-end video coding model are first frozen, and the parameters of the motion estimation module are updated based on a pixel-level reconstruction loss function and a structural similarity loss function. In the third training stage, non-key modules in the end-to-end video coding model are first frozen, and the parameters of the motion estimation module are updated based on a pixel-level reconstruction loss function, a structural similarity loss function, and a perceptual loss function.

[0054] For example, if the source domain is the natural domain and the target domain is the game domain, the motion patterns and texture distribution of the target domain differ significantly from those of natural videos, making motion estimation more difficult. Therefore, the motion estimation module is a key module affecting performance. The pixel-level reconstruction loss function calculates the loss, such as mean squared error (MSE), by comparing the generated image frame (i.e., the model output image frame) with the original input image frame pixel by pixel, ensuring basic pixel reconstruction capability. The structural similarity loss function calculates the difference between the model output image frame and the original input image frame in the feature space, constraining structural similarity through the feature extraction network. The perceptual loss function compares the semantic similarity between the image frame and the original input image frame, thereby maintaining visual perception quality.

[0055] The parameter update of the motion estimation module based on the pixel-level reconstruction loss function and the structural similarity loss function includes: calculating the weighted loss based on the pixel-level reconstruction loss function and the first weight coefficient corresponding to the pixel-level reconstruction loss function, the structural similarity loss function and the second weight system corresponding to the structural similarity loss function, and updating the parameters of the motion estimation module based on the comprehensive loss.

[0056] Furthermore, updating the parameters of the motion estimation module based on the pixel-level reconstruction loss function, the structural similarity loss function, and the perceptual loss function may include: calculating a comprehensive loss based on the pixel-level reconstruction loss function and the first weight coefficient corresponding to the pixel-level reconstruction loss function, the structural similarity loss function and the second weight system corresponding to the structural similarity loss function, and the perceptual loss function and the third weight coefficient corresponding to the perceptual loss function; and updating the parameters of the motion estimation module based on the comprehensive loss.

[0057] By introducing weighting coefficients, different losses can be made to be on the same order of magnitude. The weighting coefficients can be preset or learned parameters.

[0058] In an optional implementation, updating the parameters of the critical modules and non-critical modules in the end-to-end video coding model based on a preset global training loss function includes: updating the parameters of the critical modules and non-critical modules in the end-to-end video coding model based on a rate-distortion loss function. The rate-distortion loss function can be... λ is a weighted parameter, where D is the distortion term, R is the bitrate term, and λ is the weighting parameter. The distortion term D measures the difference between the original video frame and the reconstructed video frame, while the bitrate term R measures the number of bits used to represent the encoded video.

[0059] Furthermore, the parameters of key modules are updated based on the preset local training loss function corresponding to this training phase, including: updating the parameters of key modules based on the first learning rate and the preset local training loss function corresponding to this training phase; correspondingly, the parameters of the key modules and non-key modules in the end-to-end video coding model are updated based on the preset global training loss function, including: updating the parameters of the key modules and non-key modules in the end-to-end video coding model based on the second learning rate and the preset global training loss function; wherein, the initial value of the first learning rate is higher than the initial value of the second learning rate.

[0060] Furthermore, the initial learning rate can employ a combined cosine annealing or adaptive learning rate decay strategy. This allows for the use of a larger learning rate during the training of key local modules, accelerating the module's adaptation to the target domain's feature distribution; simultaneously, cosine annealing or adaptive learning rate decay strategies can be combined to prevent overfitting and maintain gradient smoothness. A smaller learning rate is used in the global fine-tuning phase to avoid disrupting the already converged parameter distribution, and a gradient clipping mechanism is employed to ensure smooth convergence of the fine-tuning process; the fine-tuning phase is typically set to short-cycle iterations to balance the ratio of local optimization to global adaptation.

[0061] In this embodiment, the end-to-end video coding model includes an encoding module and a decoding module. During the training process, video data samples from the target domain are input, and the end-to-end video coding model encodes and decodes them to obtain the output results. The loss is calculated with the input video data samples to update the parameters.

[0062] As can be seen, in the process of migrating the end-to-end video coding model from the source domain to the target domain, the embodiments of this application employ multiple training stages, gradually introducing multiple local training loss functions. Furthermore, in each training stage, non-critical modules in the end-to-end video coding model are first frozen, and the parameters of critical modules are updated. When the local training termination condition is met, the non-critical modules are unfrozen, and the end-to-end video coding model is trained globally based on a preset global training loss function. In this way, by introducing loss in stages, the model gradually learns the corresponding capabilities. Moreover, by first training the critical modules locally in each stage and then adjusting globally, the stability of the end-to-end video coding model training process can be guaranteed, and the performance of the end-to-end video coding model can be improved, thereby improving the video coding quality.

[0063] Further, see Figure 2 As shown, Figure 2 This illustration illustrates a scene transfer training method for an end-to-end video coding model, as provided in an embodiment of this application. This application offers an end-to-end video coding domain adaptation training scheme that progressively introduces loss and alternates fine-tuning. During cross-domain transfer of the end-to-end video coding model, multiple loss functions used to train the target module are introduced sequentially, with only one new loss term added at each stage, ensuring the controllability and stability of the training. Each time a new loss term is introduced, all other modules in the network are frozen, and the target module is locally trained to stabilize its feature learning. Then, the entire end-to-end network is globally fine-tuned, allowing the newly added loss information to act on the entire network through gradient propagation, promoting parameter rebalancing among modules. Whenever a new loss term is added, the two-stage process of local module training and global network fine-tuning is repeated until all layer losses have been introduced. Through this progressive optimization-feedback correction-re-optimization training logic, the end-to-end model gradually achieves intra-domain feature adaptation from local to global and from low to high layers, ensuring the optimization order of multi-layer losses and the stability of model interaction. In other words, a new training strategy is provided, which enables the model to establish a hierarchical and progressive collaborative adaptive relationship between modules and the system while introducing multi-loss optimization, so as to achieve more efficient cross-domain transfer and compression performance recovery.

[0064] First, model initialization and target module determination are performed. Source model preparation: An end-to-end video coding model trained on a source domain (e.g., a natural video dataset) is selected as the initial model. This model typically consists of multiple sub-network modules, such as motion estimation, motion compensation, entropy coding, and reconstruction modules. Target domain key module determination: In the target domain (e.g., game videos, animated videos, or specific surveillance videos), performance degradation often concentrates on specific modules. For example, in game videos, motion patterns and texture distribution differ significantly from natural videos, making motion estimation more difficult; therefore, the motion estimation module is a key module affecting performance. Other modules (such as the entropy model or reconstruction module) are initially frozen to reduce interference.

[0065] Furthermore, the loss functions are hierarchically partitioned: multiple loss functions used to train key modules are ordered according to semantic hierarchy, importance, or dependency. There is a progressive dependency between the loss terms. As the core foundation, , These are introduced gradually as supplementary layers. For example, the goal of the motion estimation module is to estimate the optical flow (i.e., pixel-wise motion vector) between the current frame and the previous frame. Since ground truth optical flow values ​​are lacking, this module is trained using the difference between the frame obtained from the estimated optical flow warping map and the current frame as the loss. Therefore, the loss function can include: Pixel-level reconstruction loss, used to ensure basic pixel restoration capabilities; Feature-level consistency loss constrains structural similarity through the feature extraction network; Perceptual loss, used to maintain the quality of visual perception. arrive From the pixel domain to the feature domain, and then to the receptive domain, As the core foundation, and High-level features and perceptual constraints are added on top of this. The hierarchical introduction of this loss function follows the principle of dependency progression.

[0066] Furthermore, a training strategy of progressively introducing loss and alternating fine-tuning is employed. The training process uses an alternating loop structure of progressively introducing loss levels, training local modules, and fine-tuning the global network. Taking the motion estimation module as an example, the training process is as follows:

[0067] Phase 1: Basic Training Phase (Introduction) (Loss), freeze parameters of all modules except critical modules; only utilize Loss training of key module networks, enabling them to... The semantic hierarchy establishes a basic fitting capability for target domain videos; when After convergence, the entire network is unfrozen, and the rate-distortion loss function is used to perform short-cycle joint fine-tuning on the entire end-to-end system (e.g., 10%–20% of the training steps) to propagate the underlying reconstructed features. Short-cycle can be understood as the percentage of training steps in a certain phase or the total number of training steps across all phases being less than a preset value.

[0068] Phase 2: Structural Features Phase (Introduction) (loss), in retention Based on a certain weight, increase The modules other than the key modules are frozen, and the target module is retrained only under the new loss constraint so that its feature output gradually approaches the target domain structure distribution. After convergence, the entire network is unfrozen again and jointly fine-tuned with rate-distortion loss so that the feature layer information can be transmitted to other modules.

[0069] Phase 3: Perception Optimization Phase (Introduction) (Loss), added based on the first two items with a certain weight. Then, the loop of local optimization and global fine-tuning is executed again to finally obtain the complete model after cross-domain optimization.

[0070] Furthermore, this embodiment incorporates training control and optimization mechanisms. Learning rate strategy: A larger learning rate is used during training of key local modules (e.g., ...). This accelerates the module's adaptation to the target domain's feature distribution; simultaneously, cosine annealing or adaptive learning rate decay strategies can be combined to prevent overfitting and maintain gradient smoothness. A smaller learning rate (e.g., ...) is used in the global fine-tuning stage. To avoid disrupting the converged parameter distribution, a gradient pruning mechanism is employed to ensure smooth convergence of the fine-tuning process. The fine-tuning phase is typically set to short-cycle iterations to balance the ratio of local optimization to global adaptation. Loss weight settings: During the training of the layer-by-layer loss function for key modules, this scheme adopts a uniform-scale weight control strategy, with each loss term (… Weights are assigned respectively when they are introduced. This ensures that the magnitudes of losses at different levels are on the same order of magnitude, avoiding situations where high-amplitude losses dominate model updates, thus maintaining the stability of the overall optimization direction.

[0071] Thus, this application provides an adaptive training scheme for end-to-end learning-based video coding models in cross-domain scenarios based on progressive loss introduction and alternating fine-tuning, belonging to the fields of video coding and deep learning technology. Addressing the challenges of training transfer and performance degradation in end-to-end video coding models when there are significant differences between the target and source domains (e.g., natural videos vs. game videos, surveillance videos vs. medical videos), and scenarios where key modules require joint optimization using multi-level losses, this application proposes a phased progressive training mechanism when introducing multi-level losses. This method, centered on a cyclical structure of progressively introducing loss levels—local module training—global network fine-tuning, achieves efficient re-adaptation and performance reconstruction of the end-to-end video coding model in multi-domain environments. By introducing losses sequentially level by level, after the model's basic reconstruction capability stabilizes, higher-level perceptual and semantic feature constraints are gradually introduced, ensuring a clear optimization order and dependency relationship between each level of loss, significantly improving convergence and final quality. Each training stage includes local module training and global model fine-tuning. This alternating fine-tuning optimization process fully considers the interactive dependencies of the end-to-end structure, ensuring that newly added loss information is fully propagated to all network parameter layers, avoiding performance inconsistencies caused by isolated local training. This method achieves target domain adaptation through only stepwise fine-tuning, resulting in better subjective visual consistency and a more stable training process. The scheme provided in this application does not depend on a specific network structure and can be widely applied to multi-module learning models such as end-to-end video coding systems and image compression networks, exhibiting high versatility and scalability. Experiments show that this scheme outperforms schemes that introduce different loss functions for local fine-tuning at different stages followed by global fine-tuning.

[0072] Further, see Figure 3 As shown in the figure, this application provides a video encoding method, including the following steps.

[0073] Step S21: Obtain the video to be encoded in the target domain.

[0074] Step S22: Input the video to be encoded into the target end-to-end video coding model to obtain the encoding result output by the target end-to-end video coding model, wherein the target end-to-end video coding model is trained by the end-to-end video coding model training method disclosed in the foregoing embodiments.

[0075] This embodiment deploys the target end-to-end video coding model on the encoding end, which can be a server. The model encodes the video to be encoded, requiring the current frame to be encoded based on the reconstructed data of the previous frame. Therefore, the target end-to-end video coding model is deployed on the encoding end. On the decoding end, the decoding module of the target end-to-end video coding model is deployed to decode the encoded result. That is, during training, the original video data is input into the model, resulting in decoded data. The loss is calculated based on the original video data and the decoded data to update the model parameters. The model encodes first and then decodes, including both an encoding module and a decoding module. In application, the complete model is deployed on the server, and the decoding model is deployed on the client. The server needs to encode the current frame based on the decoded data of the previous frame.

[0076] This application employs multiple training stages during the transfer of an end-to-end video coding model from the source domain to the target domain. Multiple local training loss functions are gradually introduced. In each training stage, non-critical modules in the end-to-end video coding model are first frozen, and the parameters of critical modules are updated. When the local training termination condition is met, the non-critical modules are unfrozen, and the end-to-end video coding model is trained globally based on a preset global training loss function. This phased introduction of loss allows the model to gradually learn the corresponding capabilities. Furthermore, by first training critical modules locally and then adjusting globally in each stage, the stability of the end-to-end video coding model training process is ensured, and the performance of the end-to-end video coding model is improved. Using this trained end-to-end video coding model for video coding can improve video coding quality.

[0077] Further, see Figure 4 As shown, this application provides an end-to-end video coding model training device, which includes the following modules.

[0078] The acquisition module 11 is used to acquire an end-to-end video coding model, wherein the end-to-end video coding model is trained from a video dataset in the source domain;

[0079] Training module 12 is used to train the end-to-end video coding model using multiple training stages and a video dataset of the target domain. In each training stage, non-critical modules in the end-to-end video coding model are first frozen, and the parameters of critical modules are updated based on the preset local training loss function corresponding to the training stage. When the local training termination condition is met, the non-critical modules are unfrozen, and the parameters of the critical modules and non-critical modules in the end-to-end video coding model are updated based on the preset global training loss function. When the multiple training stages are completed, the current end-to-end video coding model is determined as the target end-to-end video coding model.

[0080] Among the multiple training stages, the preset local training loss function for non-first training stages includes the preset local training loss function of the previous training stage and the newly added local training loss function in the current training stage.

[0081] Specifically, the training module 12 can be used to: when the key module is a motion estimation module, the multiple training stages include a first training stage, a second training stage, and a third training stage; in the first training stage, the non-key modules in the end-to-end video coding model are first frozen, and the parameters of the motion estimation module are updated based on the pixel-level reconstruction loss function; in the second training stage, the non-key modules in the end-to-end video coding model are first frozen, and the parameters of the motion estimation module are updated based on the pixel-level reconstruction loss function and the structural similarity loss function; in the third training stage, the non-key modules in the end-to-end video coding model are first frozen, and the parameters of the motion estimation module are updated based on the pixel-level reconstruction loss function, the structural similarity loss function, and the perceptual loss function.

[0082] Furthermore, the training module 12 can be specifically used to: calculate a comprehensive loss based on the pixel-level reconstruction loss function and the first weight coefficient corresponding to the pixel-level reconstruction loss function, the structural similarity loss function and the second weight system corresponding to the structural similarity loss function, the perceptual loss function and the third weight coefficient corresponding to the perceptual loss function; and update the parameters of the motion estimation module based on the comprehensive loss.

[0083] Furthermore, the training module 12 can be specifically used to update the parameters of the key modules and non-key modules in the end-to-end video coding model based on the rate-distortion loss function.

[0084] Furthermore, the training module 12 can be specifically used to: update the parameters of key modules based on a first learning rate and a preset local training loss function corresponding to the training stage; update the parameters of the key modules and non-key modules in the end-to-end video coding model based on a second learning rate and a preset global training loss function; wherein the initial value of the first learning rate is higher than the initial value of the second learning rate.

[0085] As can be seen, in the process of migrating the end-to-end video coding model from the source domain to the target domain, the embodiments of this application employ multiple training stages, gradually introducing multiple local training loss functions. Furthermore, in each training stage, non-critical modules in the end-to-end video coding model are first frozen, and the parameters of critical modules are updated. When the local training termination condition is met, the non-critical modules are unfrozen, and the end-to-end video coding model is trained globally based on a preset global training loss function. In this way, by introducing loss in stages, the model gradually learns the corresponding capabilities. Moreover, by first training the critical modules locally in each stage and then adjusting globally, the stability of the end-to-end video coding model training process can be guaranteed, and the performance of the end-to-end video coding model can be improved, thereby improving the video coding quality.

[0086] See Figure 5 As shown in the figure, this application discloses an electronic device 20, including a processor 21 and a memory 22; wherein, the memory 22 is used to store a computer program; the processor 21 is used to execute the computer program, the end-to-end video coding model training method and / or video coding method disclosed in the foregoing embodiments.

[0087] For details regarding the training method of the end-to-end video coding model and / or the specific process of the video coding method, please refer to the relevant content disclosed in the foregoing embodiments, which will not be repeated here.

[0088] Furthermore, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk, or optical disk, and the storage method can be temporary storage or permanent storage.

[0089] In addition, the electronic device 20 also includes a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26; wherein, the power supply 23 is used to provide operating voltage for the various hardware devices on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0090] Furthermore, embodiments of this application also disclose a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the end-to-end video coding model training method and / or video coding method disclosed in the foregoing embodiments.

[0091] For details regarding the training method of the end-to-end video coding model and / or the specific process of the video coding method, please refer to the relevant content disclosed in the foregoing embodiments, which will not be repeated here.

[0092] Furthermore, embodiments of this application provide a computer program product, including a computer program / instruction, which, when executed by a processor, implements the end-to-end video coding model training method and / or video coding method disclosed in the foregoing embodiments.

[0093] For details regarding the training method of the end-to-end video coding model and / or the specific process of the video coding method, please refer to the relevant content disclosed in the foregoing embodiments, which will not be repeated here.

[0094] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0095] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0096] The foregoing has provided a detailed description of the end-to-end video coding model training and video coding method, apparatus, and device provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. An end-to-end video coding model training method, characterized in that, include: Obtain an end-to-end video coding model, wherein the end-to-end video coding model is trained from a video dataset in the source domain; The end-to-end video coding model is trained using multiple training stages and a video dataset from the target domain. In each training stage, non-critical modules in the end-to-end video coding model are frozen, and the parameters of critical modules are updated based on a preset local training loss function corresponding to that training stage. When the local training termination condition is met, the non-critical modules are unfrozen, and the parameters of both critical and non-critical modules in the end-to-end video coding model are updated based on a preset global training loss function. When the multiple training stages are completed, the current end-to-end video coding model is determined as the target end-to-end video coding model. Among the multiple training stages, the preset local training loss function for non-first training stages includes the preset local training loss function of the previous training stage and the newly added local training loss function in the current training stage.

2. The end-to-end video coding model training method according to claim 1, characterized in that, When the key module is a motion estimation module, the multiple training stages include a first training stage, a second training stage, and a third training stage. In the first training phase, non-critical modules in the end-to-end video coding model are frozen, and the parameters of the motion estimation module are updated based on the pixel-level reconstruction loss function. In the second training phase, the non-critical modules in the end-to-end video coding model are first frozen, and the parameters of the motion estimation module are updated based on the pixel-level reconstruction loss function and the structural similarity loss function. In the third training phase, non-critical modules in the end-to-end video coding model are first frozen, and the parameters of the motion estimation module are updated based on the pixel-level reconstruction loss function, structural similarity loss function, and perceptual loss function.

3. The end-to-end video coding model training method according to claim 2, characterized in that, The motion estimation module's parameters are updated based on pixel-level reconstruction loss function, structural similarity loss function, and perceptual loss function, including: The comprehensive loss is calculated based on the pixel-level reconstruction loss function and the first weight coefficient corresponding to the pixel-level reconstruction loss function, the structural similarity loss function and the second weight system corresponding to the structural similarity loss function, the perceptual loss function and the third weight coefficient corresponding to the perceptual loss function; The parameters of the motion estimation module are updated based on the comprehensive loss.

4. The end-to-end video coding model training method according to claim 2, characterized in that, The parameters of the key modules and non-key modules in the end-to-end video coding model are updated based on a preset global training loss function, including: The parameters of the critical and non-critical modules in the end-to-end video coding model are updated based on the rate-distortion loss function.

5. The end-to-end video coding model training method according to claim 1, characterized in that, Based on the preset local training loss function corresponding to this training phase, the parameters of key modules are updated, including: The parameters of key modules are updated based on the first learning rate and the preset local training loss function corresponding to this training phase. Accordingly, the parameters of the key modules and non-key modules in the end-to-end video coding model are updated based on a preset global training loss function, including: The parameters of the key modules and non-key modules in the end-to-end video coding model are updated based on the second learning rate and the preset global training loss function. The initial value of the first learning rate is higher than the initial value of the second learning rate.

6. A video encoding method, characterized in that, include: Obtain the video to be encoded in the target domain; The video to be encoded is input into the target end-to-end video coding model to obtain the encoding result output by the target end-to-end video coding model, wherein the target end-to-end video coding model is trained by the end-to-end video coding model training method according to any one of claims 1 to 5.

7. An end-to-end video coding model training device, characterized in that, include: An acquisition module is used to acquire an end-to-end video coding model, wherein the end-to-end video coding model is trained from a video dataset in the source domain; The training module is used to train the end-to-end video coding model using multiple training stages and a video dataset of the target domain. In each training stage, the non-critical modules in the end-to-end video coding model are frozen first, and the parameters of the critical modules are updated based on the preset local training loss function corresponding to the training stage. When the local training termination condition is met, the non-critical modules are unfrozen, and the parameters of the critical modules and non-critical modules in the end-to-end video coding model are updated based on the preset global training loss function. When the multiple training stages are completed, the current end-to-end video coding model is determined as the target end-to-end video coding model. Among the multiple training stages, the preset local training loss function for non-first training stages includes the preset local training loss function of the previous training stage and the newly added local training loss function in the current training stage.

8. An electronic device, characterized in that, Includes memory and processor, wherein: The memory is used to store computer programs; The processor is configured to execute the computer program to implement the end-to-end video coding model training method as described in any one of claims 1 to 5, and / or the video coding method as described in claim 6.

9. A computer-readable storage medium, characterized in that, Used to store computer programs, wherein the computer programs, when executed by a processor, implement the end-to-end video coding model training method as described in any one of claims 1 to 5, and / or the video coding method as described in claim 6.

10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instruction is executed by the processor, it implements the end-to-end video coding model training method as described in any one of claims 1 to 5, and / or the video coding method as described in claim 6.

Images