Video coding adaptive optimization method based on generative adversarial network

Abstract

This invention discloses an adaptive optimization method for video encoding and decoding based on generative adversarial networks, comprising: S1: acquiring video stream data and preprocessing it; S2: extracting basic image features and encoding features from the video stream; then calculating cross-modal fusion features and region enhancement features, and finally calculating the optimized encoding parameter set; S3: encoding the preprocessed video stream data to obtain a continuous bitstream conforming to the H.264 standard; S4: calculating the initial decoded video frame sequence and defect region features, inputting them into a timing correction generator to obtain the repaired image, and then outputting the repair optimization signal; S5: optimizing the generator parameters based on the repair optimization signal, and outputting the repaired video frame sequence. This invention can solve the problems of insufficient accuracy in motion vector prediction and blurring / ghosting after decoding in traditional methods for dynamic image encoding and decoding.

Description

Technical Field

[0001] This invention relates to the field of video encoding and decoding technology, and in particular to an adaptive optimization method for video encoding and decoding based on generative adversarial networks. Background Technology

[0002] H.264 encoding technology, as one of the mainstream standards in the current video encoding and decoding field, is widely used in many important scenarios such as streaming media transmission, surveillance video storage, and mobile video playback due to its high-efficiency compression performance. Its core achieves efficient compression and transmission of video data through key steps such as macroblock motion estimation, intra / inter-frame prediction, and residual quantization.

[0003] In traditional H.264 encoding schemes, motion vector prediction often relies on traditional greedy search algorithms such as diamond search and hexagonal search. Although these algorithms have low computational complexity, they are difficult to accurately capture the macroblock motion trajectory of fast-moving targets in dynamic scenes, which can easily lead to motion vector prediction errors and thus affect encoding efficiency and image quality preservation. On the decoding end, entropy decoding, inverse quantization, inverse transform, and inter-frame prediction reconstruction are mainly performed according to the H.264 standard process. There is a lack of special optimization mechanisms for dynamic scenes, which makes dynamic scenes generally have defects such as blur, ghosting, and color blocks after decoding. Lost details cannot be effectively restored, and the inter-frame coherence is poor.

[0004] With the increasing penetration of deep learning technology in video encoding and decoding, existing solutions often employ models such as Convolutional Neural Networks (CNN) and Generative Adversarial Networks (GAN) to optimize encoding parameters or the decoding and repair process. For example, CNN models are used to optimize the prediction accuracy of motion vectors, or GAN models are used to complete the details of the decoded defective frames. However, these solutions still have significant drawbacks: deep learning models at the encoding end often focus more on the macroscopic balance between overall bitrate and image quality, and are insufficient in the fine-grained calculation of macroblock motion trajectories in dynamic scenes. They fail to form targeted optimization strategies, resulting in motion vector prediction accuracy that is still insufficient to meet the encoding requirements of dynamic images. Repair models at the decoding end mostly focus on repairing defects in single-frame images, and do not adequately consider the temporal correlation between frames in dynamic scenes. The repair effect fails to be specifically adapted to the needs of dynamic scenes, making it difficult to solve problems such as blur, ghosting, and poor inter-frame continuity. Lost details are also difficult to be effectively restored. Summary of the Invention

[0005] In view of this, the present invention aims to provide an adaptive optimization method for video encoding and decoding based on generative adversarial networks, so as to solve the problems of insufficient accuracy of motion vector prediction and blurring after decoding in traditional methods in dynamic picture encoding and decoding.

[0006] An adaptive optimization method for video encoding and decoding based on generative adversarial networks includes: S1: Acquire video stream data and preprocess it to obtain preprocessed video stream data and preprocessed motion trajectory data; S2: Based on the preprocessed video stream data and the preprocessed motion trajectory data, extract the basic features of the video stream image and the basic features of the encoding; then perform cross-modal temporal fusion through the image-encoding fusion unit to obtain cross-modal fusion features; perform key region enhancement through a dynamic region gating mechanism to obtain region enhancement features; finally, calculate the optimized encoding parameter set. S3: Encode the preprocessed video stream data according to the optimized encoding parameter set to obtain a continuous bitstream that conforms to the H.264 standard specification; S4: Based on the continuous bitstream and region enhancement features conforming to the H.264 standard, calculate the initial decoded video frame sequence and defect region features, and input them into the timing correction generator to obtain the repaired image; input the repaired image and the preprocessed video stream data into the dual-verification branch discriminator to output the repair optimization signal; The timing correction generator extracts inter-frame timing information by fusing defect region features from the current frame and previous and next frames; then, combined with timing constraint weights, it repairs the defect regions of the initial decoded video frame through generator operations in a generative adversarial network, generating a repaired image. The dual-verification branch discriminator calculates the restoration score of the real image and the restoration score of the repaired image through the intra-frame restoration verification branch, and calculates the coherence score of the real image and the coherence score of the repaired image through the inter-frame coherence verification branch. It then combines the above four scores to output a repair optimization signal. S5: Based on the repair and optimization signal, optimize the generator parameters and output the repaired video frame sequence.

[0007] Furthermore, step S1 also includes: S11: Acquire video stream data and perform noise reduction processing using Gaussian filtering and mean filtering to obtain preprocessed video stream data; the video stream data consists of unencoded continuous image frames; S12: The preprocessed video stream data is encoded using H.264 encoding to obtain the encoded video stream data; S13: The encoded video stream data is processed using a sliding window-based mean filtering method to smooth the parameter transitions between frames, resulting in preprocessed motion trajectory data, including: macroblock motion vector matrix, residual coefficient matrix, QP value, reference frame index, and frame type identifier for each frame.

[0008] Furthermore, the specific process of key region enhancement through dynamic region gating mechanism in step S2 includes: based on the basic features of the video stream image, the maximum value is obtained after processing by the Sobel operator; the result obtained by multiplying the maximum value by the scaling factor is compared with the region detail mask threshold; a region detail mask matrix is ​​generated using an indicator function; after performing a convolution operation on the region detail mask matrix, the dynamic region gating weight is calculated using the Sigmoid function; the cross-modal fusion features are combined with the dynamic region gating weight to complete the key region enhancement processing of the cross-modal fusion features, thereby obtaining the region enhancement features.

[0009] Furthermore, step S2 also includes: S21: Based on the preprocessed video stream data and the preprocessed motion trajectory data, extract the basic features of the video stream image and the basic features of the encoding, respectively. The calculation method is as follows: ; ; ; ; ; in, Based on the convolution kernel, Initialize for Gaussian, For edge convolution kernels, To enhance the convolution kernel, The convolution kernel fusion coefficients, Here are the basic features of the video stream image in frame i, where i is the frame index. It is a convolutional layer. For the preprocessed video stream data of the i-th frame, For convolution bias terms, The basic coded features of the i-th frame, For splicing operations, Let be the macroblock motion vector matrix of the i-th frame. For ReLU function, Let be the residual coefficient matrix of the i-th frame; S22: Based on the basic features of the video stream image and the basic features of the coding, cross-modal temporal fusion is performed through the image-coding fusion unit to obtain cross-modal fusion features. The calculation method is as follows: ; ; in, For the image-coded alignment features of the i-th frame, It is a multilayer perceptron. To add element by element, For the cross-modal fusion features of the i-th frame, For Hadama accumulation, For the Softmax function, It is a one-dimensional convolutional layer. For the cross-modal fusion features of the (i-1)th frame, For the cross-modal fusion features of the first frame, Image-coded alignment features for frame 1; S23: Based on cross-modal fusion features, key regions are enhanced using a dynamic region gating mechanism to obtain region enhancement features. The calculation method is as follows: ; ; ; in, Let be the region detail mask matrix of the i-th frame. For indicator functions, To obtain the maximum value, For Sobel operators, For the region detail mask threshold, Let be the dynamic region gating weight for the i-th frame. For the Sigmoid function, For the region enhancement features of the i-th frame; S24: Based on the region enhancement features, and combined with the reference frame index and frame type identifier of the current frame, calculate the optimized coding parameter set. The calculation method is as follows: ; ; ; ; ; in, The simplified attention weights for the i-th frame, This is the frame type identifier for the i-th frame. Let be the optimized macroblock motion vector matrix of the i-th frame. For the region enhancement features of the reference frame corresponding to the i-th frame, Let be the optimized residual coefficient matrix of the i-th frame. For the optimized encoding parameter set, To ensure that the packaging process conforms to the H.264 format, These are the QP value and reference frame index of the i-th frame in the preprocessed motion trajectory data, respectively.

[0010] It should be further explained that dynamic scenes are rich in moving targets, have frequent changes in details, and have strong temporal correlations. The core difficulty lies in how to accurately identify key detail areas in dynamic scenes, avoid ineffective enhancement of irrelevant areas, and ensure that the enhanced features can be adapted to the encoding parameter optimization and decoding repair process. If the key area is mislocated or the enhancement method is inappropriate, it will directly lead to problems such as reduced encoding efficiency and increased defects in the decoded image. To address the aforementioned issues, this invention constructs a dynamic region gating mechanism, achieving key region localization and targeted enhancement through multi-stage collaborative processing. First, the Sobel operator is used to process the basic features of the video stream image, capturing edge detail information as the basis for identifying key regions in dynamic scenes. Then, by taking the maximum value and comparing it with a region detail mask threshold, a region detail mask matrix is ​​generated using an indicator function to filter out significant detail regions in the dynamic scene, eliminating interference from irrelevant background regions and providing a clear target range for subsequent feature enhancement. Next, a convolution operation is performed on the region detail mask matrix, and the dynamic region gating weights are calculated using the Sigmoid function. Finally, cross-modal fusion features are combined with the dynamic region gating weights, and through element-wise weighted processing, targeted enhancement of key region features is achieved, improving the effectiveness and accuracy of the features. Existing technologies for video feature enhancement mostly employ a globally uniform enhancement approach or set enhancement regions based on fixed rules. This lack of adaptability to dynamic scenes often results in insufficient enhancement of critical regions and excessive enhancement of non-critical regions, leading to weak targeted enhancement features that cannot effectively support subsequent encoding and decoding optimization. In contrast, the dynamic region gating mechanism of this invention can adaptively generate region detail mask matrices and gating weights based on the actual feature state of the dynamic scene, achieving the location and enhancement of critical regions. This avoids the blindness of global enhancement and the limitations of fixed region enhancement, making the enhanced region features more consistent with the actual needs of encoding and decoding in dynamic scenes. This provides a high-quality feature foundation for subsequent optimization of encoding parameter sets and the repair of decoding defects.

[0011] It should be further explained that the video coding scenarios in which the optimized coding parameter set is applied must be compatible with the H.264 standard format. At the same time, it must meet the differentiated coding requirements of different frame types (one-way prediction frames, two-way prediction frames) in dynamic scenes. How to improve the accuracy of macroblock motion vector prediction and the adaptability of residual coefficients while ensuring compliance with standard specifications, and solve the problem that it is difficult to balance coding efficiency, image quality preservation and format compatibility in traditional parameter calculation. If the parameter optimization is not targeted, it will lead to insufficient coding compression ratio or increased defects in the decoded image in dynamic scenes. This invention first uses region enhancement features as its core foundation, combining the reference frame index and frame type identifier of the current frame to provide a scene adaptation basis for parameter optimization, ensuring that the optimization direction aligns with the actual encoding requirements of dynamic scenes. Second, by simplifying attention weights and incorporating frame type adaptation logic, it assigns differentiated weight coefficients to unidirectional and bidirectional prediction frames, making attention allocation more suitable for the encoding characteristics of different frame types and avoiding adaptation deviations caused by uniform processing. Subsequently, for the macroblock motion vector matrix, combined with the region enhancement features of the reference frame, it uses multilayer perceptron processing and Sigmoid... Function activation generates correction values ​​to adjust the original motion vectors, improving the accuracy of motion trajectory capture. Simultaneously, the residual coefficient matrix is ​​optimized by processing the region enhancement features of the current frame through convolution operations and the ReLU function, avoiding negative distortion in the residual data and enhancing the effectiveness of data compression. Finally, a packaging operation integrates the optimized macroblock motion vector matrix, residual coefficient matrix, and original effective parameters (QP value, reference frame index, frame type identifier) ​​into a parameter set conforming to the H.264 standard, ensuring both targeted parameter optimization and compatibility with standard formats. In existing technologies, the calculation of encoding parameter sets mostly relies on traditional search algorithms to generate macroblock motion vectors, and the calculation of residual coefficients also often uses fixed rules. These methods fail to fully consider the feature differences of dynamic scenes and the encoding requirements of frame types, and lack effective linkage with the previous feature processing stage, resulting in insufficient parameter accuracy and difficulty in achieving a balance between encoding compression ratio and image quality. In contrast, the optimized encoding parameter set of this invention relies on region enhancement features, incorporates frame type adaptation and reference frame association logic, and achieves parameter optimization by simplifying the synergistic effect of attention weights and activation functions. This not only improves the accuracy of macroblock motion vectors and residual coefficients, but also strictly follows the H.264 standard through standardized packaging operations, effectively solving the problem of difficulty in balancing parameter accuracy and format compatibility in existing technologies. At the same time, it provides a high-quality parameter foundation for subsequent encoding stages, achieving improvements in encoding efficiency and image quality preservation.

[0012] Furthermore, step S3 also includes: S31: Adapt and map the optimized encoding parameter set to the parameter interface of the H.264 standard encoder, and use the parameters in the optimized encoding parameter set as the encoding parameters of the H.264 standard encoder; S32: For the preprocessed video stream data, call the H.264 standard encoder frame by frame. Based on the optimized encoding parameter set, perform macroblock motion estimation, residual data quantization adjustment, intra-frame and inter-frame prediction calculations on each frame to obtain the frame-level data after encoding each frame. S33: Entropy encoding and bitstream encapsulation are performed on the frame-level data after each frame is encoded according to the H.264 standard syntax structure to obtain a continuous bitstream that conforms to the H.264 standard specification.

[0013] Furthermore, step S4 also includes: S41: Based on the continuous bitstream conforming to the H.264 standard and the region enhancement features, complete the decoding and defect region feature extraction to obtain the initial decoded video frame sequence and defect region features. The calculation method is as follows: ; ; ; in, This is the initial sequence of decoded video frames. For H.264 standard decoding operations, To produce a continuous bitstream that conforms to the H.264 standard specification, Let be the dynamic defect region mask matrix of the i-th frame. For indicator functions, It is a convolutional layer. For the region enhancement features of the i-th frame, Calculated for the mean. Let be the defect region features of the i-th frame. For the i-th frame image in the initial decoded video frame sequence, For Hadamah accumulation; S42: Input the defect region features and the initial decoded video frame sequence into the timing correction generator, fuse the inter-frame timing information to generate the repaired image, and calculate the following: ; ; ; in, Let i be the temporal features of the defects in the i-th frame. It is a one-dimensional convolutional layer. For splicing operations, These are the defect region features of the (i-1)th and (i+1)th frames, respectively. When i=1, , Let be the temporal constraint weights for the i-th frame. For the Sigmoid function, It is a multilayer perceptron. The image after restoration is the i-th frame. To add element by element, For generator operations in generative adversarial networks; S43: Input the repaired image and the preprocessed video stream data into the dual-verification branch discriminator and output the repair and optimization signal.

[0014] It should be further explained that the decoded video frames are prone to defects such as blur, ghosting, and color blocks, and the temporal correlation between frames is strong. The core difficulty lies in how to repair the defects of a single frame while taking into account the temporal continuity between frames, so as to avoid problems such as screen breakage and discontinuous motion trajectory after repair. To address the aforementioned issues, this invention constructs a temporal correction generator, achieving both single-frame defect repair and inter-frame coherence assurance. The temporal correction generator takes defect region features and the initial decoded video frame sequence as input, relying on the previously extracted defect region features to locate the defective parts to be repaired, avoiding resource waste or inaccurate repair caused by indiscriminate repair. Secondly, by splicing defect region features from the current frame and preceding and following frames, and then processing them through a one-dimensional convolutional layer, temporal features of the defects are extracted, capturing the evolution patterns and correlation information of defects between frames in dynamic scenes, breaking the limitations of traditional generators that only process one frame. Subsequently, a multilayer perceptron is used to process the temporal features of the defects, and a sigmoid function is used to generate temporal constraint weights to dynamically adapt to the correlation strength of defects between frames, providing temporal guidance for the repair process. Finally, the preliminary repair results obtained from the basic generator in the generative adversarial network are added element-wise with the weighted results of the temporal constraint weights and defect region features, achieving targeted repair of defective regions. Each step is complementary and progressively enhances the process, solving the single-frame defect problem while ensuring the consistency of motion trajectories between frames. Traditional generative adversarial networks (GANs) generators, when processing video decoding and restoration, mostly adopt a single-frame independent restoration mode, focusing only on the defect filling of a single frame image without considering the temporal correlation between frames in dynamic scenes. They lack a mechanism to capture the evolution law of inter-frame defects, resulting in improved quality of individual frames after restoration, but poor inter-frame coherence and easy image jumps. In contrast, the temporal correction generator of this invention integrates inter-frame temporal information and introduces temporal constraint weights to guide the restoration process. While achieving accurate restoration of single-frame defects, it effectively ensures the temporal coherence between frames. The restored image can restore lost details and maintain the smoothness of motion trajectories in dynamic scenes, significantly improving the decoding and restoration quality of dynamic images.

[0015] Furthermore, step S43 also includes: S431: Based on the restored image and the preprocessed video stream data, calculate the restoration score of the real image and the restoration score of the restored image through the intra-frame restoration verification branch. The calculation method is as follows: ; ; in, Let be the restoration score of the real image in the i-th frame. Let be the restoration score of the repaired image in frame i. For ReLU function, This represents the preprocessed video stream data of the i-th frame; S432: Based on the restored image and the preprocessed video stream data, the coherence scores of the real image and the restored image are calculated through the inter-frame coherence check branch. The calculation method is as follows: ; ; 1; in, The coherence score for the real image in the i-th frame is given. Let be the coherence score of the repaired image in frame i. This is the preprocessed video stream data for the (i-1)th frame. This is the repaired image of the (i-1)th frame; S433: Calculate the restoration optimization signal based on the restoration score of the real image, the restoration score of the restored image, the coherence score of the real image, and the coherence score of the restored image. The calculation method is as follows: ; in, This is the repair and optimization signal for the i-th frame.

[0016] It should be further explained that the core feature of the dynamic image decoding and repair verification scenario applied by the dual-verification branch discriminator is that it is necessary to simultaneously perform dual verification on the single-frame detail restoration degree and inter-frame temporal continuity of the repaired image, and the verification result needs to be converted into a repair optimization signal to guide the generator to iterative optimization. If only the single-frame restoration effect is focused on and the inter-frame continuity is ignored, the optimization signal will not be able to solve the problem of inter-frame breakage after dynamic image repair. If only the temporal correlation is emphasized, the repair quality of single-frame details will be sacrificed, making it difficult to achieve comprehensive and accurate verification feedback. To address the aforementioned issues, this invention designs a dual-verification branch discriminator. First, the intra-frame restorative quality verification branch assesses the restoration quality of a single frame image. Through the collaborative processing of convolution operations and the ReLU function, features are extracted and enhanced from both the preprocessed video stream data (real image) and the restored image. Then, the Sigmoid function transforms the features into a restorative score in the 0-1 range, quantifying the quality of detail restoration at the single-frame level and providing a basis for verifying the single-frame restoration effect. Second, the inter-frame coherence verification branch focuses on inter-frame correlation in dynamic scenes, employing one-dimensional convolution operations on adjacent frames (…). Temporal features are extracted and processed from adjacent frames of the real image and adjacent frames of the restored image. The ReLU function is used to enhance effective features, and the Sigmoid function is used to quantize the scores to capture the continuity differences of motion trajectories between frames, thus overcoming the limitations of single-frame verification. Finally, by splicing the differences in single-frame restoration scores and the differences in inter-frame continuity scores, and processing them through a multilayer perceptron and standardizing them with the Sigmoid function, a restoration optimization signal is generated. The verification results of the two branches complement each other and work synergistically, so that the optimization signal reflects both single-frame and inter-frame problems existing in the restoration process, providing guidance for subsequent generator optimization. In existing technologies, traditional discriminators in video restoration scenarios mostly focus only on verifying the authenticity of a single frame image, neglecting the inter-frame temporal continuity requirements unique to dynamic scenes. This results in one-sided verification results, and the generated optimization signals can only guide the generator to optimize the single-frame restoration effect, failing to address dynamic defects such as inter-frame breaks and ghosting, thus affecting the overall restoration quality. In contrast, the dual-verification-branch discriminator of this invention innovatively sets up two independent and collaborative verification branches: intra-frame and inter-frame. It can accurately verify the detail restoration of a single-frame restored image and effectively capture the differences in inter-frame temporal continuity, achieving comprehensive verification in two dimensions. The generated restoration optimization signals are more targeted, guiding the temporal correction generator to simultaneously optimize single-frame defects and inter-frame continuity issues, significantly improving the overall restoration effect of dynamic scenes and solving the technical pain point of one-sided verification in traditional discriminators. In terms of integration and fusion, this invention also overcomes the core difficulty of insufficient adaptability of traditional discriminators' single verification branch to dynamic image restoration requirements. Its innovation lies in organically integrating single-frame verification and inter-frame verification into an integrated dual-branch verification architecture, and realizing the conversion of verification results into restoration optimization signals. In the prior art, a single verification branch is difficult to fully match the dual restoration requirements of the timing correction generator, and there is a lack of coordination between the verification and restoration stages. However, the dual-verification branch discriminator of this invention, through the differentiated information formed by dual-branch verification, generates restoration optimization signals that can adapt to the optimization requirements of the generator after integration and processing, realizing deep coordination between the verification and restoration stages. At the same time, it breaks through the technical limitations of traditional discriminators' single-dimensional verification, enabling the verification process to fully support dynamic image decoding and restoration, and improving the collaborative efficiency and reliability of the entire encoding and decoding optimization scheme.

[0017] Furthermore, step S5 also includes: optimizing the parameters of the timing correction generator using the Adam optimizer based on the repair optimization signal to obtain optimized generator parameters; using the timing correction generator, iteratively processing the initial decoded video frames based on the optimized generator parameters and outputting the repaired images; arranging each repaired image in frame index order to obtain a repaired video frame sequence.

[0018] Compared with the prior art, the beneficial effects of the present invention are as follows: (1) To address the issues of insufficient accuracy in motion vector prediction, blurring, ghosting, and poor inter-frame coherence in dynamic image encoding and decoding, this invention strengthens key region features through a dynamic region gating mechanism during the encoding stage, thereby obtaining an optimized encoding parameter set. This not only improves the prediction accuracy of macroblock motion vectors in dynamic scenes but also ensures that the encoded bitstream conforms to the H.264 standard. During the decoding stage, the defective regions are accurately located based on the region enhancement features. The timing correction generator integrates inter-frame timing information to perform targeted repairs on the defective regions. The dual-check branch discriminator outputs an optimized signal through dual checks of single-frame reproducibility and inter-frame coherence, continuously optimizing the generator parameters and finally outputting a repaired video frame sequence with coherent inter-frames. This achieves the synergistic linkage between parameter optimization at the encoding end and defect repair at the decoding end. While ensuring compatibility with the H.264 standard, it significantly improves the overall quality of dynamic image encoding and decoding, adapting to the needs of multiple scenarios such as streaming media transmission and monitoring storage.

[0019] (2) To address the problems of insufficient accuracy in key region identification and lack of dynamic adaptability in enhancement methods during dynamic image encoding and decoding, which lead to decreased encoding efficiency and increased defects in the decoded image, this invention innovatively constructs a dynamic region gating mechanism. First, it captures the edge details of the basic features of the video stream image based on the Sobel operator. Then, it generates a region detail mask matrix through threshold comparison and indicator function to filter key detail regions and eliminate background interference. Subsequently, it obtains dynamic region gating weights through convolution operation and Sigmoid function. Finally, it combines them with cross-modal fusion features in a weighted manner to achieve key region feature enhancement. Compared with the blindness and limitations of existing technologies that use global unified enhancement or fixed rule enhancement, this invention can adaptively generate a mask matrix and gating weights based on the actual features of the dynamic image, improve the effectiveness and accuracy of features, provide a high-quality feature foundation for subsequent encoding parameter optimization and decoding defect repair, and accurately adapt to the encoding and decoding needs of dynamic scenes.

[0020] (3) In addressing the problem that traditional parametric encoding methods in H.264 encoding of dynamic images struggle to balance macroblock motion vector prediction accuracy and residual coefficient adaptability, leading to increased image defects after decoding, this invention innovatively constructs an optimized encoding parameter set. Based on region enhancement features, combined with reference frame index and frame type identifier, it integrates frame type differentiation logic by simplifying attention weights, and assigns adaptive weights to unidirectional / bidirectional prediction frames. For the macroblock motion vector matrix, this invention combines the region enhancement features of the reference frame, generates correction values ​​through a multilayer perceptron and Sigmoid function, and improves motion trajectory capture accuracy. For the residual coefficient matrix, this invention avoids negative distortion and enhances compression effectiveness through convolution and ReLU function processing. Finally, it integrates optimized parameters and original effective parameters through a packaging operation to form a parameter set that conforms to the H.264 standard. Compared with the limitations of traditional search algorithms and fixed rules in existing technologies, this invention achieves synergy between parameter accuracy and standard compatibility, improving encoding efficiency and image quality preservation.

[0021] (4) In response to the problem that the decoded video frames have single-frame defects such as blurring and ghosting, and that traditional repair methods are difficult to balance single-frame defect repair with inter-frame temporal continuity, which easily leads to discontinuous images, this invention innovatively constructs a temporal correction generator. The temporal correction generator takes the defect region features and the initial decoded video frame sequence as input. First, it locates the parts to be repaired based on the defect region features. Then, by splicing the defect region features of the current frame and the frames before and after, it extracts the temporal features of the defects through a one-dimensional convolutional layer, captures the evolution law of inter-frame defects, and breaks the limitation of single-frame processing of traditional generators. Subsequently, it uses a multilayer perceptron to process the temporal features of the defects and generates temporal constraint weights through the Sigmoid function to dynamically adapt the correlation strength of inter-frame defects. Finally, it adds the preliminary repair results of the basic generator in the generative adversarial network with the weighted results of the temporal constraint weights and defect region features to achieve targeted repair. Compared with the mode of traditional generators that repair single frames independently and ignore inter-frame temporal correlation, this invention not only accurately repairs single-frame defects and restores lost details, but also ensures the smoothness of inter-frame motion trajectories, significantly improving the quality of dynamic image decoding and repair.

[0022] (5) In the dynamic image decoding and repair verification, the traditional discriminator only focuses on single-frame verification, which cannot take into account the detail restoration of a single frame and the temporal continuity between frames. This results in a one-sided optimization signal and makes it difficult to solve dynamic defects such as inter-frame breaks. The present invention innovatively designs a dual-verification branch discriminator. It achieves comprehensive verification through two independent and collaborative branches: the intra-frame restoration verification branch processes the real image and the restored image through convolution and ReLU function, extracts enhancement features and quantifies the restoration score through the Sigmoid function to accurately evaluate the single-frame repair quality; the inter-frame continuity verification branch processes adjacent frames with one-dimensional convolution, extracts temporal features and quantifies the continuity score to capture the differences in motion trajectory between frames; finally, the two types of score differences are spliced ​​together and a repair optimization signal is generated through a multilayer perceptron and the Sigmoid function, which reflects both single-frame and inter-frame problems. Compared with the one-sided verification of the traditional discriminator, the present invention achieves comprehensive verification in two dimensions, and the optimization signal is more targeted. It can guide the generator to simultaneously optimize single-frame defects and inter-frame continuity, and significantly improve the overall dynamic image repair effect. Attached Figure Description

[0023] Figure 1 A flowchart illustrating an adaptive optimization method for video encoding and decoding based on generative adversarial networks provided by this invention; Figure 2 The image is obtained using the conventional video encoding and decoding method provided by this invention.

[0024] Figure 3 The image is obtained by the video encoding and decoding method based on generative adversarial networks provided by this invention. Detailed Implementation

[0025] The present invention will be further described below with reference to the accompanying drawings, but this is not intended to limit the present invention in any way. Any modifications or substitutions made based on the teachings of the present invention shall fall within the protection scope of the present invention.

[0026] Example 1: An adaptive optimization method for video encoding and decoding based on generative adversarial networks, such as... Figure 1 As shown, it includes the following steps: S1: Acquire video stream data and preprocess it to obtain preprocessed video stream data and preprocessed motion trajectory data, including: S11: Acquire video stream data and perform noise reduction processing using Gaussian filtering and mean filtering to obtain preprocessed video stream data; the video stream data consists of unencoded continuous image frames; S12: The preprocessed video stream data is encoded using H.264 encoding to obtain the encoded video stream data; S13: The encoded video stream data is processed using a sliding window-based mean filtering method to smooth the parameter transitions between frames, resulting in preprocessed motion trajectory data, including: macroblock motion vector matrix, residual coefficient matrix, QP value, reference frame index, and frame type identifier for each frame.

[0027] S2: Based on the preprocessed video stream data and preprocessed motion trajectory data, extract the basic image features and basic coding features of the video stream, respectively; then perform cross-modal temporal fusion through the image-code fusion unit to obtain cross-modal fusion features; perform key region enhancement through a dynamic region gating mechanism to obtain region enhancement features; finally, calculate the optimized coding parameter set, including: S21: Based on the preprocessed video stream data and the preprocessed motion trajectory data, extract the basic features of the video stream image and the basic features of the encoding, respectively. The calculation method is as follows: ; ; ; ; ; in, Based on the convolution kernel, Initialize for Gaussian, For edge convolution kernels, To enhance the convolution kernel, The convolution kernel fusion coefficients, Here are the basic features of the video stream image in frame i, where i is the frame index. It is a convolutional layer. For the preprocessed video stream data of the i-th frame, For convolution bias terms, The basic coded features of the i-th frame, For splicing operations, Let be the macroblock motion vector matrix of the i-th frame. For ReLU function, Let be the residual coefficient matrix of the i-th frame; S22: Based on the basic features of the video stream image and the basic features of the coding, cross-modal temporal fusion is performed through the image-coding fusion unit to obtain cross-modal fusion features. The calculation method is as follows: ; ; in, For the image-coded alignment features of the i-th frame, It is a multilayer perceptron. To add element by element, For the cross-modal fusion features of the i-th frame, For Hadama accumulation, For the Softmax function, It is a one-dimensional convolutional layer. For the cross-modal fusion features of the (i-1)th frame, For the cross-modal fusion features of the first frame, Image-coded alignment features for frame 1; S23: Based on cross-modal fusion features, key regions are enhanced using a dynamic region gating mechanism to obtain region enhancement features. The calculation method is as follows: ; ; ; in, Let be the region detail mask matrix of the i-th frame. For indicator functions, To obtain the maximum value, For Sobel operators, For the region detail mask threshold, Let be the dynamic region gating weight for the i-th frame. For the Sigmoid function, For the region enhancement features of the i-th frame; S24: Based on the region enhancement features, and combined with the reference frame index and frame type identifier of the current frame, calculate the optimized coding parameter set. The calculation method is as follows: ; ; ; ; ; in, The simplified attention weights for the i-th frame, This is the frame type identifier for the i-th frame. Let be the optimized macroblock motion vector matrix of the i-th frame. For the region enhancement features of the reference frame corresponding to the i-th frame, Let be the optimized residual coefficient matrix of the i-th frame. For the optimized encoding parameter set, To ensure that the packaging process conforms to the H.264 format, These are the QP value and reference frame index of the i-th frame in the preprocessed motion trajectory data, respectively.

[0028] Specifically, for scenarios with abrupt changes in inter-frame features in dynamic scenes (such as rapid switching of moving targets or scene transitions), this invention also provides a temporal-enhanced cross-modal fusion calculation method to replace step S22. The calculation method is as follows: ; ; ; in, For the image-coded joint temporal features of the i-th frame, The basic features of the video stream image in the (i-1)th frame are as follows: The basic coded features of the (i-1)th frame, Let be the cross-modal alignment feature of the i-th frame.

[0029] S3: Encode the preprocessed video stream data according to the optimized encoding parameter set to obtain a continuous bitstream conforming to the H.264 standard, including: S31: Adapt and map the optimized encoding parameter set to the parameter interface of the H.264 standard encoder, and use the parameters in the optimized encoding parameter set as the encoding parameters of the H.264 standard encoder; S32: For the preprocessed video stream data, call the H.264 standard encoder frame by frame. Based on the optimized encoding parameter set, perform macroblock motion estimation, residual data quantization adjustment, intra-frame and inter-frame prediction calculations on each frame to obtain the frame-level data after encoding each frame. S33: Entropy encoding and bitstream encapsulation are performed on the frame-level data after each frame is encoded according to the H.264 standard syntax structure to obtain a continuous bitstream that conforms to the H.264 standard specification.

[0030] S4: Based on the continuous bitstream and region enhancement features conforming to the H.264 standard, calculate the initial decoded video frame sequence and defect region features, and input them into the timing correction generator to obtain the repaired image; input the repaired image and preprocessed video stream data into a dual-verification branch discriminator to output a repair optimization signal, including: S41: Based on the continuous bitstream conforming to the H.264 standard and the region enhancement features, complete the decoding and defect region feature extraction to obtain the initial decoded video frame sequence and defect region features. The calculation method is as follows: ; ; ; in, This is the initial sequence of decoded video frames. For H.264 standard decoding operations, To produce a continuous bitstream that conforms to the H.264 standard specification, Let be the dynamic defect region mask matrix of the i-th frame. For indicator functions, It is a convolutional layer. For the region enhancement features of the i-th frame, Calculated for the mean. Let be the defect region features of the i-th frame. For the i-th frame image in the initial decoded video frame sequence, For Hadamah accumulation; S42: Input the defect region features and the initial decoded video frame sequence into the timing correction generator, fuse the inter-frame timing information to generate the repaired image, and calculate the following: ; ; ; in, Let i be the temporal features of the defects in the i-th frame. It is a one-dimensional convolutional layer. For splicing operations, These are the defect region features of the (i-1)th and (i+1)th frames, respectively. When i=1, , Let be the temporal constraint weights for the i-th frame. For the Sigmoid function, It is a multilayer perceptron. The image after restoration is the i-th frame. To add element by element, For generator operations in generative adversarial networks; S43: Input the repaired image and preprocessed video stream data into a dual-verification branch discriminator, and output a repair optimization signal, including: S431: Based on the restored image and the preprocessed video stream data, calculate the restoration score of the real image and the restoration score of the restored image through the intra-frame restoration verification branch. The calculation method is as follows: ; ; in, Let be the restoration score of the real image in the i-th frame. Let be the restoration score of the repaired image in frame i. For ReLU function, This represents the preprocessed video stream data of the i-th frame; S432: Based on the restored image and the preprocessed video stream data, the coherence scores of the real image and the restored image are calculated through the inter-frame coherence check branch. The calculation method is as follows: ; ; 1; in, The coherence score for the real image in the i-th frame is given. Let be the coherence score of the repaired image in frame i. This is the preprocessed video stream data for the (i-1)th frame. This is the repaired image of the (i-1)th frame; S433: Calculate the restoration optimization signal based on the restoration score of the real image, the restoration score of the restored image, the coherence score of the real image, and the coherence score of the restored image. The calculation method is as follows: ; in, This is the repair and optimization signal for the i-th frame.

[0031] S5: Based on the repaired and optimized signal, optimize the generator parameters and output the repaired video frame sequence, including: Based on the repair and optimization signal, the Adam optimizer is used to optimize the parameters of the timing correction generator to obtain the optimized generator parameters; using the generator, the initial decoded video frames are iteratively processed based on the optimized generator parameters and the repaired images are output; the repaired images are arranged in order of frame index to obtain the repaired video frame sequence.

[0032] For example, in video encoding and decoding scenarios involving dynamic moving figures (with fast arm swings), the effectiveness of traditional video encoding and decoding methods and the video encoding and decoding method based on generative adversarial networks of this invention is compared and verified; among them... Figure 2 The image obtained using traditional video encoding and decoding methods shows that the hand area is noticeably blurred due to motion blur. Figure 3 The image obtained by the method of this invention can clearly present the details of the hand, and the file size of the two is consistent. This invention effectively improves the detail reproduction of fast-moving areas in dynamic images without increasing the size of the encoded bitstream, and solves the motion blur defect of traditional methods in dynamic scenes.

[0033] It should be noted that the sequence numbers of the above embodiments of the present invention are merely for descriptive purposes and do not represent the superiority or inferiority of the embodiments. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, apparatus, article, or method. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, apparatus, article, or method that includes that element.

[0034] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0035] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A method for video coding adaptive optimization based on generative adversarial networks, characterized in that, Includes the following steps: S1: Acquire video stream data and preprocess it to obtain preprocessed video stream data and preprocessed motion trajectory data; S2: Based on the preprocessed video stream data and the preprocessed motion trajectory data, extract the basic features of the video stream image and the basic features of the encoding, respectively; Then, cross-modal temporal fusion is performed through an image-code fusion unit to obtain cross-modal fusion features; Key regions are enhanced using a dynamic region gating mechanism to obtain region enhancement features; Finally, the optimized encoding parameter set is calculated; The specific process of key region enhancement through dynamic region gating in step S2 includes: based on the basic features of the video stream image, the maximum value is obtained after processing by the Sobel operator; the result of multiplying the maximum value by a scaling factor is compared with the region detail mask threshold; a region detail mask matrix is ​​generated using an indicator function; after performing a convolution operation on the region detail mask matrix, dynamic region gating weights are calculated using the Sigmoid function; the cross-modal fusion features are combined with the dynamic region gating weights to complete the key region enhancement processing of the cross-modal fusion features, resulting in region enhancement features; S3: Encode the preprocessed video stream data according to the optimized encoding parameter set to obtain a continuous bitstream that conforms to the H.264 standard specification; S4: Based on the continuous bitstream and region enhancement features conforming to the H.264 standard, calculate the initial decoded video frame sequence and defect region features, and input them into the timing correction generator to obtain the repaired image; input the repaired image and the preprocessed video stream data into the dual-verification branch discriminator to output the repair optimization signal; The timing correction generator extracts inter-frame timing information by fusing defect region features from the current frame and previous and next frames; then, combined with timing constraint weights, it repairs the defect regions of the initial decoded video frame through generator operations in a generative adversarial network, generating a repaired image. The dual-verification branch discriminator calculates the restoration score of the real image and the restoration score of the repaired image through the intra-frame restoration verification branch, and calculates the coherence score of the real image and the coherence score of the repaired image through the inter-frame coherence verification branch. It then combines the above four scores to output a repair optimization signal. S5: Based on the repair and optimization signal, optimize the generator parameters and output the repaired video frame sequence.

2. The GAN-based video coding adaptive optimization method of claim 1, wherein, Step S1 includes: S11: Acquire video stream data and perform noise reduction processing using Gaussian filtering and mean filtering to obtain preprocessed video stream data; the video stream data consists of unencoded continuous image frames; S12: The preprocessed video stream data is encoded using H.264 encoding to obtain the encoded video stream data; S13: The encoded video stream data is processed using a sliding window-based mean filtering method to smooth the parameter transitions between frames, resulting in preprocessed motion trajectory data, including: macroblock motion vector matrix, residual coefficient matrix, QP value, reference frame index, and frame type identifier for each frame.

3. The video encoding / decoding adaptive optimization method based on generative adversarial networks according to claim 1, characterized in that, Step S2 includes: S21: Based on the preprocessed video stream data and the preprocessed motion trajectory data, extract the basic features of the video stream image and the basic features of the encoding, respectively. The calculation method is as follows: ； ； ； ； ； in, Based on the convolution kernel, Initialize for Gaussian, For edge convolution kernels, To enhance the convolution kernel, The convolution kernel fusion coefficients, Here are the basic features of the video stream image in frame i, where i is the frame index. It is a convolutional layer. For the preprocessed video stream data of the i-th frame, For convolution bias terms, The basic coded features of the i-th frame, For splicing operations, Let be the macroblock motion vector matrix of the i-th frame. For ReLU function, Let be the residual coefficient matrix of the i-th frame; S22: Based on the basic features of the video stream image and the basic features of the coding, cross-modal temporal fusion is performed through the image-coding fusion unit to obtain cross-modal fusion features. The calculation method is as follows: ； ； ； in, For the image-coded alignment features of the i-th frame, It is a multilayer perceptron. To add element by element, For the cross-modal fusion features of the i-th frame, For Hadama accumulation, For the Softmax function, It is a one-dimensional convolutional layer. For the cross-modal fusion features of the (i-1)th frame, For the cross-modal fusion features of the first frame, Image-coded alignment features for frame 1; S23: Based on cross-modal fusion features, key regions are enhanced using a dynamic region gating mechanism to obtain region enhancement features. The calculation method is as follows: ； ； ； in, Let be the region detail mask matrix of the i-th frame. For indicator functions, To obtain the maximum value, For Sobel operators, For the region detail mask threshold, Let be the dynamic region gating weight for the i-th frame. For the Sigmoid function, For the region enhancement features of the i-th frame; S24: Based on the region enhancement features, and combined with the reference frame index and frame type identifier of the current frame, calculate the optimized coding parameter set. The calculation method is as follows: ； ； ； ； ； in, The simplified attention weights for the i-th frame, This is the frame type identifier for the i-th frame. Let be the optimized macroblock motion vector matrix of the i-th frame. For the region enhancement features of the reference frame corresponding to the i-th frame, Let be the optimized residual coefficient matrix of the i-th frame. For the optimized encoding parameter set, To ensure that the packaging process conforms to the H.264 format, These are the QP value and reference frame index of the i-th frame in the preprocessed motion trajectory data, respectively.

4. The video encoding / decoding adaptive optimization method based on generative adversarial networks according to claim 3, characterized in that, Step S3 includes: S31: Adapt and map the optimized encoding parameter set to the parameter interface of the H.264 standard encoder, and use the parameters in the optimized encoding parameter set as the encoding parameters of the H.264 standard encoder; S32: For the preprocessed video stream data, call the H.264 standard encoder frame by frame. Based on the optimized encoding parameter set, perform macroblock motion estimation, residual data quantization adjustment, intra-frame and inter-frame prediction calculations on each frame to obtain the frame-level data after encoding each frame. S33: Entropy encoding and bitstream encapsulation are performed on the frame-level data after each frame is encoded according to the H.264 standard syntax structure to obtain a continuous bitstream that conforms to the H.264 standard specification.

5. The video encoding / decoding adaptive optimization method based on generative adversarial networks according to claim 4, characterized in that, Step S4 includes: S41: Based on the continuous bitstream conforming to the H.264 standard and the region enhancement features, complete the decoding and defect region feature extraction to obtain the initial decoded video frame sequence and defect region features. The calculation method is as follows: ； ； ； in, This is the initial sequence of decoded video frames. For H.264 standard decoding operations, To produce a continuous bitstream that conforms to the H.264 standard specification, Let be the dynamic defect region mask matrix of the i-th frame. For indicator functions, It is a convolutional layer. For the region enhancement features of the i-th frame, Calculated for the mean. Let be the defect region features of the i-th frame. For the i-th frame image in the initial decoded video frame sequence, For Hadamah accumulation; S42: Input the defect region features and the initial decoded video frame sequence into the timing correction generator, fuse the inter-frame timing information to generate the repaired image, and calculate the following: ； ； ； in, Let i be the temporal features of the defects in the i-th frame. It is a one-dimensional convolutional layer. For splicing operations, These are the defect region features of the (i-1)th and (i+1)th frames, respectively. When i=1, , Let be the temporal constraint weights for the i-th frame. For the Sigmoid function, It is a multilayer perceptron. The image after restoration is the i-th frame. To add element by element, For generator operations in generative adversarial networks; S43: Input the repaired image and the preprocessed video stream data into the dual-verification branch discriminator and output the repair and optimization signal.

6. The video encoding / decoding adaptive optimization method based on generative adversarial networks according to claim 5, characterized in that, Step S43 includes: S431: Based on the restored image and the preprocessed video stream data, calculate the restoration score of the real image and the restoration score of the restored image through the intra-frame restoration verification branch. The calculation method is as follows: ； ； in, Let be the restoration score of the real image in the i-th frame. Let be the restoration score of the repaired image in frame i. For ReLU function, This represents the preprocessed video stream data of the i-th frame; S432: Based on the restored image and the preprocessed video stream data, the coherence scores of the real image and the restored image are calculated through the inter-frame coherence check branch. The calculation method is as follows: ； ； 1； in, The coherence score for the real image in the i-th frame is given. Let be the coherence score of the repaired image in frame i. This is the preprocessed video stream data for the (i-1)th frame. This is the repaired image of the (i-1)th frame; S433: Calculate the restoration optimization signal based on the restoration score of the real image, the restoration score of the restored image, the coherence score of the real image, and the coherence score of the restored image. The calculation method is as follows: ； in, This is the repair and optimization signal for the i-th frame.

7. The video encoding / decoding adaptive optimization method based on generative adversarial networks according to claim 6, characterized in that, Step S5 includes: optimizing the parameters of the timing correction generator using the Adam optimizer based on the repair optimization signal to obtain optimized generator parameters; using the timing correction generator, iteratively processing the initial decoded video frames based on the optimized generator parameters and outputting the repaired images; arranging each repaired image in frame index order to obtain the repaired video frame sequence.

Images