Video coding and decoding adaptive optimization method based on generative adversarial network

Abstract

The invention discloses a video coding and decoding adaptive optimization method based on a generative adversarial network. The method comprises the following steps: S1, collecting video stream data and preprocessing the video stream data; s2, respectively extracting video stream image basic features and coding basic features; calculating a cross-modal fusion feature and a region enhancement feature, and finally calculating an optimized coding parameter set; s3, encoding the preprocessed video stream data to obtain a continuous code stream conforming to the H.264 standard specification; s4, calculating an initial decoding video frame sequence and defect area features, inputting the initial decoding video frame sequence and the defect area features into a time sequence correction generator to obtain a repaired image, and outputting a repair optimization signal; and S5, optimizing generator parameters according to the repair optimization signal, and outputting a repaired video frame sequence. According to the method, the problems of insufficient motion vector prediction accuracy and fuzzy smear after decoding in dynamic picture coding and decoding of a traditional method can be solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video coding, and in particular to a video coding adaptive optimization method based on a generative adversarial network. BACKGROUND

[0002] As one of the mainstream standards in the current video coding field, H.264 coding technology is widely used in many important scenarios such as streaming media transmission, monitoring video storage, mobile video playback, etc. due to its efficient compression performance. Its core realizes efficient compression and transmission of video data through key steps such as macroblock motion estimation, intra / inter prediction, residual quantization, etc.

[0003] In the traditional H.264 coding scheme, motion vector prediction mainly depends on traditional greedy search algorithms such as diamond search and hexagon search. Although such algorithms have low computational complexity, they are difficult to accurately capture the macroblock motion trajectory of fast moving targets in dynamic scenes, which easily leads to motion vector prediction deviation, and further affects the coding efficiency and image quality preservation effect. The decoding end mainly performs entropy decoding, inverse quantization, inverse transformation and inter-frame prediction reconstruction according to the H.264 standard process, and lacks a special optimization mechanism for dynamic pictures, so that the decoded dynamic pictures generally have defects such as blur, ghosting, color blocks, and the lost details cannot be effectively restored, and the inter-frame coherence is poor.

[0004] With the penetration of deep learning technology in the field of video coding, existing schemes mostly use convolutional neural networks (CNN), generative adversarial networks (GAN) and other models to optimize the encoding parameters or the decoding repair process, such as optimizing the prediction accuracy of motion vectors through CNN models, or using GAN models to complete the details of the decoded defective frames, but such schemes still have obvious drawbacks: the deep learning model at the encoding end often focuses more on the macroscopic balance of the overall code rate and image quality, and the fine calculation of the macroblock motion trajectory in dynamic scenes is insufficient, and targeted optimization strategies cannot be formed, resulting in that the prediction accuracy of the motion vector is still difficult to meet the coding needs of dynamic pictures; the repair model at the decoding end mostly focuses on the defect repair of single-frame images, and the inter-frame temporal correlation in dynamic scenes is not fully considered, and the repair effect cannot be adaptively adapted to the needs of dynamic scenes, making it difficult to solve problems such as blur, ghosting and poor inter-frame coherence, and the lost details are also difficult to effectively restore. SUMMARY

[0005] Therefore, the present application aims to provide a video coding adaptive optimization method based on a generative adversarial network to solve the problems of insufficient motion vector prediction accuracy and blur and ghosting after decoding in the traditional method in dynamic picture coding.

[0006] A video coding adaptive optimization method based on a generative adversarial network comprises: S1: collect video stream data and pre-process to obtain pre-processed video stream data and pre-processed motion trajectory data; S2: extract video stream image basic features and encoding basic features from the pre-processed video stream data and the pre-processed motion trajectory data, respectively; then perform cross-modal temporal fusion through an image-encoding fusion unit to obtain cross-modal fusion features; perform key area enhancement through a dynamic area gating mechanism to obtain area-enhanced features; and finally calculate an optimized encoding parameter set; S3: encode the pre-processed video stream data according to the optimized encoding parameter set to obtain continuous code streams conforming to H.264 standard specifications; S4: calculate initial decoded video frame sequences and defect area features from the continuous code streams conforming to H.264 standard specifications and the area-enhanced features, and input them into a temporal correction generator to obtain repaired images; input the repaired images and the pre-processed video stream data into a double-check branch discriminator to output repair optimization signals; The temporal correction generator extracts inter-frame temporal information by fusing defect area features of the current frame and the previous and next frames; then, in combination with temporal constraint weights, repairs the defect areas of the initial decoded video frames through generator operations in a generative adversarial network to generate repaired images; The double-check branch discriminator calculates restoration scores of real images and restoration scores of repaired images through an intra-frame restoration check branch, calculates continuity scores of real images and continuity scores of repaired images through an inter-frame continuity check branch, and then outputs repair optimization signals in combination with the four scores; S5: optimize generator parameters according to the repair optimization signals and output repaired video frame sequences.

[0007] Further, the S1 step further comprises: S11: collect video stream data and perform denoising processing using Gaussian filtering and mean filtering to obtain pre-processed video stream data; the video stream data is continuous image frames that have not been encoded; S12: encode the pre-processed video stream data using an H.264 encoding method to obtain encoded video stream data; S13: process the encoded video stream data using a mean filtering method based on a sliding window to smooth inter-frame parameter jumps to obtain pre-processed motion trajectory data, including: macroblock motion vector matrices, residual coefficient matrices, QP values, reference frame indices, and frame type identifiers of each frame.

[0008] Further, the specific process of key region enhancement in the S2 step through the dynamic region gating mechanism includes: based on the video stream image basic feature, taking the maximum value after processing by the Sobel operator, comparing the result obtained by multiplying the maximum value with a proportional coefficient with a region detail mask threshold, and generating a region detail mask matrix using an indicator function; after performing convolution operation on the region detail mask matrix, the dynamic region gating weight is calculated through the Sigmoid function; the cross-modal fusion feature is combined with the dynamic region gating weight to complete the key region enhancement processing of the cross-modal fusion feature, and the region enhanced feature is obtained.

[0009] Further, the S2 step further includes: S21: according to the preprocessed video stream data and the preprocessed motion trajectory data, video stream image basic features and encoding basic features are extracted respectively, and the calculation method is: ; ; ; ; ; Wherein, is a basic convolution kernel, is a Gaussian initialization, is an edge convolution kernel, is an enhanced convolution kernel, is a convolution kernel fusion coefficient, is the video stream image basic feature of the i-th frame, i is the frame number index, is a convolution layer, is the preprocessed video stream data of the i-th frame, is a convolution bias term, is the encoding basic feature of the i-th frame, is a splicing operation, is the macroblock motion vector matrix of the i-th frame, is a ReLU function, is the residual coefficient matrix of the i-th frame; S22: according to the video stream image basic feature and the encoding basic feature, cross-modal temporal fusion is performed through an image-encoding fusion unit to obtain a cross-modal fusion feature, and the calculation method is: ; ; Wherein, is the image-encoding alignment feature of the i-th frame, is a multi-layer perception, is an element-wise addition, is the cross-modal fusion feature of the i-th frame, is a Hadamard product, is a Softmax function, is a one-dimensional convolutional layer, is the cross-modal fusion feature of the i-1-th frame, is the cross-modal fusion feature of the 1-th frame, is the image-encoding alignment feature of the 1-th frame; S23: According to the cross-modal fusion feature, the key region enhancement is performed through a dynamic region gating mechanism to obtain a region enhanced feature, and the calculation method is: ; ; ; wherein, is a region detail mask matrix of the i-th frame, is an indicator function, is a maximum value, is a Sobel operator, is a region detail mask threshold value, is a dynamic region gating weight of the i-th frame, is a Sigmoid function, is the region enhanced feature of the i-th frame; S24: According to the region enhanced feature, the reference frame index and the frame type identifier of the current frame are combined to calculate an optimized encoding parameter set, and the calculation method is: ; ; ; ; ; wherein, is a simplified attention weight of the i-th frame, is a frame type identifier of the i-th frame, is an optimized macroblock motion vector matrix of the i-th frame, is the region enhanced feature of the reference frame corresponding to the i-th frame, is an optimized residual coefficient matrix of the i-th frame, is an optimized encoding parameter set, is a packing operation in compliance with the H.264 format, are the QP value and the reference frame index of the i-th frame in the preprocessed motion trajectory data, respectively.

[0010] It needs to be further explained that the motion target in the dynamic picture is rich, the detail changes frequently and the time sequence correlation is strong, the core difficulty is how to accurately identify the key detail area in the dynamic scene, avoid invalid enhancement of irrelevant areas, and ensure that the enhanced features can adapt to the encoding parameter optimization and decoding repair process, if the key area positioning deviation or the enhancement method is improper, it will directly lead to the problems of decreased encoding efficiency, increased picture defects after decoding, etc. To solve the above problems, the dynamic area gating mechanism is constructed, and the key area positioning and targeted enhancement are realized through the cooperation of multiple links; first, the Sobel operator is used to process the basic features of the video stream image, and the edge detail information in the image is captured as the basis for identifying the key area in the dynamic scene; then, the maximum value is taken and compared with the region detail mask threshold, and the region detail mask matrix is generated combined with the indicator function to filter out the detail areas with important significance in the dynamic picture and exclude the interference of irrelevant background areas, providing a clear target range for subsequent feature enhancement; then, the convolution operation is performed on the region detail mask matrix, and the dynamic area gating weight is calculated through the Sigmoid function; finally, the cross-modal fusion features and the dynamic area gating weight are combined, and the weighted processing is performed element by element to realize the targeted enhancement of the key area features and improve the effectiveness and accuracy of the features. The prior art mostly adopts the global uniform enhancement method or sets the enhancement area based on fixed rules when processing video feature enhancement, which lacks adaptability to dynamic scenes, often has the problems of insufficient enhancement of key areas and excessive enhancement of non-key areas, resulting in weak targetedness of the enhanced features and inability to well support the optimization of subsequent encoding and decoding links; in contrast, the dynamic area gating mechanism of the present application can adaptively generate the region detail mask matrix and the gating weight based on the actual feature state of the dynamic picture, realize the positioning and enhancement of the key area, avoid the blindness of global enhancement and the limitations of fixed area enhancement, and make the enhanced area enhancement features more meet the actual needs of encoding and decoding in dynamic scenes, providing a high-quality feature basis for the calculation of the optimized encoding parameter set and the repair of decoding defects.

[0011] It needs to be further explained that the video encoding scene to which the optimized encoding parameter set is applied needs to adapt to the compatibility requirements of the H.264 standard format, and should also meet the differentiated encoding needs of different frame types (single prediction frame, bi-directional prediction frame) in dynamic pictures, how to improve the accuracy of macroblock motion vector prediction and the adaptability of residual coefficients while ensuring compliance with standard specifications, solve the problem that the traditional parameter calculation is difficult to balance encoding efficiency, picture quality preservation and format compatibility, and if the parameter optimization lacks targetedness, it will lead to insufficient encoding compression ratio or increased picture defects after decoding in dynamic scenes; The present application firstly takes the region enhancement feature as the core basis, combines the reference frame index and the frame type identification of the current frame, provides scene adaptation basis for parameter optimization, and ensures that the optimization direction is consistent with the actual coding requirements of the dynamic picture; secondly, by calculating the simplified attention weight and integrating the frame type adaptation logic, the single prediction frame and the bi-directional prediction frame are given differentiated weight coefficients, so that the attention allocation is more consistent with the coding characteristics of different frame types, and the adaptation deviation caused by uniform processing is avoided; then, for the macro block motion vector matrix, the region enhancement feature of the reference frame is combined, the multi-layer perception machine processing and the Sigmoid function activation are performed, the correction amount is generated, the adjustment of the original motion vector is realized, and the accuracy of motion trajectory capture is improved; at the same time, the residual coefficient matrix is optimized, the region enhancement feature of the current frame is processed through convolution operation and ReLU function, the negative distortion of residual data is avoided, and the effectiveness of data compression is enhanced; finally, the optimized macro block motion vector matrix, residual coefficient matrix and original effective parameters (QP value, reference frame index, frame type identification) are integrated into a parameter set conforming to the H.264 standard through packing operation, which not only ensures the pertinence of parameter optimization, but also takes into account the compatibility of the standard format. In the prior art, the calculation of the coding parameter set mostly relies on the traditional search algorithm to generate the macro block motion vector, and the residual coefficient calculation mostly adopts the fixed rule, without fully combining the feature differences of the dynamic scene and the coding requirements of the frame type, and lacking effective linkage with the previous feature processing link, which leads to insufficient parameter accuracy, and it is difficult to balance between coding compression ratio and picture quality; compared with the prior art, the optimized coding parameter set of the present application relies on the region enhancement feature, integrates the frame type adaptation and the reference frame association logic, realizes parameter optimization through the synergistic effect of the simplified attention weight and the activation function, not only improves the accuracy of the macro block motion vector and the residual coefficient, but also strictly follows the H.264 standard through the standardized packing operation, effectively solves the problem that the parameter accuracy and the format compatibility are difficult to be considered in the prior art, and at the same time provides a high-quality parameter basis for the subsequent coding link, realizes the improvement of the coding efficiency and the picture quality preservation.

[0012] Further, the S3 step further comprises: S31: adaptively mapping the optimized coding parameter set to the parameter interface of the H.264 standard encoder, and using the parameters in the optimized coding parameter set as the coding parameters of the H.264 standard encoder; S32: calling the H.264 standard encoder according to the frame for the preprocessed video stream data, and based on the optimized coding parameter set, sequentially performing macro block motion estimation, residual data quantization adjustment, intra-frame and inter-frame prediction calculation on each frame of image to obtain the frame-level data after coding of each frame; S33: Entropy encoding and code stream encapsulation are performed on the encoded frame-level data according to the H.264 standard syntax structure, to obtain continuous code streams conforming to the H.264 standard specification.

[0013] Further, the S4 step further comprises: S41: Decoding and defect region feature extraction are completed according to the continuous code streams conforming to the H.264 standard specification and the region enhancement features, to obtain an initial decoded video frame sequence and defect region features, and the calculation manner is as follows: ; ; ; Wherein, is the initial decoded video frame sequence, is the H.264 standard decoding operation, is the continuous code stream conforming to the H.264 standard specification, is the dynamic defect region mask matrix of the i-th frame, is an indicator function, is a convolution layer, is the region enhancement feature of the i-th frame, is mean calculation, is the defect region feature of the i-th frame, is the i-th image in the initial decoded video frame sequence, is a Hadamard product; S42: The defect region feature and the initial decoded video frame sequence are input into a timing correction generator, inter-frame timing information is fused to generate a repaired image, and the calculation manner is as follows: ; ; ; Wherein, is the defect timing feature of the i-th frame, is a one-dimensional convolution layer, is a splicing operation, are the defect region features of the i-1-th and i+1-th frames respectively, when i=1, , is the timing constraint weight of the i-th frame, is a Sigmoid function, is a multi-layer perception, is the repaired image of the i-th frame, is element-wise addition, is a generator operation in a generative adversarial network; S43: input the repaired image and the preprocessed video stream data into a double-check branch discriminator to output a repair optimization signal.

[0014] It should be further explained that the decoded video frame is prone to defects such as blur, ghosting, color blocks, and the like, and the inter-frame temporal correlation is strong, and the core difficulty lies in how to repair single-frame defects while taking into account the inter-frame temporal continuity, and avoiding the problem of picture breakage and incoherent motion trajectory after repair; To solve the above problems, the present application constructs a temporal correction generator to realize single-frame defect repair and inter-frame continuity guarantee; the temporal correction generator takes the defect region features and the initial decoded video frame sequence as input, relies on the defect region features extracted in the early stage to locate the defect part to be repaired, avoiding the problem of resource waste or inaccurate repair caused by indiscriminate repair; secondly, the defect region features of the current frame and the previous and next frames are spliced, and then the defect temporal features are extracted through a one-dimensional convolution layer processing, so as to capture the evolution law and correlation information of the defects in the inter-frame under dynamic scenes, breaking the limitation of single-frame processing of the traditional generator; then, the multi-layer perception machine is used to process the defect temporal features, and the Sigmoid function is used to generate the temporal constraint weight to dynamically adapt the correlation strength of the inter-frame defects and provide temporal guidance for the repair process; finally, the preliminary repair result obtained by the basic generator in the generative adversarial network is added element by element with the weighted result of the temporal constraint weight and the defect region features, to realize the targeted repair of the defect region, and the functions of each link are complementary and progressive, which not only solves the single-frame defect problem, but also guarantees the consistency of the inter-frame motion trajectory; In the traditional generator in the generative adversarial network, a single-frame independent repair mode is mostly adopted when processing video decoding repair, only the defect completion of single-frame image is concerned, the inter-frame temporal correlation under dynamic scenes is not considered, and the mechanism for capturing the evolution law of inter-frame defects is also lacking, resulting in that although the quality of the repaired video frame is improved, the inter-frame continuity is poor, and the problem of picture jumping is prone to occur; compared with the above, the temporal correction generator of the present application introduces the temporal constraint weight to guide the repair process by fusing the inter-frame temporal information, which not only realizes the accurate repair of single-frame defects, but also effectively guarantees the inter-frame temporal continuity, so that the repaired image can not only restore the lost details, but also maintain the smoothness of the motion trajectory under dynamic scenes, and the decoding repair quality of dynamic pictures is significantly improved.

[0015] Further, the S43 step further comprises: S431: according to the repaired image and the preprocessed video stream data, calculate the restoration score of the real image and the restoration score of the repaired image through the intra-frame restoration check branch, and the calculation method is: ; ; wherein, is the restoration score of the real image of the i-th frame, is the restoration score of the repaired image of the i-th frame, is the ReLU function, is the pre-processed video stream data of the i-th frame; S432: According to the repaired image and the pre-processed video stream data, the continuity score of the real image and the continuity score of the repaired image are calculated through the inter-frame continuity checking branch, and the calculation method is: ; ; 1; wherein, is the continuity score of the real image of the i-th frame, is the continuity score of the repaired image of the i-th frame, is the pre-processed video stream data of the i-1-th frame, is the repaired image of the i-1-th frame; S433: According to the restoration score of the real image, the restoration score of the repaired image, the continuity score of the real image, and the continuity score of the repaired image, the repair optimization signal is calculated, and the calculation method is: ; wherein, is the repair optimization signal of the i-th frame.

[0016] It needs to be further explained that the dynamic picture decoding repair checking scene applied by the double checking branch discriminator has the core feature that the single-frame detail restoration degree and the inter-frame temporal continuity of the repaired image need to be checked at the same time, and the checking results need to be converted into a repair optimization signal to guide the generator to iteratively optimize. If only the single-frame restoration effect is concerned and the inter-frame continuity is ignored, the optimization signal cannot solve the inter-frame breakage problem after dynamic picture repair. If only the temporal correlation is focused on, the repair quality of the single-frame detail will be sacrificed, and it is difficult to achieve comprehensive and accurate checking feedback. To solve the above problems, the application designs a double-check branch discriminator; first, through the intra-frame restoration check branch, the repair quality of single frame image is checked, through the cooperative processing of convolution operation and ReLU function, the pre-processed video stream data (real image) and the repaired image are respectively subjected to feature extraction and strengthening, and then the features are converted into 0-1 interval restoration score through Sigmoid function, so as to quantize the advantages and disadvantages of single frame level detail restoration, and provide basis for checking the single frame repair effect; secondly, through the inter-frame continuity check branch, the inter-frame correlation in dynamic scene is focused, one-dimensional convolution operation is adopted to extract and process the time sequence features of adjacent frames (real image adjacent frames, repaired image adjacent frames), and ReLU function is combined to strengthen the effective features and Sigmoid function is combined to quantify the score, so as to capture the continuity difference of inter-frame motion track and make up for the limitation of single frame check; finally, the difference of single frame restoration score and the difference of inter-frame continuity score are spliced, and the repaired optimization signal is generated through the processing of multilayer perception machine and the standardization of Sigmoid function, the check results of the two branches are complementary and synergistic, so that the optimization signal can reflect the single frame and inter-frame problems existing in the repair process at the same time, and provide guidance for subsequent generator optimization. In the prior art, the traditional discriminator mostly focuses on the authenticity check of single frame image in the video repair scene, ignores the inter-frame time sequence continuity demand of dynamic picture, and the check result is one-sided, so that the generated optimization signal can only guide the generator to optimize the single frame repair effect, and cannot solve the inter-frame breakage, ghosting and other dynamic defect problems, thereby affecting the overall repair quality; compared with the prior art, the double-check branch discriminator of the application innovatively sets two independent and cooperative check branches of intra-frame and inter-frame, which can accurately check the detail restoration degree of single frame repair image, and effectively capture the inter-frame time sequence continuity difference, realize the comprehensive check of double dimensions, and the generated repair optimization signal is more targeted, which can guide the time sequence correction generator to optimize the single frame defect and inter-frame continuity problem at the same time, significantly improve the overall repair effect of dynamic picture, and solve the technical pain point of one-sided check of traditional discriminator; In terms of integration and fusion, the application also breaks through the core difficulty of the traditional discriminator single verification branch and the insufficient adaptability of dynamic picture repair requirements. The innovation lies in the organic integration of single-frame verification and inter-frame verification into a two-branch verification architecture, and the transformation of verification results and repair optimization signals. In the prior art, the single verification branch cannot fully match the dual repair requirements of the timing correction generator, and the verification link and the repair link are out of sync. The dual verification branch discriminator of the application generates repair optimization signals that can adapt to the optimization requirements of the generator through the differentiated information formed by the dual verification after integration processing, realizing the deep collaboration of the verification link and the repair link, breaking through the technical limitations of the traditional discriminator single-dimensional verification, and enabling the verification process to fully support dynamic picture decoding and repair, thereby improving the collaborative efficiency and reliability of the entire coding and decoding optimization scheme.

[0017] Further, the S5 step further comprises: optimizing the parameters of the timing correction generator using an Adam optimizer according to the repair optimization signal to obtain optimized generator parameters; using the timing correction generator to iteratively process the initial decoded video frames based on the optimized generator parameters and output repaired images; and arranging the repaired images in frame index order to obtain a repaired video frame sequence.

[0018] Compared with the prior art, the application has the following advantages: (1) To solve the problems of insufficient motion vector prediction accuracy, blurring, trailing and poor inter-frame coherence after decoding in dynamic picture coding and decoding, the application strengthens key area features in the coding stage through a dynamic area gating mechanism, thereby obtaining an optimized coding parameter set, which improves the prediction accuracy of macroblock motion vectors in dynamic scenes and ensures that the coding bitstream conforms to the H.264 standard specification; in the decoding stage, the area enhancement features are used to accurately locate the defect area, the timing correction generator fuses inter-frame timing information to repair the defect area, the dual verification branch discriminator outputs optimization signals through single-frame restoration and inter-frame coherence double verification, continuously optimizes the generator parameters, and finally outputs an inter-frame coherent repaired video frame sequence; the application realizes the collaborative linkage of coding end parameter optimization and decoding end defect repair, ensures the compatibility of the H.264 standard specification, significantly improves the overall quality of dynamic picture coding and decoding, and adapts to multiple scene requirements such as streaming media transmission and monitoring storage.

[0019] (2) In view of the problems of insufficient accuracy of key area identification in dynamic picture coding and decoding, lack of dynamic adaptability of enhancement mode, and increase of picture defects after decoding, the application innovatively constructs a dynamic region gating mechanism. First, the edge details of the basic features of the video stream image are captured based on the Sobel operator, and then the region detail mask matrix is generated through threshold comparison and indicator function. The key detail area is screened and background interference is excluded. Then, the dynamic region gating weight is obtained through convolution operation and Sigmoid function. Finally, it is combined with the cross-modal fusion feature weighting to realize the key area feature enhancement. Compared with the blindness and limitations of the existing technology of global unified enhancement or fixed rule enhancement, the application can adaptively generate mask matrix and gating weight based on the actual characteristics of dynamic pictures, improve the effectiveness and accuracy of the features, provide high-quality feature basis for subsequent encoding parameter optimization and decoding defect repair, and accurately adapt to the coding needs of dynamic scenes.

[0020] (3) In view of the problems of traditional parameter coding method in dynamic picture H.264 coding, it is difficult to consider the macroblock motion vector prediction accuracy and residual coefficient adaptability, which leads to the increase of picture defects after decoding, the application innovatively constructs an optimized coding parameter set; based on the enhanced features of the region, combined with the reference frame index and frame type identifier, the frame type differentiation logic is introduced by simplifying the attention weight, and the adaptive weight is allocated for the single / double prediction frame; for the macroblock motion vector matrix, the application combines the enhanced features of the reference frame, generates the correction amount through the multi-layer perception and Sigmoid function, and improves the motion trajectory capture accuracy; for the residual coefficient matrix, the application avoids negative distortion through convolution and ReLU function processing, enhances the compression effectiveness, and finally integrates the optimized parameters and original effective parameters through packing operation to form a parameter set conforming to H.264 standard; compared with the limitations of the existing technology of traditional search algorithm and fixed rule, the application realizes the coordination of parameter accuracy and standard compatibility, improves the coding efficiency and picture quality reservation effect.

[0021] (4) In view of the single frame defects such as blur and ghosting of the decoded video frame, and the traditional repair method is difficult to balance the single frame defect repair and the inter-frame temporal continuity, which is easy to cause the problem of discontinuous picture, the application innovatively constructs a temporal correction generator; the temporal correction generator takes the defect area feature and the initial decoded video frame sequence as input, first relies on the defect area feature to locate the part to be repaired, then splices the defect area features of the current frame and the previous and next frames, extracts the defect temporal feature through a one-dimensional convolution layer, captures the evolution law of inter-frame defects, and breaks the single frame processing limitation of the traditional generator; then the multi-layer perception machine is used to process the defect temporal feature, the temporal constraint weight is generated through the Sigmoid function, and the correlation strength of the inter-frame defect is dynamically adapted; finally, the preliminary repair result of the basic generator in the generative adversarial network is added to the weighted result of the temporal constraint weight and the defect area feature, so that the targeted repair is realized; compared with the traditional generator which independently repairs the single frame and ignores the inter-frame temporal correlation, the application not only accurately repairs the single frame defect and restores the lost details, but also guarantees the smoothness of the inter-frame motion trajectory, and significantly improves the dynamic picture decoding repair quality.

[0022] (5) In view of the dynamic picture decoding repair verification, the traditional discriminator only focuses on single frame verification, cannot balance the single frame detail restoration degree and inter-frame temporal continuity, leads to one-sided optimization signal, and is difficult to solve the problem of inter-frame breakage and other dynamic defects, the application innovatively designs a double verification branch discriminator; comprehensive verification is realized through two independent and cooperative branches: the intra-frame restoration verification branch processes the real image and the repaired image through convolution and ReLU function, extracts the reinforced features and quantifies the restoration score through the Sigmoid function, accurately evaluates the single frame repair quality; the inter-frame continuity verification branch adopts one-dimensional convolution to process adjacent frames, extracts temporal features and quantifies continuity scores, and captures the difference of inter-frame motion trajectory; finally, the differences of the two types of scores are spliced, the multi-layer perception machine and the Sigmoid function are used to generate the repair optimization signal, which reflects the single frame and inter-frame problems at the same time; compared with the one-sided verification of the traditional discriminator, the application realizes comprehensive verification in double dimensions, the optimization signal is more targeted, can guide the generator to optimize the single frame defect and the inter-frame continuity at the same time, and significantly improves the overall repair effect of the dynamic picture. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 A flowchart of a video coding and decoding adaptive optimization method based on a generative adversarial network is provided for the application. Figure 2 An image obtained by a traditional video coding and decoding method is provided for the application.

[0024] Figure 3 An image obtained by a video coding and decoding method based on a generative adversarial network is provided for the application. DETAILED DESCRIPTION

[0025] The application will be further described below with reference to the drawings, but the application is not limited in any way by the following description, any transformation or replacement based on the teaching of the application shall fall within the protection scope of the application.

[0026] Embodiment 1: a video coding adaptive optimization method based on a generative adversarial network, as shown in the figure, comprising the following steps: Figure 1 S1: collecting video stream data and performing preprocessing to obtain preprocessed video stream data and preprocessed motion trajectory data, comprising: S11: collecting video stream data and performing denoising processing by using Gaussian filtering and mean filtering to obtain preprocessed video stream data; the video stream data is continuous image frames without coding; S12: encoding the preprocessed video stream data by using an H.264 encoding mode to obtain encoded video stream data; S13: processing the encoded video stream data by using a mean filtering method based on a sliding window to smooth the parameter jump between frames to obtain preprocessed motion trajectory data, comprising: a macroblock motion vector matrix, a residual coefficient matrix, a QP value, a reference frame index and a frame type identifier of each frame.

[0027] S2: extracting video stream image basic features and encoding basic features according to the preprocessed video stream data and the preprocessed motion trajectory data; then performing cross-modal time sequence fusion through an image-encoding fusion unit to obtain cross-modal fusion features; performing key area enhancement through a dynamic area gating mechanism to obtain area enhancement features; finally calculating an optimized encoding parameter set, comprising: S21: extracting video stream image basic features and encoding basic features according to the preprocessed video stream data and the preprocessed motion trajectory data, the calculation method being: ; ; ; ; ; wherein, is a basic convolution kernel, is Gaussian initialization, is an edge convolution kernel, is an enhancement convolution kernel, is a convolution kernel fusion coefficient, is a video stream image basic feature of an i-th frame, i is a frame number index, is a convolution layer, is preprocessed video stream data of an i-th frame,​ is a convolution bias term, is an encoded base feature of the i-th frame, is a concatenation operation, is a macroblock motion vector matrix of the i-th frame, is a ReLU function, is a residual coefficient matrix of the i-th frame; S22: According to the image base feature and the encoded base feature of the video stream, cross-modal temporal fusion is performed through an image-encoding fusion unit to obtain cross-modal fusion features, and the calculation manner is: ; ; wherein, is an image-encoding alignment feature of the i-th frame, is a multi-layer perception, is an element-wise addition, is a cross-modal fusion feature of the i-th frame, is a Hadamard product, is a Softmax function, is a one-dimensional convolution layer, is a cross-modal fusion feature of the i-1-th frame, is a cross-modal fusion feature of the 1st frame, is an image-encoding alignment feature of the 1st frame; S23: According to the cross-modal fusion feature, key region enhancement is performed through a dynamic region gating mechanism to obtain region enhanced features, and the calculation manner is: ; ; ; wherein, is a region detail mask matrix of the i-th frame, is an indicator function, is a maximum value, is a Sobel operator, is a region detail mask threshold value, is a dynamic region gating weight of the i-th frame, is a Sigmoid function, is a region enhanced feature of the i-th frame; S24: According to the region enhanced feature, the reference frame index and the frame type identifier of the current frame are combined to calculate an optimized encoding parameter set, and the calculation manner is: ; ; wherein, is the simplified attention weight of the i-th frame, is the frame type identifier of the i-th frame, is the optimized macroblock motion vector matrix of the i-th frame, is the region enhanced feature of the corresponding reference frame of the i-th frame, is the optimized residual coefficient matrix of the i-th frame, is the optimized encoding parameter set, is the packing operation following the H.264 format, are the QP value and the reference frame index of the i-th frame in the preprocessed motion trajectory data, respectively.

[0028] In particular, for scenes with inter-frame feature mutations (such as fast moving target switching and scene jumping) in dynamic pictures, the application further provides a time sequence enhanced cross-modal fusion calculation method for replacing the S22 step, and the calculation method is: wherein, is the image-encoding joint time sequence feature of the i-th frame, is the video stream image base feature of the i-th-1 frame, is the encoding base feature of the i-th-1 frame, is the cross-modal alignment feature of the i-th frame.

[0029] S3: encoding the preprocessed video stream data according to the optimized encoding parameter set to obtain a continuous code stream conforming to the H.264 standard specification, including: S31: adapting and mapping the optimized encoding parameter set to the parameter interface of the H.264 standard encoder, and using the parameters in the optimized encoding parameter set as the encoding parameters of the H.264 standard encoder; S32: calling the H.264 standard encoder by frame for the preprocessed video stream data, and based on the optimized encoding parameter set, sequentially performing macroblock motion estimation, residual data quantization adjustment, and intra-frame and inter-frame prediction calculation on each frame of image to obtain each frame of encoded frame-level data; S33: entropy encoding and code stream packaging the each frame of encoded frame-level data according to the H.264 standard syntax structure to obtain a continuous code stream conforming to the H.264 standard specification.

[0030] ​​​​​​S4: According to the continuous code stream and the region enhancement feature conforming to the H.264 standard specification, the initial decoded video frame sequence, the defect region feature are calculated, and input into the timing correction generator to obtain the repaired image; the repaired image and the preprocessed video stream data are input into the double-check branch discriminator to output the repair optimization signal, including: S41: According to the continuous code stream and the region enhancement feature conforming to the H.264 standard specification, the decoding and the defect region feature extraction are completed to obtain the initial decoded video frame sequence and the defect region feature, and the calculation method is: ; ; ; Among them, is the initial decoded video frame sequence, is the H.264 standard decoding operation, is the continuous code stream conforming to the H.264 standard specification, is the dynamic defect region mask matrix of the i-th frame, is an indicator function, is a convolution layer, is the region enhancement feature of the i-th frame, is the mean value calculation, is the defect region feature of the i-th frame, is the i-th frame image in the initial decoded video frame sequence, is the Hadamard product; S42: The defect region feature and the initial decoded video frame sequence are input into the timing correction generator to generate the repaired image by fusing the inter-frame timing information, and the calculation method is: ; ; ; Among them, is the defect timing feature of the i-th frame, is a one-dimensional convolution layer, is a splicing operation, are the defect region features of the i-1th and i+1th frames respectively, when i=1, , is the timing constraint weight of the i-th frame, is a Sigmoid function, is a multi-layer perception, is the repaired image of the i-th frame, is an element-wise addition, is a generator operation in a generative adversarial network; S43: input the repaired image and the preprocessed video stream data into the double-check branch discriminator, and output a repair optimization signal, including: S431: according to the repaired image and the preprocessed video stream data, calculate the restoration score of the real image and the restoration score of the repaired image through the intra-frame restoration check branch, and the calculation method is: ; ; wherein, is the restoration score of the real image of the i-th frame, is the restoration score of the repaired image of the i-th frame, is a ReLU function, is the preprocessed video stream data of the i-th frame; S432: according to the repaired image and the preprocessed video stream data, calculate the continuity score of the real image and the continuity score of the repaired image through the inter-frame continuity check branch, and the calculation method is: ; ; 1; wherein, is the continuity score of the real image of the i-th frame, is the continuity score of the repaired image of the i-th frame, is the preprocessed video stream data of the i-1-th frame, is the repaired image of the i-1-th frame; S433: according to the restoration score of the real image, the restoration score of the repaired image, the continuity score of the real image, and the continuity score of the repaired image, calculate the repair optimization signal, and the calculation method is: ; wherein, is the repair optimization signal of the i-th frame.

[0031] S5: according to the repair optimization signal, optimize the generator parameters, and output the repaired video frame sequence, including: According to the repair optimization signal, the parameters of the timing correction generator are optimized by using the Adam optimizer, and the optimized generator parameters are obtained; the repaired image is output by iteratively processing the initial decoded video frame based on the optimized generator parameters using the generator; the repaired image of each frame is arranged in order of frame index to obtain the repaired video frame sequence.

[0032] For example, for the video coding scene of dynamic moving figures (the arm swing speed is fast), the effects of the traditional video coding method and the video coding method based on the generative adversarial network of the application are compared and verified; wherein, Figure 2 For the image obtained by the traditional video coding method, it can be seen that the hand region presents obvious blur state due to motion blur problem; Figure 3 For the image obtained by the method of the application, the hand details can be clearly presented, and the file sizes of the two are consistent; the application effectively improves the detail restoration degree of the fast motion region in the dynamic picture under the premise of ensuring that the coding stream size is not increased, and solves the motion blur defect of the traditional method in the dynamic scene.

[0033] It should be noted that the above-mentioned serial numbers of the embodiments of the application are only for description, and do not represent the advantages and disadvantages of the embodiments. And the terms "include", "contain" or any other variant thereof in this paper are intended to cover non-exclusive inclusion, so that the process, device, article or method including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or includes the elements inherent to such process, device, article or method. Without more limitations, the element defined by the statement "including a" does not exclude the presence of another identical element in the process, device, article or method including the element.

[0034] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment method can be realized by means of software and necessary general hardware platform, of course, it can also be realized by hardware, but in many cases the former is a better embodiment. Based on such understanding, the technical solutions of the application can be embodied in the form of software product, which is stored in a storage medium (such as ROM / RAM, magnetic disc, optical disc) as described above, including a plurality of instructions for making a terminal device (which can be a mobile phone, computer, server or network device, etc.) execute the method described in the plurality of embodiments of the application.

[0035] The above is only the preferred embodiment of the application, and does not limit the patent scope of the application, and any equivalent structure or equivalent flow transformation made by using the content of the specification and drawings of the application, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the application.

Claims

1. A video encoding / decoding adaptive optimization method based on generative adversarial networks, characterized in that, Includes the following steps: S1: Acquire video stream data and preprocess it to obtain preprocessed video stream data and preprocessed motion trajectory data; S2: Based on the preprocessed video stream data and the preprocessed motion trajectory data, extract the basic features of the video stream image and the basic features of the encoding, respectively; Then, cross-modal temporal fusion is performed through an image-code fusion unit to obtain cross-modal fusion features; Key regions are enhanced using a dynamic region gating mechanism to obtain region enhancement features; Finally, the optimized encoding parameter set is calculated; S3: Encode the preprocessed video stream data according to the optimized encoding parameter set to obtain a continuous bitstream that conforms to the H.264 standard specification; S4: Based on the continuous bitstream and region enhancement features conforming to the H.264 standard, calculate the initial decoded video frame sequence and defect region features, and input them into the timing correction generator to obtain the repaired image; input the repaired image and the preprocessed video stream data into the dual-verification branch discriminator to output the repair optimization signal; The timing correction generator extracts inter-frame timing information by fusing defect region features from the current frame and previous and next frames; then, combined with timing constraint weights, it repairs the defect regions of the initial decoded video frame through generator operations in a generative adversarial network, generating a repaired image. The dual-verification branch discriminator calculates the restoration score of the real image and the restoration score of the repaired image through the intra-frame restoration verification branch, and calculates the coherence score of the real image and the coherence score of the repaired image through the inter-frame coherence verification branch. It then combines the above four scores to output a repair optimization signal. S5: Based on the repair and optimization signal, optimize the generator parameters and output the repaired video frame sequence.

2. The video encoding / decoding adaptive optimization method based on generative adversarial networks according to claim 1, characterized in that, Step S1 includes: S11: Acquire video stream data and perform noise reduction processing using Gaussian filtering and mean filtering to obtain preprocessed video stream data; the video stream data consists of unencoded continuous image frames; S12: The preprocessed video stream data is encoded using H.264 encoding to obtain the encoded video stream data; S13: The encoded video stream data is processed using a sliding window-based mean filtering method to smooth the parameter transitions between frames, resulting in preprocessed motion trajectory data, including: macroblock motion vector matrix, residual coefficient matrix, QP value, reference frame index, and frame type identifier for each frame.

3. The video encoding / decoding adaptive optimization method based on generative adversarial networks according to claim 1, characterized in that, The specific process of key region enhancement through dynamic region gating in step S2 includes: based on the basic features of the video stream image, the maximum value is obtained after processing by the Sobel operator; the result of multiplying the maximum value by a scaling factor is compared with the region detail mask threshold; a region detail mask matrix is ​​generated using an indicator function; after performing a convolution operation on the region detail mask matrix, dynamic region gating weights are calculated using the Sigmoid function; the cross-modal fusion features are combined with the dynamic region gating weights to complete the key region enhancement processing of the cross-modal fusion features, resulting in region enhancement features.

4. The video encoding / decoding adaptive optimization method based on generative adversarial networks according to claim 3, characterized in that, Step S2 includes: S21: Based on the preprocessed video stream data and the preprocessed motion trajectory data, extract the basic features of the video stream image and the basic features of the encoding, respectively. The calculation method is as follows: ； ； ； ； ； in, Based on the convolution kernel, Initialize for Gaussian, For edge convolution kernels, To enhance the convolution kernel, The convolution kernel fusion coefficients, Here are the basic features of the video stream image in frame i, where i is the frame index. It is a convolutional layer. For the preprocessed video stream data of the i-th frame, For convolution bias terms, The basic coded features of the i-th frame, For splicing operations, Let be the macroblock motion vector matrix of the i-th frame. For ReLU function, Let be the residual coefficient matrix of the i-th frame; S22: Based on the basic features of the video stream images and the basic features of the coding, cross-modal temporal fusion is performed through the image-code fusion unit to obtain cross-modal fusion features. The calculation method is as follows: ； ； ； in, For the image-coded alignment features of the i-th frame, It is a multilayer perceptron. To add element by element, For the cross-modal fusion features of the i-th frame, For Hadama accumulation, For the Softmax function, It is a one-dimensional convolutional layer. For the cross-modal fusion features of the (i-1)th frame, For the cross-modal fusion features of the first frame, Image-coded alignment features for frame 1; S23: Based on cross-modal fusion features, key regions are enhanced using a dynamic region gating mechanism to obtain region enhancement features. The calculation method is as follows: ； ； ； in, Let be the region detail mask matrix of the i-th frame. For indicator functions, To obtain the maximum value, For Sobel operators, For the region detail mask threshold, Let be the dynamic region gating weight for the i-th frame. For the Sigmoid function, For the region enhancement features of the i-th frame; S24: Based on the region enhancement features, and combined with the reference frame index and frame type identifier of the current frame, calculate the optimized coding parameter set. The calculation method is as follows: ； ； ； ； ； in, The simplified attention weights for the i-th frame, This is the frame type identifier for the i-th frame. Let be the optimized macroblock motion vector matrix of the i-th frame. For the region enhancement features of the reference frame corresponding to the i-th frame, Let be the optimized residual coefficient matrix of the i-th frame. For the optimized encoding parameter set, To comply with the H.264 format packaging operation, These are the QP value and reference frame index of the i-th frame in the preprocessed motion trajectory data, respectively.

5. The video encoding / decoding adaptive optimization method based on generative adversarial networks according to claim 3, characterized in that, Step S3 includes: S31: Adapt and map the optimized encoding parameter set to the parameter interface of the H.264 standard encoder, and use the parameters in the optimized encoding parameter set as the encoding parameters of the H.264 standard encoder; S32: For the preprocessed video stream data, call the H.264 standard encoder frame by frame. Based on the optimized encoding parameter set, perform macroblock motion estimation, residual data quantization adjustment, intra-frame and inter-frame prediction calculations on each frame to obtain the frame-level data after encoding each frame. S33: Entropy encoding and bitstream encapsulation are performed on the frame-level data after each frame is encoded according to the H.264 standard syntax structure to obtain a continuous bitstream that conforms to the H.264 standard specification.

6. The video encoding / decoding adaptive optimization method based on generative adversarial networks according to claim 5, characterized in that, Step S4 includes: S41: Based on the continuous bitstream conforming to the H.264 standard and the region enhancement features, complete the decoding and defect region feature extraction to obtain the initial decoded video frame sequence and defect region features. The calculation method is as follows: ； ； ； in, This is the initial sequence of decoded video frames. For H.264 standard decoding operations, To produce a continuous bitstream that conforms to the H.264 standard specification, Let be the dynamic defect region mask matrix of the i-th frame. For indicator functions, It is a convolutional layer. For the region enhancement features of the i-th frame, Calculated for the mean. Let be the defect region features of the i-th frame. For the i-th frame image in the initial decoded video frame sequence, For Hadamah accumulation; S42: Input the defect region features and the initial decoded video frame sequence into the timing correction generator, fuse the inter-frame timing information to generate the repaired image, and calculate the following: ； ； ； in, Let i be the temporal features of the defects in the i-th frame. It is a one-dimensional convolutional layer. For splicing operations, These are the defect region features of the (i-1)th and (i+1)th frames, respectively. When i=1, , Let be the temporal constraint weights for the i-th frame. For the Sigmoid function, It is a multilayer perceptron. The image after restoration is the i-th frame. To add element by element, For generator operations in generative adversarial networks; S43: Input the repaired image and the preprocessed video stream data into the dual-verification branch discriminator and output the repair and optimization signal.

7. The video encoding / decoding adaptive optimization method based on generative adversarial networks according to claim 6, characterized in that, Step S43 includes: S431: Based on the restored image and the preprocessed video stream data, calculate the restoration score of the real image and the restoration score of the restored image through the intra-frame restoration verification branch. The calculation method is as follows: ； ； in, Let be the restoration score of the real image in the i-th frame. Let be the restoration score of the repaired image in frame i. For ReLU function, This represents the preprocessed video stream data of the i-th frame; S432: Based on the restored image and the preprocessed video stream data, the coherence scores of the real image and the restored image are calculated through the inter-frame coherence check branch. The calculation method is as follows: ； ； 1； in, The coherence score for the real image in the i-th frame is given. Let be the coherence score of the repaired image in frame i. This refers to the preprocessed video stream data of the (i-1)th frame. This is the repaired image of the (i-1)th frame; S433: Calculate the restoration optimization signal based on the restoration score of the real image, the restoration score of the restored image, the coherence score of the real image, and the coherence score of the restored image. The calculation method is as follows: ； in, This is the repair and optimization signal for the i-th frame.

8. The video encoding / decoding adaptive optimization method based on generative adversarial networks according to claim 7, characterized in that, Step S5 includes: optimizing the parameters of the timing correction generator using the Adam optimizer based on the repair optimization signal to obtain optimized generator parameters; using the timing correction generator, iteratively processing the initial decoded video frames based on the optimized generator parameters and outputting the repaired images; arranging each repaired image in frame index order to obtain the repaired video frame sequence.

Images