Video coding method based on double-layer condition enhanced normalized stream compressor

Abstract

The invention discloses a video coding method based on a double-layer conditional enhancement normalized stream compressor. The video coding method comprises the following steps: S1, inputting a reference frame into a frame interpolator to generate a conditional frame of a basic layer conditional enhancement normalized stream compressor; s2, down-sampling the input high-resolution frame into a low-resolution frame by a down-sampling network; s3, recovering the decoded low-resolution frame of the base layer into a full-resolution frame by the super-resolution reconstruction network; s4, merging the video frame information of the base layer and the predicted forward and reverse motion field information into a high-quality prediction frame; s5, generating a binary skip mask according to the predicted forward and reverse motion field information and the prediction frame; and S6, performing high-quality coding on the target frame by a conditional enhancement normalized stream compressor. According to the method, a low-resolution video frame compressor replaces a full-resolution motion compressor, a motion information compressor is omitted, potential samples needing to be transmitted are reduced through the skip mode coding technology, and the calculation complexity is greatly reduced.

Description

Technical Field

[0001] The invention belongs to the technical field of video processing and encoding, and in particular relates to a video encoding method based on a double-layer conditional enhanced normalized stream compressor. Background Art

[0002] The two key components of the current mainstream video codecs are motion coding (including motion estimation and compensation) and residual image coding. Motion coding is mainly used to reduce temporal redundancy, while residual coding is optimized for spatial redundancy, so this structure is often called "hybrid coding". However, hybrid coding also has certain limitations. First, it relies on traditional motion estimation methods to generate motion vectors, which often require a lot of computing resources and cannot capture the nonlinear motion of objects in complex scenes. In addition, residual coding can only reduce the spatial redundancy of local images, and has weak processing capabilities for global or non-local information, resulting in unsatisfactory compression effects in high-dynamic scenes or pictures with rich details. As video resolution increases, traditional block partitioning strategies (such as the block mode in HEVC) provide higher coding efficiency, but also face more bit overhead that needs to be processed.

[0003] In recent years, with the development of deep learning technology, video compression has made breakthrough progress. So far, most learning codecs still follow the structure of the classic hybrid coding system, which consists of two main parts: motion coding and residual image coding. It is generally believed that accurate motion compensation is an effective way to reduce video temporal redundancy, and image coding technology is used to encode only the remaining unpredictable pixels. However, accurately describing the motion field of objects of arbitrary shapes requires a large number of bits.

[0004] With the advancement of neural networks, we can get more accurate video prediction results without transmitting a lot of bits, so only the unpredictable pixels need to be transmitted. Usually, the positions of unpredictable pixels are sparse, and the bit overhead of transmitting these precise position information is very large. To solve this problem, a new guidance strategy is needed to avoid transmitting motion information and position information at the same time, and instead transmit the unpredictable pixel information through a two-layer structure. Summary of the invention

[0005] The purpose of the present invention is to provide a video encoding method based on a double-layer conditional enhanced normalized stream compressor, aiming to solve the problems raised in the background technology.

[0006] To achieve the above-mentioned purpose, the present invention provides the following technical solutions: A video encoding method based on a double-layer conditional enhanced normalized stream compressor comprises the following steps: S1, inputting the reference frame into the frame interpolator to generate the conditional frame of the base layer conditional enhanced normalized stream compressor; S2, the downsampling network downsamples the input high-resolution frame to a low-resolution frame, and the conditional enhanced normalized flow compressor compresses the low-resolution frame; S3, the super-resolution reconstruction network restores the decoded base layer low-resolution frame to a full-resolution frame; S4, inputting the video frame information of the base layer and the predicted forward and reverse motion field information into the multi-frame merging network in the enhancement layer to merge them into a high-quality predicted frame; S5, a skip mask generator generates a binary skip mask according to the predicted forward and reverse motion field information and the predicted frame; S6. The conditionally enhanced normalized stream compressor encodes the target frame with high quality according to the binary skip mask.

[0007] As a further limitation of the technical solution of the present invention, step S1 includes the following steps: Input reference frame , and a step length ranging from 0 to 1, according to which the number of intermediate frames to be synthesized is determined; Use the intermediate flow network to estimate the intermediate flow information of the input frame; Based on the estimated intermediate flow information, the reference frame is backward-warped to obtain a preliminary estimate of the intermediate frame; The warped frames are fused using a fusion network to generate the final interpolated frame ; As a further limitation of the technical solution of the present invention, step S2 specifically includes the following steps: Input frame With interpolated frames Input to the downsampling network, and after downsampling processing, we get and ; Base layer conditional enhancement normalized stream compressor downsampled target frame As input, Encoding the conditional signal, i.e. converting the image data into a more compact representation; Using conditional normalization (a technique that adjusts normalization parameters based on a conditional signal), The weights and biases of the convolutional layer and other related Normalize the relevant parameters to make the encoder better adapt to the characteristics of the input data, thereby improving the efficiency and quality of encoding; Output compressed image data .

[0008] As a further limitation of the technical solution of the present invention, the specific operation of downsampling is: Input frame With interpolated frames Input to the downsampling network; Use two convolutional layers to process the input image. The stride of these two convolutional layers is set to 2 (the stride refers to the step size of the convolution kernel moving on the image). When the stride is 2, the convolution kernel moves two pixels each time, thereby achieving image downsampling with a downsampling factor of 4. The residual block directly adds the input to the output by introducing a shortcut connection (also called a skip connection), maintains feature integrity, and fuses features from different layers through shortcut connections, enhancing the reuse of features. Output downsampled frames and , the resolution of the video frame is reduced to 1 / 4 of the original, and the width and height of the image are reduced to 1 / 2 of the original, reducing the amount of image data.

[0009] As a further limitation of the technical solution of the present invention, step S3 specifically includes the following steps: The super-resolution reconstruction network receives the decoded base layer low-resolution frame as input; The convolutional layer extracts features from the input low-resolution frame to capture key information such as edges and textures in the image, providing necessary feature representation for the subsequent reconstruction process. The extracted low-resolution features are mapped to the high-resolution feature space using a deep learning model. During this process, sub-pixel convolution is used to achieve feature upsampling and amplification, completing the conversion from low-resolution features to high-resolution images. Output a preliminary high-resolution image and use post-processing operations to further optimize the reconstruction results; Finally, after the super-resolution reconstruction processing operation, the super-resolution reconstruction network outputs a full-resolution frame that is close to or better than the original high-resolution frame.

[0010] As a further limitation of the technical solution of the present invention, the post-processing operation specifically includes the following steps: De-noising, sharpening and color correction are performed on the preliminary high-resolution images that have some defects or do not conform to the details of the real scene; Image noise is removed by using a mean filter. First, a small rectangular window is selected, which will be used to calculate the neighborhood average of the current pixel. For each pixel in the image, the average value of all pixel values ​​in its neighborhood is then calculated. Finally, the calculated average value is assigned to the current pixel as the denoised pixel value, and the denoised image is output. Laplace operator sharpening is used to improve the detail clarity and edge recognition ability of the image. First, a Laplace convolution kernel is constructed to perform a second-order differential operation on each pixel in the image to capture areas where the grayscale value changes dramatically. Then, the convolution kernel is applied to the image, and the second-order derivative value of each pixel is calculated through convolution, which reflects the rate of change of the surrounding grayscale value. Subsequently, the grayscale value of each pixel in the image is adjusted according to these second-order derivative values, that is, the second-order derivative value is added to the original pixel value, and normalized to ensure that the pixel value is between 0 and 255, thereby enhancing edges and details. Finally, the adjusted image is weighted summed with the original image, and the final sharpened image is obtained by superposition.

[0011] Color correction is achieved by adjusting image parameters such as brightness, contrast, and saturation, ultimately obtaining a further optimized and reconstructed image.

[0012] As a further limitation of the technical solution of the present invention, step S4 specifically includes the following steps: Get the predicted forward and backward motion field information from the base layer frame interpolator and ; Base layer video frame information and predicted forward and reverse motion field information , Input multi-frame merging network, which is designed with two output channels with weight maps, these weight maps are used to combine information from different video frames; Generate a weight map based on the similarities and differences between the input video frames and the feature representations learned by the network to determine the contribution of each input video frame to the final merged output; Use these weight maps to combine the input video frames, multiply the pixel value of each input video frame by the corresponding weight, and then add the weighted pixel values ​​to generate the final combined output; Output a high-quality predicted frame.

[0013] As a further limitation of the technical solution of the present invention, the forward and reverse motion field information predicted by the base layer frame interpolator is obtained. and , specifically including the following steps: Pass the reference frame from the decoded frame buffer and input frame interpolator; Analyze previous and next frames and The frame interpolator computes the motion vector field between them, representing how objects move between these frames; The motion field between frames is calculated using the optical flow method. The optical flow method estimates the motion vector by tracking the movement of pixels in the frame, which represents the movement of pixels from the reference frame to the reference frame. The position change to the current frame; The interpolator not only generates To the motion field of the current frame , and also generates To the motion field of the current frame These two sets of motion fields can be used to simultaneously predict the changes between previous and next frames and interpolate more accurately; The generated motion field information is not only used to generate interpolated frames , and will be further passed to the subsequent decoding module and multi-frame merging network to enhance the prediction and processing of the current frame.

[0014] As a further limitation of the technical solution of the present invention, step S5 specifically includes the following steps: The predicted forward and reverse motion field information and The merged frame is input to the skip mask generator; After the input stage, the skip mask generator extracts the features of the input data through a series of convolutional layers and residual blocks. The convolutional layer captures the local features in the input data, including edges, textures, shapes, etc. by applying convolution operations; the residual block solves the gradient vanishing problem in deep networks by introducing skip connections, retaining part of the information of the input data, and combining it with the features processed by the convolutional layer to generate a richer feature representation; The output range is scaled to between 0 and 1 using the Sigmoid function. The Sigmoid function is a non-linear activation function that maps any real value to the range between 0 and 1. This allows the output of the network to be interpreted as a probability or confidence, indicating the likelihood of each pixel or region being skipped (not transmitted).

[0015] Use the normalization operation to convert the output of the Sigmoid function into a binary skip mask The normalization operation converts the probability value of each pixel or region to 0 or 1, thereby generating a binary skip mask. In this binary skip mask, pixels or regions with a value of 0 represent samples that should be skipped (not transmitted), while pixels or regions with a value of 1 represent samples that should be transmitted and processed.

[0016] As a further limitation of the technical solution of the present invention, step S6 specifically includes the following steps: According to the binary skip mask , divide the current frame into a skip area and a high-quality coding area; For skipped areas, there is no need to re-encode the pixel information of these areas. These pixels can be directly restored from the reference frame, reducing the amount of data transmitted for these areas. For some skipped areas with small changes but not completely static, a low bit rate encoding method is selected to transmit only the most basic feature information; For areas that require high-quality encoding, the conditional enhanced normalized flow codec is used for processing. The conditional enhanced normalized flow uses a regularized flow mechanism to encode the complex parts of the current frame (high-quality areas) into a latent representation through a series of reversible transformations, which allows more efficient compression while preserving the details of the area; During the decoding process, the pixels of the skipped area will be directly restored from the previous and next reference frames. This motion compensation-based strategy takes advantage of the similarity between frames in the video sequence and avoids unnecessary repeated transmission. The low-quality encoded part restores these areas by receiving low-quality features. Although the accuracy is low, it is enough to maintain the visual consistency of these areas. After receiving the latent representation of the high-quality area, the conditional enhanced normalized stream decoder decodes the latent representation back to the original image data through a series of inverse operations (the opposite of the encoding process) to restore the details of the high-quality area. These areas are encoded with high quality to ensure that the decoded image is consistent with the original frame, especially in the foreground, moving objects or areas with rich details, the reconstructed visual effect is very close to the original input; In the decoding stage, the multi-frame merging network combines the decoding result of the current frame with the information of the reference frame to further optimize the quality of the reconstructed frame. By integrating the information of multiple reference frames, the network generates the final decoded frame. , minimizing the loss of detail.

[0017] Compared with the prior art, the present invention has the following beneficial effects: 1. The dual-layer conditional enhanced normalized stream video codec proposed by the present invention for efficient video decoding does not need to transmit motion information, and achieves the goal of video compression through a two-layer image coding structure. The bottom layer provides downsampled unpredictable pixel information, avoiding the complexity of motion estimation and compensation in the traditional hybrid coding structure, thereby significantly reducing the computational complexity.

[0018] 2. By introducing a skip mask generator and skip mode coding, the model only transmits important potential samples in the enhancement layer, which greatly reduces bit consumption and improves coding efficiency. This strategy effectively reduces the bit rate while maintaining image quality.

[0019] 3. Compared with the traditional hybrid coding system, the present invention saves about 45% and 27% of multiplication and addition operations in the encoding and decoding processes respectively, greatly reducing the demand for computing resources, especially at the encoding end. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 The internal structure diagram of the double-layer conditionally enhanced normalized stream video encoder provided by an embodiment of the present invention is shown.

[0022] Figure 2 A schematic structural diagram of a basic layer component provided by an embodiment of the present invention is shown.

[0023] Figure 3 A schematic diagram of the structure of a multi-frame merging network provided by an embodiment of the present invention is shown.

[0024] Figure 4 A schematic structural diagram of a skip mask generator provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0026] Understandably, existing hybrid coding techniques face several significant problems in video compression. First, hybrid coding relies on motion estimation and compensation to reduce temporal redundancy, but this process is computationally complex, especially at HD and UHD resolutions, which limits the encoding speed. Second, residual coding is used to reduce spatial redundancy, but it mainly relies on local block partitioning strategies and has difficulty handling complex or global image features, especially in high-dynamic scenes or areas with rich details, and the compression effect is often not ideal. Finally, in order to accurately describe the changes in the motion field of objects, hybrid coding requires the transmission of a large number of bits, especially in complex scenes. Standards such as HEVC use multiple block partitioning strategies to define areas with shared motion vectors, but accurately encoding these areas will significantly increase bit consumption.

[0027] To solve the above problems, an embodiment of the present invention provides a new method for deep video encoding and decoding that supports multiple computational complexities. The specific steps of the method are as follows: Figure 1 The internal structure diagram of the double-layer conditionally enhanced normalized stream video encoder provided by an embodiment of the present invention is shown.

[0028] S1, inputting the reference frame into the frame interpolator to generate the conditional frame of the base layer conditional enhanced normalized stream compressor; In an embodiment of the present invention, the reference frame is input , and a step size ranging from 0 to 1, according to which the number of intermediate frames to be synthesized is determined; the intermediate flow network is used to estimate the intermediate flow information of the input frame; based on the estimated intermediate flow information, the reference frame is backward warped (warping refers to transforming pixels in an image or video by moving pixels from their original positions to new positions to transform an image or frame.) to obtain a preliminary estimate of the intermediate frame; the warped frame is fused using a fusion network to generate the final intermediate frame; the intra-frame interpolator outputs a high-quality interpolated frame .

[0029] S2, the downsampling network downsamples the input high-resolution frame to a low-resolution frame, and the conditional enhanced normalized flow compressor compresses the low-resolution frame; In this embodiment, the input frame With interpolated frames Input to the downsampling network, and after downsampling processing, we get and ; Base layer conditional enhancement normalized stream compressor downsampled target frame As input, Encode the conditional signal, that is, convert the image data into a more compact representation; use conditional normalization technology (a technology that adjusts the normalization parameters according to the conditional signal). The weights and biases of the convolutional layer and other related Normalize the relevant parameters to make the encoder better adapt to the characteristics of the input data, thereby improving the efficiency and quality of encoding; after the processing of the encoding network and the adjustment of the conditional normalization technology, the base layer conditional enhanced normalized stream compressor finally outputs the compressed image data . Figure 2 The schematic diagram of the structure of the base layer component downsampling network provided by the embodiment of the present invention is shown. Specifically: Input frame With interpolated frames The input is sent to the downsampling network, which contains convolutional layers and residual blocks. Two convolutional layers are used to process the input image. The step sizes of the two convolutional layers are set to 2 (the step size refers to the step size of the convolution kernel moving on the image). When the step size is 2, the convolution kernel moves two pixels each time, thereby achieving image downsampling with a downsampling factor of 4. The residual block introduces a shortcut connection (also called a jump connection) to add the input directly to the output, thereby avoiding the gradient vanishing problem that may occur in deep networks, maintaining feature integrity, and fusing features of different layers through shortcut connections to enhance the feature reuse capability. The output is a downsampled frame. and The resolution of the video frame is reduced to 1 / 4 of the original, and the width and height of the image are reduced to 1 / 2 of the original, reducing the amount of image data and facilitating more efficient storage or transmission.

[0030] S3, the super-resolution reconstruction network restores the decoded base layer low-resolution frame to a full-resolution frame; In this embodiment, Figure 2 The figure shows a structural diagram of a super-resolution reconstruction network of a base layer component provided by an embodiment of the present invention, in which Conv2D refers to a two-dimensional convolutional layer; ConvTranspose2D refers to a two-dimensional deconvolutional layer; c refers to the number of channels of the convolutional layer; k refers to the size of the convolution kernel; s is the stride of the convolution; PReLU and ReLU refer to activation functions; and ResBlock refers to a residual block. Specifically, the super-resolution reconstruction network receives the decoded base layer low-resolution frame as input; the input low-resolution frame is subjected to feature extraction through the convolution layer, which captures key information such as edges and textures in the image, and provides necessary feature representation for the subsequent reconstruction process; based on the feature extraction, the extracted low-resolution features are mapped to the high-resolution feature space using a deep learning model. In this process, sub-pixel convolution is used to achieve feature upsampling and amplification, completing the conversion from low-resolution features to high-resolution images; after feature mapping and amplification, the network outputs a preliminary high-resolution image, and post-processing operations are used to further optimize the reconstruction results; finally, after super-resolution reconstruction processing, the super-resolution reconstruction network outputs a full-resolution frame that is close to or better than the original high-resolution frame.

[0031] S4, inputting the video frame information of the base layer and the predicted forward and reverse motion field information into the multi-frame merging network in the enhancement layer to merge them into a high-quality predicted frame; In this embodiment, Figure 3 A schematic diagram of the structure of a multi-frame merging network provided by an embodiment of the present invention is shown. and Refers to the inter-frame motion field information; M m 、M 1 m 、M 2 m 、M 3 m Refers to the multi-frame merging weight map; Refinememt refers to the refinement module, which is used to refine the merged image to improve the quality of the final frame. Specifically, the predicted forward and reverse motion field information is obtained from the base layer frame interpolator and ; Video frame information of the base layer and predicted forward and reverse motion field information , Input a multi-frame merging network, which is designed with two output channels with weight maps. These weight maps are used to combine information from different video frames. Based on the similarities and differences between the input video frames and the feature representations learned by the network, a weight map is generated to determine the contribution of each input video frame in the final merged output. These weight maps are used to combine the input video frames, multiplying the pixel values ​​of each input video frame with the corresponding weights, and then adding the weighted pixel values ​​to generate the final merged output. Output a high-quality predicted frame. This predicted frame combines the video frame information from the base layer and the enhancement layer, while retaining important motion details and clear background information, which can be used for subsequent video processing tasks.

[0032] S5, a skip mask generator generates a binary skip mask according to the predicted forward and reverse motion field information and the predicted frame; In this embodiment, Figure 4 A schematic diagram of the structure of a skip mask generator provided by an embodiment of the present invention is shown, in which: , They refer to the mean parameter and standard deviation parameter of the input frame variable generated by the decoder; μ and σ represent the Gaussian distribution parameters generated from the conditional enhanced normalized stream compressor, with standard mean and standard deviation respectively; cat represents the feature concatenation operation, which concatenates multiple feature maps in the channel dimension; LeakyReLU represents a ReLU activation function with leakage, which allows some negative values ​​to pass; Sigmoid represents an activation function that compresses the output to between 0 and 1, which is used to generate a probability map of the skip mask; Round refers to the rounding operation, which discretizes the generated mask value into a binary value, which is used to determine the final skip mask. Specifically, the predicted forward and reverse motion field information is and The merged frame is input to the skip mask generator; the convolution operation is applied through the convolution layer to capture the local features in the input data, including edges, textures, shapes, etc.; the residual block solves the gradient vanishing problem in deep networks by introducing skip connections, retaining part of the input data information, and combining it with the features processed by the convolution layer to generate a richer feature representation; the output range is scaled to between 0 and 1 using the Sigmoid function; the output of the Sigmoid function is converted into a binary skip mask using the normalization operation .

[0033] S6. The conditionally enhanced normalized stream compressor encodes the target frame with high quality according to the binary skip mask.

[0034] In this embodiment, according to the binary skip mask , the current frame is divided into two different areas: the skipped area and the high-quality coding area; for the skipped area, there is no need to re-encode the pixel information of these areas. These pixels can be directly restored from the reference frame, reducing the amount of data transmitted for these areas. For some skipped areas with small changes but not completely static, a low-bitrate encoding method is selected to transmit only the most basic feature information; for areas that require high-quality coding, the conditional enhanced normalized stream codec is used for processing. The conditional enhanced normalized stream uses the regularized stream mechanism to encode the complex part of the current frame (high-quality area) into a potential representation through a series of reversible transformations, which allows more efficient compression while retaining the details of the area; during the decoding process, the pixels of the skipped area will be directly restored from the previous and next reference frames. This motion compensation-based strategy exploits the similarity between frames in the video sequence and avoids unnecessary repeated transmission. The low-quality coded parts restore these areas by receiving low-quality features. Although the accuracy is low, it is enough to maintain the visual consistency of these areas; after receiving the potential representation of the high-quality area, the conditional enhanced normalized stream decoder decodes the potential representation back to the original image data through a series of inverse operations (opposite to the encoding process) to restore the details of the high-quality area. These areas are encoded with high quality to ensure that the decoded image is consistent with the original frame. Especially in the foreground, moving objects or areas with rich details, the reconstructed visual effect is very close to the original input. In the decoding stage, the multi-frame merging network combines the decoding result of the current frame with the information of the reference frame to further optimize the quality of the reconstructed frame. By integrating the information of multiple reference frames, the decoded frame is finally generated. .

[0035] The results of this invention effectively reduce bit consumption, achieve more efficient compression rate, reduce hardware and energy requirements, and are very suitable for bandwidth-constrained scenarios such as streaming media and mobile devices. And while maintaining high compression efficiency, it ensures high-quality decoded frames, which is suitable for scenarios with high image quality requirements such as high-definition television broadcasting and video on demand.

[0036] It should be understood that, although each step in the flow chart of each embodiment of the present invention is shown in sequence according to the indication of the arrow, these steps are not necessarily performed in sequence according to the order indicated by the arrow. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be performed in other orders. Moreover, at least a portion of the steps in each embodiment may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these sub-steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a portion of other steps or sub-steps or stages of other steps.

[0037] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0038] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

Claims

1. A video coding method based on a two-layer conditional enhanced normalized flow compressor, It is characterized in that The following steps are involved: S1, inputting the reference frame into the frame interpolator to generate the conditional frame of the base layer conditional enhanced normalized stream compressor; S2, the downsampling network downsamples the input high-resolution frame to a low-resolution frame, and the conditional enhanced normalized flow compressor compresses the low-resolution frame; S3, the super-resolution reconstruction network restores the decoded base layer low-resolution frame to a full-resolution frame; S4, inputting the video frame information of the base layer and the predicted forward and reverse motion field information into the multi-frame merging network in the enhancement layer to merge them into a high-quality predicted frame; S5, a skip mask generator generates a binary skip mask according to the predicted forward and reverse motion field information and the predicted frame; S6. The conditionally enhanced normalized stream compressor encodes the target frame with high quality according to the binary skip mask.

2. A video coding method based on a double-layer conditional enhanced normalized flow compressor as claimed in claim 1, It is characterized in that Step S1 specifically includes the following steps: Input reference frame , and a step length ranging from 0 to 1, according to which the number of intermediate frames to be synthesized is determined; Use the intermediate flow network to estimate the intermediate flow information of the input frame; Based on the estimated intermediate flow information, the reference frame is backward warped to obtain a preliminary estimate of the intermediate frame; The warped frames are fused using a fusion network to generate the final intermediate frame. Intra-frame interpolator outputs high-quality interpolated frames , the image information will be used as the conditional signal for the base layer conditional enhancement normalized stream compressor.

3. A video coding method based on a double-layer conditional enhanced normalized flow compressor as claimed in claim 1, It is characterized in that Step S2 specifically includes the following steps: Input frame With interpolated frames Input to the downsampling network, and after downsampling processing, we get and ; Base layer conditional enhancement normalized stream compressor downsampled target frame As input, Encoding the conditional signal to convert the image data into a more compact representation; During the encoding process, conditional adaptive normalization technology is used to The weights and biases of the convolutional layer and other related Normalize the relevant parameters to make the encoder better adapt to the characteristics of the input data, thereby improving the efficiency and quality of encoding; After the processing of the encoding network and the adjustment of the conditional normalization technology, the base layer conditional enhanced normalized stream compressor finally outputs the compressed image data ,The compressed data usually exists in the form of a bit stream and can be used for subsequent storage, transmission or further processing.

4. A video coding method based on a double-layer conditional enhanced normalized stream compressor as claimed in claim 3, It is characterized in that The specific process of the downsampling process is as follows: Input frame With interpolated frames Input to the downsampling network, which contains convolutional layers and residual blocks; Use two convolutional layers to process the input image. The stride of these two convolutional layers is set to 2. When the stride is 2, the convolution kernel moves two pixels each time, thereby achieving image downsampling with a downsampling factor of 4. The residual block directly adds the input to the output by introducing shortcut connections, and fuses the features of different layers through shortcut connections; Output downsampled frames and .

5. A video coding method based on a double-layer conditional enhanced normalized flow compressor as claimed in claim 1, It is characterized in that Step S3 specifically includes the following steps: The super-resolution reconstruction network receives the decoded base layer low-resolution frame as input; The convolutional layer extracts features from the input low-resolution frame. The convolutional layer captures the edge and texture information in the image and provides the necessary feature representation for the subsequent reconstruction process. Based on feature extraction, a deep learning model is used to map the extracted low-resolution features to a high-resolution feature space. In this process, sub-pixel convolution is used to achieve feature upsampling and amplification, completing the conversion from low-resolution features to high-resolution images. After feature mapping and amplification, the network outputs a preliminary high-resolution image, and post-processing operations are used to further optimize the reconstruction results; Finally, after super-resolution reconstruction processing, the super-resolution reconstruction network outputs a full-resolution frame that is close to or better than the original high-resolution frame.

6. A video coding method based on a double-layer conditional enhanced normalized flow compressor as claimed in claim 5, It is characterized in that The post-processing operation to further optimize the reconstruction result specifically includes the following steps: De-noising, sharpening and color correction are performed on the preliminary high-resolution images that have some defects or do not conform to the details of the real scene; Image noise is removed by using a mean filter. First, a small rectangular window is selected, which will be used to calculate the neighborhood average of the current pixel. For each pixel in the image, the average value of all pixel values ​​in its neighborhood is then calculated. Finally, the calculated average value is assigned to the current pixel as the denoised pixel value, and the denoised image is output. Laplace operator sharpening is used to improve the detail clarity and edge recognition ability of the image. First, the second-order differential operation is performed on each pixel in the image by constructing a Laplace convolution kernel to capture the area where the grayscale value changes dramatically. Then, the convolution kernel is applied to the image, and the second-order derivative value of each pixel is obtained by convolution calculation. This value reflects the rate of change of the surrounding grayscale value. Subsequently, the grayscale value of each pixel in the image is adjusted according to these second-order derivative values, that is, the second-order derivative value is added to the original pixel value, and normalized to ensure that the pixel value is between 0 and 255, thereby enhancing the edges and details. Finally, the adjusted image is weighted summed with the original image, and the final sharpened image is superimposed. Color correction is achieved by adjusting image parameters such as brightness, contrast, and saturation, ultimately obtaining a further optimized and reconstructed image.

7. A video encoding method based on a double-layer conditional enhanced normalized flow compressor as claimed in claim 1, It is characterized in that Step S4 specifically includes the following steps: Get the predicted forward and backward motion field information from the base layer frame interpolator and ; Base layer video frame information and predicted forward and reverse motion field information , Input multi-frame merging network, which is designed with two output channels with weight maps, these weight maps are used to combine information from different video frames; Generate a weight map based on the similarities and differences between the input video frames and the feature representations learned by the network to determine the contribution of each input video frame to the final merged output; Use these weight maps to combine the input video frames, multiply the pixel value of each input video frame by the corresponding weight, and then add the weighted pixel values ​​to generate the final combined output; Output a high-quality predicted frame.

8. A video encoding method based on a double-layer conditional enhanced normalized flow compressor as claimed in claim 7, It is characterized in that The forward and reverse motion field information is obtained from the base layer frame interpolator and , specifically including the following steps: Pass the reference frame from the decoded frame buffer and Input frame interpolator, the reference frame is relative to the current frame The previous and next frames; By analyzing the previous and next frames and The frame interpolator computes the motion vector field between them, which represents how objects move between these frames. Used to describe the previous frame To current frame sports; The optical flow method is used to calculate the motion field between frames. The optical flow method estimates the motion vector by tracking the movement of pixels in the frame, which represents the movement of pixels from the reference frame to the reference frame. The position change to the current frame; The interpolator not only generates To the motion field of the current frame , and also generates To the motion field of the current frame ; The generated motion field information is used to generate interpolated frames .

9. A video coding method based on a double-layer conditional enhanced normalized flow compressor as claimed in claim 1, It is characterized in that Step S5 specifically includes the following steps: The predicted forward and reverse motion field information and The merged frame is input to the skip mask generator; After the input stage, the skip mask generator extracts the features of the input data through convolutional layers and residual blocks. The convolutional layer captures local features in the input data, including edges, textures, shapes, etc. by applying convolution operations; the residual block solves the gradient vanishing problem in deep networks by introducing skip connections, retaining part of the information of the input data and combining it with the features processed by the convolutional layer to generate a richer feature representation; After the feature extraction stage, the skip mask generator uses a sigmoid function to scale the output range to between 0 and 1; Finally, the skip mask generator uses a normalization operation to convert the output of the sigmoid function into a binary skip mask. , the normalization operation converts the probability value of each pixel or region to 0 or 1, thereby generating a binary skip mask in which pixels or regions with a value of 0 represent samples that should be skipped, while pixels or regions with a value of 1 represent samples that should be transmitted and processed.

10. A video encoding method based on a double-layer conditional enhanced normalized flow compressor as claimed in claim 1, It is characterized in that Step S6 specifically includes the following steps: According to the binary skip mask , divide the current frame into two different areas: Skip area: The part with a value of 1 in the mask represents those areas that do not need to be processed finely and do not require a large amount of data to describe; High-quality coding area: The part with a value of 0 in the mask represents the part of the image that needs to retain details; For skipped areas, there is no need to re-encode the pixel information of these areas, and these pixels can be directly restored from the reference frame; for skipped areas that are not completely still, a low-bit-rate encoding method is selected to transmit only the most basic feature information; For areas that require high-quality encoding, the conditional enhanced normalized flow codec is used for processing. The conditional enhanced normalized flow uses a regularized flow mechanism to encode the high-quality area of ​​the current frame into a latent representation through a reversible transformation, which allows more efficient compression while preserving the details of the area. During the decoding process, the pixels of the skipped area will be directly restored from the previous and next reference frames, and the low-quality encoded part will restore these areas by receiving low-quality features to maintain the visual consistency of these areas; After receiving the latent representation of the high-quality area, the conditional enhanced normalized stream decoder decodes the latent representation back to the original image data through the inverse operation to restore the details of the high-quality area; In the decoding stage, the multi-frame merging network combines the decoding result of the current frame with the information of the reference frame to further optimize the quality of the reconstructed frame.

Images