Video intelligent compression system based on generative AI super-resolution reconstruction

Abstract

The application relates to the technical field of video processing, in particular to a video intelligent compression system based on generative AI super-resolution reconstruction, which comprises a data processing unit, which is used for carrying out content analysis and key frame extraction processing on low-resolution video data and original video data to obtain a video key frame sequence and a non-key frame feature flow; a corresponding semantic mask is generated according to the video key frame sequence; a super-resolution reconstruction unit is used for inputting the non-key frame feature flow, the semantic mask and control parameter data into a pre-trained generative super-resolution model to carry out super-resolution reconstruction and obtain high-resolution super-resolution reconstruction video data; residual data between the original video data and the super-resolution reconstruction video data is calculated, and a texture complexity mapping diagram is determined based on the residual data. The super-resolution reconstruction is carried out through the generative super-resolution model, the visual quality of the video can be significantly improved, and the finally output video can retain more details and definition under high resolution.

Description

Technical Field

[0001] This invention relates to the field of video processing technology, specifically to a video intelligent compression system based on generative AI super-resolution reconstruction. Background Technology

[0002] Currently, traditional systems typically rely on fixed encoding algorithms, making it difficult to dynamically and adaptively adjust to different video content and scenes. This results in the failure to fully utilize relevant control parameters when processing low-resolution video data, leading to a waste of storage and transmission data. Furthermore, many traditional compression methods sacrifice video visual quality, especially in terms of detail preservation and clarity, in pursuit of high compression ratios. As a result, the reconstructed video often suffers from problems such as blurriness, noise, and distortion, affecting the viewing experience.

[0003] Furthermore, traditional systems typically use time intervals or fixed thresholds for keyframe extraction, lacking in-depth analysis of scene dynamics. This can easily lead to the omission of important information or the processing of redundant data, reducing overall data processing efficiency. In addition, traditional encoders often cannot automatically select appropriate encoding modes, such as low latency or high fidelity, based on content characteristics. This means that the encoding effect cannot reach its optimal state in different application scenarios, affecting the performance of real-time video streams and online live broadcasts. Summary of the Invention

[0004] To achieve the above objectives, the present invention provides the following technical solution: a video intelligent compression system based on generative AI super-resolution reconstruction, comprising: The data acquisition unit is used to acquire raw video data, perform downsampling and basic encoding on the raw video data, and obtain low-resolution video data and control parameter data. The data processing unit is used to perform content parsing and keyframe extraction on low-resolution video data and raw video data to obtain video keyframe sequences and non-keyframe feature streams; and to generate corresponding semantic masks based on the video keyframe sequences. The super-resolution reconstruction unit is used to input non-key frame feature streams, semantic masks, and control parameter data into a pre-trained generative super-resolution model to perform super-resolution reconstruction and obtain high-resolution super-resolution reconstructed video data; it also calculates the residual data between the original video data and the super-resolution reconstructed video data and determines the texture complexity map based on the residual data. The data completion unit is used to perform generative completion on the high-frequency details in the residual data based on the generative repair model and the texture complexity map, so as to obtain optimized high-resolution video data. The feature fusion unit is used to fuse the optimized high-resolution video data with the video keyframe sequence to obtain a fused reconstructed video frame sequence; based on the control parameter data and the fused reconstructed video frame sequence, the corresponding encoding mode is selected, and entropy encoding is performed on the fused reconstructed video frame sequence to obtain the encoded binary bitstream; The intelligent compression unit is used to encapsulate the encoded binary bitstream to obtain the final intelligent compressed bitstream corresponding to the original video data.

[0005] Preferably, the original video data is downsampled and fundamentally encoded to obtain low-resolution video data and control parameter data, including: The original video data is spatially compressed according to a preset downsampling ratio to generate low-resolution basic video data. Scene feature vectors, motion vector information, and tone mapping parameters are extracted from the original video data, and these parameters are used as the basic data for control parameters. The low-resolution basic video data is synchronized and correlated with the basic control parameter data to obtain low-resolution video data and control parameter data.

[0006] Preferably, content parsing and keyframe extraction are performed on the low-resolution video data and the original video data to obtain a video keyframe sequence and a non-keyframe feature stream, including: Dynamic scene analysis is performed on the raw video data, and the feature differences of low-resolution video data are combined to obtain frames with significant scene changes; Frames with significant scene changes are identified as video keyframes, and high-dimensional features corresponding to the keyframes are extracted to obtain a video keyframe sequence. The remaining frames in the original video data, excluding keyframes, are subjected to feature dimensionality reduction processing. Combined with the corresponding frame features of the low-resolution video data, a low-dimensional non-keyframe feature stream is obtained.

[0007] Preferably, a semantic mask is generated based on the video keyframe sequence, including: Input the sequence of keyframes from the video into the semantic segmentation model to obtain the object category probability map for each keyframe; Set a preset saliency threshold, and mark the areas in the object category probability map that exceed the threshold as salient object areas, and the areas that are below the threshold as background areas; Morphological dilation is performed on the salient object region to obtain a binary semantic mask that covers the entire object.

[0008] Preferably, the non-keyframe feature stream, semantic mask, and control parameter data are input into a pre-trained generative super-resolution model for super-resolution reconstruction, resulting in high-resolution super-resolution reconstructed video data, including: Analyze the scene feature vectors in the control parameter data to determine the network weights of the generative super-resolution model corresponding to the current video scene; The non-keyframe feature stream and semantic mask are input into the generative super-resolution model, and upsampled and feature-enhanced according to the determined network weights to generate initial high-resolution video data. Temporal consistency alignment is performed on the initial high-resolution video data based on the motion vector information in the control parameter data to obtain high-resolution super-resolution reconstructed video data.

[0009] Preferably, the residual data between the original video data and the super-resolution reconstructed video data is calculated, and a texture complexity mapping is determined based on the residual data, including: Pixel-level subtraction is performed on the original video data and the super-resolution reconstructed video data to obtain residual data; The residual data is traversed using a preset sliding window, and the pixel variance value within each window is calculated. The pixel variance values ​​calculated for each window are normalized to generate a texture complexity map.

[0010] Preferably, based on the generative inpainting model, and combined with the texture complexity map, generative inpainting is performed on the high-frequency details in the residual data to obtain optimized high-resolution video data, including: The residual data and super-resolution reconstructed video data are input into the encoder of the generative inpainting model to extract residual feature maps; Based on the numerical distribution of the texture complexity map, the generative repair model is controlled to perform corresponding processing in different regions: smoothing and denoising in the first numerical region, and generating high-frequency details in the second numerical region. The high-frequency compensation details are generated by the decoder of the generative inpainting model and then superimposed onto the super-resolution reconstructed video data to obtain optimized high-resolution video data.

[0011] Preferably, the optimized high-resolution video data is fused with the video keyframe sequence to obtain a fused and reconstructed video frame sequence, including: Extract detailed features from optimized high-resolution video data and high-dimensional features from video keyframe sequences; We perform weighted fusion of detailed features and high-dimensional features to retain the core features of keyframes and the detailed features of optimized data. The fused features are then reassembled into a frame sequence to obtain the fused and reconstructed video frame sequence.

[0012] Preferably, based on the control parameter data and the fused and reconstructed video frame sequence, a corresponding encoding mode is selected, and entropy encoding is performed on the fused and reconstructed video frame sequence to obtain the encoded binary bitstream, including: The complexity of analyzing motion vector information in control parameter data and fusing and reconstructing video frame sequences determines whether the encoding mode is low-latency encoding mode or high-fidelity encoding mode. In low-latency coding mode, only the residual information between the fused and reconstructed video frame sequence and the video keyframe is entropy encoded. If it is a high-fidelity coding mode, joint entropy coding is performed on the compressed representation of motion vectors, residual information and semantic masks in the fused and reconstructed video frame sequence.

[0013] Preferably, the encoded binary bitstream is encapsulated to obtain the final intelligent compressed bitstream corresponding to the original video data, including: The entropy-encoded binary code stream is associated and matched with the control parameter data; The associated binary stream and control parameter data are packaged into a package that conforms to the network transmission format; The integrity of the package is verified. Once the verification is successful, the final intelligent compressed bitstream is obtained.

[0014] Compared with the prior art, the beneficial effects of the present invention are: (1) By combining low-resolution video data and control parameter data, the system can effectively reduce the amount of data that needs to be stored and transmitted, thereby improving the compression efficiency of video encoding; and by performing super-resolution reconstruction through a generative super-resolution model, the visual quality of the video can be significantly improved, so that the final output video retains more details and clarity at high resolution. (2) Through scene dynamic analysis and feature extraction, the system can more accurately identify key frames, thereby reducing the processing of redundant information and improving the overall data processing efficiency; and by using video content characteristics and control parameter data, the system can intelligently select suitable encoding modes, such as low latency or high fidelity, so as to achieve the best encoding effect in different application scenarios. (3) By generating corresponding semantic masks, the system can understand video content, thereby more adaptively handling different types of scenes and objects during video reconstruction and compression, improving the accuracy of visual effects; and by performing integrity verification on the final intelligent compressed bitstream, the reliability of data during transmission is ensured, avoiding playback problems caused by data corruption. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of the overall system architecture in one embodiment of the present invention.

[0016] In the diagram: 1. Data acquisition unit; 2. Data processing unit; 3. Super-resolution reconstruction unit; 4. Data completion unit; 5. Feature fusion unit; 6. Intelligent compression unit. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] Example 1, please refer to Figure 1 This invention provides a technical solution: a video intelligent compression system based on generative AI super-resolution reconstruction, comprising: Data acquisition unit 1 is used to acquire raw video data, perform downsampling processing and basic encoding on the raw video data, and obtain low-resolution video data and control parameter data. Data processing unit 2 is used to perform content parsing and keyframe extraction on low-resolution video data and raw video data to obtain video keyframe sequences and non-keyframe feature streams; and to generate corresponding semantic masks based on the video keyframe sequences. Super-resolution reconstruction unit 3 is used to input non-key frame feature streams, semantic masks and control parameter data into a pre-trained generative super-resolution model to perform super-resolution reconstruction and obtain high-resolution super-resolution reconstructed video data; calculate the residual data between the original video data and the super-resolution reconstructed video data, and determine the texture complexity mapping based on the residual data; Data completion unit 4 is used to perform generative completion on the high-frequency details in the residual data based on the generative repair model and the texture complexity map, so as to obtain optimized high-resolution video data. Feature fusion unit 5 is used to fuse the optimized high-resolution video data with the video keyframe sequence to obtain a fused reconstructed video frame sequence; according to the control parameter data and the fused reconstructed video frame sequence, the corresponding encoding mode is selected, and entropy encoding is performed on the fused reconstructed video frame sequence to obtain the encoded binary bitstream; The intelligent compression unit 6 is used to encapsulate the encoded binary bitstream to obtain the final intelligent compressed bitstream corresponding to the original video data.

[0019] It should be noted that the data acquisition unit is mainly responsible for acquiring the raw video data, which is the starting point of the entire video processing flow. After acquiring the raw video, it will be downsampled to facilitate subsequent processing and reduce the amount of computation. Downsampling is like reducing the size of the video, reducing the resolution of the video, and obtaining low-resolution video data. At the same time, control parameter data will also be generated. This parameter data is like an instruction manual for the subsequent processing, which will guide other units on how to operate. The data processing unit processes both low-resolution video data and raw video data simultaneously. On one hand, it performs content parsing and keyframe extraction. Keyframes are like important shots in a video, summarizing the main content of the video. By extracting keyframes, a video keyframe sequence is obtained, and the remaining part is the non-keyframe feature stream. On the other hand, it generates corresponding semantic masks based on the extracted video keyframe sequence. A semantic mask can be understood as a mark added to the keyframe to identify different semantic regions in the keyframe, such as people and backgrounds. The super-resolution reconstruction unit inputs the non-keyframe feature stream, semantic mask, and previously generated control parameter data into the pre-trained generative super-resolution model; the generative inpainting model includes an input module, a feature extraction module, a feature fusion module, a data reconstruction module, and an output module. The input module is responsible for receiving residual data and texture complexity maps from the super-resolution reconstruction unit; the residual data reflects the differences between the original video data and the super-resolution reconstructed video data, and contains information that needs to be repaired and completed. The texture complexity map identifies the texture complexity of different regions in the video, providing an important reference for the restoration process. The input module preprocesses this input data, such as performing normalization operations, to adjust the data to a numerical range suitable for the model to process, thereby improving the model's training efficiency and restoration effect. The feature extraction module typically consists of multiple convolutional layers and pooling layers. Convolutional layers slide their kernels across the input data to extract features at different levels. For example, shallow convolutional layers can extract local features such as edges and textures from residual data, while deeper convolutional layers can capture more abstract and global features. Pooling layers are used to reduce the dimensionality of the feature maps, reduce computation, and enhance the model's translation invariance. After being processed by the encoder, the input data is transformed into a low-dimensional feature representation. Introducing an attention mechanism during feature extraction allows the model to focus more on important feature regions in the residual data. For example, for regions with high texture complexity, the model will allocate more attention resources to extract features from these regions more accurately. The attention mechanism can be implemented by calculating the correlation between different locations in the feature map and then weighting and fusing features according to the correlation weights to highlight important features. The feature fusion module fuses the features extracted by the encoder with the texture complexity map. The texture complexity map is combined with the feature map output by the encoder in a specific way (such as concatenation, multiplication, etc.), so that the model can fully consider the texture complexity of different regions in the subsequent repair process. For example, the texture complexity map is multiplied element-wise with the feature map, so the weight of regions with higher texture complexity in the feature map will be increased accordingly, and the model will pay more attention to the generation of details in these regions during repair. In the data reconstruction module, the decoder corresponds to the encoder. The decoder consists of multiple deconvolutional layers (or transposed convolutional layers) and upsampling layers. The deconvolutional layers are used to gradually restore the low-dimensional feature representation to high-dimensional data, while the upsampling layers increase the resolution of the data. During the decoding process, the model gradually generates restored high-frequency detail data based on the fused feature information; each deconvolution and upsampling operation combines the feature information of the previous layer to continuously refine the restoration result. High-frequency details are often the fine and rapidly changing parts of a video, such as wrinkles on a person's face or the edges of objects. By completing these details, we can obtain optimized high-resolution video data, which can further improve the video quality. To better preserve the original information of the input data and improve the accuracy of the reconstruction, residual connections can be introduced into the decoder. The features of certain layers in the encoder are directly added to the features of the corresponding layers in the decoder, so that the model can refer to the original feature information when reconstructing the data, thus avoiding information loss and gradient vanishing problems. The output module performs post-processing on the data generated by the decoder, such as performing inverse normalization to restore the data to its original numerical range; at the same time, it performs certain smoothing processing on the output data to remove possible noise and artifacts, thereby obtaining optimized high-resolution video data and completing the generative completion of high-frequency details in the residual data. For example, in the restoration of old movies, the original movies have problems such as low resolution, blurry images, scratches and noise due to their age. After the super-resolution reconstruction unit is processed, the super-resolution reconstructed video data is obtained, but some high-frequency details may still be missing. Generative inpainting models can use residual data and texture complexity maps to accurately fill in these missing high-frequency details. For example, for high-frequency details such as subtle changes in facial expressions of characters in movies and textures on clothing, the model can identify the complexity of these areas based on the texture complexity map, and then extract relevant information from the residual data for repair and completion. The feature fusion unit fuses the optimized high-resolution video data with the previously extracted video keyframe sequence. This step integrates the important scenes of the video with the optimized ordinary scenes, making the entire video more consistent in content and quality, resulting in a fused and reconstructed video frame sequence. Then, based on the control parameter data and the fused and reconstructed video frame sequence, an appropriate encoding mode is selected, and entropy encoding is performed on the fused and reconstructed video frame sequence to obtain the encoded binary bitstream. Entropy encoding is an efficient compression encoding method that can reduce the amount of data. The intelligent compression unit encapsulates the entropy-encoded binary stream. Encapsulation is like packaging the compressed video data, adding necessary header information, identifiers, etc., to make it a complete final intelligent compressed stream that can be played and transmitted on various devices. This stream corresponds to the original video data, but the data volume is greatly reduced.

[0020] In an optional embodiment, the original video data is downsampled and fundamentally encoded to obtain low-resolution video data and control parameter data, including: The original video data is spatially compressed according to a preset downsampling ratio to generate low-resolution basic video data. Scene feature vectors, motion vector information, and tone mapping parameters are extracted from the original video data, and these parameters are used as the basic data for control parameters. The low-resolution basic video data is synchronized and correlated with the basic control parameter data to obtain low-resolution video data and control parameter data.

[0021] It should be noted that downsampling is an operation that reduces the resolution of data. In video processing, it mainly targets the spatial dimensions of the video (i.e., the width and height of the image). The preset downsampling ratio determines the degree to which the original video data is compressed spatially. For example, if the downsampling ratio is 2:1, it means that every other pixel is sampled in both the horizontal and vertical directions. This will reduce the width and height of the video frame to half of their original size, significantly reducing the overall data volume and generating low-resolution base video data. Generating low-resolution video data can reduce the amount of data in subsequent processing and improve processing efficiency; at the same time, in some application scenarios with high real-time requirements, low-resolution video can be transmitted and preliminarily analyzed faster. Scene feature vectors are an abstract representation of a scene in a video, containing various feature information about the scene, such as the scene type, the main objects in the scene, and the scene layout. These features are extracted from the raw video data and converted into vector form for subsequent processing and analysis. Video is composed of a series of consecutive frames, and objects move between different frames; motion vector information is used to describe this motion, which records the direction and magnitude of displacement of each pixel or image block between adjacent frames; Tone mapping is the process of adjusting the color and brightness range of a video. Related parameters are used to control the effect of tone mapping; these parameters can affect the overall color tone, contrast, saturation and other visual characteristics of the video. Combining the extracted scene feature vectors, motion vector information, and tone mapping parameters forms the basic data for control parameters; this data will provide important reference information for subsequent video processing. The low-resolution basic video data and the control parameter basic data are interrelated. The control parameter basic data is a supplementary description of the low-resolution basic video data. Through synchronous association, it can be ensured that the corresponding control parameters can be accurately used according to the specific content of the low-resolution video data in subsequent processing. After synchronization and association, the low-resolution basic video data and the control parameter basic data are respectively formed into low-resolution video data and control parameter data; these two parts of data will be used as a whole for subsequent video processing, such as super-resolution reconstruction and video encoding.

[0022] In an optional embodiment, content parsing and keyframe extraction are performed on low-resolution video data and raw video data to obtain a video keyframe sequence and a non-keyframe feature stream, including: Dynamic scene analysis is performed on the raw video data, and the feature differences of low-resolution video data are combined to obtain frames with significant scene changes; Frames with significant scene changes are identified as video keyframes, and high-dimensional features corresponding to the keyframes are extracted to obtain a video keyframe sequence. The remaining frames in the original video data, excluding keyframes, are subjected to feature dimensionality reduction processing. Combined with the corresponding frame features of the low-resolution video data, a low-dimensional non-keyframe feature stream is obtained.

[0023] It should be noted that scene dynamic analysis is an in-depth exploration of how different scenes in the original video change over time; by analyzing the content of each frame in the video, information such as objects, human activities, and background changes in the scene is identified, and the dynamic evolution of this information between different frames is observed. Low-resolution video data is obtained by downsampling and other processing of the original video. It is compressed in the spatial dimension, but retains the basic structure and general content of the original video. By comparing the scene dynamic analysis results of the original video with the features of the low-resolution video data, the differences between the two can be found. Because low-resolution video data loses some detailed information, when the scene in the original video changes significantly, this change may not be obvious in the low-resolution video data. However, through comparative analysis, it is still possible to find those frames that show significant scene changes in the original video and also have certain feature differences in the low-resolution video data. Identify the frames where the scene changes most significantly; these frames are the frames with significant scene changes. They usually mark the switching of important scenes, the occurrence of key events, or major changes in the content of the scene in the video. Video keyframes are representative frames in a video that can summarize the main content and key plot points of the video. Frames with significant scene changes are identified as video keyframes because these frames can accurately reflect the changes in the scene in the video, which is of great significance for understanding the overall structure and content of the video. High-dimensional features provide a more comprehensive and in-depth description of keyframe content; they not only include visual information such as color, texture, and shape, but may also involve semantic information such as the category of objects and the type of scene in the keyframe; extracting these high-dimensional features from keyframes can more accurately characterize the features of the keyframes. Arranging all the identified keyframes in chronological order in the original video creates a video keyframe sequence; this sequence can concisely display the main content and plot development of the video, facilitating subsequent video processing and analysis. Non-key frames in the original video contain a large amount of data, but they may contain many redundant and unnecessary features. Feature dimensionality reduction is a process that reduces the feature dimension of these non-key frames by removing features that have little impact on understanding the video content and retaining the most important features. This can reduce the complexity of data processing and improve processing efficiency. Although the corresponding frames of low-resolution video data have lower resolution, they also contain the basic feature information of the original video frames. By combining the non-key frame features after feature dimensionality reduction with the corresponding frame features of low-resolution video data, we can make full use of the information of low-resolution video data and further optimize the feature representation of non-key frames. After the above processing, what is obtained is a low-dimensional non-keyframe feature stream; this feature stream describes the feature information of non-keyframes in the original video in a low-dimensional form, which can more efficiently perform subsequent video processing, such as video retrieval and video classification.

[0024] In an optional embodiment, generating a corresponding semantic mask based on a sequence of video keyframes includes: Input the sequence of keyframes from the video into the semantic segmentation model to obtain the object category probability map for each keyframe; Set a preset saliency threshold, and mark the areas in the object category probability map that exceed the threshold as salient object areas, and the areas that are below the threshold as background areas; Morphological dilation is performed on the salient object region to obtain a binary semantic mask that covers the entire object.

[0025] It should be noted that the keyframe sequence extracted from the video is input into a semantic segmentation model; the model's task is to analyze each keyframe, identify the objects within it, and assign a category label to each pixel; these classification results will be output in the form of a probability map, representing the probability that each pixel belongs to a certain category; After obtaining the object category probability map, a preset saliency threshold can be set; this threshold is used to distinguish between salient object regions and background regions; specifically, pixels in the probability map that exceed the threshold will be marked as salient object regions, while pixels that are below the threshold will be marked as background regions. Morphological dilation is performed on the regions marked as salient objects. Morphological dilation is an image processing technique that can be used to expand the region of an object in a binary image, making the edges of the object more complete and continuous. The purpose of this step is to ensure that the salient object region can cover the entire object, rather than just being limited to the central part of the object. Morphological dilation will produce a binary semantic mask; this mask clearly shows which parts are salient objects (represented by white) and which parts are background (represented by black); this semantic mask will play an important role in the video compression process, helping to optimize the compression efficiency of the background while preserving the details of salient objects.

[0026] In an optional embodiment, non-keyframe feature streams, semantic masks, and control parameter data are input into a pre-trained generative super-resolution model for super-resolution reconstruction to obtain high-resolution super-resolution reconstructed video data, including: Analyze the scene feature vectors in the control parameter data to determine the network weights of the generative super-resolution model corresponding to the current video scene; The non-keyframe feature stream and semantic mask are input into the generative super-resolution model, and upsampled and feature-enhanced according to the determined network weights to generate initial high-resolution video data. Temporal consistency alignment is performed on the initial high-resolution video data based on the motion vector information in the control parameter data to obtain high-resolution super-resolution reconstructed video data.

[0027] It should be noted that scene feature vectors are extracted from the control parameter data; these feature vectors contain key information about the current video scene, such as scene type, lighting conditions, and motion features; this information will help select the network weights of the generative super-resolution model that are suitable for the current scene. Based on the scene feature vector obtained from the analysis, the network weights of the generative super-resolution model that best matches the current video scene will be determined. Different scenes may affect the details and textures of the image, so using appropriate network weights can improve the super-resolution effect. The previously extracted non-keyframe feature stream and semantic mask are input into the generative super-resolution model; based on the determined network weights, the model will upsample and enhance the features of the input data; the upsampling process not only increases the resolution of the image, but also improves the details and clarity of the image through feature enhancement techniques; To ensure that the generated high-resolution video data maintains consistency in the temporal dimension, it is necessary to perform temporal consistency alignment on the initial high-resolution video data based on the motion vector information in the control parameter data. The motion vector provides displacement information between frames, which can help to properly align each frame and avoid visual discontinuity caused by motion.

[0028] In an optional embodiment, calculating residual data between the original video data and the super-resolution reconstructed video data, and determining a texture complexity map based on the residual data, includes: Pixel-level subtraction is performed on the original video data and the super-resolution reconstructed video data to obtain residual data; The residual data is traversed using a preset sliding window, and the pixel variance value within each window is calculated. The pixel variance values ​​calculated for each window are normalized to generate a texture complexity map.

[0029] It should be noted that pixel-level subtraction is performed between the original video data and the super-resolution reconstructed video data. The purpose of this operation is to calculate the difference between the two, i.e., the residual data. The residual data contains details and information that were not captured during the super-resolution reconstruction process, and these details are usually important parts that affect the visual experience. Set a preset sliding window on the obtained residual data and traverse the entire residual map; this sliding window is a fixed-size area that moves on the residual data and calculates the pixel variance value within the window at each position; variance is a statistic that reflects the distribution of pixel values ​​within the window. The larger the variance, the more complex the details of the changes in that area. During the traversal of each window, the variance of all pixels within the window is calculated; this process helps identify which regions in the residual data have more texture and detail variations, and thus determine the complexity of these regions. The pixel variance values ​​calculated from all windows are normalized to generate a texture complexity map. The purpose of normalization is to transform the variance values ​​of different windows into a uniform range to facilitate subsequent analysis and visualization. The texture complexity map can intuitively represent the level of detail in different regions of the video.

[0030] In an optional embodiment, generative inpainting model is used to generatively complete the high-frequency details in the residual data using a texture complexity map, resulting in optimized high-resolution video data, including: The residual data and super-resolution reconstructed video data are input into the encoder of the generative inpainting model to extract residual feature maps; Based on the numerical distribution of the texture complexity map, the generative repair model is controlled to perform corresponding processing in different regions. Smoothing and denoising are performed in the first numerical region, and high-frequency detail generation is performed in the second numerical region. The texture complexity of the first numerical region is higher than that of the second numerical region. The high-frequency compensation details are generated by the decoder of the generative inpainting model and then superimposed onto the super-resolution reconstructed video data to obtain optimized high-resolution video data.

[0031] It should be noted that the residual data and the video data reconstructed by super-resolution are input together into the encoder of the generative inpainting model. At this stage, the encoder analyzes the input data, extracts key feature information, and forms the so-called residual feature map. This feature map contains important information about the areas of missing details in the video and the contextual information related to these areas. Based on the previously generated texture complexity map, the system will control the generative inpainting model to execute different processing strategies in different regions. The texture complexity map shows the complexity of each region in the video. For regions with low complexity, the model will perform smoothing and denoising to reduce unnecessary noise. In regions with high complexity, the model will focus on generating high-frequency details to enhance the image's clarity and realism. After feature extraction and region control are completed, the decoder of the generative inpainting model will generate high-frequency compensated details based on the feature maps extracted by the encoder; this means that the model will focus on generating those fine details that are missing in the super-resolution reconstructed video data; this process is to ensure that the final video is not only high-resolution, but also more visually appealing and realistic. The generated high-frequency compensation details are superimposed onto the super-resolution reconstructed video data to form an optimized high-resolution video. This superimposition process ensures that the generated new details can effectively fill the previous gaps, and the final output video will be clearer and more detailed.

[0032] In an optional embodiment, the optimized high-resolution video data is fused with the video keyframe sequence to obtain a fused reconstructed video frame sequence, including: Extract detailed features from optimized high-resolution video data and high-dimensional features from video keyframe sequences; We perform weighted fusion of detailed features and high-dimensional features to retain the core features of keyframes and the detailed features of optimized data. The fused features are then reassembled into a frame sequence to obtain the fused and reconstructed video frame sequence.

[0033] It should be noted that detailed features are extracted from the optimized high-resolution video. These detailed features mainly reflect important information such as texture, color, and lighting changes in the video. At the same time, high-dimensional features are also extracted from the keyframe sequence of the video. These high-dimensional features usually contain the overall structural information of each frame, including motion patterns, scene changes, and object layout. After extracting detailed features and high-dimensional features, the next step is to perform weighted fusion. The purpose of this step is to effectively combine the two types of features in order to retain the core features in the keyframe while enhancing the detailed features in the optimized data. Weighted fusion means assigning different weights according to the importance of the features, thereby ensuring that the key information and detailed features of the keyframe are presented reasonably in the final output. The fused features are reassembled into a frame sequence to generate a fused and reconstructed video frame sequence. This process involves integrating the fused features in chronological order of the video to generate a coherent and smooth video sequence. The purpose of the reassembly is to ensure that the transitions between frames are natural during playback and to maintain the overall smoothness and visual consistency of the video.

[0034] In an optional embodiment, based on control parameter data and the fused and reconstructed video frame sequence, a corresponding encoding mode is selected, and entropy encoding is performed on the fused and reconstructed video frame sequence to obtain an encoded binary bitstream, including: The complexity of analyzing motion vector information in control parameter data and fusing and reconstructing video frame sequences determines whether the encoding mode is low-latency encoding mode or high-fidelity encoding mode. In low-latency coding mode, only the residual information between the fused and reconstructed video frame sequence and the video keyframe is entropy encoded. If it is a high-fidelity coding mode, joint entropy coding is performed on the compressed representation of motion vectors, residual information and semantic masks in the fused and reconstructed video frame sequence.

[0035] It should be noted that the analysis of control parameter data includes motion vector information and the complexity of the fused and reconstructed video frame sequence. Motion vectors are key elements in video coding that describe the movement of objects, telling how objects move in the frame. The complexity of the fused and reconstructed video frame sequence reflects the level of detail, texture complexity, and changes between each frame and the preceding and following frames. Based on the results of the above analysis, the system will decide whether to use a low-latency encoding mode or a high-fidelity encoding mode: Low-latency coding mode: suitable for scenarios that require fast transmission and are less concerned with absolute image quality; in this mode, only the residual information between the fused and reconstructed video frame sequence and the keyframes needs to be entropy encoded; this method reduces latency in the encoding process and improves the efficiency of real-time transmission. High-fidelity encoding mode: suitable for application scenarios with high image quality requirements; in this mode, in addition to encoding the residual information, joint entropy encoding is also performed on the compressed representation of motion vectors and semantic masks, which can more comprehensively preserve the important information and details in the video; In low-latency coding mode, the system only needs to entropy encode the residual information between the fused and reconstructed video frame sequence and the keyframes; the residual information refers to the difference between the current frame and the previous keyframes. The advantage of doing this is that it can significantly reduce the amount of data that needs to be encoded, thereby improving the transmission speed. In high-fidelity coding mode, the system not only encodes residual information, but also includes compressed representations of motion vectors and semantic masks. Motion vectors help record the trajectory of objects, while semantic masks provide contextual information about the image content, such as the separation of background and foreground, which can further improve coding efficiency and quality.

[0036] In an optional embodiment, the encoded binary bitstream is encapsulated to obtain a final intelligent compressed bitstream corresponding to the original video data, including: The entropy-encoded binary code stream is associated and matched with the control parameter data; The associated binary stream and control parameter data are packaged into a package that conforms to the network transmission format; The integrity of the package is verified. Once the verification is successful, the final intelligent compressed bitstream is obtained.

[0037] It should be noted that entropy coding is a data compression technique that effectively reduces the overall data volume by using short codes for frequently occurring parts of the data and long codes for less frequently occurring parts. In the video compression process, the entropy-coded binary bitstream contains processed motion vectors, residual information, and semantic masks. Association matching refers to integrating these encoded data with the corresponding control parameter data. The control parameter data usually includes keyframe information, timestamps, resolution, frame rate, etc., which are crucial for decoding and playing the video. Therefore, the association matching process can ensure that the decoder can correctly understand and reconstruct the video content during processing. After completing the association matching, the next step is to package this information into a packet for transmission over the network. The packet needs to follow a specific format so that it can be correctly parsed by the receiving end. Generally, the packet may contain the following parts: The header information contains basic information such as the version, length, and type of the packet, which helps the receiver identify and process the data; the entropy-encoded binary bitstream is the compressed video data, which is the effective data stream after entropy encoding; the control parameter data refers to all control information related to the video, such as timestamps and resolutions; Ensuring data integrity is crucial in network transmission; integrity verification is typically achieved by calculating checksums or hash values; when the sender packages data, it calculates a unique identifier (such as a checksum) and appends it to the end of the package; the receiver performs the same verification process upon receiving the data to ensure that the received data is consistent with the original data. After integrity verification, if everything is normal, the final package becomes an intelligent compressed bitstream; this bitstream contains both entropy-encoded video data and control parameter data, enabling efficient transmission and decoding over the network.

[0038] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited thereto. Various changes can be made within the scope of knowledge possessed by those skilled in the art without departing from the spirit of the present invention.

Claims

1. A video intelligent compression system based on generative AI super-resolution reconstruction, characterized in that, include: The data acquisition unit is used to acquire raw video data, perform downsampling and basic encoding on the raw video data, and obtain low-resolution video data and control parameter data. The process involves downsampling and basic encoding of the original video data to obtain low-resolution video data and control parameter data, including: The original video data is spatially compressed according to a preset downsampling ratio to generate low-resolution basic video data. Scene feature vectors, motion vector information, and tone mapping parameters are extracted from the original video data, and these parameters are used as the basic data for control parameters. The low-resolution basic video data and the control parameter basic data are synchronized and associated to obtain low-resolution video data and control parameter data; The data processing unit is used to perform content parsing and keyframe extraction on low-resolution video data and raw video data to obtain video keyframe sequences and non-keyframe feature streams; and to generate corresponding semantic masks based on the video keyframe sequences. This includes generating a semantic mask based on the video keyframe sequence, including: Input the sequence of keyframes from the video into the semantic segmentation model to obtain the object category probability map for each keyframe; Set a preset saliency threshold, and mark the areas in the object category probability map that exceed the threshold as salient object areas, and the areas that are below the threshold as background areas; Morphological dilation is performed on the salient object region to obtain a binary semantic mask that covers the entire object. The super-resolution reconstruction unit is used to input non-key frame feature streams, semantic masks, and control parameter data into a pre-trained generative super-resolution model to perform super-resolution reconstruction and obtain high-resolution super-resolution reconstructed video data; it also calculates the residual data between the original video data and the super-resolution reconstructed video data and determines the texture complexity map based on the residual data. The data completion unit is used to perform generative completion on the high-frequency details in the residual data based on the generative repair model and the texture complexity map, so as to obtain optimized high-resolution video data. The feature fusion unit is used to fuse the optimized high-resolution video data with the video keyframe sequence to obtain a fused reconstructed video frame sequence; based on the control parameter data and the fused reconstructed video frame sequence, the corresponding encoding mode is selected, and entropy encoding is performed on the fused reconstructed video frame sequence to obtain the encoded binary bitstream; The intelligent compression unit is used to encapsulate the encoded binary bitstream to obtain the final intelligent compressed bitstream corresponding to the original video data.

2. The video intelligent compression system based on generative AI super-resolution reconstruction according to claim 1, characterized in that, Content parsing and keyframe extraction are performed on low-resolution video data and raw video data to obtain video keyframe sequences and non-keyframe feature streams, including: Dynamic scene analysis is performed on the raw video data, and the feature differences of low-resolution video data are combined to obtain frames with significant scene changes; Frames with significant scene changes are identified as video keyframes, and high-dimensional features corresponding to the keyframes are extracted to obtain a video keyframe sequence. The remaining frames in the original video data, excluding keyframes, are subjected to feature dimensionality reduction processing. Combined with the corresponding frame features of the low-resolution video data, a low-dimensional non-keyframe feature stream is obtained.

3. The video intelligent compression system based on generative AI super-resolution reconstruction according to claim 2, characterized in that, Non-keyframe feature streams, semantic masks, and control parameter data are input into a pre-trained generative super-resolution model for super-resolution reconstruction, resulting in high-resolution super-resolution reconstructed video data, including: Analyze the scene feature vectors in the control parameter data to determine the network weights of the generative super-resolution model corresponding to the current video scene; The non-keyframe feature stream and semantic mask are input into the generative super-resolution model, and upsampled and feature-enhanced according to the determined network weights to generate initial high-resolution video data. Temporal consistency alignment is performed on the initial high-resolution video data based on the motion vector information in the control parameter data to obtain high-resolution super-resolution reconstructed video data.

4. The video intelligent compression system based on generative AI super-resolution reconstruction according to claim 3, characterized in that, Calculate the residual data between the original video data and the super-resolution reconstructed video data, and determine the texture complexity mapping based on the residual data, including: Pixel-level subtraction is performed on the original video data and the super-resolution reconstructed video data to obtain residual data; The residual data is traversed using a preset sliding window, and the pixel variance value within each window is calculated. The pixel variance values ​​calculated for each window are normalized to generate a texture complexity map.

5. The video intelligent compression system based on generative AI super-resolution reconstruction according to claim 4, characterized in that, Based on the generative inpainting model, and combined with the texture complexity map, high-frequency details in the residual data are generatively incomplete to obtain optimized high-resolution video data, including: The residual data and super-resolution reconstructed video data are input into the encoder of the generative inpainting model to extract residual feature maps; Based on the numerical distribution of the texture complexity map, the generative repair model is controlled to perform corresponding processing in different regions. Smoothing and denoising are performed in the first numerical region, and high-frequency detail generation is performed in the second numerical region. The texture complexity of the first numerical region is higher than that of the second numerical region. The high-frequency compensation details are generated by the decoder of the generative inpainting model and then superimposed onto the super-resolution reconstructed video data to obtain optimized high-resolution video data.

6. The video intelligent compression system based on generative AI super-resolution reconstruction according to claim 5, characterized in that, The optimized high-resolution video data is fused with the video keyframe sequence to obtain a fused and reconstructed video frame sequence, including: Extract detailed features from optimized high-resolution video data and high-dimensional features from video keyframe sequences; We perform weighted fusion of detailed features and high-dimensional features to retain the core features of keyframes and the detailed features of optimized data. The fused features are then reassembled into a frame sequence to obtain the fused and reconstructed video frame sequence.

7. The video intelligent compression system based on generative AI super-resolution reconstruction according to claim 6, characterized in that, Based on the control parameter data and the fused and reconstructed video frame sequence, the corresponding encoding mode is selected, and entropy encoding is performed on the fused and reconstructed video frame sequence to obtain the encoded binary bitstream, including: The complexity of analyzing motion vector information in control parameter data and fusing and reconstructing video frame sequences determines whether the encoding mode is low-latency encoding mode or high-fidelity encoding mode. In low-latency coding mode, only the residual information between the fused and reconstructed video frame sequence and the video keyframe is entropy encoded. If it is a high-fidelity coding mode, joint entropy coding is performed on the compressed representation of motion vectors, residual information and semantic masks in the fused and reconstructed video frame sequence.

8. The video intelligent compression system based on generative AI super-resolution reconstruction according to claim 7, characterized in that, The encoded binary stream is encapsulated to obtain the final intelligent compressed stream corresponding to the original video data, including: The entropy-encoded binary code stream is associated and matched with the control parameter data; The associated binary stream and control parameter data are packaged into a package that conforms to the network transmission format; The integrity of the package is verified. Once the verification is successful, the final intelligent compressed bitstream is obtained.

Images