Semantic-driven real scene video super-division method and system

Abstract

The invention provides a semantic-driven real scene video super-division method and system, and belongs to the technical field of video data. Understanding the obtained low-resolution video, and converting the low-resolution video into a hierarchical semantic representation form; channel self-attention and channel mutual attention are alternately used for the obtained low-resolution video to extract intra-frame features and inter-frame features, fusion is carried out, aligned features are obtained through motion compensation operation, and coarse-grained video restoration is carried out according to the aligned features; adding Gaussian noise to the coarse-grained restored video to obtain an intermediate state of a current diffusion step, converting the intermediate state to a semantic equivalent space, and decoding the potential vector to generate a refined frame; and performing deep convolution fusion on the coarse-grained restored video and the refined frame to generate a final super-resolution reconstruction frame. According to the method, the reconstruction fidelity and reality sense of the video in the complex degraded scene are improved based on hierarchical semantic representation, and the definition and reality sense of texture details in the reconstructed video are enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of video data processing, and in particular to a semantically driven real-scene video super-resolution method and system. Background Art

[0002] Deep learning-based video super-resolution algorithms have designed numerous sophisticated alignment modules to leverage inter-frame pixel variations and long-range video information, achieving breakthroughs on simple, fixed downsampled datasets. However, existing algorithms still face limitations when faced with the complex and unknown degradation processes found in real-world scenes. For example, videos shot in low-light environments often exhibit significant noise, and during transmission, operations like quantization introduce compression artifacts such as blocking artifacts. These complex degradation issues not only reduce the accuracy of inter-frame alignment, but also negatively impact the quality of subsequent frame reconstructions due to artifacts not eliminated during long-range information propagation. To address these issues, blind video super-resolution methods incorporate blur kernel and noise estimation to account for the varying types and intensities of degradation in videos. BlindVSR simultaneously estimates blur kernels and motion vectors for low-resolution videos in an end-to-end manner. SelfBlindVSR further incorporates a self-supervised approach, generating auxiliary data from the original low-resolution video to reduce the need for paired datasets for model training. Differently, RealBasicVSR constructs a complex video degradation reduction model during training to improve its generalization capabilities. It also inserts a preprocessing module before the BasicVSR network to reduce artifacts in low-resolution videos. RealViformer reveals that the channel self-attention mechanism is less sensitive to artifacts and utilizes a channel squeeze-excitation mechanism and covariance-based attention coefficient scaling to avoid interference from noise and blur. MGLD, conditioned on the motion information of low-resolution videos, adds a one-dimensional convolution in the temporal dimension to the pre-trained VAE decoder, reducing the impact of the inherent randomness of the diffusion process and the domain difference between the latent space and pixel space on temporal consistency between video frames. Upscale-A-Video locally inserts 3D convolution and temporal self-attention layers into the U-Net and VAE decoder to learn dependencies between video frames. It also introduces an optical flow-guided recurrent propagation module, effectively extending its temporal perception range without increasing training parameters. DiffVSR uses linear interpolation of latent features of overlapping frames and a noise rescaling mechanism to address discontinuity and jitter in the generated video. SeedVR introduces a variable-size attention window design at the boundaries of spatial and temporal dimensions, overcoming the size limitations of traditional window-based attention mechanisms. STAR inserts a local information enhancement module before the global attention module and introduces a dynamic frequency loss function to adaptively adjust the focus on high- and low-frequency components at different diffusion steps. This reduces artifacts while enhancing the fidelity of restoration, generating videos of higher spatiotemporal quality.

[0003] Due to the reduced information available in low-quality videos, discriminative real-world video super-resolution algorithms struggle to reconstruct realistic and clear texture details. Diffusion-based real-world video super-resolution algorithms damage video frames during the diffusion process, making it difficult to directly integrate with pixel-space frame alignment methods. Furthermore, the randomness of the diffusion process poses a greater challenge to video temporal consistency. Summary of the Invention

[0004] The purpose of the present invention is to provide a semantic-driven real-scene video super-resolution method and system to solve at least one technical problem existing in the above-mentioned background technology. The present invention aims to combine the advantages of discriminant and generative video super-resolution algorithms, and improve the reconstruction fidelity and realism of videos in complex degraded scenes based on hierarchical semantic representation. On the basis of the discriminant video super-resolution algorithm processing low-quality videos, the generative video super-resolution algorithm further enhances the clarity and realism of texture details in the reconstructed video. In addition, in order to avoid artifacts in the reconstructed video caused by the randomness of the generative algorithm, the results of the discriminant algorithm and the generative algorithm are fused from coarse to fine, so as to obtain a high-quality video with consistent content and more realistic and natural texture. Text can only provide a macroscopic description of the image, lacking fine-grained information such as location and appearance. Therefore, it is difficult to give full play to the generation ability of the text-based image diffusion model by relying solely on text prompts.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] In a first aspect, the present invention provides a semantically driven real-scene video super-resolution method, comprising:

[0007] Understand the acquired low-resolution video and convert it into a hierarchical semantic representation;

[0008] For the acquired low-resolution video, channel self-attention and channel mutual attention are used alternately to extract intra-frame features and inter-frame features, and then fused. The aligned features are obtained through motion compensation operation, and the coarse-grained video restoration is performed based on the aligned features.

[0009] Add Gaussian noise to the coarse-grained restored video, obtain the intermediate state of the current diffusion step, transform to the semantically equivalent space, and decode the latent vector to generate the refined frame;

[0010] The coarse-grained restored video and the refined frame are fused through deep convolution to generate the final super-resolution reconstructed frame.

[0011] As a further limitation of the first aspect of the present invention, alternatingly using channel self-attention and channel mutual attention to extract intra-frame features and inter-frame features includes: channel self-attention first models intra-frame correlation to optimize features; shallow features are divided into two different groups of features along the time dimension; the two groups of features are subjected to channel mutual attention respectively to fuse temporal information to obtain enhanced features; the enhanced features are cascaded along the temporal dimension and the channel dimension, and then further fused through a fully connected feedforward neural network; the fused features are aligned, and the aligned and fused features are convolved and sub-pixel convolution is performed to coarsely restore the video.

[0012] As a further limitation of the first aspect of the present invention, adjacent frames are aligned in combination with a block alignment operation, where the block alignment operation averages the optical flows estimated in the local area and uses the image block as the basic operation object for inter-frame alignment, so that the motion estimation information from the previous frame to the next frame is more robust to noise and artifact interference; thereafter, the aligned features are obtained through a motion compensation operation.

[0013] As a further limitation of the first aspect of the present invention, event tracking is performed based on a referential attention mechanism so that consistent semantic information between adjacent frames can flow along the temporal dimension, including: semantic tokens are divided into foreground and background according to the different referential contents, and only semantic information that persists in consecutive frames is regarded as a valuable reference condition; semantic tokens between adjacent frames establish semantic associations through referential cross-attention operations to detect content changes between frames; spatiotemporal information of video clips is collected through cascaded temporal convolution and attention mechanisms, and semantic tokens are weightedly fused along the temporal dimension to generate more comprehensive semantic tokens.

[0014] As a further limitation of the first aspect of the present invention, the restored video is used as a query, the refined frames are used as key indexes and calculation values ​​to provide high-frequency information, and the dot product attention matrix is ​​used to determine the details with high credibility; the low-resolution frame and the restored video frame are spliced ​​and mapped to a modulation mask of the attention matrix through a linear transformation to further enhance the consistency with the low-resolution video; finally, the final super-resolution reconstructed frame is generated through deep convolution fusion.

[0015] As a further limitation of the first aspect of the present invention, the semantic-aware loss transforms the video and the corresponding text description into a Lorentz hyperbolic space via an exponential mapper.

[0016] In a second aspect, the present invention provides a semantically driven real-scene video super-resolution system, comprising:

[0017] The semantic parsing module is used to understand the acquired low-resolution video and convert it into a hierarchical semantic representation;

[0018] The Fidelity Content Restoration Network is used to extract intra-frame and inter-frame features from the acquired low-resolution video by alternating channel self-attention and channel mutual attention, and fuse them. The aligned features are obtained through motion compensation, and the coarse-grained video restoration is performed based on the aligned features.

[0019] A realistic texture generation network, which adds Gaussian noise to the coarse-grained restored video, obtains the intermediate state of the current diffusion step, transforms to a semantically equivalent space, and decodes the latent vector to generate the refined frame;

[0020] The cross-domain generalized fusion network is used to fuse the coarse-grained restored video and the refined frame through deep convolution to generate the final super-resolution reconstructed frame.

[0021] In a third aspect, the present invention provides a non-transitory computer-readable storage medium, which is used to store computer instructions. When the computer instructions are executed by a processor, the semantically driven real-scene video super-resolution method as described in the first aspect is implemented.

[0022] In a fourth aspect, the present invention provides a computer device comprising a memory and a processor, wherein the processor and the memory communicate with each other, the memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the semantically driven real-scene video super-resolution method as described in the first aspect.

[0023] In a fifth aspect, the present invention provides an electronic device comprising: a processor, a memory, and a computer program; wherein the processor is connected to the memory, and the computer program is stored in the memory. When the electronic device is running, the processor executes the computer program stored in the memory so that the electronic device executes instructions for implementing the semantically driven real scene video super-resolution method as described in the first aspect.

[0024] The beneficial effects of the present invention are as follows: low-rank adaptation technology is used to introduce segmentation semantic prior information into the text-based graph diffusion model, effectively improving the quality of the reconstructed video; a semantic parsing module trained on image data of large volumes and varying quality is used to decouple low-quality videos into a hierarchical content representation consisting of instances, events, and scenes, which is used as a generation condition for the diffusion model to improve the generation quality of the diffusion model while reducing computational overhead. By aggregating the temporal information of video clips and sampling realistic textures within frames, the problem of inaccurate inter-frame alignment in pixel space is addressed; under the constraints of semantic perception loss, the hyperbolic space is optimized to encode the reconstructed video and the corresponding text description into a tree-like data structure, enhancing the video quality from coarse to fine. This demonstrates superior results in processing degraded videos of real scenes.

[0025] Additional advantages of the present invention will be more clearly given in the following description or learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0027] Figure 1 This is a flow chart of the semantically driven real-scene video super-resolution method described in an embodiment of the present invention. DETAILED DESCRIPTION

[0028] The embodiments of the present invention are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention and are not to be construed as limiting the present invention.

[0029] Those skilled in the art will understand that unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art to which this invention belongs.

[0030] It should also be understood that terms, such as those defined in commonly used dictionaries, should be understood to have a meaning consistent with their meaning in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless as defined herein.

[0031] Those skilled in the art will appreciate that, unless otherwise stated, the singular forms "a," "an," "said," and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the stated features, integers, steps, operations, elements, and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, and / or groups thereof.

[0032] In the description of this specification, reference to the terms "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Those skilled in the art may combine and integrate different embodiments or examples described in this specification, as well as features of different embodiments or examples, unless otherwise contradictory.

[0033] To facilitate understanding of the present invention, the present invention is further explained below with reference to specific embodiments in conjunction with the accompanying drawings. However, the specific embodiments do not constitute a limitation on the embodiments of the present invention.

[0034] Those skilled in the art should understand that the drawings are merely schematic diagrams of embodiments, and the components in the drawings are not necessarily necessary for implementing the present invention.

[0035] The present invention aims to explore solutions for using visual semantic information as a generation condition for the diffusion model, guiding inter-frame alignment, and improving content fidelity and naturalness of details in real-scene video super-resolution tasks. Specifically, the discriminant network first handles different types of degradation problems in low-quality videos from both global and local perspectives. Then, the generative network adds more high-frequency details to the reconstructed video while ensuring the integrity of the video's dynamic scene. Finally, the fusion network performs video fusion of different granularities in the hyperbolic space to achieve a balance between fidelity and naturalness. Video super-resolution is to reconstruct a high-resolution video with rich details and connected content through an algorithm using the spatiotemporal complementary information of consecutive frames without changing the hardware conditions. Video segmentation divides video frames into different semantic regions by analyzing the semantic correlation of pixels, while tracking the same semantic object between frames.

[0036] Example 1

[0037] In this embodiment 1, a semantic-driven real-scene video super-resolution system is first provided, which includes two parts: video semantic understanding and super-resolution reconstruction. Video semantic understanding decouples the video into a hierarchical semantic representation, and the super-resolution reconstruction mainly consists of a fidelity content restoration network, a true texture generation network, and a cross-domain generalization fusion network. The video semantic understanding part includes a semantic parsing module, which is used to understand the acquired low-resolution video and convert it into a hierarchical semantic representation. The fidelity content restoration network is used to alternately use channel self-attention and channel mutual attention to extract intra-frame features and inter-frame features from the acquired low-resolution video, and fuse them. The aligned features are obtained through motion compensation operations, and the coarse-grained video is restored based on the aligned features; the true texture generation network is used to add Gaussian noise to the coarse-grained restored video, obtain the intermediate state of the current diffusion step, convert it to a semantically equivalent space, and decode the latent vector to generate a refined frame; the cross-domain generalization fusion network is used to perform deep convolution on the coarse-grained restored video and the refined frame to generate the final super-resolution reconstructed frame.

[0038] In this embodiment, the fidelity content restoration network uses low-resolution video The spatiotemporal information in the video frame is used to improve the resolution while maximally eliminating the different degradation phenomena in the video frame, thereby obtaining a coarse-grained reconstructed frame. The Real Texture Generation Network iteratively optimizes the missing texture details in the video in the latent space. In addition, in order to avoid the jitter and flicker problems introduced by the diffusion model, the Gaussian mixture distribution estimation module and the event tracking module are used to determine the flow of semantic priors along the time dimension, thereby generating the video. Add more conditions. Finally, cross-domain generalized fusion network fusion and Generate high-resolution videos with both fidelity and realism

[0039] In this embodiment, the above-mentioned system is used to implement a semantic-driven real-scene video super-resolution method, including: understanding the acquired low-resolution video and converting it into a hierarchical semantic representation; alternately using channel self-attention and channel mutual attention to extract intra-frame features and inter-frame features of the acquired low-resolution video, and fusing them, obtaining aligned features through motion compensation operations, and coarse-grained restoration of the video based on the aligned features; adding Gaussian noise to the coarse-grained restored video to obtain the intermediate state of the current diffusion step, converting it to a semantically equivalent space, decoding the latent vector to generate a refined frame; deep convolutionally fusing the coarse-grained restored video and the refined frame to generate the final super-resolution reconstructed frame.

[0040] In low-quality videos, different types of degradation problems jointly lead to unsatisfactory visual effects, and the network only processes from a local or global receptive field, which makes it difficult to effectively deal with different degradation problems. For example, the global information of the video frame can establish the statistical characteristics of the noise distribution, while the local information helps to repair the compression artifacts caused by the block change process. In order to identify and remove the degradation phenomenon in the video frame from a global perspective, the faithful content restoration network based on the bidirectional encoder-decoder architecture alternately uses channel self-attention and channel mutual attention to extract intra-frame and inter-frame features. It can be expressed as:

[0041]

[0042] Among them, Conv(·) represents the convolutional layer that extracts shallow features, CSA(·) and CMS(·) represent channel self-attention and channel mutual attention, respectively, which have a larger receptive field than the window-based attention mechanism and are insensitive to artifacts to a certain extent. Specifically, channel self-attention first models the intra-frame correlation to optimize the feature It can be expressed as:

[0043]

[0044] Among them, LN(·) represents the layer normalization operation. Then, the shallow features are divided into two different groups along the time dimension and It can be expressed as:

[0045]

[0046] Among them, Split(·) represents the feature splitting operation in the time dimension. Subsequently, the two perform channel mutual attention to fuse the temporal information to obtain enhanced features. and It can be expressed as:

[0047]

[0048] In order to retain the important information in the current frame while fusing the temporal features, the above features are concatenated along the temporal dimension and the channel dimension, and then further fused through a fully connected feedforward neural network, which can be expressed as:

[0049]

[0050] in,"[·,·] t "and"[·,·] c" represents the cascade operation of the temporal dimension and the channel dimension respectively. In order to supplement the useful local information in the reconstruction process, the fidelity content restoration network uses convolution operations to further refine the enhanced features. Since the degradation problem in the video frame will also damage the accuracy of the optical flow information, the fidelity content restoration network combines the block alignment operation to align adjacent frames. Specifically, the block alignment operation takes the average value of the estimated optical flow in the local area and uses the image block as the basic operation object for inter-frame alignment, so that the motion estimation information from the t-1 frame to the t frame is It is more robust to noise and artifact interference and can be expressed as:

[0051]

[0052] in, Represents the motion estimation operation. After that, the aligned features are obtained through motion compensation operation. It can be expressed as:

[0053]

[0054] in, Represents the motion compensation operation. Finally, the aligned fusion features are input into the reconstruction module composed of convolutional layer and sub-pixel convolutional layer to coarsely restore the video. It can be expressed as:

[0055]

[0056] in, and Represent the reconstruction module and bilinear interpolation upsampling operation respectively.

[0057] This embodiment builds a real texture generation network based on the pre-trained latent diffusion model (LDM), and uses the encoder ε to transform the high-resolution frame into Convert to semantically equivalent space The network iteratively denoises in the latent space to generate the corresponding high-quality video, which is then passed through the decoder Reconstruct it. Specifically, in each diffusion step τ∈[1,τ max ],,realistic texture generation network is based on a predefined noise scheduler α τ and σ τ Adding Gaussian noise to high-resolution video Thus, the intermediate state of the current diffusion step is obtained U-Net denoising network∈ θ During the training process, the loss function is optimized to predict the noise, which can be expressed as:

[0058]

[0059] Where c represents a set of optional generation conditions, such as semantic cues and noise intensity. During inference, the real texture generation network directly uses the restored video is the starting state of the generation process, and the latent vector is finally decoded to generate the refined frame This embodiment uses a semantic parser module based on the Transformer architecture to understand low-resolution content and convert it into a hierarchical semantic representation. t , which can be expressed as:

[0060]

[0061] in, The introduction of the semantic parsing module inevitably creates a heavy computational burden, which largely hinders its application in resource-constrained scenarios. In order to provide a more general solution, this embodiment uses LoRA technology to adjust the specific parameters in the denoising network. Fine-tune it to have the ability to perceive semantics, which can be expressed as:

[0062]

[0063] in, Indicates the updated parameters. and Is a low-rank matrix whose dimensions satisfy r<<d1 and r<<d2. LoRA updates the parameters W in the low-dimensional subspace composed of A, while B controls the update direction of the parameters, which has the characteristics of residual low-rank decomposition.

[0064] Although similar content in adjacent frames differs in position, appearance, and scale, it still provides rich conditions for the generation of the current frame. In order to detect content changes between frames, this paper constructs an event tracking module based on the reference attention mechanism, so that the consistent semantic information between adjacent frames can flow along the temporal dimension, such as Figure 1 Specifically, semantic tokens can be divided into foreground, fg and background bg , only semantic information that persists in consecutive frames is considered as valuable reference condition C t-1→t , which can be expressed as:

[0065]

[0066] in, represents the semantic features of the mth token in the video frame, and They represent the instances that persist between the t-1th and tth frames, the newly appeared instances, and the disappeared instances, respectively. The semantic tokens between adjacent frames establish semantic associations by referring to the cross-attention operation to detect content changes between frames, which is expressed as:

[0067] O cont =RCA(O t-1 ,O t ,O t )+Hg(O t-1 ,O t )

[0068] Among them, RCA(·) and Hg(·) refer to the cross-attention operation and Hungarian matching operation respectively. The frame-by-frame instance semantic tokens lay a solid foundation for aggregating spatiotemporal information in the video. To ensure the long-term consistency of the video, the event tracking module collects the spatiotemporal information of the video clips through the cascaded temporal convolution and attention mechanism, and performs weighted fusion of the semantic tokens along the time dimension to generate a more comprehensive semantic token O c , which can be expressed as:

[0069]

[0070] Where Linear(·) represents a linear transformation and N is the number of frames in the video clip. Inspired by the Gaussian mixture model (GMM) for background modeling, the generation of the rest of the content in the video frame (such as newly appeared objects) is achieved with the help of an external memory module initialized to the network parameters, such as Figure 1 As shown in Figure 2, different targets are clustered into different semantic clusters based on semantic categories, which are sampled from mixed Gaussian distributions with different parameters. The GMM estimation module integrates external knowledge into the memory module during training, while the generation process reversely retrieves the condition C from the nearest cluster center. t , expressed as:

[0071]

[0072] in Represents a memory module that is continuously updated iteratively. μ and θ σ represents the Gaussian mean and variance parameters of each semantic cluster in the memory module. Used to measure the Wasserstein distance between two distributions.

[0073] Although the video sampled by the real texture generation network has more realistic textures, the inherent randomness of the diffusion model will also interfere with its consistency with the original low-resolution video. In order to enhance the fidelity of the reconstructed video, the cross-domain generalization fusion network fuses the rough reconstruction results of the fidelity content restoration network with the optimized video, which includes two parts: the fidelity-dominant branch and the realism-dominant branch. For query, the sampling results of the network will be generated As the key index and calculation value that provides high-frequency information, the dot product attention matrix To determine the details with high credibility. In addition, the cross-domain generalization fusion network splices the low-resolution frame and the restored video frame and maps them into the modulation mask of the attention matrix through linear transformation. To further enhance the consistency with low-resolution videos, it can be expressed as:

[0074]

[0075] Among them, “[·,·]” represents the splicing operation. Correspondingly, the realism-dominant branch exchanges the transformation sources of the query tensor, key index, and calculated value. Finally, the results of the two branches are fused through deep convolution to generate the final super-resolution reconstructed frame.

[0076] Considering the hierarchical structure of the video, this paper designs a semantic perception loss in the hyperbolic space to constrain the network optimization process. Different from the Euclidean space, the hyperbolic space has an exponential growth characteristic and is good at modeling various hierarchical concept representations, which plays a vital role in improving the quality of video generation. Therefore, the semantic perception loss First, the video and the corresponding text description are transformed into Lorentz hyperbolic space through an exponential mapper to measure and The subtle differences between them can be expressed as:

[0077]

[0078] Here, exp(·) and MLLM(·) represent the exponential function and the multimodal large language model, respectively. The latter can provide a comprehensive and detailed text description of the content and style of the video.

[0079] Example 2

[0080] This embodiment 2 provides a non-transitory computer-readable storage medium for storing computer instructions. When the computer instructions are executed by a processor, the semantically driven real scene video super-resolution method described above is implemented. The method includes:

[0081] Understand the acquired low-resolution video and convert it into a hierarchical semantic representation;

[0082] For the acquired low-resolution video, channel self-attention and channel mutual attention are used alternately to extract intra-frame features and inter-frame features, and then fused. The aligned features are obtained through motion compensation operation, and the coarse-grained video restoration is performed based on the aligned features.

[0083] Add Gaussian noise to the coarse-grained restored video, obtain the intermediate state of the current diffusion step, transform to the semantically equivalent space, and decode the latent vector to generate the refined frame;

[0084] The coarse-grained restored video and the refined frame are fused through deep convolution to generate the final super-resolution reconstructed frame.

[0085] Example 3

[0086] This embodiment 3 provides a computer device, including a memory and a processor, wherein the processor and the memory communicate with each other, the memory stores program instructions executable by the processor, and the processor invokes the program instructions to execute the semantically driven real scene video super-resolution method described above, the method comprising:

[0087] Understand the acquired low-resolution video and convert it into a hierarchical semantic representation;

[0088] For the acquired low-resolution video, channel self-attention and channel mutual attention are used alternately to extract intra-frame features and inter-frame features, and then fused. The aligned features are obtained through motion compensation operation, and the coarse-grained video restoration is performed based on the aligned features.

[0089] Add Gaussian noise to the coarse-grained restored video, obtain the intermediate state of the current diffusion step, transform to the semantically equivalent space, and decode the latent vector to generate the refined frame;

[0090] The coarse-grained restored video and the refined frame are fused through deep convolution to generate the final super-resolution reconstructed frame.

[0091] Example 4

[0092] This embodiment 4 provides an electronic device, including: a processor, a memory, and a computer program; wherein the processor is connected to the memory, and the computer program is stored in the memory. When the electronic device is running, the processor executes the computer program stored in the memory to cause the electronic device to execute instructions for implementing the semantically driven real scene video super-resolution method described above, the method including:

[0093] Understand the acquired low-resolution video and convert it into a hierarchical semantic representation;

[0094] For the acquired low-resolution video, channel self-attention and channel mutual attention are used alternately to extract intra-frame features and inter-frame features, and then fused. The aligned features are obtained through motion compensation operation, and the coarse-grained video restoration is performed based on the aligned features.

[0095] Add Gaussian noise to the coarse-grained restored video, obtain the intermediate state of the current diffusion step, transform to the semantically equivalent space, and decode the latent vector to generate the refined frame;

[0096] The coarse-grained restored video and the refined frame are fused through deep convolution to generate the final super-resolution reconstructed frame.

[0097] In summary, the semantic-driven real-scene video super-resolution method described in the embodiment of the present invention includes two parts: video semantic understanding and super-resolution reconstruction. The former models the dynamic content of the video as a hierarchical semantic representation, provides global semantic information and instance-specific semantic information, and uses low-rank adaptive technology as the generation condition of the diffusion model. The latter mainly consists of a fidelity content restoration network, a realistic texture generation network, and a cross-domain generalization fusion network. During the restoration process, the present invention solves different types of degradation problems in the video from the global and local receptive fields, and further uses the diffusion model to optimize the texture details within the frame. To this end, the present invention designs an event tracking module to dynamically associate semantically consistent inter-frame information, and combines the prior knowledge learned by the Gaussian mixture distribution estimation module as conditional information for the generation process. In addition, the present invention proposes an algorithm optimization strategy based on hyperbolic space to achieve video quality enhancement at different granularities. The real-scene video super-resolution algorithm proposed by the present invention uses low-rank adaptive technology to introduce segmentation semantic prior information into the Wensheng graph diffusion model, effectively improving the quality of the reconstructed video. The present invention utilizes a semantic parsing module trained on image data of large data volume and varying quality to decouple low-quality videos into a hierarchical content representation consisting of instances, events, and scenes, and uses this as a generation condition for a diffusion model to improve the generation quality of the diffusion model while reducing computational overhead. In order to address the problem of inaccurate inter-frame alignment in pixel space, the present invention designs an event tracking module and a Gaussian mixture distribution estimation module, which are respectively used to aggregate the temporal information of video clips and sample realistic textures within frames. In addition, the present invention optimizes in hyperbolic space under the constraint of semantic perception loss, encodes the reconstructed video and the corresponding text description into a tree-like data structure, and enhances the video quality from coarse to fine, showing more outstanding results in processing degraded videos of real scenes.

[0098] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0099] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0100] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0101] These computer program instructions can also be loaded onto a computer or other programmable data processing device, and a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide the functions for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0102] Although the above describes the specific embodiments of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solutions disclosed in the present invention without the need for creative work should be included in the scope of protection of the present invention.

Claims

1. A semantic-driven real-scene video super-resolution method, characterized by: include: Understand the acquired low-resolution video and convert it into a hierarchical semantic representation; For the acquired low-resolution video, channel self-attention and channel mutual attention are used alternately to extract intra-frame features and inter-frame features, and then fused. The aligned features are obtained through motion compensation operation, and the coarse-grained video restoration is performed based on the aligned features. Add Gaussian noise to the coarse-grained restored video, obtain the intermediate state of the current diffusion step, transform to the semantically equivalent space, and decode the latent vector to generate the refined frame; The coarse-grained restored video and the refined frame are fused through deep convolution to generate the final super-resolution reconstructed frame.

2. The semantic-driven real-scene video super-resolution method according to claim 1, characterized in that: The alternating use of channel self-attention and channel mutual attention to extract intra-frame features and inter-frame features includes: channel self-attention first models intra-frame correlation to optimize features; shallow features are divided into two different groups of features along the time dimension; the two groups of features are subjected to channel mutual attention respectively to fuse temporal information to obtain enhanced features; the enhanced features are cascaded along the temporal dimension and channel dimension, and further fused through a fully connected feedforward neural network; the fused features are aligned, and the aligned and fused features are convolved and sub-pixel convolution is performed to coarsely restore the video.

3. The semantic-driven real-scene video super-resolution method according to claim 2, characterized in that Adjacent frames are aligned in combination with a block alignment operation. The block alignment operation averages the estimated optical flow in the local area and uses the image block as the basic operation object for inter-frame alignment, making the motion estimation information from the previous frame to the next frame more robust to noise and artifact interference; then, the aligned features are obtained through a motion compensation operation.

4. The semantic-driven real-scene video super-resolution method according to claim 1, characterized in that Event tracking is performed based on the referential attention mechanism, so that the consistent semantic information between adjacent frames can flow along the temporal dimension, including: semantic tokens are divided into foreground and background according to the different referential content, and only semantic information that persists in consecutive frames is regarded as valuable reference conditions; semantic tokens between adjacent frames establish semantic associations through referential cross-attention operations to detect content changes between frames; the spatiotemporal information of video clips is collected through cascaded temporal convolution and attention mechanisms, and the semantic tokens are weightedly fused along the temporal dimension to generate more comprehensive semantic tokens.

5. The semantic-driven real-scene video super-resolution method according to claim 1, characterized in that: Taking the restored video as the query, the refined frame is used as the key index and calculation value to provide high-frequency information, and the dot product attention matrix is ​​used to determine the details with high credibility; the low-resolution frame and the restored video frame are spliced ​​and mapped into the modulation mask of the attention matrix through linear transformation to further enhance the consistency with the low-resolution video; finally, through deep convolution fusion, the final super-resolution reconstructed frame is generated.

6. The semantic-driven real-scene video super-resolution method according to claim 5, characterized in that: The semantic-aware loss transforms videos and corresponding text descriptions into Lorentz hyperbolic space via an exponential mapper.

7. A semantically driven real-scene video super-resolution system, characterized by: include: The semantic parsing module is used to understand the acquired low-resolution video and convert it into a hierarchical semantic representation; The Fidelity Content Restoration Network is used to extract intra-frame and inter-frame features from the acquired low-resolution video by alternating channel self-attention and channel mutual attention, and fuse them. The aligned features are obtained through motion compensation, and the coarse-grained video restoration is performed based on the aligned features. A realistic texture generation network, which adds Gaussian noise to the coarse-grained restored video, obtains the intermediate state of the current diffusion step, transforms to a semantically equivalent space, and decodes the latent vector to generate the refined frame; The cross-domain generalized fusion network is used to fuse the coarse-grained restored video and the refined frame through deep convolution to generate the final super-resolution reconstructed frame.

8. A non-transitory computer-readable storage medium, characterized in that The non-transitory computer-readable storage medium is used to store computer instructions, and when the computer instructions are executed by the processor, the semantic-driven real-scene video super-resolution method according to any one of claims 1 to 6 is implemented.

9. A computer device, characterized in that: It includes a memory and a processor, the processor and the memory communicate with each other, the memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the semantically driven real scene video super-resolution method as described in any one of claims 1 to 6.

10. An electronic device, characterized in that: include: A processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to execute instructions for implementing the semantically driven real scene video super-resolution method as described in any one of claims 1-6.

Images