Intelligent video repairing method and system based on spatio-temporal context reasoning

Abstract

The invention belongs to the technical field of digital media processing, and discloses an intelligent video repairing method and system based on spatio-temporal context reasoning. In order to solve the problems that in a large-area or continuous damage scene, the repairing efficiency is insufficient, dynamic inconsistency is caused due to the fact that single-frame repairing ignores time continuity, logic conflicts are caused by semantic deficiency and the like in a traditional method, a space-time multi-dimensional information fusion and reasoning framework is constructed, and a space-time context reasoning generative adversarial network (STCI-GAN) is provided. According to the method, a reference frame is obtained through multi-scale time window sampling, space, time and semantic features are extracted and fused through a semantic enhancement encoder, an initial repair frame is generated by adopting semantic constraint hierarchical repair, and finally, confrontation training optimization is performed through a multi-dimensional discriminator, so that space authenticity, time continuity, motion rationality and semantic compliance are realized. According to the method, the repair efficiency of large-area damage and continuous frame shielding is improved, the dynamic content is ensured to be real and smooth and consistent with semantics, and the method has high robustness and universality.

Description

Technical Field

[0001] This invention belongs to the field of digital media processing technology, specifically a video intelligent repair method and system based on spatiotemporal context reasoning. Background Technology

[0002] The core objective of video restoration technology is to restore and reconstruct damaged, missing, or obscured video content, ensuring it meets the requirements of visual realism and temporal continuity. This technology is widely used in fields such as film and television data restoration, surveillance video quality optimization, and multimedia content editing. Currently, mainstream video restoration technologies can be divided into two categories:

[0003] The first category is traditional restoration methods, mainly based on optical flow or patch matching techniques. Their core mechanism involves searching for similar pixel blocks in adjacent frames of the frame to be restored, thus filling in the damaged area. However, these methods have significant limitations: when the video experiences drastic motion, sudden changes in lighting, or large-area damage, the matching accuracy of similar pixel blocks drops drastically, easily leading to blurred, distorted restoration results and "ghost" artifacts. Furthermore, for occlusions at the same location in multiple consecutive frames (such as logos or watermarks), these methods are usually ineffective because adjacent frames cannot provide valid reference information.

[0004] The second category is single-frame restoration methods based on deep learning. These methods treat video frames as independent images and use deep learning models (such as U-Net and GAN) from the field of image inpainting for processing. Their core flaw lies in ignoring the temporal continuity of video: while the restoration result within a single frame may appear visually plausible, it is prone to flickering, jitter, and inconsistencies during sequential playback (e.g., the wave pattern in the restoration undergoes unnatural abrupt changes in adjacent frames), severely compromising the dynamic realism of the video.

[0005] Furthermore, existing technologies generally suffer from a critical lack of semantic understanding: traditional methods rely solely on pixel-level grayscale / texture similarity for matching and filling, while single-frame deep learning methods focus on optimizing local texture generation within a single frame. Neither establishes a semantic cognitive system for video content—it cannot identify the inherent attributes of objects (such as category, shape, and functional features), nor can it capture cross-frame semantic consistency (such as the temporal continuity of the attributes, states, and logical relationships of the same object). This results in the restoration process being limited to illogical pixel-level filling. Specifically, traditional methods cannot distinguish key semantic components of objects from background areas, easily leading to "component mismatches" (such as filling tree texture into the area of ​​a person's arm); single-frame methods are prone to "semantic paradoxes" (such as generating incomplete objects), and the attributes of the same object may abruptly change across frames (such as clothing texture changing from stripes to a solid color). In professional scenarios such as film and television restoration and surveillance video optimization, these problems may compromise the authenticity of the content or render the evidence invalid, failing to meet professional-level restoration requirements.

[0006] In summary, existing technologies either rely excessively on local spatiotemporal information, making them ill-suited for large-scale or continuous damage scenarios; or neglect temporal correlations, resulting in insufficient dynamic coherence; or lack semantic understanding capabilities, leading to logical conflicts in the repaired content. None of the aforementioned methods achieves advanced semantic understanding and logical reasoning of video content, making it difficult to meet the high-quality video repair needs in complex scenarios. Summary of the Invention

[0007] The core objective of this invention is to overcome the aforementioned shortcomings of existing video restoration technologies and provide a video intelligent restoration method and system based on spatiotemporal context reasoning. This addresses the restoration bottlenecks of traditional methods in large-area / continuous damage scenarios, the dynamic inconsistency of single-frame restoration methods, and the logical conflicts in restoration content caused by a lack of semantic understanding. It achieves efficient, high-quality, and semantically compliant restoration for various video damage scenarios. Based on deep learning video data restoration technology, it particularly focuses on using spatiotemporal context information, semantic understanding mechanisms, and generative adversarial networks to achieve intelligent reconstruction of damaged or missing frame regions in videos. It is applicable to diverse video restoration scenarios such as video watermark removal, logo removal, large-area occlusion restoration, random noise elimination, and semantic restoration of film and television content.

[0008] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0009] A video intelligent restoration method based on spatiotemporal context reasoning includes the following steps:

[0010] (1) Obtain the video sequence to be repaired and the corresponding mask of the damaged area;

[0011] (2) A multi-scale time window sampler is used to collect multiple reference frames at an exponentially increasing time interval to form a reference frame set;

[0012] (3) Spatiotemporal information encoding and fusion of the reference frame set by the separate attention encoding module to obtain a spatiotemporal context feature map; wherein, the separate attention encoding module includes a spatial attention network and a temporal attention network running in parallel, which respectively extract the internal structure and texture features of a single frame and the object motion trajectory and logical association features across frames, and perform weighted fusion.

[0013] (4) Based on the spatiotemporal context feature map, perform hierarchical repair generation:

[0014] ① Extract structural components from the spatiotemporal context feature map, and use a structure-guided network to generate a coarse structural outline and a coarse motion vector field;

[0015] ② Input the rough structural outline and rough motion vector field into the texture generator, and combine them with the spatiotemporal context feature map to generate a high-definition repair frame;

[0016] (5) Motion fusion processing: The coarse motion vector field generated by the structure-guided network and the fine optical flow field predicted by the optical flow network are fused by gated convolution to obtain the final motion field; the gated convolution fusion specifically includes:

[0017] a) The coarse motion vector field and the fine optical flow field are spliced ​​together in the channel dimension;

[0018] b) Generate pixel-by-pixel gated maps using a lightweight convolutional network, wherein the lightweight convolutional network sequentially includes Conv... 3×3 ReLU, Conv 3×3 Sigmoid is used to output a gated value that takes values ​​in the range [0,1].

[0019] c) Weighted fusion is performed based on the gating map, that is, a coarse motion vector field is preferentially used in areas with clear structure, a fine optical flow field is preferentially used in areas with rich texture and no occlusion, and the two are automatically balanced in occluded or uncertain areas. The fusion formula is as follows:

[0020] Where O represents the final motion field, and V represents the coarse motion vector field. The optical flow field is defined as a fine optical flow field, g is a gated graph, and ⊙ represents element-wise multiplication; the optical flow network is RAFT, PWC-Net, or a similar optical flow estimation network.

[0021] (6) The generated repair frames are subjected to adversarial training and optimization using a multi-dimensional discriminator module. The multi-dimensional discriminator module includes a spatial discriminator, a temporal discriminator, an optical flow discriminator, and a semantic discriminator. The quality is evaluated from the perspectives of single-frame authenticity, temporal coherence, motion regularity, and semantic logic, and the generator parameters are optimized accordingly.

[0022] (7) Output the repaired video frame sequence.

[0023] Preferably, the reference frame set for multi-scale time window sampling includes: short-term adjacent frames, occlusion boundary frames, and long-term complete pose frames; the sampling method is to select frames according to exponentially increasing time distances to simultaneously capture short-term motion information and long-term scene information.

[0024] Preferably, the spatial attention network integrates a semantic segmentation sub-network to learn structural and texture features within a single frame; the temporal attention network integrates a semantic attribute tracking module to align identical objects across frames and infer their motion trajectories and dynamic priors.

[0025] Preferably, in the hierarchical repair generation step: the structural component extraction is obtained by edge detection or low-frequency filtering of the spatiotemporal context feature map; the structural guidance network is a lightweight network that outputs a coarse structural outline and a coarse motion vector field to ensure that the skeleton, boundary and motion trend are reasonable; the texture generator is a strong generation network that fills in high-definition texture and color details under the guidance of the structural outline and motion vector field.

[0026] Preferably, the loss function of the multi-dimensional discriminator module includes: spatial discrimination loss, used to constrain the visual realism of a single frame image; temporal discrimination loss, used to constrain the motion coherence of short video clips; optical flow discrimination loss, used to constrain the optical flow field to conform to the laws of natural motion; and semantic discrimination loss, used to constrain the repaired content to conform to the semantic rules of objects and the consistency of cross-frame attributes.

[0027] Preferably, the method is applicable to the repair of pedestrian walking videos with moving mosaic occlusion, the repair of static text area damage videos caused by camera shake, and other scenarios involving large-area occlusion, continuous frame damage, random noise elimination, and film and television semantic repair.

[0028] This invention also provides a video intelligent repair system based on spatiotemporal context reasoning, comprising:

[0029] The input layer is used to receive the original video sequence and the mask for the damaged area.

[0030] A multi-scale temporal window sampler is used to acquire multi-scale reference frames;

[0031] The spatiotemporal information encoding and fusion module includes a spatial attention network, a temporal attention network, and a contextual feature fusion unit, which is used to generate spatiotemporal contextual feature maps.

[0032] A hierarchical repair generation module, including a structure-guided network and a texture generator, is used to generate high-definition repair frames;

[0033] The motion fusion submodule is used to perform gated convolution fusion of the coarse motion vector field generated by the structure-guided network and the fine optical flow field predicted by the optical flow network to obtain the final motion field. This fusion employs a lightweight convolutional network, Conv. 3×3 →ReLU→Conv 3×3 →Sigmoid, generates pixel-by-pixel gated maps and merges them according to the formula;

[0034] The multi-dimensional discriminator module includes a spatial discriminator, a temporal discriminator, an optical flow discriminator, and a semantic discriminator, which are used to perform multi-dimensional quality assessment and adversarial training on the repaired frames.

[0035] The output layer is used to output the repaired video frame sequence.

[0036] Furthermore, the optical flow network is RAFT, PWC-Net, or a similar optical flow estimation network.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] 1. Significantly improved repair efficiency for large-area damage: Traditional methods and single-frame repair methods are prone to producing blurry, distorted, or "illusionary" content that is incompatible with the environment in large-area damage scenarios. This invention obtains contextual information of distant frames through multi-scale time window sampling, and combines it with separate attention encoding and logical reasoning mechanisms to generate repair content that is structurally complete and logically coherent (such as accurately restoring a vehicle that has been obscured for multiple frames in a video based on its motion trajectory and morphological features).

[0039] 2. Excellent stability in continuous frame damage repair: For continuous frame occlusion scenarios such as watermarks and logos, existing methods often fail to repair or cause temporal flickering due to insufficient reference information. This invention uses a long-term time window sampling strategy to extract the occluded background information from distant preceding and following frames, achieving smooth and natural filling of the occluded area and ensuring the temporal stability of the repair results.

[0040] 3. Enhanced Dynamic Realism: Single-frame restoration methods often result in stiffness or discontinuous timing when restoring dynamic objects (such as water ripples, flames, and leaves). This invention, through the collaborative supervision of a time discriminator and an optical flow discriminator, forces the generator to learn realistic motion patterns, enabling the restored dynamic content to be smooth and natural, achieving seamless integration with the original video.

[0041] 4. Full assurance of semantic consistency and logical compliance: Existing technologies are prone to semantic and logical errors (such as missing object parts or attribute mutations). This invention ensures that the repaired content conforms to the semantic rules of objects (such as human body proportions and the rationality of vehicle structure), the semantic attributes are consistent across frames (such as clothing color and no mutations in object shape), and the semantics of scenes are compatible (such as no modern elements in ancient costume scenes), thus avoiding semantic conflicts at the source.

[0042] 5. Significant advantages in core technologies:

[0043] Global spatiotemporal awareness: The innovative multi-scale time window and separate attention mechanism enable the algorithm to have global contextual understanding capabilities;

[0044] Logical reasoning-based repair: The hierarchical process of "structure-guided texture generation" upgrades the repair process from traditional pixel filling to content reconstruction based on semantic logical reasoning;

[0045] Multi-dimensional quality control: The original "space-time-optical flow-semantic" four-discriminator system comprehensively guarantees the quality of repair from four dimensions: static authenticity, dynamic coherence, motion regularity and semantic logic, reaching the industry advanced (SOTA) level;

[0046] High robustness and versatility: It has excellent repair performance for various video damage scenarios such as random noise, large block occlusion, and fixed watermarks, while adapting to the semantic needs of professional scenarios such as film and television restoration and surveillance optimization. Attached Figure Description

[0047] Figure 1 This is a system architecture diagram of the algorithm of this invention;

[0048] Figure 2 This is a comparison diagram of the repair effects of the present invention. Detailed Implementation

[0049] The technical solution of the present invention will be described in detail below with reference to specific embodiments and accompanying drawings. Those skilled in the art can reproduce the present invention based on these embodiments.

[0050] This embodiment uses the restoration of pedestrian walking videos obscured by moving mosaics as an application scenario to explain the specific implementation process of the present invention in detail and verify the technical effectiveness of the present invention.

[0051] like Figure 1 As shown, the attached figures are labeled as follows:

[0052] 1. Input layer: includes the original video sequence and the damaged region mask Mt;

[0053] 2. Multi-scale Temporal Sampler: Used to acquire reference frames;

[0054] 3. Spatiotemporal Information Encoding and Fusion Module: Includes a spatial attention network (integrating a semantic segmentation sub-network), a temporal attention network (integrating a semantic attribute tracking module), and a context feature fusion unit;

[0055] 4. Hierarchical Repair Generation Module (Generator G): Includes a structure guidance network (containing a semantic structure template library) and a texture generator (containing semantic attribute constraint units);

[0056] 5. Multi-dimensional discriminator module: includes spatial discriminator DS, temporal discriminator DT, optical flow discriminator DF, and semantic discriminator D-Sem;

[0057] 6. Output layer: Repaired video frame Ft.

[0058] Data flow: Input video frames with damaged areas and reference frames → Multi-scale temporal window sampler → Spatiotemporal information encoding and fusion module (spatial attention network + semantic segmentation sub-network → temporal attention network + semantic attribute tracking module → contextual feature fusion) → Hierarchical repair generation module (structure guidance network + semantic structure template library → texture generator + semantic attribute constraint unit) → Repaired video frames → Multi-dimensional discriminator module (spatial discriminator, temporal discriminator, optical flow discriminator, semantic discriminator parallel verification) → Adversarial training optimization module feedback to adjust generator parameters → Output final repaired frames.

[0059] 1. Algorithm Steps Review

[0060] Step 1.1 - Multi-scale temporal window sampling for the frame F to be repaired t Damaged area M t Not just using adjacent frames F t-1 F t+1 Instead, it acquires reference frames using an exponentially increasing sampling window: {F t±1 ,F t±2 ,F t±4 ,F t±8 This allows for the simultaneous capture of short-term motion (person's movements) and long-term scene information (background changes).

[0061] Frame to be repaired (at time t)

[0062] Binary mask, =1 indicates a pixel damage

[0063] : No. The reference frame set obtained by sampling within a time window

[0064] For example , , ,...

[0065] : All reference frames

[0066] Ily encoder (input single frame) Spatial features)

[0067] Temporal attention encoder (input multi-frame sequence) (Time characteristics)

[0068] : Fusion weights of spatial and temporal features (learnable or adaptive)

[0069] Step 1.2 Spatial-Temporal Separated Attention Encoder: Design a Separated Attention Encoder (SAE) with two parallel branches:

[0070] Spatial attention branch: Calculate pixel correlation within a single frame → learn structural and texture features.

[0071] Temporal attention branch: Align identical / similar objects across frames → Learn motion trajectories and logical connections to form dynamic prior knowledge.

[0072] Multi-scale sampling:

[0073] Step 1.3 - Context Feature Fusion: Spatial and temporal features are weighted and fused to obtain a spatiotemporal context feature map. .

[0074] Spatial feature extraction (processed independently for each frame):

[0075] R

[0076] For frames Spatial feature map (including structure and texture).

[0077] Temporal feature extraction (cross-frame alignment and correlation)

[0078]

[0079] Dynamic prior features encoded with temporal attention (including motion trajectory and temporal consistency);

[0080] Contextual feature fusion

[0081] The spatial features S of each frame i Fusion with temporal global features T (can be done frame-by-frame or by first aggregating spatial features and then fusing).

[0082] Method A (frame-by-frame fusion):

[0083] in It is a local slice of the temporal feature corresponding to frame i or a feature mapped back to frame i (if the temporal feature is global, attention can be used to map it to each frame).

[0084] 2. Hierarchical Repair and Reasoning

[0085] Step 2.1 — Structure-Guided Repair

[0086] From spatiotemporal context feature maps Extracting structural components (e.g., through edge detection operators) or low-frequency filtering Extract structural channels).

[0087] Generate using the lightweight network G1 (structure generator):

[0088] 1. Rough structural outline C t (Semantic / Geometric Shape)

[0089] 2. Coarse motion vector field V t (Direction and amplitude of movement per pixel)

[0090] Objective: To ensure that the skeleton, boundaries, and motion trends are correct, and to maintain a reasonable layout even if textures are missing.

[0091] Step 2.2 Texture Detail Generation

[0092] Using the more powerful generator G2, input:

[0093] Complete spatiotemporal context feature map

[0094] The structural profile C output in the first stage t With sports field V t ;

[0095] Output: High-resolution textures and color details This refers to refining and repairing frames.

[0096] 2.3 Hierarchical Repair Formula

[0097] (1) Extraction of structural components

[0098]

[0099] (2) Structure-guided generation (first stage)

[0100]

[0101] Provide spatial layout / outline

[0102] Provides motion trends (which can be used for motion compensation during subsequent detail compositing).

[0103] (3) Texture detail generation (second stage)

[0104]

[0105] Here It already includes fine-grained textures and colors, and can be directly used as a final repair candidate.

[0106] 3. Multidimensional discriminator formula

[0107] (1) The spatial discriminator inputs a single frame image. (may be or real frame ):

[0108] (True probability)

[0109] Combat Loss (Space):

[0110]

[0111] (2) Input short video clips to the time discriminator

[0112] or :

[0113] Combat Loss (Time):

[0114] (3) The optical flow discriminator first calculates the optical flow field. or :

[0115]

[0116] Countermeasures against loss (optical flow):

[0117] 2.4 Overall Objective Function

[0118] Let the basic reconstruction loss (such as L1 / L2 or perception loss) be... :

[0119]

[0120] Total generator loss:

[0121] Total discriminator loss:

[0122] Alternate optimization of generator during training With discriminator

[0123] Structure guidance phase Lightweight Networks It is generated based on the structural components of the spatiotemporal context feature map, which is fast, but may lack high-precision details, especially with large errors in areas with weak texture or occlusion.

[0124] Optical flow estimation networks (such as FlowNet, PWC-Net, RAFT, etc.) can directly regress pixel-level motion from adjacent frames with high accuracy, but they are prone to errors in occlusion, motion blur, and repetitive textures.

[0125] The two can be combined Provides an initial guess / regularization for optical flow estimation, reduces the search space, and improves robustness in occluded regions; uses optical flow estimation to correct... The obvious errors improve the spatiotemporal consistency of the sports field; joint training allows the learner to output motion vectors that are closer to the real optical flow, thus making the structure guidance phase more useful.

[0126] Specifically, we will Considered as the initial motion prior, and combined with the optical flow estimation network (Parameters are) Collaborative prediction and fusion are performed, and joint optimization of multi-task losses is achieved. and .

[0127] The overall approach consists of four steps:

[0128] Dual-path motion estimation: respectively obtained (Structure guidance) and (Optical flow network output).

[0129] Motion Fusion Module: Designing a Fusion Network Combining the two into the final sports field .

[0130] Joint loss design: includes losses for optical flow truth monitoring, structural consistency, and repair quality, updated via backpropagation. , , .

[0131] Iterative refinement (optional): using Feedback is then sent to the structural guidance or repair phase to further enhance spatiotemporal consistency.

[0132] Motion Fusion Module Design

[0133] The rough motion vector field generated during the structural guidance stage Optical flow predicted by optical flow network Perform pixel-by-pixel gated fusion:

[0134] In regions with clear and reliable structures (such as strong edges and large gradients), more resources are preserved. To maintain the advantages of structural guidance;

[0135] In areas with no obstructions and rich textures, more methods are used. To utilize the high precision of optical flow networks;

[0136] In occluded or uncertain areas, gating automatically balances the two to reduce artifacts and inconsistencies.

[0137] Network Structure

[0138] enter

[0139] : Structure-guided motion vector field (2 channels, horizontal and vertical components), dimensions ;

[0140] Optical flow network output (same size) )

[0141] Feature splicing

[0142] Concatenate the two along the channel dimension:

[0143] Gated Generative Networks

[0144] Generate the gated map g using a lightweight CNN (e.g., 2-3 convolutional layers + ReLU + Sigmoid):

[0145]

[0146] Number of output channels = 1 (single channel gating value ∈ [0,1])

[0147] Sigmoid activation maps values ​​to the 0-1 range.

[0148] Gate value for each pixel Indicates to Trust ratio Indicates to The trust ratio;

[0149] Fusion computation

[0150] Equivalent to ; For element-wise multiplication (broadcast to a 2-channel vector).

[0151] Joint training strategy

[0152] 4.1 Loss Function

[0153] (1) Optical flow supervision loss (when true value is available) (area)

[0154]

[0155] For effective optical flow mask (non-occlusion, non-extrapolation region); Charbonnier robust loss,

[0156] Structural consistency loss

[0157] In structural outline High confidence edge region (Obtained from edge detection or gradient magnitude):

[0158]

[0159] The fusion results are encouraged to maintain the structure-guided motion trend in the structural region as much as possible, and to prevent optical flow noise from disrupting the skeleton motion.

[0160] Occlusion perception loss

[0161] Obstructed area (Detection of forward-backward optical flow consistency)

[0162]

[0163] In obstructed areas, structural guidance is more readily believed. This reduces the impact of random guessing in optical flow networks.

[0164] Repair quality loss

[0165] Use the integrated sports field Generate repair frames:

[0166]

[0167] Reconstruction losses:

[0168]

[0169] Total loss is

[0170] The training process is as follows:

[0171] initialization:

[0172] First, pre-train separately (Structure-guided generator) and optical flow network (Use standard optical flow datasets or video data).

[0173] Introducing the GCMF module:

[0174] Will , Gated converged network Join the training together, make it a regular practice Or make simultaneous fine adjustments.

[0175] End-to-end optimization:

[0176] calculate Backpropagation update , , as well as .

[0177] Alternating fine-tuning:

[0178] A module can be periodically frozen to stabilize training (e.g., fix it first). Adjust and Then release all the fine adjustments to the body.

[0179] Phase 3: Multi-dimensional discriminator and adversarial training

[0180] The multi-dimensional discriminator design includes three discriminators:

[0181] Spatial discriminator D s : Determine whether a single frame image is realistic and free of artifacts.

[0182] Time discriminator D t Input a short video clip (the repaired frame and several frames before and after it), determine the motion continuity, and suppress flickering / jitter.

[0183] Optical Flow Discriminator D f First, calculate the optical flow field O. t(Pre-trained or jointly learned optical flow networks can be used) to determine whether the optical flow distribution conforms to the laws of natural motion.

[0184] Adversarial training generator G=G2 (main repair network) and {D s D t D f We conduct adversarial training to ensure that the generated results closely approximate the true distribution in terms of space, time, and motion.

[0185] Example 1

[0186] Taking the repair of a pedestrian walking video that is obscured by moving mosaic as an example: Input a 10-second video sequence {F1,F2,...,F...} 300} (30fps), of which frames 100 to 120 (F 100 to F 120 The upper body of the pedestrian is obscured by a mosaic M; the goal is to repair F. 110 .

[0187] Information collection:

[0188] The system automatically uses F 110 Centered on a reference frame, such as {F 109 ,F 111} (short-term motion), {F 100 ,F 120} (occlusion boundary), and {F 90 ,F 130 (Earlier and later complete forms of pedestrians).

[0189] Encoding and Fusion

[0190] The separate attention encoder processes these reference frames. The spatial attention branch in F 90 The learning branch focuses on the texture of pedestrian clothing; the time attention branch aligns with F. 90 and F 130 By observing pedestrians in the scene, we can deduce the posture and speed at which their arms should swing at F110.

[0191] This information is encoded and fused into a spatiotemporal context feature map.

[0192] Hierarchical repair:

[0193] Structural Repair: The structure-guided network utilizes motion and pose information from the feature map to perform structural repair in F... 110 Within the mosaic area, a rough outline of the human body and a light flow field that conform to the walking posture are first generated.

[0194] Texture generation: The texture generator receives this contour and combines it with the clothing texture information in the feature map to fill the contour with realistic clothing details, lighting and colors.

[0195] Adversarial training and optimization:

[0196] The generated repair frame F 110 It is sent to the discriminator.

[0197] Spatial discriminator checks F 110 This ensures that the edges of the repaired area transition naturally without pixel artifacts.

[0198] The time discriminator examines video clips {F 108 ′,F 109 ′,F 110 ′,F 111 ′,F 112 ′}、{F 108 ′,F 109 ′,F 110 ′,F 111 ′,F 112 Ensure that the pedestrian's arm swing is continuous, without sudden jumps or pauses.

[0199] The optical flow discriminator analyzes the optical flow of the segment to ensure that its motion vector conforms to the biomechanical laws of normal human walking. If any discriminator detects a "flaw," it will send a negative feedback signal to the generator.

[0200] The generator adjusts its network parameters based on this signal and regenerates until all discriminators can no longer distinguish between real and fake.

[0201] Output:

[0202] After multiple rounds of iterative optimization, a highly realistic restored frame F was finally output, which is visually, dynamically, and logically accurate. 110 Repeat this process for all damaged frames to obtain a perfectly restored video.

[0203] Example 2

[0204] Step 0: Problem Definition and Input

[0205] Objective: To repair the video sequence {F1, F2, ..., F...} 300 (30fps, 10 seconds total) F 100 ~F 120 The area where the upper body of a pedestrian is obscured by mosaic; key example of repair F. 110 .

[0206] Given: Mosaic mask M 110 Mark F 110The pixels that are obscured.

[0207] Multi-frame reference: {F 109 , F 111} (short-term adjacent frames), {F 100 , F 120} (Occlusion boundary frame), {F 90 , F 130 (Long-term complete pose frame).

[0208] Final output: Visually, dynamically, and logically consistent repaired frames. .

[0209] Step 1: Multi-scale time window sampling

[0210] Centered on F110, reference frames are selected based on exponentially increasing time intervals:

[0211]

[0212] Objective: To capture fast movements (arm swing) in short frames and complete pose changes and background information in long frames.

[0213] Step 2: Separate attention encoding (generating spatiotemporal context feature maps)

[0214] Spatial attention branch:

[0215] Aligning identical pedestrian instances across frames (e.g., F) 90 ↔ F 130 )

[0216] Infer F 110 The position, swing speed, and trajectory of the arm at that moment form a priori dynamic posture.

[0217] Fusion:

[0218] Spatial features + temporal features → Generate a spatiotemporal context feature map

[0219] Step 3: Structure-guided generation of motion vector field (Gross motion priors)

[0220] Extracting structural components: For example, edge detection or low-frequency filtering can highlight the human body's contours and posture structure.

[0221] enter To lightweight structure generator

[0222] Output 1: Human body outline (Rough shape of the obscured area)

[0223] Output 2: Structure-guided motion vector field (2-channel vector field, describing the expected motion trend of the occluded area at F110)

[0224] It provides a reasonable skeleton and direction of motion in the occluded area (derived from long-term attitude inference).

[0225] Step 4: Optical Flow Network Prediction (Data-driven sports field)

[0226] Input adjacent frames: F 109 (Previous frame) and F 110 (Current frame, including mosaic)

[0227] Network: Optical Flow Estimation Network (such as RAFT, PWC-Net)

[0228] Output: Preliminary optical flow field (The obstructed area may be noisy or missing)

[0229] It provides high-precision motion details in unobstructed areas;

[0230] Step 5: Gated Convolution Motion Fusion (GCMF)

[0231] Feature splicing Channel dimension 4

[0232] Generate gated graph

[0233] Lightweight CNN (Conv) 3×3 →ReLU→Conv 3×3 →Sigmoid) processing

[0234] Output: Gated graph ;

[0235] meaning: Trust structure guidance (e.g., clear edges, long-term pose regions)

[0236] Trusted Optical Flow Network (such as areas with no obstructions and rich textures)

[0237] Fusion computation

[0238] , This is an element-wise multiplication (broadcast to a 2-channel vector).

[0239] Result: A final sports field with consistent time and space and a reasonable structure was obtained.

[0240] Step 6: Hierarchical Repair

[0241] Input: Spatiotemporal context feature map Human body outline Integrated sports field Lightweight network generation: A reasonable structural outline for the occluded area (ensuring the skeleton and joint positions match the walking posture), and the corresponding motion vector field (for subsequent texture alignment).

[0242] Texture generation, input , ,

[0243] Strong Generator Networks Fill: Clothing texture (from F) 90 (e.g., reference frame extraction), lighting and color details, output repair frames. ;

[0244] Step 7: Multi-dimensional adversarial training and optimization

[0245] Spatial discriminator examine Single-frame realism (natural edge transitions, no artifacts)

[0246] Time discriminator

[0247] · Input fragment {F 108 ', F 109 ', F 110 ', F 111 ', F 112 '}, to verify that the arm swing is continuous, without sudden jumps or pauses,

[0248] Countermeasures against loss feedback:

[0249] • Any discriminator detects an anomaly → the generator receives negative feedback → parameters are adjusted → generation is restarted.

[0250] Iteration: Multiple rounds of optimization until all discriminators can no longer distinguish between repaired frames and real frames.

[0251] Step 8: Output and Batch Repair

[0252] get The visual, dynamic, and logical consistency is high. Repeat Steps 3 to 7 to repair each frame in F100 to F120; synthesize the repaired complete video sequence.

[0253] Explanation of English Abbreviations

[0254] GAN (Generative Adversarial Network): A deep learning model framework consisting of a generator and a discriminator, which optimizes the model through an adversarial game between the two.

[0255] STCI-GAN (Spatio-Temporal Contextual Inference GAN): Spatio-Temporal Contextual Inference Generative Adversarial Network, which is the core algorithm model proposed in this invention;

[0256] SOTA (State-of-the-Art): The most advanced technological level in the current field;

[0257] U-Net: A convolutional neural network architecture commonly used for image segmentation and restoration tasks, named for its U-shaped network structure.

[0258] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. All modifications and improvements made by those skilled in the art without departing from the technical concept of the present invention should be included within the protection scope of the present invention.

Claims

1. A video intelligent restoration method based on spatiotemporal context reasoning, characterized in that, Includes the following steps: (1) Obtain the video sequence to be repaired and the corresponding mask of the damaged area; (2) A multi-scale time window sampler is used to collect multiple reference frames at an exponentially increasing time interval to form a reference frame set; (3) Spatiotemporal information encoding and fusion of the reference frame set by the separate attention encoding module to obtain a spatiotemporal context feature map; wherein, the separate attention encoding module includes a spatial attention network and a temporal attention network running in parallel, which respectively extract the internal structure and texture features of a single frame and the object motion trajectory and logical association features across frames, and perform weighted fusion. (4) Based on the spatiotemporal context feature map, perform hierarchical repair generation: ① Extract structural components from the spatiotemporal context feature map, and use a structure-guided network to generate a coarse structural outline and a coarse motion vector field; ② Input the rough structural outline and rough motion vector field into the texture generator, and combine them with the spatiotemporal context feature map to generate a high-definition repair frame; (5) Motion fusion processing: The coarse motion vector field generated by the structure-guided network and the fine optical flow field predicted by the optical flow network are fused by gated convolution to obtain the final motion field; the gated convolution fusion specifically includes: a) The coarse motion vector field and the fine optical flow field are spliced ​​together in the channel dimension; b) Generate pixel-by-pixel gated maps using a lightweight convolutional network, wherein the lightweight convolutional network sequentially includes Conv... 3×3 ReLU, Conv 3×3 Sigmoid is used to output a gated value that takes values ​​in the range [0,1]. c) Weighted fusion is performed based on the gated graph, that is, a coarse motion vector field is preferentially used in areas with clear structure, a fine optical flow field is preferentially used in areas with rich texture and no occlusion, and the two are automatically balanced in occluded or uncertain areas. The fusion formula is as follows: Where O represents the final motion field, and V represents the coarse motion vector field. The optical flow field is defined as a fine optical flow field, g is a gated graph, and ⊙ represents element-wise multiplication; the optical flow network is RAFT, PWC-Net, or a similar optical flow estimation network. (6) The generated repair frames are subjected to adversarial training and optimization using a multi-dimensional discriminator module. The multi-dimensional discriminator module includes a spatial discriminator, a temporal discriminator, an optical flow discriminator, and a semantic discriminator. The quality is evaluated from the perspectives of single-frame authenticity, temporal coherence, motion regularity, and semantic logic, and the generator parameters are optimized accordingly. (7) Output the repaired video frame sequence.

2. The method according to claim 1, characterized in that, The reference frame set for multi-scale time window sampling includes: short-term adjacent frames, occlusion boundary frames, and long-term complete pose frames; the sampling method is to select frames according to exponentially increasing time distances to simultaneously capture short-term motion information and long-term scene information.

3. The method according to claim 1, characterized in that, The spatial attention network integrates a semantic segmentation sub-network to learn structural and texture features within a single frame; the temporal attention network integrates a semantic attribute tracking module to align identical objects across frames and infer their motion trajectories and dynamic priors.

4. The method according to claim 1, characterized in that, In the hierarchical repair generation step: the structural component extraction is obtained by edge detection or low-frequency filtering of the spatiotemporal context feature map; the structural guidance network is a lightweight network that outputs a coarse structural outline and a coarse motion vector field to ensure that the skeleton, boundary and motion trend are reasonable; the texture generator is a strong generation network that fills in high-definition texture and color details under the guidance of the structural outline and motion vector field.

5. The method according to claim 1, characterized in that, The loss function of the multi-dimensional discriminator module includes: spatial discrimination loss, used to constrain the visual realism of a single frame image; temporal discrimination loss, used to constrain the motion coherence of short video clips; optical flow discrimination loss, used to constrain the optical flow field to conform to the laws of natural motion; and semantic discrimination loss, used to constrain the repaired content to conform to the semantic rules of objects and the consistency of cross-frame attributes.

6. The method according to claim 1, characterized in that, The method is applicable to the repair of pedestrian walking videos with moving mosaic occlusion, the repair of static text area damage videos caused by camera shake, and other scenarios involving large-area occlusion, continuous frame damage, random noise elimination, and film and television semantic repair.

7. A video intelligent repair system based on spatiotemporal context reasoning, characterized in that, include: The input layer is used to receive the original video sequence and the mask for the damaged area. A multi-scale temporal window sampler is used to acquire multi-scale reference frames; The spatiotemporal information encoding and fusion module includes a spatial attention network, a temporal attention network, and a contextual feature fusion unit, which is used to generate spatiotemporal contextual feature maps. A hierarchical repair generation module, including a structure-guided network and a texture generator, is used to generate high-definition repair frames; The motion fusion submodule is used to perform gated convolution fusion of the coarse motion vector field generated by the structure-guided network and the fine optical flow field predicted by the optical flow network to obtain the final motion field. This fusion employs a lightweight convolutional network, Conv. 3×3 →ReLU→Conv 3×3 →Sigmoid, generates pixel-by-pixel gated maps and merges them according to the formula; The multi-dimensional discriminator module includes a spatial discriminator, a temporal discriminator, an optical flow discriminator, and a semantic discriminator, which are used to perform multi-dimensional quality assessment and adversarial training on the repaired frames. The output layer is used to output the repaired video frame sequence.

8. The system according to claim 7, characterized in that, The optical flow network is RAFT, PWC-Net, or a similar optical flow estimation network.

Images