Film and television picture intelligent frame supplementing method and system based on deep learning

Abstract

The invention relates to the technical field of computer vision and video image processing, and discloses a video picture intelligent frame supplementing method and system based on deep learning, and the method comprises the steps: carrying out the feature coding and semantic segmentation of front and rear frames of a video sequence, and dividing an image into a rigid body instance region and a non-rigid body region; constructing a parameterized rigid body reference flow field by predicting geometric transformation parameters of the rigid body instance; calculating a space gradient and a semantic edge weight of the reference flow field; a motion field gradient consistency constraint is introduced when a residual optical flow is predicted, and a semantic edge weight is used for forcibly synthesizing a space gradient of an optical flow field to keep consistent with a reference flow field at the edge; and determining a pseudo depth sequence according to the semantic category, and executing hierarchical forward transformation and hole filling by using the synthetic optical flow to generate a target intermediate frame. According to the method, parameterized modeling and a gradient constraint mechanism are combined, the problems of structural deformation and obscured shielding edges in a rigid body large-displacement scene are solved, and the definition and structural integrity of a complementary frame picture are improved.

Description

Technical Field

[0001] This invention relates to the field of computer vision and video image processing technology, specifically to a method and system for intelligent frame interpolation of film and television footage based on deep learning. Background Technology

[0002] Video frame interpolation (VFI) aims to improve the video frame rate and enhance the smoothness and clarity of the picture by analyzing the spatiotemporal relationship between adjacent frames in the original video sequence and synthesizing non-existent intermediate frames. This technology is widely used in slow-motion video generation, restoration of old films and television programs, frame rate up-conversion, and video enhancement for ultra-high-definition display devices.

[0003] With the development of deep learning technology, frame interpolation methods based on convolutional neural networks (CNNs) have gradually replaced traditional block matching methods as the mainstream. Existing deep learning frame interpolation schemes typically rely on optical flow, which first uses a neural network to predict the pixel motion vector field between adjacent frames, and then uses optical flow to resample (warping) the original image to synthesize intermediate frames. Although existing optical flow-based frame interpolation algorithms have achieved good results in handling small displacement and simple texture scenes, they still have significant technical limitations when facing large displacement rigid body motion and complex occlusion scenes common in film and television.

[0004] Most current end-to-end optical flow prediction networks rely on pixel-level motion regression based on local convolutional features. This prediction mechanism, based on local receptive fields, ignores the inherent overall geometric constraints of rigid objects (such as vehicles and buildings) in the scene. When dealing with scenes of rapid motion or large displacement of rigid bodies, the network often struggles to ensure the consistency of motion among all pixels within the rigid body, leading to bending, twisting, or breaking of straight lines in the synthesized image, producing non-physical deformation artifacts. Furthermore, to ensure the smoothness of the optical flow field, existing techniques typically introduce a smoothing term into the loss function. While this reduces noise, it can also easily lead to over-smoothing of motion boundaries. At object edges, the motion vectors of the foreground are prone to aliasing with the motion vectors of the background, causing blurring of motion boundaries. Simultaneously, monocular videos lack depth information. When multiple objects occlude each other or the background is exposed, pixel brightness information alone is insufficient to accurately determine the hierarchical relationship of pixels. This results in background textures incorrectly covering the foreground during synthesis, or failing to correctly fill background holes exposed by foreground movement, severely impacting the quality of the final generated video. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a method and system for intelligent frame interpolation of film and television images based on deep learning, which solves the problems of structural distortion and blurred synthesis in occluded areas that existing video frame interpolation technologies easily produce when processing large displacement rigid body motion.

[0006] To achieve the above objectives, this invention provides the following technical solution: The first aspect of this invention provides a deep learning-based intelligent frame interpolation method for video images. This method first acquires temporally adjacent frames of a video sequence, then uses a deep neural network to extract multi-scale features of the image and performs semantic parsing. Through semantic segmentation technology, image pixels are divided into rigid body instance regions (such as vehicles and buildings) and non-rigid body regions (such as pedestrians and vegetation). Unlike traditional methods that perform uniform pixel-level optical flow prediction across the entire image, this method employs a divide-and-conquer strategy. For rigid body instances with strict geometric constraints, a parametric regression network is used to directly predict their geometric transformation parameters under three-dimensional spatial projection, including homography transformation matrices or affine transformation matrices. The system uses these transformation parameters to construct a parameterized rigid body reference flow field that conforms to the laws of physical perspective. This reference flow field has mathematically analytical smoothness within the rigid body, enabling it to accurately describe the planar motion or parallax changes of the rigid body.

[0007] Building upon this foundation, our method further calculates the spatial gradient field of the parameterized rigid body reference flow field. This gradient field characterizes the stretching, rotation, and shear trends of the rigid body motion in space. Simultaneously, a semantic edge weight map is extracted based on the semantic segmentation results to identify the boundary regions between the rigid body and the background, or between the rigid body and non-rigid objects. In generating the final optical flow, our method utilizes a deep residual network to predict an unstructured residual optical flow field and superimposes this residual optical flow field with the parameterized rigid body reference flow field.

[0008] To ensure that the rigid body edges do not deform during motion, this method introduces a motion field gradient consistency constraint mechanism. This mechanism uses a semantic edge weight map as a gating signal to force the spatial gradient of the synthesized optical flow field to remain consistent with the spatial gradient of the parameterized rigid body reference flow field in the edge region of the rigid body contour. By suppressing the tangential rate of change of the residual optical flow field at the edges, the synthesized optical flow field retains the independent motion details of the non-rigid body regions while strictly maintaining the geometric integrity of the rigid body object.

[0009] Finally, this method establishes a pseudo-depth ranking logic based on semantic categories. Pseudo-depth maps are generated according to preset priorities of semantic categories (such as foreground rigid bodies, background, etc.). During the image compositing stage, a layered forward transformation is performed using the synthetic optical flow field, and a pseudo-depth-based splatter algorithm is employed to resolve occlusion conflicts where multiple pixels are mapped to the same location, ensuring that foreground objects correctly cover background objects. For background holes exposed by foreground motion, the independent transformation results of the background layer are used to fill them, and texture details are restored via a mesh refinement network, ultimately generating a high-fidelity target intermediate frame image.

[0010] A second aspect of this invention provides a deep learning-based intelligent frame interpolation system for film and television footage. This system includes a feature extraction and semantic parsing module, a parametric rigid body motion estimation module, a residual optical flow generation module, and a depth-aware synthesis module.

[0011] The feature extraction and semantic parsing module encodes the input image sequence, generating feature maps and semantic masks that distinguish rigid and non-rigid objects. The parametric rigid body motion estimation module identifies rigid body instances based on the semantic masks, predicts geometric transformation parameters through a regression network, and constructs a parametric rigid body reference flow field reflecting the physical motion of the rigid body. The residual optical flow generation module calculates the spatial gradient and semantic edge weights of the reference flow field and predicts the residual optical flow through a deep network. This module integrates gradient consistency constraint logic during the calculation process, using edge weights to force the spatial derivative of the synthesized optical flow to approximate the analytical derivative of the reference flow field at the rigid body boundary, thereby eliminating edge artifacts. The depth-aware synthesis module generates pseudo-depth information based on semantic categories, performs layered forward transformation and hole filling capable of handling occlusion using the optical flow field and pseudo-depth, and finally outputs intermediate frame images.

[0012] This invention provides a method and system for intelligent frame interpolation of film and television footage based on deep learning. It has the following beneficial effects: 1. This invention effectively solves the problem of maintaining rigid body structures in large displacement scenarios by constructing a parameterized rigid body reference flow field based on semantic segmentation and combining it with a residual optical flow generation mechanism. This method uses homography or affine matrices to perform analytical motion modeling on rigid body instances such as vehicles and buildings, ensuring that the reference flow field strictly follows the laws of physical perspective. At the same time, it uses a deep residual network to correct only non-rigid body regions or small deformations, avoiding the structural distortion caused by traditional pure pixel-level optical flow prediction when dealing with large-amplitude motion, and improving the geometric realism of rigid body objects in the synthesized image.

[0013] 2. This invention introduces motion field gradient consistency constraints based on semantic edge weights, which improves the sharpness of moving object edges. By calculating the spatial gradient field of the reference flow field and forcing the spatial derivative of the synthesized optical flow field to approximate the analytical derivative of the reference flow field in the semantic edge region, this invention can effectively suppress erroneous smoothing or tangential abrupt changes in residual optical flow at the object contour. This mechanism ensures that the edge topology of the rigid body does not collapse or become blurred during motion, so that the synthesized intermediate frame image remains sharp at the object boundary, solving the edge blurring problem caused by optical flow overflow.

[0014] 3. This invention employs a hierarchical forward transform and hole-filling strategy based on semantic pseudo-depth, solving the problem of occlusion and hole repair in monocular video frame interpolation. By generating a pseudo-depth map through preset semantic priorities and performing forward warping in conjunction with the Softmax splashing algorithm, the system can accurately determine the occlusion relationship between pixels, preventing background pixels from mistakenly covering foreground objects. At the same time, by using independent background layer transformation to perform complementary filling on exposed areas, the system effectively repairs the holes in the image caused by object movement, ensuring the integrity of the synthesized image in terms of texture details and spatial logic. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of the method flow of the present invention; Figure 2 This is a schematic diagram of the system modules of the present invention. Detailed Implementation

[0016] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] This invention provides a deep learning-based intelligent frame interpolation system for film and television footage, which runs on a computer device. The computer device includes a processor, memory, input / output interfaces, communication interfaces, and a bus.

[0018] The processor, memory, input / output interfaces, and communication interfaces are interconnected via a bus. The processor is configured to execute computer program instructions stored in the memory to implement the various steps of the intelligent frame interpolation method for video images in this embodiment. The processor is a central processing unit, graphics processing unit, application-specific integrated circuit, or field-programmable gate array (FPGA) with data processing capabilities. During deep learning model inference, the processor preferentially employs a graphics processing unit with parallel computing capabilities.

[0019] The memory is configured to store computer program code and intermediate data generated during the implementation of this invention, including input first frame image data, second frame image data, semantic segmentation mask data, transformation matrix parameters, and generated intermediate frame image data. The memory includes volatile memory and non-volatile memory.

[0020] The input / output interfaces are configured to connect to display devices, keyboards, mice, or external storage devices to receive user-input video sequences or output interpolated video sequences. The communication interface is configured to transmit data with other computing devices over a network and receive video data to be processed from a remote server.

[0021] See attached document Figure 2 This intelligent frame interpolation system for film and television footage is configured to generate a target intermediate frame image located at the time point between the first and second frame images. The system includes a feature extraction and semantic parsing module, a parametric rigid body motion estimation module, a residual optical flow generation module, and a depth-aware synthesis module.

[0022] The feature extraction and semantic parsing module is configured to receive a first frame image and a second frame image. The module uses a convolutional neural network to encode features in the first and second frames, generating a first multi-scale feature map and a second multi-scale feature map, respectively. Simultaneously, the module performs semantic segmentation on both frames, generating a first semantic segmentation probability map and a second semantic segmentation probability map. Based on a preset category set, the module divides the pixel regions in the image into a set of rigid instance regions and a set of non-rigid regions, generating corresponding binary semantic masks.

[0023] The parametric rigid body motion estimation module is connected to the feature extraction and semantic parsing module. The parametric rigid body motion estimation module is configured to predict the homography transformation matrix from the first frame to the target intermediate frame for each rigid body instance in the set of rigid body instance regions using neural network regression. The module then uses the homography transformation matrix to project the pixel coordinates within the rigid body instance region, calculating the analytical displacement vector for each pixel within that region. Based on a binary semantic mask, the module aggregates the analytical displacement vectors of all rigid body instances to generate a reference rigid body flow field. For regions corresponding to non-rigid body regions, the module initializes the vectors at the corresponding positions in the reference rigid body flow field to either a zero vector or a global background transformation vector.

[0024] The residual optical flow generation module is connected to the feature extraction and semantic parsing module and the parameterized rigid body motion estimation module, respectively. The residual optical flow generation module is configured to receive a first multi-scale feature map, a second multi-scale feature map, and a reference rigid body flow field. The module extracts edge information from the first semantic segmentation probability map to generate a semantic edge weight map. It then calculates the spatial gradient field of the reference rigid body flow field. Finally, the module uses a deep residual network to predict the residual optical flow field and adds the reference rigid body flow field and the residual optical flow field pixel-by-pixel to obtain the synthesized optical flow field. During this process, the module constrains the residual optical flow field using the semantic edge weight map and the spatial gradient field of the reference rigid body flow field, ensuring that the spatial gradient of the synthesized optical flow field remains consistent with the spatial gradient of the reference rigid body flow field in the semantic edge region.

[0025] The depth-aware synthesis module is connected to the residual optical flow generation module. The depth-aware synthesis module is configured to set the pseudo-depth order of each pixel region based on its semantic category. Using the synthesized optical flow field, the depth-aware synthesis module performs forward transformation operations on different semantic level regions of the first frame image, mapping the pixels to the coordinate system of the target intermediate frame image. The depth-aware synthesis module uses the pseudo-depth order and Z-Buffer algorithm to handle multi-pixel overlap during the mapping process, retaining the pixel value with the earliest pseudo-depth order to generate a coarse intermediate frame image. The depth-aware synthesis module uses the first multi-scale feature map to perform contextual refinement processing on the coarse intermediate frame image, filling in hole regions and restoring texture details, finally outputting the target intermediate frame image.

[0026] See attached document Figure 1 This invention provides a method for frame interpolation of film and television images based on semantic rigid body decoupling and edge motion gradient consistency constraints. The method includes the following steps: Obtain the input image sequence and perform multi-scale feature extraction and semantic parsing: The computer device acquires the first and second frames of the video sequence to be processed. To capture image details at different scales, the device utilizes a shared-weight feature extraction network to perform multi-level convolutional operations on the first and second frames, extracting a multi-scale feature pyramid containing texture, edges, and high-dimensional contextual information. Simultaneously, the device uses a semantic segmentation network branch to perform pixel-level classification prediction on the two frames, generating corresponding semantic segmentation probability maps. Based on preset physical attribute classification rules, the system maps the semantic segmentation results to a set of rigid body instances and a set of non-rigid body instances. The rigid body instance set includes background, vehicles, buildings, and road signs, while the non-rigid body set includes pedestrians, animals, fluids, and vegetation. For each independent object identified as a rigid body, the system generates an independent binary mask for subsequent motion decoupling processing.

[0027] Constructing a parameterized rigid body reference flow field: For each region marked as a rigid body instance, the computer device does not directly predict the pixel-by-pixel optical flow vector. Instead, it predicts the global geometric transformation parameters of that instance from the first frame to the intermediate frame of the target object using a parametric regression network. Specifically, for background and planar rigid body objects, the computer device regresses and predicts the homography transformation matrix; for simple rigid body objects, it regresses and predicts the affine transformation matrix.

[0028] The computer device uses the predicted transformation matrix to perform spatial projection transformation on the pixel coordinates within the mask of each rigid body instance, calculating the analytical displacement vector induced by the geometric transformation. Subsequently, based on the distribution of the semantic mask, the computer device spatially aggregates the analytical displacement vectors of all rigid body instances to generate a reference rigid body flow field covering the entire image. In the reference rigid body flow field, the motion of the rigid body region strictly follows the projection geometric constraints, thus ensuring the topological consistency of the internal structure of the rigid body; for non-rigid body regions, the displacement values ​​in the reference rigid body flow field are initialized to zero or filled by the global background motion, serving as the starting reference for subsequent residual prediction.

[0029] Generate a residual optical flow field based on gradient consistency constraints: To capture the complex deformations of non-rigid objects and correct for minor errors in rigid body parameter estimation, the computer device further predicts the residual optical flow field superimposed on the reference rigid body flow field. In this process, the computer device first performs edge detection on the semantic segmentation probability map of the first frame image, extracting a semantic edge weight map to identify the boundary regions between the rigid body and the background, and between the rigid body and non-rigid bodies. Simultaneously, the computer device calculates the spatial gradient field of the reference rigid body flow field, which reflects the rate of change of the rigid body motion in space.

[0030] The computer system inputs multi-scale feature maps, the reference rigid body flow field, and gradient information into the residual prediction network. During the prediction process, the system introduces a motion field gradient consistency constraint mechanism. This mechanism mandates that in the high-response regions of the semantic edge weight map, the spatial gradient of the generated synthetic optical flow field (i.e., the sum of the reference flow field and the residual flow field) must be consistent with the spatial gradient of the reference rigid body flow field. Through this constraint, the tangential rate of change of the residual optical flow field at the rigid body edge is locked, preventing edge blurring due to smoothing effects or structural tearing due to excessive residuals, ultimately generating a high-precision synthetic optical flow field.

[0031] Perform depth-aware layered warping and image compositing: The computer device generates intermediate frame images of the target image using the synthesized optical flow field and semantic information. Unlike uniform interpolation of the entire image, the computer device employs a hierarchical forward transform strategy. First, the system defines a pseudo-depth order based on semantic categories, prioritizing foreground rigid bodies and foreground non-rigid bodies over the background. Subsequently, based on the synthesized optical flow field, the computer device maps the background layer, rigid body layer, and non-rigid body layer from the first frame image to the coordinate system of the intermediate frame of the target image.

[0032] During the mapping process, the computer device uses the Z-Buffer depth buffering algorithm to handle multi-pixel overlap conflicts, retains the color value of the foreground pixels according to the pseudo-depth order, and generates a coarse intermediate frame that includes occlusion relationships. For unoccluded areas, i.e., hole areas, caused by object movement, the system uses the mapping results of the background layer to fill them. Finally, the computer device inputs the coarse intermediate frame and contextual features into the mesh refinement module, uses residual learning to recover the high-frequency texture details of the image, eliminates resampling artifacts, and outputs the final patched frame image.

[0033] This section details how to extract feature representations for subsequent optical flow estimation and semantic information for motion decoupling from the input raw image sequence. Input data and network architecture fundamentals. The feature extraction and semantic parsing module receives the first temporally continuous image from the video sequence. Second frame image As input. The input image is represented in tensor form with dimensions of . ,in Represents the image height. 3 represents the image width, and 3 represents the RGB color channels.

[0034] This module employs a Siamese network architecture, comprising two encoder branches with shared weights. These two branches process data in parallel. and The encoder network consists of a series of stacked convolutional blocks, each containing a convolutional layer, a normalization layer (such as InstanceNormalization), and a non-linear activation function (such as LeakyRelU). To capture motion information of varying magnitudes and preserve image texture details, the encoder network performs downsampling operations during processing, generating a multi-scale feature pyramid with progressively decreasing resolution and progressively increasing channel count. The construction of the multi-scale feature pyramid is described below. For the input image (Similarly applies to) The encoder network outputs a... The feature set of each level is denoted as . .in, Indicates the first Feature maps of each layer. With each layer... With the increase of , the spatial resolution of the feature map decreases, and the receptive field expands.

[0035] Specifically, the first layer features It features the highest spatial resolution for capturing high-frequency texture details in images to support subsequent sharpness refinement; highest-level features It possesses the smallest spatial resolution but the largest receptive field, enabling it to capture large displacement motion information and global contextual semantics in images. These multi-scale features will be shared by subsequent parameterized rigid body motion estimation and residual optical flow generation modules.

[0036] Semantic segmentation and probabilistic graph generation: While extracting general visual features, this module performs pixel-level classification prediction of the image through a semantic segmentation branch. This branch receives high-level features from the encoder output, recovers spatial resolution through upsampling and convolution operations, and finally outputs a channel count of [number missing]. The semantic logitsMap, where The total number of predefined semantic categories.

[0037] The system applies the Softmax function to the log-odds map along the channel dimension to generate a semantic segmentation probability map. and For each pixel location in the image It belongs to the first Probability of semantic category The calculation is as follows: ; in, Indicates the first Each channel at pixel position The logarithmic probability value at that point. Indicates the first Each channel at pixel position The logarithmic probability value at that point. Natural exponential function (in) (Based on the bottom). Ultimately, the semantic label for each pixel. The category index identified as having the highest probability value: .

[0038] To achieve motion decoupling, this embodiment predefines a physical property classification rule, dividing all semantic categories into two mutually exclusive sets: a rigid body category set. Non-rigid body category set .

[0039] rigid body category set This includes objects with stable geometric structures, such as backgrounds, roads, buildings, and vehicles (cars / buses / trucks).

[0040] Non-rigid body category set This includes objects that are prone to deformation or have irregular movements, such as pedestrians, vegetation, sky, and water.

[0041] Based on the above classification, the module further generates specific binary masks. For those belonging to... The system aggregates the pixels to generate a non-rigid mask. For those belonging to For each pixel, the system not only distinguishes its semantic category, but also uses connected component analysis algorithms or instance segmentation heads to distinguish different instances of the same category (e.g., distinguishing two cars driving side by side).

[0042] The system provides each individual rigid body instance in the image. Generate a binary mask For pixel position If the pixel belongs to the first For each rigid body instance, then ,otherwise All generated masks satisfy the coverage and mutual exclusion constraints, meaning that any pixel in the image belongs to only one rigid body instance or a non-rigid body region. The output mask set... The non-rigid mask will be transmitted to the parametric rigid body motion estimation module as a spatial gating signal for applying different motion models.

[0043] This section details how to use geometric constraints to transform semantically defined rigid body objects into physically governed reference motion fields. It covers parametric regression of rigid body instances. The parametric rigid body motion estimation module receives deep semantic features output from the feature extraction module, along with a binary mask for each rigid body instance. For a set of rigid bodies... Each independent instance in (Including global background instances), the module estimates its time from the current frame moment through a parameter regression sub-network. to the target intermediate frame time Geometric transformation relationship of (0,1).

[0044] Specifically, the parametric regression subnetwork first utilizes instance masks. Spatial masking or region-of-interest pooling (ROPooling) is applied to the global feature map to extract the feature vector of the specific rigid object. This feature vector is then mapped to a set of geometric transformation parameters through a fully connected layer. In this embodiment, for rigid objects with significant perspective changes (such as roads or buildings), the network predicts a homography matrix with 8 degrees of freedom. For objects that only undergo planar motion or have a distant background, the network prediction includes an affine transformation matrix with 6 degrees of freedom. This matrix describes the projection transformation of the plane containing the rigid body in three-dimensional space.

[0045] Coordinate projection transformation and analytical displacement calculation: In obtaining an instance Transformation matrix Then, the module performs a coordinate grid-based projection transformation. First, the system constructs a standard pixel coordinate grid with the same resolution as the input image. For any pixel position coordinate in the grid... Extend it to homogeneous coordinate form .

[0046] Module utilizes matrix Perform matrix multiplication on the homogeneous coordinates to calculate the target homogeneous coordinates of the pixel under the rigid body motion model. ; in, These are the actual coordinates of the image in two-dimensional space. These homogeneous components are added to facilitate matrix operations. These are the homogeneous coordinate components obtained after matrix operations.

[0047] To obtain the actual coordinates in Euclidean space, the system performs perspective division, dividing the first two transformed components by the third component. This step is crucial for accurately simulating the visual effect of objects appearing larger when closer or smaller when farther away, caused by camera zooming or changes in object depth. Normalized target coordinates. The calculation is as follows: ; Based on the coordinate differences before and after the transformation, the system calculates the analytical motion vector corresponding to the rigid body instance. : ; This calculation process ensures that the motion vectors of all pixels belonging to the same rigid body are geometrically strictly coplanar, fundamentally eliminating local distortions within the object.

[0048] Spatial aggregation and initialization of the reference flow field: After calculating the independent motion vector fields of all rigid body instances, the module aggregates them into a complete full-map reference rigid body flow field based on mutually exclusive semantic masks. For each pixel position in the image The system detects the semantic category to which it belongs.

[0049] If the pixel belongs to the th A rigid body instance (i.e.) Then directly calculate the... Assigned to the reference flow field .

[0050] If the pixel belongs to a non-rigid region (i.e.) The system initializes the location using the transformation parameters of the global background. This means that non-rigid objects (such as pedestrians) are assigned a motion tendency consistent with the camera background in the reference flow field. This initialization strategy provides the subsequent residual network with a zero-point reference that includes the camera's own motion (Egomotion), allowing the residual network to focus only on predicting the independent motion of non-rigid objects relative to the background (such as the walking motion of pedestrians), rather than learning absolute motion, thereby reducing the learning difficulty and improving prediction accuracy.

[0051] The final generated reference rigid body flow field It is a dense two-dimensional vector field that appears as a smooth manifold that conforms to the perspective law within the rigid body region. At the edge of the rigid body, there is a motion discontinuity caused by the depth difference of the object. This flow field is passed to the residual optical flow generation module as the basic input.

[0052] This section details how to use the analytical properties of rigid body motion to constrain unstructured neural network predictions, thereby preserving the topological structure of rigid body edges while retaining non-rigid details.

[0053] Extraction and construction of semantic edge weight graph: To determine the spatial scope of the gradient consistency constraint, the residual optical flow generation module first extracts edge features from the input semantic segmentation probability map. The module receives the semantic segmentation probability map of the first frame image and performs a convolution operation on the binary mask of the rigid body instance using a spatial gradient operator (such as the Sobel operator or the Scharr operator) to calculate the semantic boundary response.

[0054] Specifically, the system calculates the first derivatives of the rigid body mask in the horizontal and vertical directions, and uses the magnitude of the gradient as the edge intensity at that location. To cover potential blurred areas or optical flow overflow areas at the edges of moving objects, the system performs morphological dilation on the calculated edge intensity map, generating a semantic edge weight map with a certain width, denoted as... .exist In the diagram, high response values ​​correspond to the boundary regions between rigid bodies and the background or between rigid bodies and non-rigid objects, while response values ​​within rigid bodies or flat background regions approach zero. This weighted map will serve as a spatial gating signal for subsequent constraint mechanisms.

[0055] Calculation of the spatial gradient field of the reference rigid body flow field: Reference rigid body flow field It is generated by homography or affine matrices, therefore it naturally possesses mathematical smoothness and continuity (i.e., analytical differentiability) within the rigid body connected domain. The module relates to the reference rigid body flow field. Perform numerical differentiation to calculate its spatial gradient field. .

[0056] Define a two-dimensional vector field ,in, and These are the two component functions of a vector. Corresponding to The components of the axis (horizontal direction), Corresponding to The components of the axis (vertical direction), The transpose is represented, and its spatial gradient field contains partial derivative information in both the horizontal and vertical directions: ; in, Indicates at point The gradient at that point, the first line: contains about and The partial derivatives, second line: contain about and The partial derivatives of .

[0057] The gradient field It accurately describes the stretching, rotation, and shearing trends of a rigid body during spatial motion and is the kinematic fingerprint of a rigid body's physical structure.

[0058] Prediction and Synthesis of Residual Optical Flow Field The module utilizes a deep residual flow network to predict minute motion components other than rigid body transformations. This network receives the following data as a concatenation of input channels: First and second multi-scale feature maps: provide visual texture matching cues.

[0059] Reference rigid body flow field : Provides initial estimates of the motion.

[0060] Spatial gradient field of reference rigid body flow field Provides structural guidance information for motion.

[0061] Deep residual networks output a residual optical flow field through a series of convolutional and deconvolutional layers. The residual flow field is used to capture the independent motion of non-rigid objects (such as the limb swing of a pedestrian) and to correct projection errors caused by the rigid body plane assumption (such as the small parallax of a curved car body).

[0062] Final synthetic optical flow field The calculation is a pixel-by-pixel superposition of the reference flow field and the residual flow field: ; in, Indicates at point The final vector value, This is the fundamental vector field, at point... The initial value at that location, Indicates at point The change or increment at a point.

[0063] Motion field gradient consistency constraint mechanism at semantic edges: To prevent residual optical flow field Introducing erroneous smoothing or abrupt changes at the edges of a rigid body can disrupt its contour integrity. This system introduces gradient consistency constraints during residual generation.

[0064] The core logic of this constraint mechanism lies in: in semantic edge regions (i.e. High response region), synthetic optical flow field The local spatial rate of change should strictly follow the reference rigid body flow field. The rate of change. This means the residual optical flow field The spatial gradient at the edge should be close to zero or remain smooth, and should not produce abrupt tangential jumps.

[0065] The system optimizes the generation quality of the synthesized optical flow field by minimizing the following constraints: ; in, Indicates the location Constraint loss at the location, This is a weighting function. and They represent the final vector field and the fundamental vector field at their respective positions. gradient at, This represents the L2 norm.

[0066] Will Substituting the definition into the above equation, this constraint is equivalent to weighted suppression of the gradient of the residual flow field: ; During the inference phase, this constraint is manifested as a feature attention mechanism in the network structure, that is, utilizing... Feature channels The generated features are gated to force the residual flow to maintain tangential continuity at the rigid body boundaries. In this way, the system ensures that even in scenes with large-scale motion, the edges of rigid objects can still move as a whole like a rigid body, eliminating edge trailing or jelly deformation phenomena common in traditional methods, while allowing rich nonlinear motion details to be preserved inside the rigid body or in non-rigid body regions.

[0067] This section details how to use synthetic optical flow field and semantic depth information to accurately map the pixels of the source frame image to the target time and synthesize the final high-quality intermediate frame image.

[0068] Definition and generation of semantic pseudo-depth order: The depth-aware synthesis module first establishes pixel-level depth prior information to address the pixel occlusion problem that inevitably occurs during forward warping. Since monocular videos lack true physical depth information, this system employs a pseudo-depth ranking strategy based on semantic categories.

[0069] The module pre-stores a semantic category priority lookup table for each semantic category. Assign a depth priority value Generally speaking, foreground objects have a higher priority than background objects. The module generates a pseudo-depth map with the same resolution as the image based on the semantic segmentation probability map of the first frame. For each pixel location in the image Its pseudo-depth value is defined as: ; in, Represents the original image. This is the semantic label for that pixel. This is the pseudo-depth map. Used to determine which pixel should be displayed on top when multiple source pixels are mapped to the same target location.

[0070] Layered forward transform and soft splash synthesis: The module utilizes a synthetic optical flow field Perform a layered forward transform. Unlike the traditional backward transform, the forward transform transforms the source image... pixel position Based on motion vectors Move to target image Non-grid point positions in To handle non-integer coordinates and many-to-one mapping conflicts, this embodiment employs a pseudo-depth-based Softmax Splatting algorithm. The system defines an importance metric function. This function is related to pseudo-depth. Positive correlation is used to assign higher composition weights to foreground pixels. For example, using an exponential function: ,in This is the depth discrimination coefficient.

[0071] For any integer pixel coordinate on the target image Its pixel value By aggregating all source pixels projected into the neighborhood of that location The calculation yielded: ; in, Indicates the source location Image values ​​at that location, It is a weighted function. This represents the L1 norm.

[0072] In the above formula, the denominator term is used to normalize the weights. This mechanism ensures higher pseudo-depth when background and foreground pixels are mapped to the same target location simultaneously (i.e., occlusion occurs). The foreground pixels will dominate in the weighted average, thus correctly covering the background pixels and preventing objects from clipping through or ghosting.

[0073] Hole filling complements the background hierarchy: Due to object motion, background areas that were originally occluded in the source image may become visible in the target frame. Forward transformation will cause these areas to appear in the target frame. Undefined holes (Holes) have appeared.

[0074] To fill these gaps, the module employs a semantic layering strategy for complementary processing. The system extracts the background layer (i.e., the region with the semantic category "Background") from the first frame image separately, and performs independent warping operations using the affine transformation parameters of the background layer or the baseline flow field to generate a complete background prediction map. During the synthesis stage, for In regions where the sum of weights is zero (i.e., passive pixel projection), the module directly uses... The pixel values ​​are used for filling. This semantically layered filling strategy effectively utilizes the global motion consistency of the background to correct visual gaps behind moving objects.

[0075] Contextual refinement and final imaging: The coarse intermediate frame generated by the above steps Although geometrically correct, high-frequency details may be lost due to resampling operations, or artifacts may exist at fill boundaries. Therefore, the module sets up a grid refinement subnetwork.

[0076] This subnetwork receives the following data as input coarse intermediate frames. .

[0077] The first multi-scale feature map (i.e., context feature) after being transformed by the same warping operation.

[0078] Synthetic optical flow field The mesh refinement subnetwork employs an hourglass architecture based on the U-Net structure, learning the residual texture information of the image through multiple convolutional layers. The network outputs a residual image. Add it to the coarse intermediate frame to obtain the final target intermediate frame. ; Through this refinement step, the system restores the image's clarity and smooths the transition edges between the hole-filled area and the surrounding pixels, completing the entire frame interpolation process.

[0079] See attached document Figure 2 This section maps the above method steps to specific device functional modules, and elaborates on the internal composition and data interaction logic of each module to support the device-type claims of this invention.

[0080] The feature extraction and semantic parsing module is the system's perceptual front end, configured to receive the first and second frames of the time series image. This module integrates a parallel-processing dual-stream encoder unit and a semantic segmentation unit. The dual-stream encoder unit contains a series of weighted convolutional and downsampling layers to convert the input RGB image data into high-dimensional feature tensors. These feature tensors not only encode the image's color and brightness information but also encode the image's local texture and global structural information through multi-level receptive fields, forming a first multi-scale feature map and a second multi-scale feature map.

[0081] Simultaneously, the semantic segmentation unit utilizes a fully convolutional network structure to perform pixel-by-pixel classification of the input image. This unit internally stores predefined physical attribute classification rules, enabling it to identify rigid objects (such as vehicles and buildings) and non-rigid objects (such as pedestrians and water flow) in the image. The feature extraction and semantic parsing module generates a semantic segmentation probability map based on the classification results and further generates binary instance masks for each independent rigid object and a global mask for non-rigid regions through thresholding and connected component analysis. These mask signals serve as spatial gating control signals for subsequent modules, determining which motion modeling strategy will be used for different image regions.

[0082] The parametric rigid body motion estimation module is connected to the feature extraction and semantic parsing module, primarily used to construct a reference flow field that conforms to the laws of physical perspective. This module internally contains a parametric regression sub-network and a geometric projection transformation unit. The parametric regression sub-network is configured to extract the corresponding image features for each rigid body instance selected by the mask, and then predict the geometric transformation parameters of that instance in 3D space motion through fully connected layers. For different rigid body types, this module can adaptively output either a homography transformation matrix or an affine transformation matrix.

[0083] The geometric projection transformation unit is configured to perform matrix-operation-based coordinate mapping. This unit constructs a standard pixel coordinate grid and uses the predicted transformation matrix to project the coordinates of each pixel within the rigid body instance mask, calculating the analytical displacement vector of that pixel caused by the rigid body motion. The parameterized rigid body motion estimation module further includes a flow field aggregation unit, which concatenates the analytical displacement vectors of all rigid body instances into a complete reference rigid body flow field based on the spatial distribution of the semantic mask. For regions not covered by the rigid body mask, the flow field aggregation unit initializes their motion vectors to zero or the motion components of the global background, thereby outputting a dense vector field containing the prior knowledge of the rigid body structure.

[0084] The residual optical flow generation module, connected to both the feature extraction and semantic parsing module and the parameterized rigid body motion estimation module, is the core computational unit of this system. This module aims to predict unstructured residual motion through deep learning and correct the prediction results using geometric constraints. Internally, this module includes an edge detection unit, a gradient calculation unit, and a deep residual network unit. The edge detection unit uses a differential operator to convolve the semantic segmentation probability map, extracting a semantic edge weight map reflecting the rigid body contour and the object's boundary. The gradient calculation unit performs spatial differentiation on the input baseline rigid body flow field, calculating the spatial gradient field reflecting the rate of change of rigid body motion.

[0085] The deep residual network unit receives multi-scale feature maps, a reference rigid body flow field, and its spatial gradient field as input, and predicts the residual optical flow field through a multi-layer convolutional neural network. During network inference, this module integrates gradient consistency constraint logic, which uses the semantic edge weight map and the spatial gradient field of the reference rigid body flow field to gate the generation of the residual optical flow field. Specifically, in the semantic edge region, this constraint logic forces the spatial derivative of the synthesized optical flow field (i.e., the sum of the reference flow field and the residual flow field) to approach the analytic derivative of the reference rigid body flow field, thereby ensuring that the rigid body edge maintains the geometric integrity during motion and avoiding non-physical deformation.

[0086] The depth-aware synthesis module is connected to the residual optical flow generation module to generate the final target intermediate frame image. This module internally includes a depth sorting unit, a hierarchical splatter transformation unit, and a mesh refinement unit. The depth sorting unit, based on the input semantic category label, queries a pre-defined priority list and assigns a pseudo-depth value to each pixel in the image, constructing a pseudo-depth map. The hierarchical splatter transformation unit utilizes the synthesized optical flow field and the pseudo-depth map to perform a forward warping operation. This unit employs the SoftmaxSplatting algorithm, using pseudo-depth values ​​to address occlusion issues when multiple pixels overlap during the mapping of source pixels to the target coordinate system, ensuring that foreground objects correctly cover background objects.

[0087] In addition, the depth-sensing synthesis module is equipped with hole-filling logic, which uses the independent transformation results of the background layer to fill the background areas exposed by foreground motion. Finally, the mesh refinement unit receives the transformed coarse image and the original context features, and utilizes a residual learning machine.

Claims

1. A deep learning-based intelligent frame filling method for movie pictures, characterized in that, Includes the following steps: The first and second frame images in the video sequence are acquired. By performing feature encoding and semantic segmentation on the first and second frame images, a multi-scale feature map, a semantic segmentation probability map, and a semantic segmentation mask are generated. Based on the semantic segmentation mask, the first frame image is divided into rigid body instance regions and non-rigid body regions. The geometric transformation parameters of each rigid body instance region from the time of the first frame image to the time of the target intermediate frame image are predicted by a parameter regression network. A parameterized rigid body reference flow field covering the entire image is constructed based on the geometric transformation parameters. The spatial gradient field is calculated by performing spatial differentiation on the parameterized rigid body reference flow field, and the semantic edge weight map is extracted based on the semantic segmentation probability map. The multi-scale feature map, the parameterized rigid body reference flow field, and the spatial gradient field are input into a deep residual network to predict the residual optical flow field. A synthetic optical flow field is generated by superimposing the parameterized rigid body reference flow field and the residual optical flow field. In the process of predicting the residual optical flow field, a motion field gradient consistency constraint is introduced, and the spatial gradient of the synthetic optical flow field is constrained to be consistent with the spatial gradient of the parameterized rigid body reference flow field through the semantic edge weight map. The pseudo-depth order of each pixel region is determined based on the semantic segmentation probability map. The target intermediate frame image is generated by performing a layered forward transform and hole filling on the first frame image using the synthetic optical flow field and the pseudo-depth order.

2. The deep learning-based intelligent frame interpolation method for film pictures according to claim 1, characterized in that, The step of dividing the first frame image into rigid body instance regions and non-rigid body regions based on the semantic segmentation mask includes: By using preset physical attribute classification rules, the semantic categories in the semantic segmentation probability map are divided into a rigid body category set and a non-rigid body category set; the rigid body category set includes background, vehicle, building and road sign objects, and the non-rigid body category set includes pedestrian, animal, fluid and vegetation objects. For pixels belonging to the set of rigid body categories, an independent rigid body instance mask is generated by utilizing connected component analysis; for pixels belonging to the set of non-rigid body categories, a non-rigid body region mask is generated.

3. The deep learning-based intelligent frame interpolation method for film pictures according to claim 1, characterized in that, The steps for constructing a parameterized rigid body reference flow field covering the entire graph based on the geometric transformation parameters include: For each of the rigid body instance regions, a projection transformation matrix is ​​constructed using the geometric transformation parameters, and the analytical displacement vector induced by the rigid body motion is calculated by projecting the pixel coordinates within the rigid body instance region using the projection transformation matrix. Based on the spatial distribution of the semantic segmentation mask, the analytical displacement vectors of all rigid body instance regions are aggregated into the same vector field; For the vector field position corresponding to the non-rigid body region, the parameterized rigid body reference flow field is obtained by initializing the analytical displacement vector to a zero vector or by calculating the background motion vector using the geometric transformation parameters of the global background.

4. The deep learning-based intelligent frame interpolation method for film and television pictures according to claim 1, characterized in that, The geometric transformation parameters include homography transformation matrix and affine transformation matrix; The step of predicting the geometric transformation parameters of each rigid body instance region from the time of the first frame image to the time of the target intermediate frame image using a parametric regression network includes: If the semantic category corresponding to the rigid body instance region is background or planar rigid body structure, then the homography transformation matrix containing 8 degrees of freedom is obtained by regression prediction calculation. If the semantic category corresponding to the rigid body instance region is a non-planar simple rigid body structure, then the affine transformation matrix containing 6 degrees of freedom is obtained by regression prediction calculation.

5. The intelligent frame interpolation method for film and television images based on deep learning according to claim 1, characterized in that, The steps for extracting a semantic edge weight map from the semantic segmentation probability map include: The edge response intensity is calculated by performing a convolution operation on the rigid body instance boundaries in the semantic segmentation probability map using the spatial gradient operator. The semantic edge weight map is generated by performing morphological dilation on the edge response intensity. The semantic edge weight map is used to identify the boundary region between rigid body and background, as well as the boundary region between rigid body and non-rigid body.

6. The intelligent frame interpolation method for film and television images based on deep learning according to claim 1, characterized in that, The step of introducing motion field gradient consistency constraints includes: Construct a constraint loss function, which is used to characterize the difference norm between the spatial gradient of the synthetic optical flow field and the spatial gradient of the parameterized rigid body reference flow field; By using the semantic edge weight map as a weighting coefficient to weight the constraint loss function, the local spatial rate of change of the residual optical flow field in the high response region of the semantic edge weight map approaches zero, thereby maintaining the continuity of the tangential motion of the rigid body edge.

7. The intelligent frame interpolation method for film and television images based on deep learning according to claim 1, characterized in that, The steps for determining the pseudo-depth order of each pixel region based on the semantic segmentation probability map include: A preset semantic category priority list is provided, in which the foreground rigid body category and the foreground non-rigid body category have higher priority than the background category; By querying the semantic category priority list based on the semantic category label of each pixel in the first frame image, a pseudo-depth value is assigned to each pixel, generating a pseudo-depth map consistent with the image resolution.

8. The intelligent frame interpolation method for film and television images based on deep learning according to claim 1, characterized in that, The steps for generating a target intermediate frame image by performing a layered forward transform and hole filling on the first frame image using the synthetic optical flow field and the pseudo-depth order include: Using the synthetic optical flow field, the pixels of the first frame image are mapped to the target coordinate system; The Softmax splashing algorithm is used to handle the overlapping conflict of multiple source pixels mapped to the same target position. The synthesis weight of each source pixel is calculated through the pseudo-depth map, and the pixel information with the largest pseudo-depth value is retained to generate a coarse intermediate frame image. The background layer in the first frame image is used to fill the hole areas in the rough intermediate frame image caused by foreground motion; The coarse intermediate frame image after padding and the multi-scale feature map are received using a mesh refinement network. Texture details are recovered through residual learning, and the target intermediate frame image is output.

9. The intelligent frame interpolation method for film and television images based on deep learning according to claim 1, characterized in that, The steps for obtaining the spatial gradient field by performing spatial differentiation on the parameterized rigid body reference flow field include: The spatial gradient field, comprising four components, is calculated by taking the partial derivatives of the horizontal and vertical components of the parameterized rigid body reference flow field in the horizontal and vertical directions, respectively. The spatial gradient field characterizes the stretching, rotation, and shearing trends of the rigid body motion in space.

10. A deep learning-based intelligent frame interpolation system for film and television footage, characterized in that, The method applied to the intelligent frame interpolation method for film and television images based on deep learning as described in any one of claims 1-9 includes: The feature extraction and semantic parsing module is used to acquire the first frame image and the second frame image in the video sequence. By performing feature encoding and semantic segmentation on the first frame image and the second frame image, a multi-scale feature map, a semantic segmentation probability map, and a semantic segmentation mask are generated. The parameterized rigid body motion estimation module, which is connected to the feature extraction and semantic parsing module, is used to divide the first frame image into rigid body instance regions and non-rigid body regions based on the semantic segmentation mask, predict the geometric transformation parameters of each rigid body instance region through a parameter regression network, and construct a parameterized rigid body reference flow field covering the entire image based on the geometric transformation parameters. The residual optical flow generation module, connected to the feature extraction and semantic parsing module and the parameterized rigid body motion estimation module, is used to calculate the spatial gradient field by performing spatial differentiation on the parameterized rigid body reference flow field, and extract the semantic edge weight map based on the semantic segmentation probability map; predict the residual optical flow field using a deep residual network, and generate a synthetic optical flow field by superimposing the parameterized rigid body reference flow field and the residual optical flow field; and introduce a motion field gradient consistency constraint in the process of predicting the residual optical flow field, using the semantic edge weight map to constrain the spatial gradient of the synthetic optical flow field to be consistent with the spatial gradient of the parameterized rigid body reference flow field. A depth-sensing synthesis module, connected to the residual optical flow generation module, is used to determine the pseudo-depth order of each pixel region based on the semantic segmentation probability map, and to generate a target intermediate frame image by performing a layered forward transformation and hole filling on the first frame image using the synthesized optical flow field and the pseudo-depth order.

Images