Animation frame interpolation method and system based on image data

Abstract

The invention discloses an animation frame interpolation method and system based on image data, and relates to the technical field of data processing. Touch interaction parameters are collected, a key interface frame is obtained when the estimated pixel displacement is larger than a visual fluency displacement threshold value, initial motion optical flow features are mapped, an interaction control active area is extracted, and the interaction control active area is extracted; extracting a local structure feature of the key interface frame, extracting a global layout context dependent feature under the guidance of the active area of the interactive control, and fusing the local structure feature and the global layout context dependent feature to generate an interface context content feature; and inputting a multi-mode splicing feature formed by splicing the initial motion optical flow feature and the interface context content feature into a joint selective state space model, calculating a correlation weight, fusing a joint space fusion feature, predicting an optical flow residual error to generate a refined target motion optical flow, and twisting a key interface frame based on the refined target motion optical flow to obtain a target motion optical flow. And calculating complexity to synthesize an intermediate transition frame, and outputting the intermediate transition frame to a display panel for rendering.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to an animation frame interpolation method and system based on image data. Background Technology

[0002] With the rapid development of mobile internet and touch-screen smart terminals, human-computer interaction has evolved from basic tapping operations to highly continuous, gesture-dependent swiping and dragging interactions. In these high-frequency interaction scenarios, users have placed stringent demands on the responsiveness of the interface, the smoothness of animations, and the fidelity of visuals.

[0003] Currently, Chinese invention application number 202511324246.9 discloses a method, device, terminal, storage medium, and product for displaying keyframe animation. By pre-configuring dynamically updated attribute expressions for animation elements, and based on the scene information of the current terminal or the user's interface interaction, the specific attribute values ​​of the animation elements in the start and end frames are calculated. Then, a traditional interpolation algorithm is used to rigidly transition the beginning and end attribute values ​​to generate keyframe animation. Existing technologies have the following shortcomings: First, rule-based numerical interpolation lacks pixel-level depth understanding and struggles to handle complex interface deformations. In real-world, high-frequency interaction scenarios such as rapid scrolling and complex element occlusion and overlap, simple numerical interpolation easily leads to screen fragmentation, component clipping, and unnatural motion trajectories, failing to achieve high-fidelity visual transitions. Second, traditional deep learning visual models are limited by global attention mechanisms, resulting in significant computational and power consumption barriers. To address the image quality deficiencies of traditional interpolation, the industry has begun to explore the introduction of large-scale deep learning visual models for pixel-level redrawing. However, existing large-scale visual models generally rely on global attention matrix calculations, with computational complexity increasing quadratically with sequence length. The extremely high screen resolution and extremely short response time requirements of mobile devices can lead to rapid overflow of device memory, a sharp increase in the power consumption of the underlying GPU and heat generation, which cannot meet the stringent requirements of real-time performance and low power consumption for mobile interface interaction. Third, the disconnect between pure visual computing and physical operation intentions can easily produce interaction artifacts and tearing. Existing pure visual AI frame interpolation models are usually in a state of blind guessing, relying solely on the visual feature differences between two consecutive frames to deduce optical flow, completely ignoring the core human-computer interaction principle that physical operation must precede visual feedback. This can easily produce huge optical flow prediction deviations under extreme interaction conditions such as rapid dragging or sudden stops, resulting in ghosting, ghosting, and interface distortion tearing in the generated intermediate frames. Summary of the Invention

[0004] The technical problem solved by this invention is that existing technologies are unable to cope with the large screen changes caused by users’ rapid touch in real time under limited device computing power. Traditional frame interpolation models not only cause device overheating and lag due to excessive computational complexity, but also cannot synchronously perceive the user’s dynamic physical operation intentions and interface content deformation. As a result, severe screen tearing, frame drops and ghosting are inevitably produced when scrolling or dragging quickly, resulting in an extremely disjointed human-computer interaction experience.

[0005] To solve the above-mentioned technical problems, the present invention provides the following technical solution: an animation frame interpolation method based on image data, comprising the following steps: Step S1: Collect touch interaction parameters, obtain two key interface frames before and after the current interaction scene, and map the touch interaction parameters to the initial motion optical flow features. Step S2: Extract the local structural features of the two key interface frames before and after, perform multi-directional selective scanning on the two key interface frames before and after, and extract the global layout context dependency features with the spatial position information of the active area of ​​the interactive control as the prior guiding condition. Combine the local structural features and the global layout context dependency features to obtain the interface context content features. Step S3: The initial motion optical flow features and interface context content features are concatenated into multimodal concatenated features and input into the joint selective state space model for scanning. The correlation weights are calculated, and the scanned features are weighted and fused to obtain joint space fusion features. The optical flow residuals are predicted based on the joint space fusion features to obtain the refined target motion optical flow. Step S4: Based on the refined target motion optical flow, spatial distortion is performed on the two key interface frames before and after, and combined with joint spatial fusion features, an intermediate transition frame is synthesized while maintaining linear computational complexity, and output to the display panel for rendering.

[0006] Preferably, step S1 includes the following sub-steps: Step S101: Collect the user's touch trajectory data through the underlying driver interface of the touch screen, extract the direction information and speed information from the touch trajectory data to form touch interaction parameters, and calculate the estimated pixel displacement between adjacent frames based on the product of the speed information and the current refresh cycle of the system. Step S102: Dynamically compare the estimated pixel displacement with the preset visual smoothness displacement threshold. When the estimated pixel displacement is detected to be greater than the preset visual smoothness displacement threshold, a high frame rate interpolation mode is triggered, and a retrieval instruction is sent to the frame buffer queue of the system rendering engine to extract the current display frame and the next pre-rendered frame that are aligned with the touch interaction parameters on the timestamp, which are used as the two key interface frames before and after.

[0007] Preferably, step S1 further includes the following sub-steps: Step S103: Obtain the mapping relationship between the physical resolution of the touch screen and the display area. Combine the time interval parameter between the two key interface frames, construct a spatiotemporal mapping model from the touch physical coordinate system to the image pixel coordinate system, and determine the active area of ​​the interactive control corresponding to the touch trajectory data in the two key interface frames. Using a spatiotemporal mapping model, the direction and velocity information are normalized into a reference displacement vector in the corresponding pixel coordinate system. The reference displacement vector is broadcast and filled into the active area of ​​the interactive control to generate a two-dimensional pixel displacement vector field. The two-dimensional pixel displacement vector field includes the foreground motion state and the background static state. The two-dimensional pixel displacement vector field is used as the initial motion optical flow feature to guide the matching of two key interface frames before and after.

[0008] Preferably, step S2 includes the following sub-steps: Step S201: Input the two key interface frames before and after into the context learning module. In the local feature branch of the context learning module, through parallel deep separable convolutional layers with different preset convolutional kernel scales, extract local spatial features of the multi-scale receptive field of the two key interface frames before and after, respectively, to obtain multi-scale feature maps. Then, stitch and aggregate the multi-scale feature maps along the channel dimension to obtain local structural features. In step S202, the two key interface frames are synchronously input into the global feature branch that runs parallel to the local feature branch. Through image block embedding operation, the two key interface frames are converted into four corresponding one-dimensional serialized features according to the image block traversal paths of horizontal forward scanning, horizontal reverse scanning, vertical forward scanning, and vertical reverse scanning, respectively. The four corresponding one-dimensional serialized features are then input into the first selective state space model for multi-directional selective scanning.

[0009] Preferably, step S2 further includes the following sub-steps: Step S203: During the multi-directional selective scanning process, the spatial location information of the active area of ​​the interactive control is obtained, and according to the input dependency dynamic parameterization mechanism in the first selective state space model where parameters are dynamically generated with input data, the spatial location information is used as a priori guiding condition to dynamically allocate information retention weights and forgetting gate weights in the state transition process to each feature element in the one-dimensional serialized feature. The forgetting gate weights are used to suppress feature responses located outside the active area of ​​interactive controls in one-dimensional serialization features, and to filter redundant features in the background noise area in two key interface frames before and after. Using the aforementioned information, weights are retained to perform global context modeling only on the active areas of interactive controls, and global layout context dependency features are extracted. Step S204: The local structural features and the global layout context-dependent features are concatenated in the channel dimension, and the concatenated features are compressed in the channel and fused with cross-channel information through a preset 1×1 convolutional layer to obtain the interface context content features.

[0010] Preferably, step S3 includes the following sub-steps: Step S301: Map the initial motion optical flow features and interface context content features to a unified feature space and flatten them into one-dimensional serialized features. Then, concatenate the flattened initial motion optical flow features and interface context content features along the channel dimension to obtain multimodal concatenated features. At the same time, perform sequence flipping on the multimodal concatenated features to generate corresponding reverse multimodal concatenated features. Step S302: Input the multimodal splicing features and the corresponding reverse multimodal splicing features into the joint selective state space model, and use the joint selective state space model to perform a forward scan of the multimodal splicing features and the reverse multimodal splicing features. The results of the forward scan are added to the results of the reverse scan after re-flipping and alignment, and the updated motion feature branch and the updated content feature branch are separated along the channel dimension.

[0011] Preferably, step S3 further includes the following sub-steps: Step S303: Perform global average pooling on the updated motion feature branch and content feature branch respectively through the channel attention mechanism to generate channel feature vectors that represent the correlation between the estimated pixel displacement and the deformation of the interface content. Then, use the activation function to map the channel feature vectors into correlation weights. The correlation weights are multiplied element-wise with the corresponding motion feature branches and content feature branches. The weighted motion feature branches and content feature branches are then reassembled and projected through a linear layer to output the joint spatial fusion features. Step S304: Input the joint spatial fusion features into a preset residual prediction network layer to predict the optical flow residual used to correct the dynamic displacement deviation; The optical flow residual and the initial motion optical flow features are fused element-wise to output a refined target motion optical flow.

[0012] Preferably, step S4 includes the following sub-steps: Step S401: Determine the time step parameter corresponding to the intermediate transition frame to be generated in the time series. Based on the linear interactive motion assumption, use the time step parameter to perform proportional time scale scaling calculation on the refined target motion optical flow to generate forward intermediate optical flow and backward intermediate optical flow from the two key interface frames to the time node where the intermediate transition frame is located. The forward intermediate optical flow and backward intermediate optical flow together constitute the target intermediate motion vector field. Step S402: Based on the target intermediate motion vector field, perform forward and reverse spatial distortion operations on the pixels of the two key interface frames before and after, respectively, and align and map the pixels of the two key interface frames before and after to the time node where the intermediate transition frame is located, thereby generating the first candidate distortion frame and the second candidate distortion frame.

[0013] Preferably, step S4 further includes the following sub-steps: Step S403: Input the joint spatial fusion features into a preset mask mapping convolutional layer for channel compression and activation processing, and decode and output a dynamic mask weight matrix for pixel-level fusion. The first candidate warped frame and the second candidate warped frame are weighted and combined using the dynamic mask weight matrix to obtain the initial synthesized interface frame. Step S404: The initial synthesized interface frame and the joint spatial fusion feature are spliced ​​together in the channel dimension and then input into the preset frame synthesis network; Based on the implicit memory mechanism of the selective state space model configured inside the frame synthesis network for the temporal signal, the explicit calculation of the global attention feature matrix is ​​replaced by the latent state mapping of a one-dimensional function. While maintaining linear computational complexity, the joint spatial fusion feature is used to reconstruct local details and repair interaction artifacts of the initial synthesized interface frame, generating an intermediate transition frame that conforms to the current physical inertia of the interaction. The intermediate transition frame is then synchronously output to the rendering engine of the display panel for image rendering.

[0014] An animation frame interpolation system based on image data includes a touch guidance module, a modeling module, a fusion correction module, and an intermediate compositing module; The touch guidance is used to collect touch interaction parameters, obtain key interface frames when the estimated pixel displacement is greater than the visual smoothness displacement threshold, map the initial motion optical flow features and extract the active area of ​​the interactive control. The modeling is used to extract local structural features of key interface frames through multi-scale depth separable convolutional layers, extract global layout context-dependent features by using a first selective state space model guided by the active area of ​​interactive controls, and fuse local structural features and global layout context-dependent features to generate interface context content features. The fusion correction is used to input the multimodal splicing feature of the initial motion optical flow feature and the interface context content feature into the joint selective state space model, calculate the correlation weight based on the estimated pixel displacement to fuse the joint space fusion feature, and generate the refined target motion optical flow from the predicted optical flow residual. The intermediate synthesis is used to synthesize intermediate transition frames with linear computational complexity based on the refined target motion optical flow distortion key interface frames and combined with joint spatial fusion features.

[0015] The beneficial effects of this invention are as follows: The active area of ​​the acquired interactive control is injected into the selective state space model as a priori guiding condition. Redundant features in the background noise area are accurately filtered by dynamically allocating gating weights. The correlation weights are calculated by combining the estimated pixel displacement to perform deep fusion of multimodal stitching features. The optical flow residual is predicted to correct the deviation of rapid sliding. The time-consuming global attention calculation is abandoned. Under the condition of maintaining linear computational complexity and extremely low computing power, intermediate transition frames are generated in real time with no background ghosting artifacts and pixel-level alignment with the user's real interactive motion inertia in physical space deformation. Attached Figure Description

[0016] Figure 1 A flowchart illustrating the steps of an animation frame interpolation method based on image data, provided in one embodiment of the present invention; Figure 2 This is a basic flowchart of an animation frame interpolation system based on image data, provided as an embodiment of the present invention. Detailed Implementation

[0017] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0018] Example 1, referring to Figure 1 This paper provides an animation frame interpolation method based on image data, which includes the following steps: Step S1: Collect touch interaction parameters, and when the estimated pixel displacement between adjacent frames is greater than the preset visual smoothness displacement threshold, obtain the two key interface frames before and after the current interaction scene, and map the touch interaction parameters into initial motion optical flow features.

[0019] Step S2: Extract local structural features of two key interface frames before and after by multi-scale depth separable convolutional layers, perform multi-directional selective scanning of the two key interface frames before and after by the first selective state space model, and extract global layout context dependency features by using the spatial location information of the active area of ​​the interactive control as a priori guiding condition. Combine local structural features and global layout context dependency features to obtain interface context content features.

[0020] Step S3: The initial motion optical flow features and interface context content features are concatenated into multimodal concatenation features and input into the joint selective state space model for scanning. Based on the estimated pixel displacement, the correlation weight is calculated, and the scanned features are weighted and fused to obtain joint spatial fusion features. The optical flow residual is predicted based on the joint spatial fusion features to obtain the refined target motion optical flow.

[0021] Step S4: Based on the refined target motion optical flow, spatial distortion is performed on the two key interface frames before and after, and combined with joint spatial fusion features, an intermediate transition frame is synthesized while maintaining linear computational complexity, and output to the display panel for rendering.

[0022] This invention injects the active area of ​​the acquired interactive control as a priori guiding condition into the selective state space model. By dynamically allocating gating weights, it accurately filters redundant features in background noise areas. It also combines the estimated pixel displacement to calculate the correlation weights and deeply fuse the multimodal stitching features. It predicts the optical flow residual to correct the deviation of rapid sliding. It abandons the time-consuming global attention calculation. While maintaining extremely low computational power overhead with linear computational complexity, it generates intermediate transition frames in real time that are free of background ghosting artifacts and are pixel-level aligned with the user's real interactive motion inertia in terms of physical space deformation.

[0023] Step S1 includes the following sub-steps: Step S101: Collect the user's touch trajectory data through the underlying driver interface of the touch screen, extract the direction information and speed information from the touch trajectory data to form touch interaction parameters, and calculate the estimated pixel displacement between adjacent frames based on the product of the speed information and the current refresh cycle of the system.

[0024] Step S102: Dynamically compare the estimated pixel displacement with the preset visual smoothness displacement threshold.

[0025] When the estimated pixel displacement is detected to be greater than the preset visual smoothness displacement threshold, the high frame rate interpolation mode is triggered, and a retrieval instruction is sent to the frame buffer queue of the system rendering engine to extract the current display frame and the next pre-rendered frame that are aligned with the touch interaction parameters on the timestamp, which are used as the two key interface frames before and after.

[0026] Step S103: Obtain the mapping relationship between the physical resolution of the touch screen and the display area. Combine the time interval parameter between two key interface frames to construct a spatiotemporal mapping model from the touch physical coordinate system to the image pixel coordinate system, and determine the active area of ​​the interactive control corresponding to the touch trajectory data in the two key interface frames.

[0027] Using a spatiotemporal mapping model, the direction and velocity information are normalized into a reference displacement vector in the corresponding pixel coordinate system. The reference displacement vector is broadcast and filled into the active area of ​​the interactive control to generate a two-dimensional pixel displacement vector field. The two-dimensional pixel displacement vector field includes the foreground motion state and the background static state. The two-dimensional pixel displacement vector field is used as the initial motion optical flow feature to guide the matching of two key interface frames before and after.

[0028] In one specific implementation, the user's touch trajectory data is collected in real time through the underlying driver interface of the terminal's touch screen (such as InputReader in Android or UITouch in iOS). The touch trajectory data includes at least a two-dimensional physical coordinate sequence of the touch point and the timestamp information corresponding to the touch point. From the touch trajectory data, the instantaneous direction vector and instantaneous velocity scalar are extracted through differential calculation, and the instantaneous direction vector and instantaneous velocity scalar together constitute the touch interaction parameters.

[0029] It is important to note that in step S101, the estimated pixel displacement is calculated based on the product of the speed information and the current system refresh cycle. This converts the instantaneous speed of the user's finger in physical space (in millimeters per second) into the expected displacement distance of the interface controls in pixel space between two adjacent frames (in pixels per frame). This conversion process is accomplished using the current system refresh cycle (i.e., the fixed time interval between two adjacent frames, for example, approximately 16.6 milliseconds for a 60Hz refresh rate), enabling the subsequent threshold judgment mechanism to establish a strict mathematical relationship directly with the underlying image pixel space.

[0030] In practice, the estimated pixel displacement is dynamically compared with a preset visual smoothness displacement threshold. This threshold is based on the physiological principles of visual persistence and the visual resolution limit. When the pixel displacement of an interface element between two adjacent frames exceeds the visual resolution limit, the user will perceive screen stuttering and lag. Considering the screen pixel density of mainstream smart mobile terminals and the user's normal viewing distance, the optimal range for the visual smoothness displacement threshold is set to 1 to 3 pixels. For typical terminal devices with a screen pixel density of approximately 400 PPI, the optimal value is 2 pixels. By using this preset threshold, the system can accurately trigger a high frame rate interpolation mode the instant the estimated pixel displacement exceeds the human eye's visual smoothness tolerance. Simultaneously, it strictly avoids performing invalid pixel interpolation calculations at extremely low motion rates, maximizing the saving of the terminal device's computational power consumption while ensuring absolute visual smoothness for the human eye.

[0031] When the estimated pixel displacement is detected to be greater than the preset visual smoothness displacement threshold, the system triggers a high frame rate interpolation mode. After triggering the high frame rate interpolation mode, the system actively sends a retrieval command to the frame buffer queue of the system rendering engine to extract the current display frame and the next pre-rendered frame that are aligned with the current touch interaction parameters in terms of timestamp, respectively serving as the two key interface frames before and after the current frame. The retrieval command mechanism ensures that the acquired key interface frames are image frames that are synchronized in real time with the user's actual physical operations, avoiding the key frame timing misalignment problem caused by rendering command lag in traditional solutions.

[0032] In practical implementation, the system first acquires the physical resolution parameters of the touch screen and the pixel resolution parameters of the display area, establishing a spatial mapping relationship from physical coordinates to pixel coordinates. Simultaneously, it combines this with the fixed time interval parameter between two consecutive key interface frames to construct a spatiotemporal mapping model from the touch physical coordinate system to the image pixel coordinate system. Based on the landing point position of the collected touch trajectory data on the screen, and combined with the interface layout hierarchy information of the current interaction scenario, the system determines the pixel region corresponding to the interactive control directly affected by the touch event in the two consecutive key interface frames, defining this pixel region as the active region of the interactive control. The active region of the interactive control is typically a connected component in a two-dimensional image, corresponding to the set of interactive interface elements where physical interaction actually occurs. Through the spatiotemporal mapping model, the system normalizes the direction and velocity information into a reference displacement vector aligned with the device's pixel resolution. This reference displacement vector accurately represents the ideal motion trajectory desired by the user's finger.

[0033] The system copies the reference displacement vector and assigns it to each individual pixel within the active area of ​​the interactive control. The physical basis of the broadcast fill operation is that when the user drags or slides the interactive control, all pixels contained in the interactive control should move as a whole in a quasi-rigid motion. Therefore, all pixels within the active area of ​​the interactive control share the same reference displacement vector.

[0034] In a specific embodiment, the broadcast padding operation can be formally expressed as: Let This is the set of pixel coordinates of the active area of ​​the interactive control. This is the reference displacement vector calculated using the spatiotemporal mapping model. The output is the initial motion optical flow characteristic. The mathematical expression is: ; in, This represents a stationary vector with zero displacement. and These represent the horizontal and vertical pixel coordinates in the image pixel coordinate system, respectively.

[0035] After the broadcast fill operation, the system generates a complete two-dimensional pixel displacement vector field. In this field, each pixel within the active area of ​​the interactive control is assigned a non-zero displacement vector, representing the foreground motion state, while pixels outside the active area are assigned a zero vector, representing the background static state. The initial motion optical flow features not only possess clear physical semantics directly driven by user physical touch, but also naturally distinguish between the moving foreground and the static background, providing absolutely accurate spatial prior information for subsequent deep learning networks. This significantly improves the noise resistance and computational efficiency of subsequent feature extraction.

[0036] Step S2 includes the following sub-steps: Step S201: Input the two key interface frames before and after into the context learning module. In the local feature branch of the context learning module, through parallel deep separable convolutional layers with different preset convolutional kernel scales, extract the local spatial features of the receptive field of the two key interface frames before and after, respectively, to obtain multi-scale feature maps. Then, stitch and aggregate the multi-scale feature maps along the channel dimension to obtain local structural features.

[0037] It should be noted that the context learning module proposed in this invention is a parallel dual-branch multimodal feature extraction network structure, not a traditional single convolutional layer or single attention layer. Specifically, the input of the context learning module receives serialized or tensorized key interface frame data, and is divided into two parallel feature extraction branches within the module: The local feature branch incorporates at least three depthwise separable convolutional layers of different scales, with preferred kernel sizes of 3×3, 5×5, and 7×7. Different kernel sizes provide differentiated receptive fields: a 3×3 receptive field captures subtle UI control edges or fine text outlines, while 5×5 and 7×7 receptive fields capture larger card views or sliding deformations of large color blocks. These output feature maps at different scales are then concatenated along the channel dimension to aggregate local structural features. The introduction of depthwise separable convolution ensures that parameter explosion does not occur during multi-scale extraction, significantly reducing the memory overhead of mobile devices.

[0038] The global feature branch and the local feature branch are completely parallel. The global feature branch uses a selective state space model to perform multi-directional scanning and establish long-distance spatial topological dependencies.

[0039] The output of the context learning module has a feature alignment and fusion layer, such as a 1×1 convolutional layer, which is used to fuse the outputs of the global feature branch and the local feature branch. Through the parallel architecture, the context learning module enables the underlying network to simultaneously and efficiently understand what a button in the interactive interface looks like and the relative position of a button in the entire screen layout.

[0040] In step S202, the two key interface frames are synchronously input into the global feature branch that runs parallel to the local feature branch. Through image block embedding operation, the two key interface frames are converted into four corresponding one-dimensional serialized features according to the image block traversal paths of horizontal forward scanning, horizontal reverse scanning, vertical forward scanning, and vertical reverse scanning, respectively. The four corresponding one-dimensional serialized features are then input into the first selective state space model for multi-directional selective scanning.

[0041] It should be noted that converting the two key interface frames into four corresponding one-dimensional serialized features is specifically achieved through image patch embedding technology, using a size of... The 2D key interface frame is divided into multiple non-overlapping local image blocks, and the size of each image block is set to [size missing]. By using a linear projection layer to map two-dimensional image patches into one-dimensional feature vectors, a length of [missing value] is obtained. A one-dimensional sequence, wherein, In order to enable the first selective state space model to fully perceive the two-dimensional spatial topology of the image, the present invention flattens the generated image block matrix along the top left to bottom right (horizontal forward direction), bottom right to top left (horizontal reverse direction), top right to bottom left (vertical forward direction), and bottom left to top right (vertical reverse direction), respectively, to generate four corresponding one-dimensional serialization features with different spatial traversal paths.

[0042] In step S203, during the multi-directional selective scanning process, the spatial location information of the active area of ​​the interactive control is obtained. Based on the input dependency dynamic parameterization mechanism in the first selective state space model where parameters are dynamically generated with the input data, the spatial location information is used as a priori guiding condition to dynamically allocate information retention weights and forgetting gate weights to each feature element in the one-dimensional serialized feature during the state transition process.

[0043] By utilizing forgetting gate weights to suppress feature responses located outside the active area of ​​interactive controls in one-dimensional serialized features, redundant features in background noise areas are filtered out in two key interface frames.

[0044] By using information retention weights, global context modeling is performed only on the active areas of interactive controls to extract global layout context dependency features.

[0045] It should be noted that, in order to maintain the linear computational complexity of the state-space model... In step S203 of this invention, instead of employing a computationally expensive cross-attention mechanism to fuse touch priors, a gated bias modulation mechanism is designed, specifically including: Convert the active area of ​​the 2D interactive control to a size of [size missing]. A binary space mask matrix is ​​used, with active regions masked as 1 and inactive regions as 0. To align with the dimension of the feature tensor after image patch embedding, a kernel size and stride of 1 are used. The max pooling operation downsamples the initial binary mask matrix space to The resolution is used to obtain the downsampling mask matrix.

[0046] Following the same four traversal paths as image feature serialization (horizontal forward and reverse, and vertical forward and reverse), the downsampling mask matrix is ​​synchronously flattened into four lengths. A one-dimensional mask sequence, wherein, In the first selective state-space model, when generating the state transition matrix and projection matrix, a one-dimensional mask sequence is used as a bias multiplier in the forget gate calculation formula within the model. Through this hardware-friendly element-wise multiplication operation, the model can be forced to shut down feature responses in inactive regions while retaining adaptive learning capabilities for local gradient information and motion saliency in active regions. The input-dependent dynamic parameterization mechanism employed in the first selective state-space model of this invention is the core underlying mechanism for selectively filtering features. In traditional linear time-invariant state-space models, the parameter matrix of the continuous-time state equation is static. However, in the input-dependent dynamic parameterization mechanism of this invention, the discretization step size... and state projection matrix and These are no longer static parameters, but rather derived from the one-dimensional sequential input features at the current time step. Generated in real time through a linear mapping network; specifically, for each input element... Dynamically generate state projection matrix State projection matrix and step size .

[0047] In this embodiment, when introducing prior guidance on the active area of ​​the interactive control, the acquired spatial location information is converted into a binary spatial mask matrix. And after being flattened, it acts as a modulation coefficient for the step size. During the generation process, for example Due to step size The discretized system state transition matrix is ​​determined. The degree of decay, when one of the pixels is located in the non-active area ( When the step size is dynamically generated, Forced to be set to 0, causing the feature input at the corresponding time moment to be completely ignored and forgotten by the mechanism, while the input located in the active interaction zone ( When the first selective state-space model generates the retained weights according to the conventional input-dependent parameterization mechanism, it achieves precise suppression of the feature response of the background noise region.

[0048] Step S204: The local structural features and the global layout context-dependent features are concatenated in the channel dimension, and the concatenated features are compressed in the channel and fused with cross-channel information through a preset 1×1 convolutional layer to obtain interface context content features that combine the local edge texture details of key interface frames with the global spatial position topology.

[0049] This invention addresses the pain point of feature extraction being easily affected by background interference. It proposes to inject the spatial location information of the active area of ​​the interactive control as a priori guiding condition into the first selective state space model, use forgetting gating weights to accurately filter redundant features in background noise areas, extract global layout context dependency features, and fuse them with local structural features extracted by depthwise separable convolutional layers to obtain interface context content features. This completely eliminates background interference and perfectly balances local details and global dependencies.

[0050] Step S3 includes the following sub-steps: Step S301: Map the initial motion optical flow features and interface context content features to a unified feature space and flatten them into one-dimensional serialized features. Then, concatenate the flattened initial motion optical flow features and interface context content features along the channel dimension to obtain multimodal concatenated features. At the same time, perform sequence flipping on the multimodal concatenated features to generate corresponding reverse multimodal concatenated features.

[0051] Step S302: Input the multimodal splicing features and the corresponding reverse multimodal splicing features into the joint selective state space model, and use the joint selective state space model to perform a forward scan of the multimodal splicing features and a forward scan of the reverse multimodal splicing features.

[0052] The results of the forward scan are added to the results of the reverse scan after re-flipping and alignment, and the updated motion feature branch and the updated content feature branch are separated along the channel dimension.

[0053] Step S303: Perform global average pooling on the updated motion feature branch and content feature branch respectively through the channel attention mechanism. Concatenate the pooling results of the two branches along the channel dimension and input them into the shared fully connected layer to generate channel feature vectors that represent the correlation between the estimated pixel displacement and the deformation of the interface content. Use the activation function to map the channel feature vectors into correlation weights.

[0054] The relevance weights are multiplied element-wise with the corresponding motion feature branches and content feature branches. The weighted motion feature branches and content feature branches are then reassembled and projected through a linear layer to output the joint spatial fusion features.

[0055] Step S304: Input the joint spatial fusion features into the preset residual prediction network layer to predict the optical flow residual used to correct the dynamic displacement deviation.

[0056] The optical flow residual and the initial motion optical flow characteristics are fused element-wise by addition and feedback compensation to eliminate the optical flow estimation error caused by the high-speed interactive motion and output a refined target motion optical flow.

[0057] It should be noted that the residual prediction network layer is a lightweight network consisting of three convolutional layers. Its input is the joint spatial fusion feature, and its output is an optical flow residual tensor with the same shape as the initial motion optical flow feature. The physical rationale for setting up the residual prediction network layer is that the generated initial motion optical flow feature is based solely on the spatial mapping of touch parameters, without considering the deformation constraints of the interface content itself, such as the continuity of image texture edges and occlusion relationships. Relying solely on the initial optical flow for spatial distortion can result in edge breaks or texture misalignment. By predicting residuals from the joint spatial fusion feature, the network can learn the optical flow correction amount under content constraints, thereby compensating for the physical limitations of purely touch-driven optical flow.

[0058] In one specific embodiment, the residual prediction network layer employs three 3×3 convolutional layers with 256, 128, and 2 channels respectively. Each convolutional layer is followed by a normalization layer and a ReLU activation function. The final layer outputs 2 channels, corresponding to the optical flow residuals in the x and y directions. The optical flow residuals are fused with the initial motion optical flow features through element-wise addition to achieve feedback compensation. The compensated refined target motion optical flow simultaneously includes macroscopic motion information driven by touch and microscopic correction information under content constraints, which can eliminate optical flow estimation errors caused by rapid interactive motion.

[0059] Existing frame interpolation methods suffer from two problems: first, motion estimation and content feature processing are separated, with motion optical flow relying solely on pixel brightness gradients, leading to breaks at the edges of interface controls; second, bidirectional temporal modeling has high computational complexity, making it difficult to respond to high-speed touch in real time. This invention innovatively concatenates the initial optical flow driven by touch with interface content features into multimodal features. It achieves linearly complex bidirectional temporal modeling through a joint selective state-space model, and utilizes the channel attention mechanism of a shared fully connected layer to force learning the cross-modal correlation between estimated pixel displacement and interface content deformation. Then, a residual prediction network layer predicts optical flow residuals from the fused features to compensate for the initial optical flow. This allows the refined target motion optical flow to simultaneously possess touch physical accuracy and content geometric constraints, generating intermediate transition frames with complete edges and no artifacts in high frame rate interactive scenarios. Furthermore, the computational complexity is linearly related to the sequence length, meeting the requirements of real-time rendering.

[0060] Step S4 includes the following sub-steps: Step S401: Determine the time step parameter corresponding to the intermediate transition frame to be generated in the time series. Based on the linear interactive motion assumption, use the time step parameter to perform proportional time scale scaling calculation on the refined target motion optical flow to generate forward intermediate optical flow and backward intermediate optical flow from the two key interface frames to the time node where the intermediate transition frame is located. The forward intermediate optical flow and backward intermediate optical flow together constitute the target intermediate motion vector field.

[0061] It should be noted that this invention introduces the linear interactive motion assumption because in modern touch-enabled smart devices, the native rendering frame rate is typically 60Hz or higher, and the time interval between two adjacent key interface frames is usually less than 16.6 milliseconds. Within this extremely short time window, the acceleration or deceleration effect of the user's finger swipe is extremely small, and the physical displacement of control elements in the interface can be approximately equivalent to uniform linear motion. Based on this assumption, when it is necessary to generate the previous key frame... and the next keyframe Intermediate transition frames (wherein the time step parameter) When this is done, the optical flow of the refined target motion can be observed. (indicating from) point to The complete optical flow is calculated using linear scaling over time to obtain the intermediate optical flow in two directions, including the forward intermediate optical flow. and backward intermediate optical flow .

[0062] Forward intermediate optical flow is used to transfer the previous keyframe The pixel towards the center moment The mathematical expression for the forward intermediate optical flow after performing forward alignment mapping is: ; Backward intermediate optical flow is used to transfer the next keyframe The pixel towards the center moment After performing reverse alignment mapping, the mathematical expression for the backward intermediate optical flow is: ; Therefore, scaling the optical flow linearly based on the time step parameter not only conforms to the objective physical laws of UI interaction, but also obtains a high-precision intermediate motion vector field with extremely low computational cost.

[0063] Step S402: Based on the target intermediate motion vector field, perform forward and reverse spatial distortion operations on the pixels of the two key interface frames before and after, respectively, and align and map the pixels of the two key interface frames before and after to the time node where the intermediate transition frame is located, thereby generating the first candidate distortion frame and the second candidate distortion frame.

[0064] Step S403: Input the joint spatial fusion features into a preset mask mapping convolutional layer for channel compression and activation processing, and decode the output to create a dynamic mask weight matrix for pixel-level fusion.

[0065] The first candidate distorted frame and the second candidate distorted frame are weighted and combined using a dynamic mask weight matrix to obtain the initial synthesized interface frame.

[0066] It should be noted that, in order to extract weights for image fusion from the high-dimensional joint spatial fusion features, this invention employs a lightweight mask mapping convolutional layer. Specifically, the system uses a 1×1 convolutional layer to compress the multi-channel fusion features into a single-channel feature map, and then normalizes the pixel values ​​of the single-channel feature map to a normal value using a Sigmoid activation function. Within the specified range, the dynamic mask weight matrix is ​​accurately decoded. Each weight value in the dynamic mask weight matrix represents the confidence level of the first candidate distorted frame compared to the second candidate distorted frame at the current pixel coordinates, thus perfectly solving the problem of interface element occlusion and ghosting during fast scrolling.

[0067] Step S404: The initial synthesized interface frame and the joint spatial fusion features are spliced ​​together in the channel dimension and then input into the preset frame synthesis network.

[0068] Based on the implicit memory mechanism of the selective state space model configured inside the frame synthesis network for the temporal signal, the explicit calculation of the global attention feature matrix is ​​replaced by the latent state mapping of a one-dimensional function. While maintaining linear computational complexity, the joint spatial fusion feature is used to reconstruct local details and repair interaction artifacts of the initial synthesized interface frame, generating an intermediate transition frame that conforms to the current physical inertia of the interaction. The intermediate transition frame is then synchronously output to the rendering engine of the display panel for image rendering.

[0069] It should be noted that the frame synthesis network is the final decoding network used to output the final image. Structurally, the frame synthesis network consists of a pre-concatenation feature concatenation layer, a mid-concatenation selective state-space model, and a post-concatenation pixel-level reconstruction layer (such as a deconvolution or pixelShuffle layer) connected in series. The frame synthesis network receives a coarse initial synthesis interface frame and high-dimensional joint space fusion features as joint inputs, and utilizes the one-dimensional state equation of the selective state-space model. conduct This reduces the complexity of feature updates and eliminates the time-consuming overhead of Transformer. Global attention matrix calculation ensures real-time performance and low power consumption when mobile devices perform high frame rate interpolation.

[0070] Example 2, refer to Figure 2 This paper provides an animation frame interpolation system based on image data, including a touch guidance module, a modeling module, a fusion correction module, and an intermediate compositing module.

[0071] Touch guidance is used to collect touch interaction parameters. When the estimated pixel displacement is greater than the visual smoothness displacement threshold, key interface frames are obtained, the initial motion optical flow features are mapped, and the active area of ​​the interactive control is extracted.

[0072] Modeling is used to extract local structural features of key interface frames through multi-scale depth separable convolutional layers. Global layout context-dependent features are extracted by using a first selective state space model guided by the active area of ​​interactive controls. Local structural features and global layout context-dependent features are fused to generate interface context content features.

[0073] The fusion correction is used to input the multimodal splicing features, which are the initial motion optical flow features and the interface context content features, into the joint selective state space model. Based on the estimated pixel displacement, the correlation weights are calculated to fuse the joint spatial fusion features, and the predicted optical flow residuals are used to generate refined target motion optical flow.

[0074] Intermediate synthesis is used to synthesize intermediate transition frames with linear computational complexity by combining joint spatial fusion features based on the refined target motion optical flow distortion key interface frames.

[0075] This invention addresses the pain points of pure visual interpolation deviating from physical intent and the extremely high computational cost of large models. It extracts active areas of interactive controls through a touch guidance module, and the modeling module uses a first selective state space model to extract global layout context-dependent features guided by these active areas. A fusion correction module calculates correlation weights based on estimated pixel displacements and generates refined target motion optical flow from predicted optical flow residuals. An intermediate synthesis module combines joint spatial fusion features to synthesize intermediate transition frames while maintaining linear computational complexity. This solves the problems of interface distortion tearing and motion artifacts caused by blind extrapolation in pure visual algorithms. While maintaining extremely low computational cost with linear computational complexity, it achieves pixel-level precise alignment between physical operation intent and terminal visual rendering, rendering intermediate transition frames in real-time that are free from background noise and ghosting interference and perfectly match the user's instantaneous sliding inertia. This greatly improves the visual fidelity and system smoothness of human-computer interaction on high frame rate mobile screens.

[0076] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the protection scope of the present invention.

Claims

1. An animation frame interpolation method based on image data, characterized in that, Includes the following steps: Step S1: Collect touch interaction parameters, obtain two key interface frames before and after the current interaction scene, and map the touch interaction parameters to the initial motion optical flow features. Step S2: Extract the local structural features of the two key interface frames before and after, perform multi-directional selective scanning on the two key interface frames before and after, and extract the global layout context dependency features with the spatial position information of the active area of ​​the interactive control as the prior guiding condition. Combine the local structural features and the global layout context dependency features to obtain the interface context content features. Step S3: The initial motion optical flow features and interface context content features are concatenated into multimodal concatenated features and input into the joint selective state space model for scanning. The correlation weights are calculated, and the scanned features are weighted and fused to obtain joint space fusion features. The optical flow residuals are predicted based on the joint space fusion features to obtain the refined target motion optical flow. Step S4: Based on the refined target motion optical flow, spatial distortion is performed on the two key interface frames before and after, and combined with joint spatial fusion features, an intermediate transition frame is synthesized while maintaining linear computational complexity, and output to the display panel for rendering.

2. The animation frame interpolation method based on image data as described in claim 1, characterized in that, Step S1 includes the following sub-steps: Step S101: Collect the user's touch trajectory data through the underlying driver interface of the touch screen, extract the direction information and speed information from the touch trajectory data to form touch interaction parameters, and calculate the estimated pixel displacement between adjacent frames based on the product of the speed information and the current refresh cycle of the system. Step S102: Dynamically compare the estimated pixel displacement with the preset visual smoothness displacement threshold. When the estimated pixel displacement is detected to be greater than the preset visual smoothness displacement threshold, a high frame rate interpolation mode is triggered, and a retrieval instruction is sent to the frame buffer queue of the system rendering engine to extract the current display frame and the next pre-rendered frame that are aligned with the touch interaction parameters on the timestamp, which are used as the two key interface frames before and after.

3. The animation frame interpolation method based on image data as described in claim 2, characterized in that, Step S1 further includes the following sub-steps: Step S103: Obtain the mapping relationship between the physical resolution of the touch screen and the display area. Combine the time interval parameter between the two key interface frames, construct a spatiotemporal mapping model from the touch physical coordinate system to the image pixel coordinate system, and determine the active area of ​​the interactive control corresponding to the touch trajectory data in the two key interface frames. Using a spatiotemporal mapping model, the direction and velocity information are normalized into a reference displacement vector in the corresponding pixel coordinate system. The reference displacement vector is broadcast and filled into the active area of ​​the interactive control to generate a two-dimensional pixel displacement vector field. The two-dimensional pixel displacement vector field includes the foreground motion state and the background static state. The two-dimensional pixel displacement vector field is used as the initial motion optical flow feature to guide the matching of two key interface frames before and after.

4. The animation frame interpolation method based on image data as described in claim 3, characterized in that, Step S2 includes the following sub-steps: Step S201: Input the two key interface frames before and after into the context learning module. In the local feature branch of the context learning module, through parallel deep separable convolutional layers with different preset convolutional kernel scales, extract local spatial features of the multi-scale receptive field of the two key interface frames before and after, respectively, to obtain multi-scale feature maps. Then, stitch and aggregate the multi-scale feature maps along the channel dimension to obtain local structural features. In step S202, the two key interface frames are synchronously input into the global feature branch that runs parallel to the local feature branch. Through image block embedding operation, the two key interface frames are converted into four corresponding one-dimensional serialized features according to the image block traversal paths of horizontal forward scanning, horizontal reverse scanning, vertical forward scanning, and vertical reverse scanning, respectively. The four corresponding one-dimensional serialized features are then input into the first selective state space model for multi-directional selective scanning.

5. The animation frame interpolation method based on image data as described in claim 4, characterized in that, Step S2 further includes the following sub-steps: Step S203: During the multi-directional selective scanning process, the spatial location information of the active area of ​​the interactive control is obtained, and according to the input dependency dynamic parameterization mechanism in the first selective state space model where parameters are dynamically generated with input data, the spatial location information is used as a priori guiding condition to dynamically allocate information retention weights and forgetting gate weights in the state transition process to each feature element in the one-dimensional serialized feature. The forgetting gate weights are used to suppress feature responses located outside the active area of ​​interactive controls in one-dimensional serialization features, and to filter redundant features in the background noise area in two key interface frames before and after. Using the aforementioned information, weights are retained to perform global context modeling only on the active areas of interactive controls, and global layout context dependency features are extracted. Step S204: The local structural features and the global layout context-dependent features are concatenated in the channel dimension, and the concatenated features are compressed in the channel and fused with cross-channel information through a preset 1×1 convolutional layer to obtain the interface context content features.

6. The animation frame interpolation method based on image data as described in claim 5, characterized in that, Step S3 includes the following sub-steps: Step S301: Map the initial motion optical flow features and interface context content features to a unified feature space and flatten them into one-dimensional serialized features. Then, concatenate the flattened initial motion optical flow features and interface context content features along the channel dimension to obtain multimodal concatenated features. At the same time, perform sequence flipping on the multimodal concatenated features to generate corresponding reverse multimodal concatenated features. Step S302: Input the multimodal splicing features and the corresponding reverse multimodal splicing features into the joint selective state space model, and use the joint selective state space model to perform a forward scan of the multimodal splicing features and the reverse multimodal splicing features. The results of the forward scan are added to the results of the reverse scan after re-flipping and alignment, and the updated motion feature branch and the updated content feature branch are separated along the channel dimension.

7. The animation frame interpolation method based on image data as described in claim 6, characterized in that, Step S3 further includes the following sub-steps: Step S303: Perform global average pooling on the updated motion feature branch and content feature branch respectively through the channel attention mechanism to generate channel feature vectors that represent the correlation between the estimated pixel displacement and the deformation of the interface content. Then, use the activation function to map the channel feature vectors into correlation weights. The correlation weights are multiplied element-wise with the corresponding motion feature branches and content feature branches. The weighted motion feature branches and content feature branches are then reassembled and projected through a linear layer to output the joint spatial fusion features. Step S304: Input the joint spatial fusion features into a preset residual prediction network layer to predict the optical flow residual used to correct the dynamic displacement deviation; The optical flow residual and the initial motion optical flow features are fused element-wise to output a refined target motion optical flow.

8. The animation frame interpolation method based on image data as described in claim 7, characterized in that, Step S4 includes the following sub-steps: Step S401: Determine the time step parameter corresponding to the intermediate transition frame to be generated in the time series. Based on the linear interactive motion assumption, use the time step parameter to perform proportional time scale scaling calculation on the refined target motion optical flow to generate forward intermediate optical flow and backward intermediate optical flow from the two key interface frames to the time node where the intermediate transition frame is located. The forward intermediate optical flow and backward intermediate optical flow together constitute the target intermediate motion vector field. Step S402: Based on the target intermediate motion vector field, perform forward and reverse spatial distortion operations on the pixels of the two key interface frames before and after, respectively, and align and map the pixels of the two key interface frames before and after to the time node where the intermediate transition frame is located, thereby generating the first candidate distortion frame and the second candidate distortion frame.

9. The animation frame interpolation method based on image data as described in claim 8, characterized in that, Step S4 further includes the following sub-steps: Step S403: Input the joint spatial fusion features into a preset mask mapping convolutional layer for channel compression and activation processing, and decode and output a dynamic mask weight matrix for pixel-level fusion. The first candidate warped frame and the second candidate warped frame are weighted and combined using the dynamic mask weight matrix to obtain the initial synthesized interface frame. Step S404: The initial synthesized interface frame and the joint spatial fusion feature are spliced ​​together in the channel dimension and then input into the preset frame synthesis network; Based on the implicit memory mechanism of the selective state space model configured inside the frame synthesis network for the temporal signal, the explicit calculation of the global attention feature matrix is ​​replaced by the latent state mapping of a one-dimensional function. While maintaining linear computational complexity, the joint spatial fusion feature is used to reconstruct local details and repair interaction artifacts of the initial synthesized interface frame, generating an intermediate transition frame that conforms to the current physical inertia of the interaction. The intermediate transition frame is then synchronously output to the rendering engine of the display panel for image rendering.

10. An animation frame interpolation system based on image data, applied in an animation frame interpolation method based on image data as described in any one of claims 1-9, characterized in that, It includes a touch guidance module, a modeling module, a fusion correction module, and an intermediate compositing module; The touch guidance is used to collect touch interaction parameters, obtain key interface frames when the estimated pixel displacement is greater than the visual smoothness displacement threshold, map the initial motion optical flow features and extract the active area of ​​the interactive control. The modeling is used to extract local structural features of key interface frames through multi-scale depth separable convolutional layers, extract global layout context-dependent features by using a first selective state space model guided by the active area of ​​interactive controls, and fuse local structural features and global layout context-dependent features to generate interface context content features. The fusion correction is used to input the multimodal splicing feature of the initial motion optical flow feature and the interface context content feature into the joint selective state space model, calculate the correlation weight based on the estimated pixel displacement to fuse the joint space fusion feature, and generate the refined target motion optical flow from the predicted optical flow residual. The intermediate synthesis is used to synthesize intermediate transition frames with linear computational complexity based on the refined target motion optical flow distortion key interface frames and combined with joint spatial fusion features.

Images