A video motion magnification method, system, device and storage medium

Abstract

The application discloses a video motion amplification method, system, device and storage medium, and relates to the technical field of computer vision and image processing, and comprises the following steps: converting a video sequence to be amplified into a continuous frame sequence; inputting a current frame and a reference frame in the frame sequence into a video motion amplification model to obtain a motion amplified image; wherein the video motion amplification model comprises a shallow feature extraction module and a motion feature extraction module, the motion feature extraction module comprises a down-sampling module, a deep feature extraction module, a controller, a decoder and an up-sampling module. By adopting a state space model as the core architecture of the deep feature extraction module and the decoder, the model can realize long-distance dependence modeling on an image feature sequence with linear calculation complexity, and the processing efficiency of video motion amplification is significantly improved.

Description

Technical Field

[0001] This invention relates to the field of computer vision and image processing technology, and in particular to a video motion magnification method, system, device and storage medium. Background Technology

[0002] Video motion magnification technology is often figuratively called a "motion microscope," aiming to reveal minute dynamic changes in video sequences that are imperceptible to the naked eye, such as subtle physiological signs in the human body (e.g., breathing, pulse) and minute vibrations in building structures. Since the amplitude of such minute movements is typically at the sub-pixel level, and their intensity is often on the same order of magnitude as imaging noise, the core technical challenge in this field lies in how to effectively amplify motion signals while avoiding the simultaneous amplification of background noise and maintaining the integrity of object edges and texture structures in the image.

[0003] Early research typically relied on an Euler perspective, employing spatial multi-scale decomposition techniques such as Laplacian pyramids or directional pyramids to break down video frames into different spatial frequency bands. Motion signals within specific frequency ranges were then extracted and amplified using pre-designed temporal filters to achieve motion amplification for specific wavebands. With the development of deep learning, researchers began using convolutional neural networks to model motion amplification as a data-driven task. These methods no longer depend on manually designed filters but instead build learning networks that automatically learn decoupled representations of motion and appearance features from the data. To adapt to motion features at different scales, some schemes further introduce multi-scale network architectures or axis-based processing strategies, aiming to extract and recover motion details within local receptive fields through convolutional operations.

[0004] To overcome the limitations of convolutional networks in modeling long-range dependencies, recent research has begun to employ architectures based on self-attention mechanisms. These methods utilize attention mechanisms to establish global associations between image features, attempting to integrate global contextual information during processing. By capturing long-range dependencies between features, these methods aim to identify and amplify motion patterns with overall geometric consistency, addressing the problem that local operations struggle to maintain the overall structure of objects.

[0005] While existing video motion magnification techniques have achieved some degree of magnification of minute movements, they still have significant technical limitations when dealing with complex scenes or smaller motion scenarios. First, methods based on convolutional neural networks are limited by the inherent local receptive field of convolution operations, lacking the ability to model long-distance dependencies. When processing motions with overall geometric consistency, such as object rotation or global translation, it is difficult to maintain global geometric stability, often resulting in inconsistent magnification effects across different regions of the image. This leads to local distortion and spatial artifacts, causing the magnified video to lose its physical realism.

[0006] The self-attention mechanism-based approach, introduced to address the long-distance dependency problem, provides global perception capabilities but faces a severe computational efficiency bottleneck. Its computational complexity increases quadratically with image resolution, resulting in extremely high consumption of video memory resources and slow inference speed, making it difficult to meet the real-time requirements of high-resolution video processing. Summary of the Invention

[0007] In view of the deficiencies of the existing technology, the present invention provides a video motion magnification method, system, device and storage medium, which solves the existing problems.

[0008] The present invention adopts the following technical solution: In a first aspect, the present invention provides a video motion magnification method, comprising the following steps: Convert the video sequence to be enlarged into a continuous frame sequence; The current frame and reference frame in the frame sequence are input into the video motion magnification model to obtain a motion magnified image; wherein, the video motion magnification model includes a shallow feature extraction module and a motion feature extraction module, and the motion feature extraction module includes a downsampling module, a deep feature extraction module, a manipulator, a decoder and an upsampling module; The shallow feature extraction module extracts features from the reference frame and the current frame to obtain shallow features of the reference frame and the current frame. The downsampling module downsamples the shallow features of the reference frame and the current frame to obtain feature maps of the reference frame and the current frame. The deep feature extraction module encodes the feature maps of the reference frame and the current frame using a state-space model to obtain features of the reference frame and the current frame. The manipulator subtracts and amplifies the features of the reference frame and the current frame to obtain a feature sequence containing amplified motion information. The decoder decodes the feature sequence using a state-space model to obtain decoded features. The upsampling module upsamples the decoded features to obtain motion magnification features. A motion magnification image is generated based on the motion magnification features.

[0009] Preferably, before the deep feature extraction module encodes the reference frame feature map and the current frame feature map using the state space model, it performs serialization and position encoding on the reference frame feature map and the current frame feature map to obtain serialized reference frame features and serialized current frame features. The deep feature extraction module includes a state space model, a normalization layer, and a fully connected layer. The deep feature extraction module encodes the serialized reference frame features and the serialized current frame features to obtain intermediate features. The normalization layer normalizes the intermediate features, and the fully connected layer maps the normalized intermediate features to obtain the reference frame features and the current frame features.

[0010] Preferably, the video motion magnification model further includes a static feature extraction module, a stitching module, and a fusion module; The static feature extraction module generates a static mask based on the difference between the shallow features of the reference frame and the shallow features of the current frame. The shallow features of the reference frame are then weighted using the static mask to generate static baseline features. The stitching module stitches together motion amplification features and static baseline features to obtain a fused feature tensor; The fusion module reconstructs the fusion feature tensor to obtain a motion magnified image.

[0011] Preferably, the static feature extraction module includes a subtraction module, a truncation module, a nonlinear mapping module, a static masking module, a weighting module, a correction module, and a generation module; The difference module subtracts the shallow features of the reference frame and the shallow features of the current frame to obtain the difference feature; The truncation module truncates the difference features to obtain the differential features; The nonlinear mapping module performs feature extraction and dimensionality compression on the differential features to obtain the mapped differential features. The static masking module performs convolution on the mapped difference features to obtain a static mask; the static mask value represents the probability that each pixel belongs to the static background. The weighting module performs element-wise multiplication of the static mask and the shallow features of the reference frame to obtain the static correction amount; The generation module performs in-depth feature refinement on the static correction amount to obtain static baseline features.

[0012] Preferably, the truncation module performs truncation processing on the difference features, as shown below: ; In the formula, Value truncation function, As a distinguishing feature, As a reference frame shallow features, These are the shallow features of the current frame, where min is the minimum value and max is the maximum value. The static mask is shown below: ; In the formula, For the Sigmoid function, For bias terms, The difference features after mapping For static masking, Conv 1×1 It is a 1×1 convolutional layer.

[0013] Preferably, the manipulator includes a difference module, a convolution module, an amplification module, and a transformation module; The difference module subtracts the features of the reference frame from the features of the current frame to obtain motion difference features; The convolutional module extracts motion features from the motion difference features to obtain motion features; The amplification module multiplies the motion feature by the amplification factor to obtain the amplified motion feature; The transformation module performs a nonlinear transformation on the amplified motion features to obtain a feature sequence containing amplified motion information.

[0014] Preferably, before inputting the current frame and reference frame in the frame sequence into the video motion magnification model, reflection filling is performed on the reference frame and the current frame according to a preset downsampling rate so that their length and width pixel values ​​can be divided by the downsampling rate.

[0015] In a second aspect, the present invention provides a video motion magnification system, comprising: The conversion module is used to convert the video sequence to be enlarged into a continuous frame sequence; The input module is used to input the current frame and the reference frame in the frame sequence into the video motion magnification model to obtain a motion magnification image; wherein, the video motion magnification model includes a shallow feature extraction module and a motion feature extraction module, and the motion feature extraction module includes a downsampling module, a deep feature extraction module, a manipulator, a decoder and an upsampling module; The processing module is used to extract features from the reference frame and the current frame through the shallow feature extraction module to obtain shallow features of the reference frame and the current frame; the downsampling module downsamples the shallow features of the reference frame and the current frame to obtain feature maps of the reference frame and the current frame; the depth feature extraction module encodes the feature maps of the reference frame and the current frame through a state space model to obtain features of the reference frame and the current frame; the manipulator subtracts and amplifies the features of the reference frame and the current frame to obtain a feature sequence containing amplified motion information; the decoder decodes the feature sequence through a state space model to obtain decoded features; the upsampling module upsamples the decoded features to obtain motion magnification features; and a motion magnification image is generated based on the motion magnification features.

[0016] Thirdly, the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the above-described video motion magnification method.

[0017] Fourthly, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described video motion magnification method.

[0018] Compared with the prior art, the above-mentioned at least one technical solution adopted by the present invention can achieve the following beneficial effects: This invention employs a state-space model as the core architecture of the deep feature extraction module and decoder, enabling the model to model long-distance dependencies of image feature sequences with linear computational complexity. While obtaining the global receptive field, it effectively overcomes the technical problems of high memory consumption and slow inference speed caused by the quadratic increase in computational complexity with image resolution in methods based on self-attention mechanisms, thus significantly improving the processing efficiency of video motion magnification. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating the specific execution of a video motion magnification method according to the present invention. Figure 2 This is a schematic diagram of the overall architecture of the video motion magnification model of the present invention; Figure 3 This is a schematic diagram of the structure of the visual state space core unit used inside the depth feature extractor and state space perception module of the present invention. Figure 4 This is a detailed structural diagram of the static feature refiner of the present invention; Figure 5 This is a schematic diagram of the manipulator of the present invention for performing feature differentiation and motion enhancement; Figure 6 This is a schematic diagram comparing the indicators of the method of the present invention with those of existing methods. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] Furthermore, standard self-attention mechanisms tend to establish correlations between all input features, lacking the ability to selectively filter signals. When dealing with tasks involving magnifying minute motions, it is often difficult to effectively distinguish between spatially coherent real motion and randomly distributed high-frequency imaging noise. This results in the background noise being indiscriminately amplified while the motion signal is magnified, severely reducing the signal-to-noise ratio of the image.

[0023] To address the problems of low computational efficiency, poor preservation of global geometric structure, and difficulty in suppressing background noise in the existing technologies, this invention aims to provide a video motion magnification method based on a state-space model. The main technical problem this invention addresses is: how to achieve efficient capture and magnification of the global geometric structure of minute moving objects in a video while maintaining linear computational complexity, while intelligently distinguishing motion signals from background noise, thereby generating high-fidelity, artifact-free, and structurally intact magnified videos.

[0024] The process of the video motion amplification method based on the state-space model is as follows: Figure 1 As shown, this method processes video frames using a dual-stream architecture. To more clearly illustrate the feature transformation process in the network, the number of channels in the input video frame is defined as 3 (RGB), and the preset feature channel dimension is [value missing]. (This embodiment takes) The specific execution steps are as follows:

[0025] Step 1: Video Sequence Acquisition and Frame Conversion. Acquire the video sequence to be amplified (e.g., vibration video of an industrial rotating rotor) and convert it into a continuous frame sequence. .

[0026] Step 2: Reference Frame and Current Frame Selection. For any frame sequence, select one frame as the reference frame (denoted as ). For non-vigorous motion, the initial frame is selected; for vigorous motion, the frame preceding the current frame is typically selected, depending on the specific situation. The current frame is selected as the frame at the current moment (denoted as...). ).

[0027] Step 3: Preprocessing and Size Adaptation. Detect the size of the input image and, based on the preset downsampling rate (selected as 8 times in this embodiment), perform preprocessing on the reference frame. and the current frame Perform reflection filling so that the length and width pixel values ​​are divisible by the downsampling rate to obtain the adapted input frame.

[0028] Step 4: Shallow Feature Extraction. Construct a shallow feature extraction network consisting of 3×3 convolutional layers with shared weights. Input the adapted reference frame and the current frame into this network, mapping the image from the red-green-blue color space to a high-dimensional feature space, and outputting the shallow features of the reference frame. and shallow features of the current frame .

[0029] Step 5: Calculate and truncate differential features, proceed to the static feature refinement process, and calculate the shallow features of the current frame. Shallow features of the reference frame absolute value of the difference To prevent outliers from interfering with and to normalize the motion intensity, the difference features are numerically truncated, limiting their range to a preset interval, thus obtaining the truncated difference features. The truncation function is defined as follows:

[0030] ; In the formula, This is a value-truncation function. Its function is to truncate the absolute value. Limits the value to a specified minimum and maximum value.

[0031] Step 6: Nonlinear mapping of differential features, truncating the differential features The input is fed into a shallow differential processing unit, which contains a "3×3 convolution-corrected linear unit-3×3 convolution" structure. This structure is used to extract high-frequency texture information from the differential features and then compress the features to an intermediate dimension to recover the mapped differential features. .

[0032] Step 7: Static mask generation, using a mask generation network to process the mapped differential features. A pixel-level static mask is generated using a 1×1 convolutional layer and a sigmoid activation function. The value of this mask represents the probability that each pixel belongs to a static background (the closer to 1, the more likely it is to be a static background). The formula is as follows:

[0033] ; in, For the sigmoid function. Specifically, the bias term. It is initialized to a positive value (2.0 in this example).

[0034] Step 8: Weighting of static background, using the generated static mask Shallow features of the reference frame Element-wise multiplication is performed to suppress features in moving regions and forcibly preserve features in static background regions, resulting in the static correction amount. .

[0035] Step 9: Static baseline feature generation.

[0036] The static correction amount obtained in step 8 As input, the data is fed into a deep feature refinement module consisting of residual blocks and convolutional layers. This module performs further nonlinear mapping and depth processing on the input features to filter out residual noise and enhance background texture representation, directly outputting the final high-fidelity static baseline features for fusion. This feature preserves the complete structural information of the reference frame and effectively eliminates motion interference. The formula is:

[0037] ; in, This represents the composite feature mapping function formed by the residual block and subsequent convolutional layers.

[0038] Step 10: Large stride downsampling, parallel processing of depth motion flow, using convolutional layers with equal stride and downsampling rate to directly process shallow features. and Downsampling is performed to obtain a low-resolution feature map. and This reduces the computational load of subsequent processing and expands the receptive field.

[0039] Step 11: Serialization and Position Encoding. The two-dimensional downsampled feature map is flattened into a one-dimensional feature sequence through block embedding. Learnable absolute position encoding parameters are introduced, and the position encoding size is adjusted by bicubic interpolation according to the actual resolution of the current feature map before being superimposed on the one-dimensional feature sequence to obtain the serialized reference frame features and the serialized current frame features.

[0040] Step 12: Visual state-space encoding, constructing a deep feature extractor containing N layers (N=12 in this example) into which the serialized features are input. The formula is:

[0041] ; in, These are discretized state parameters. They are not fixed values, but rather determined by continuous parameters. Combined with time scale parameters It is derived from (the transformation relationship is as follows) ). The time scale parameter (step size) is a parameter that is based on the current input. The dynamically predicted variables are used to convert continuous parameters Discretize into This allows for selective focusing on different information streams, and It is dynamically predicted based on the input content; For the current moment The hidden state is used to store context history information; For the previous moment The hidden state; For the current moment The input feature sequence (i.e., the image features after serialization). This is the output projection matrix, used to map the hidden states to output features; For the current moment The output characteristics.

[0042] The deep feature extraction module includes a state-space model, a normalization layer, and a fully connected layer. The state-space model performs recursive state updates and feature encoding on the serialized reference frame features and the serialized current frame features. It models key information in the feature sequence through a selective scanning mechanism to capture long-distance dependencies and extract global structural representations. The normalization layer normalizes the distribution of intermediate features output by the state-space model to reduce numerical fluctuations between different channels and positions and improve the stability of feature representation. The fully connected layer performs linear mapping and channel recombination on the normalized features to form a deep feature representation suitable for subsequent motion difference construction and motion manipulation processing, resulting in the encoded reference frame features and the encoded current frame features.

[0043] This deep feature extractor employs a selective scanning mechanism based on a state-space model, effectively capturing global geometric structure information of the image while maintaining linear computational complexity. The specific data processing and correspondence are as follows: the serialized reference frame features and the serialized current frame features obtained from previous processing are fed into the extractor as independent inputs. Through state evolution and deep feature mapping within the model, encoded reference frame features containing global context information are output. Features of the current frame after encoding .

[0044] Step 13: Motion Manipulation and Amplification, Constructing the Manipulator, Calculating Motion Difference Features Motion features are extracted using a large-kernel 7×7 convolution and then multiplied by an amplification factor. After residual connection and convolution processing, the features are added back to the current frame features. In this process, a feature sequence containing amplified motion information is obtained. .

[0045] Step 14: Utilize the feature sequence containing amplified motion information obtained in Step 13. The input state-space decoding module performs decoding processing. This module includes a state-space model, a normalization layer, and a fully connected layer. The state-space model performs state propagation and contextual reorganization on the feature sequence containing amplified motion information to compensate for spatial correlation information that may be weakened during motion amplification and to restore the structural consistency of the features. The normalization layer regularizes the distribution of intermediate features after state propagation to improve the numerical stability of the decoding process. The fully connected layer performs linear projection and semantic fusion on the normalized features to form a decoded feature representation suitable for image reconstruction.

[0046] The state-space decoding module utilizes a state-space model to compensate for and reconstruct the global contextual information in the magnified feature sequence, enhancing the target motion response while maintaining the consistency of the overall image structure and local details. After processing by subsequent reconstruction layers, the decoded features generate the motion-magnified image. This decoding process reduces structural distortion and artifacts introduced during motion manipulation and magnification, improving the visual continuity and detail fidelity of the output.

[0047] Step 15: Subpixel upsampling. Using the upsampling module, subpixel convolution technology is employed to decode the features. The channel count is rearranged to spatial resolution, and its size is restored to be consistent with the shallow features, resulting in high-resolution motion magnification features. .

[0048] Step 16: Dual-stream feature stitching, combining the static baseline features obtained in Step 9. Numerical range truncation is performed to enhance stability, and then it is compared with the motion amplification feature obtained in step 15. The channel dimensions are spliced ​​together to form a fused feature tensor.

[0049] Step 17: Fusion and Reconstruction. Construct a fusion and reconstruction module. First, fuse the spliced ​​features through convolutional layers. Then, introduce a group normalization layer (channels are divided into 32 groups) to replace the traditional layer normalization. Finally, further refine the features through residual blocks.

[0050] Step 18: Image generation. The features processed by the reconstruction module are input to and output to the convolutional layer, and the number of channels is mapped back to RGB channels. Finally, the final motion-enlarged image is generated through the hyperbolic tangent activation function (Tanh). Repeat the above steps for each frame in the video sequence to generate the complete magnified video.

[0051] This invention employs a visual state-space model to construct the core network architecture. Leveraging its unique selective scanning mechanism and long-distance dependency modeling capabilities, it can achieve a global receptive field while maintaining linear computational complexity (computational load is proportional to sequence length). This allows the model to consistently maintain the spatial coherence and geometric consistency of various parts of an object when processing sub-pixel-level motion magnification, effectively avoiding image structure distortion or breakage caused by uneven local magnification. This significantly enhances the physical realism of magnified videos, substantially reduces memory consumption, and improves inference speed.

[0052] Furthermore, this invention designs a static feature refiner that adaptively generates pixel-level static masks by calculating feature differences. This mechanism is equivalent to building a soft gating system at the feature level, which can intelligently distinguish between slightly moving foregrounds and static backgrounds, forcing the model to strictly preserve the original information of the static background while amplifying motion signals. This design effectively overcomes the shortcomings of existing technologies that have to amplify background noise in order to amplify motion, significantly eliminating ripples and artifacts in the background region and improving the signal-to-noise ratio of the image.

[0053] A video motion amplification network architecture based on a state-space model.

[0054] The details of the network architecture used to perform the method described in Example 1 are highlighted. Figure 2 As shown, the network adopts a dual-stream architecture, including a parallel deep motion stream branch and a static feature refinement stream branch.

[0055] The internal structure of the deep feature extractor and state space awareness module: such as Figure 3 As shown, the depth feature extractor (and subsequent state space perception module) in this embodiment consists of multiple stacked "visual state space modules".

[0056] To address the limitations of the receptive field and high computational complexity of self-attention mechanisms in traditional convolutional neural networks, this module employs a state-space model (SSM) at its core. Specifically, the input features are first processed through layer normalization, and then divided into two paths:

[0057] The main path (which is the path after downsampling and passing through a deep feature extractor) undergoes feature dimension transformation through a linear mapping layer (step 4) and extracts local features through a one-dimensional convolutional layer; then it enters the core state-space model unit. In the state-space model unit, a zero-order preserved discretization strategy is used to transform continuous state-space parameters into discrete parameters. Crucially, this embodiment introduces a selective scanning mechanism, enabling the step size parameter to be dynamically predicted based on the content of the input data, thereby achieving adaptive modeling of non-stationary video signals. The bypass path, through linear mapping and activation functions, ultimately performs element-wise multiplication with the output of the main path (gating mechanism) to control the information flow.

[0058] This structural design enables the network to capture long-range spatiotemporal dependencies with linear computational complexity (proportional to sequence length), thereby maintaining the consistency of the global geometry when processing whole frames of images.

[0059] Detailed construction of the static feature refiner: as follows Figure 4 As shown, to accurately separate motion from the background, the static feature refiner does not employ a complex deep network, but is instead designed as a lightweight difference-aware structure. This module first receives features from the reference frame. and current frame features The difference is calculated using a subtractor. The difference features are sequentially passed through a convolutional layer, an activation layer, and another convolutional layer (i.e., a "convolution-activation-convolution" bottleneck structure) to extract high-frequency texture differences. Finally, a 1×1 convolutional layer compresses the channels to 1, and a sigmoid activation function is used to output the numerical values. A static mask is applied between the reference frames. This mask acts directly on the features of the original reference frame, filtering out interference from moving regions through multiplication operations.

[0060] Motion enhancement mechanisms of the manipulator, such as Figure 5 As shown, the manipulator is the core component for achieving motion amplification. It receives the encoded features and first calculates the difference between the features of two frames. To accommodate motion at different scales, the differential features are passed through a 7×7 convolutional layer with a large receptive field. Subsequently, a user-preset magnification factor is introduced. The differential features are linearly amplified. To prevent artifacts caused by linear amplification, the amplified features are then passed through a nonlinear correction module containing residual connections (containing two 3×3 convolutional layers and an activation layer in between). Finally, the corrected motion increment is superimposed back into the current frame features.

[0061] This invention innovatively employs a visual state-space model as the core backbone in the deep motion flow branch. It utilizes state-space equations and discretized parameters to construct a feature extraction network and introduces a selective scanning mechanism to model long-distance dependencies in image feature sequences. This key innovation primarily addresses the technical problems in existing technologies, such as the geometric distortion caused by the limited receptive field of convolutional neural networks when processing object rotation or overall translation, and the slow inference speed and high memory consumption due to the quadratic increase in computational complexity of the Transformer architecture with image resolution. Through this technical solution, this invention can obtain a global receptive field while maintaining linear computational complexity, thus preserving the spatial coherence and geometric consistency of various parts of the object throughout sub-pixel-level motion magnification. This significantly improves video processing efficiency while effectively ensuring the physical realism of the magnified video.

[0062] This application designs a parallel static feature refinement stream branch. By calculating the absolute difference between the shallow features of the current frame and the reference frame, and after numerical truncation and mapping using a "convolution-activation-convolution" structure, a pixel-level static mask is generated using the Sigmoid activation function. This mask is then used to apply soft-gated weighting to the features of the reference frame. This key innovation solves the technical problem of existing technologies being unable to effectively distinguish between spatially coherent real motion and randomly distributed high-frequency imaging noise when amplifying minute motion signals, resulting in the synchronous amplification of background noise and the generation of ripple artifacts. Through this technical solution, the model can intelligently identify and suppress the signal response of static background regions, forcibly preserving the original texture information of the background. This effectively eliminates background artifacts while significantly amplifying the target motion, significantly improving the signal-to-noise ratio of the generated image.

[0063] This application constructs a feature space manipulator that incorporates nonlinear residual correction. Specifically, after linearly amplifying the feature differences, the manipulator performs nonlinear transformation correction through residual blocks and numerical truncation before feature fusion. This key innovation primarily addresses the technical problem that traditional methods easily introduce unnatural artifacts and cause image distortion when performing simple linear amplification in the feature space. Through this technical solution, nonlinear correction smooths the feature abrupt changes generated during the amplification process, ensuring that the synthesized motion conforms to physical laws, resulting in a more natural and realistic amplified image in terms of detail and texture.

[0064] Example To verify the effectiveness of the video motion amplification method based on the state-space model proposed in this invention, this embodiment conducted quantitative tests on a synthetic dataset containing translation, rotation, and noise interference, and compared it with existing mainstream technologies.

[0065] 1. The experiment selected two representative test scenarios: Scenario 1: Simulating sub-pixel-level minute displacements of objects in the horizontal and vertical directions. Scenario 2: Simulating sub-pixel and minute displacements caused by object translation and rotation, which is a scenario that is difficult to handle with existing technology. The evaluation metric used is Peak Signal-to-Noise Ratio (PSNR, in dB, the higher the better).

[0066] 2. Comparison and analysis of results, as follows: Figure 6 As shown, the comparison results of the method of the present invention with three existing technologies in terms of PSNR are presented. Comparison Model 1: represents the traditional handmade feature method; Comparison Model 2: represents the learning method based on convolutional neural networks; Comparison Model 3: represents the existing state-of-the-art method based on sub-attention mechanism.

[0067] Experimental data show that in scenario one, the PSNR of the proposed method reaches 21.32 dB, significantly better than the 16.16 dB of the comparative model 2; in the more challenging scenario two, the PSNR of the proposed method reaches 21.35 dB, better than the 20.25 dB of the comparative model 3. This demonstrates that the proposed method, utilizing a state-space model, can effectively overcome the local receptive field limitations of convolutional networks, maintaining structural integrity and avoiding image distortion when processing motions with overall geometric relationships, such as rotation. Simultaneously, thanks to the introduction of a static feature thinner, background noise is effectively suppressed, significantly improving the signal-to-noise ratio.

[0068] Based on the same concept, the present invention also provides a video motion amplification system, including a conversion module, an input module and a processing module.

[0069] The conversion module is used to convert a video sequence to be enlarged into a continuous frame sequence.

[0070] The input module is used to input the current frame and the reference frame in the frame sequence into the video motion magnification model to obtain a motion magnified image. The video motion magnification model includes a shallow feature extraction module and a motion feature extraction module. The motion feature extraction module includes a downsampling module, a deep feature extraction module, a manipulator, a decoder, and an upsampling module.

[0071] The processing module extracts features from the reference frame and the current frame using a shallow feature extraction module to obtain shallow features of the reference frame and the current frame. The downsampling module downsamples the shallow features of the reference frame and the current frame to obtain feature maps of the reference frame and the current frame. The depth feature extraction module encodes the feature maps of the reference frame and the current frame using a state-space model to obtain features of the reference frame and the current frame. The manipulator subtracts and amplifies the features of the reference frame and the current frame to obtain a feature sequence containing amplified motion information. The decoder decodes the feature sequence using a state-space model to obtain decoded features. The upsampling module upsamples the decoded features to obtain motion-magnified features. Finally, a motion-magnified image is generated based on these motion-magnified features.

[0072] The present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the above-described video motion magnification method.

[0073] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-described video motion amplification method.

[0074] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0075] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A video motion magnification method, characterized in that, Includes the following steps: Convert the video sequence to be enlarged into a continuous frame sequence; The current frame and reference frame in the frame sequence are input into the video motion magnification model to obtain a motion magnified image; wherein, the video motion magnification model includes a shallow feature extraction module and a motion feature extraction module, and the motion feature extraction module includes a downsampling module, a deep feature extraction module, a manipulator, a decoder and an upsampling module; The shallow feature extraction module extracts features from the reference frame and the current frame to obtain shallow features of the reference frame and the current frame. The downsampling module downsamples the shallow features of the reference frame and the current frame to obtain feature maps of the reference frame and the current frame. The deep feature extraction module encodes the feature maps of the reference frame and the current frame using a state-space model to obtain features of the reference frame and the current frame. The manipulator subtracts and amplifies the features of the reference frame and the current frame to obtain a feature sequence containing amplified motion information. The decoder decodes the feature sequence using a state-space model to obtain decoded features. The upsampling module upsamples the decoded features to obtain motion magnification features. A motion magnification image is generated based on the motion magnification features.

2. The video motion magnification method as described in claim 1, characterized in that, Before encoding the reference frame feature map and the current frame feature map using the state space model, the deep feature extraction module performs serialization and position encoding on the reference frame feature map and the current frame feature map to obtain serialized reference frame features and serialized current frame features. The deep feature extraction module includes a state space model, a normalization layer, and a fully connected layer. The deep feature extraction module encodes the features of the serialized reference frame and the features of the serialized current frame to obtain intermediate features. The normalization layer normalizes the intermediate features, and the fully connected layer maps the normalized intermediate features to obtain the reference frame features and the current frame features.

3. The video motion magnification method as described in claim 1, characterized in that, The video motion magnification model also includes a static feature extraction module, a stitching module, and a fusion module; The static feature extraction module generates a static mask based on the difference between the shallow features of the reference frame and the shallow features of the current frame. The shallow features of the reference frame are then weighted using the static mask to generate static baseline features. The stitching module stitches together motion amplification features and static baseline features to obtain a fused feature tensor; The fusion module reconstructs the fusion feature tensor to obtain a motion magnified image.

4. The video motion magnification method as described in claim 3, characterized in that, The static feature extraction module includes a difference module, a truncation module, a nonlinear mapping module, a static masking module, a weighting module, a correction module, and a generation module; The difference module subtracts the shallow features of the reference frame and the shallow features of the current frame to obtain the difference feature; The truncation module truncates the difference features to obtain the differential features; The nonlinear mapping module performs feature extraction and dimensionality compression on the differential features to obtain the mapped differential features. The static masking module performs convolution on the mapped difference features to obtain a static mask; the static mask value represents the probability that each pixel belongs to the static background. The weighting module performs element-wise multiplication of the static mask and the shallow features of the reference frame to obtain the static correction amount; The generation module performs in-depth feature refinement on the static correction amount to obtain static baseline features.

5. The video motion magnification method as described in claim 4, characterized in that, The truncation module performs truncation processing on the difference features, as shown below: ； In the formula, Value truncation function, As a distinguishing feature, As a reference frame shallow features, These are the shallow features of the current frame, where min is the minimum value and max is the maximum value. The static mask is shown below: ； In the formula, For the Sigmoid function, For bias terms, The difference features after mapping For static masking, Conv 1×1 It is a 1×1 convolutional layer.

6. The video motion magnification method as described in claim 1, characterized in that, The manipulator includes a difference module, a convolution module, an amplification module, and a transformation module; The difference module subtracts the features of the reference frame from the features of the current frame to obtain motion difference features; The convolutional module extracts motion features from the motion difference features to obtain motion features; The amplification module multiplies the motion feature by the amplification factor to obtain the amplified motion feature; The transformation module performs a nonlinear transformation on the amplified motion features to obtain a feature sequence containing amplified motion information.

7. The video motion magnification method as described in claim 1, characterized in that, Before inputting the current frame and reference frame in the frame sequence into the video motion magnification model, reflection filling is performed on the reference frame and the current frame according to the preset downsampling rate so that their length and width pixel values ​​can be divided by the downsampling rate.

8. A video motion magnification system, characterized in that, include: The conversion module is used to convert the video sequence to be enlarged into a continuous frame sequence; The input module is used to input the current frame and the reference frame in the frame sequence into the video motion magnification model to obtain a motion magnification image; wherein, the video motion magnification model includes a shallow feature extraction module and a motion feature extraction module, and the motion feature extraction module includes a downsampling module, a deep feature extraction module, a manipulator, a decoder and an upsampling module; The processing module is used to extract features from the reference frame and the current frame through the shallow feature extraction module to obtain shallow features of the reference frame and the current frame; the downsampling module downsamples the shallow features of the reference frame and the current frame to obtain feature maps of the reference frame and the current frame; the depth feature extraction module encodes the feature maps of the reference frame and the current frame through a state space model to obtain features of the reference frame and the current frame; the manipulator subtracts and amplifies the features of the reference frame and the current frame to obtain a feature sequence containing amplified motion information; the decoder decodes the feature sequence through a state space model to obtain decoded features; the upsampling module upsamples the decoded features to obtain motion magnification features; and a motion magnification image is generated based on the motion magnification features.

9. A computer device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the video motion magnification method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the video motion magnification method according to any one of claims 1-7.

Images