Motion amplification method, device and storage medium based on event camera

Abstract

The present invention relates to a motion amplification method, device, and storage medium based on an event camera. The method comprises: providing an optical splitter, an event camera, and an RGB camera, with the output of the optical splitter connected to the event camera and the RGB camera, respectively, to form a dual-camera imaging system; acquiring event signals via the event camera and image signals via the RGB camera; processing the acquired event and image signals through an encoder, an amplifier, and a decoder, respectively. The encoder is configured to model the features of the event and image signals and is equipped with a second-order recursive propagation module and a channel attention fusion module; the amplifier is configured to amplify the motion amplitude detected by the features output by the encoder; and the decoder is configured to model the features of the amplifier back into the image space to obtain a final motion-amplified image. Compared with existing technologies, the present invention can economically perform motion amplification and significantly improve the effectiveness of high-frequency motion amplification.

Description

Technical Field

[0001] The present invention relates to the technical field of motion magnification, and in particular to a motion magnification method, device and storage medium based on an event camera. Background Art

[0002] In the real world, many tiny movements are imperceptible to humans, yet they are crucial for industrial and medical applications. These movements are characterized by small amplitudes, ranging from micrometers to meters, and high frequencies, from a few hertz to megahertz. Motion amplification technology helps visualize these imperceptible movements, enabling us to detect vibrations, remotely measure vital signs, and analyze the physical properties of materials.

[0003] The existing technical solutions are as follows:

[0004] In high-frequency, small motion amplification, spectral aliasing occurs if the camera system's Nyquist frequency is lower than the highest natural frequency of the motion. This limitation makes conventional camera systems unable to handle such applications. Therefore, previous image-based motion amplification methods have traditionally relied on specialized high-speed cameras. However, these high-speed cameras are characterized by high cost, large memory requirements, and limited spatial resolution, which prevents them from being widely used. To address the limitations of high-speed cameras, a dual-shutter system has been introduced. However, this method requires an active light source and is mainly suitable for motion detection within a limited area, which restricts the application scenarios and limits its applicability. Therefore, effectively and conveniently detecting and amplifying low-amplitude, high-frequency motion remains a challenge. Summary of the Invention

[0005] The present invention aims to overcome the drawbacks of the prior art, which requires extremely high frame rate (even greater than 1000) video capture to detect high-frequency motion. Current methods rely on expensive, specialized high-speed cameras and active light sources, which limits their application scenarios. The present invention provides a motion amplification method, device, and storage medium based on an event camera.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] A motion magnification method based on an event camera comprises the following steps:

[0008] A spectrometer, an event camera, and an RGB camera are provided, and the output end of the spectrometer is connected to the event camera and the RGB camera respectively, so as to form a dual-camera imaging system;

[0009] Event signals are collected through an event camera, and image signals are collected through an RGB camera;

[0010] The collected event signals and image signals are processed through an encoder, an amplifier, and a decoder, respectively. The encoder is used to model the features of the event signals and image signals and is equipped with a second-order recursive propagation module and a channel attention fusion module. The second-order recursive propagation module is used to form a second-order recursive propagation network with the event signals or image signals at adjacent moments to perform feature modeling; the channel attention fusion module is used to fuse the features of the event signals and image signals;

[0011] The amplifier is used to amplify the motion amplitude detected by the features output by the encoder; the decoder is used to model the features of the amplifier back into the image space to obtain a final motion-amplified image.

[0012] Furthermore, the encoder includes an image signal encoding branch and an event signal encoding branch, wherein the image signal encoding branch is used to model the image signal to obtain image features, wherein the image features include texture features and shape features;

[0013] The second-order recursive propagation module and the channel attention fusion module are both set in the event signal encoding branch to model the event signal and fuse image features to construct motion features;

[0014] The amplifier amplifies the shape feature based on the motion feature, and then combines it with the texture feature and inputs it into the decoder.

[0015] Furthermore, the amplifier is also provided with a time filter, which is used to perform pixel-by-pixel Fourier transform on the motion features of each frame along the time dimension, extract the required motion frequency using a bandpass filter, and apply inverse Fourier transform to return it to the time domain features of the amplifier.

[0016] Furthermore, the event signal input into the event signal encoding branch is processed in sequence by a second-order recursive propagation module and a channel attention fusion module to construct motion features.

[0017] Furthermore, the input of each of the second-order recursive propagation modules includes the event signal or image signal at the current moment, and the output of the second-order recursive propagation module at the previous two moments.

[0018] Furthermore, the event camera and the RGB camera are both subjected to geometric calibration and time synchronization processing.

[0019] The present invention also provides a motion amplification device based on an event camera, comprising:

[0020] The output of the optical splitter is connected to the event camera and the RGB camera respectively;

[0021] An event camera, used to collect event signals;

[0022] RGB camera, used to collect image signals;

[0023] An encoder for modeling the features of event signals and image signals, and comprising a second-order recursive propagation module and a channel attention fusion module. The second-order recursive propagation module is used to form a second-order recursive propagation network with event signals or image signals at adjacent moments to perform feature modeling; the channel attention fusion module is used to fuse the features of event signals and image signals;

[0024] An amplifier for amplifying the motion amplitude detected by the features output by the encoder;

[0025] The decoder is used to model the features of the upscaler back to the image space to obtain the final motion upscaled image.

[0026] Furthermore, the encoder includes an image signal encoding branch and an event signal encoding branch, wherein the image signal encoding branch is used to model the image signal to obtain image features, wherein the image features include texture features and shape features;

[0027] The second-order recursive propagation module and the channel attention fusion module are both set in the event signal encoding branch to model the event signal and fuse image features to construct motion features;

[0028] The amplifier amplifies the shape feature based on the motion feature, and then combines it with the texture feature and inputs it into the decoder.

[0029] Furthermore, the amplifier is also provided with a time filter, which is used to perform pixel-by-pixel Fourier transform on the motion features of each frame along the time dimension, extract the required motion frequency using a bandpass filter, and apply inverse Fourier transform to return it to the time domain features of the amplifier.

[0030] The present invention further provides a computer-readable storage medium having a computer program stored thereon, and the computer program is used by a processor to execute the above method.

[0031] Compared with the prior art, the present invention has the following advantages:

[0032] (1) This invention uses both an event camera and an RGB camera, leveraging the ultra-high resolution of the event camera and the dense spatiotemporal features of the RGB camera to economically amplify motion. Based on this, a motion method algorithm for event cameras is designed. By designing a second-order recursive propagation module and a temporal filter, the method significantly improves the performance of high-frequency motion amplification and can accurately restore motion frequency.

[0033] (2) Traditional motion amplification solutions based on professional high-speed cameras cannot be widely used due to their high hardware cost, large memory requirements, and limited spatial resolution. Active light source-based methods avoid dependence on professional high-speed cameras, but their dependence on active light sources and the fact that they can only detect high-frequency motion within a specific range limit their application scenarios.

[0034] To address this issue, this paper proposes a dual-camera imaging system combining an event camera and an RGB camera for motion amplification. This design addresses the hardware-level dependence of traditional motion amplification algorithms on expensive specialized high-speed cameras and active light sources. By leveraging the temporally dense event signals, the proposed dual-camera imaging system can economically detect and amplify high-frequency motion.

[0035] (3) This invention aims to address the problem of long-distance temporal modeling caused by excessive interpolation of frames during motion amplification. For example, for RGB at a 20 fps rate, in order to detect 512 Hz motion, at least 50 frames need to be interpolated, which is much higher than previous interpolation algorithms.

[0036] To address this issue, a second-order recursive propagation module, based on the dual-camera hardware architecture, was proposed. This module not only considers the relationship between the current state and the previous state, but also the relationship between the states of the two previous moments, improving the network's ability to model long-range features. Excessive interpolation of frames poses challenges in long-range temporal modeling. This method enhances its ability to detect and amplify high-frequency motion, improving the effect of motion amplification.

[0037] (4) In order to solve the problem of aliasing between small motions and noise of event cameras in motion amplification, the present invention proposes the use of a time filter to reduce the impact of noise in the event signal on small motions, thereby greatly reducing the disturbance caused by noise during the amplification process. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 Schematic diagram of the overall network structure of a motion amplification method based on an event camera provided in an embodiment of the present invention;

[0039] Figure 2 A schematic diagram of the structure of a second-order recursive propagation module (SRP) provided in an embodiment of the present invention;

[0040] Figure 3 Schematic diagram of the structure of a channel attention fusion module (CAF) provided in an embodiment of the present invention;

[0041] Figure 4 This is a schematic structural diagram of a 256 Hz tuning fork provided in an embodiment of the present invention;

[0042] Figure 5A schematic diagram of a motion amplification result without a time filter provided in an embodiment of the present invention;

[0043] Figure 6 A schematic diagram of a motion amplification result using a traditional recursive propagation module provided in an embodiment of the present invention;

[0044] Figure 7 A schematic diagram of a motion amplification result using the method of the present invention is provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0045] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations.

[0046] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention as claimed, but rather merely represents selected embodiments of the present invention. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without creative effort shall fall within the scope of protection of the present invention.

[0047] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings.

[0048] Example 1

[0049] like Figure 1 As shown, this embodiment provides a motion magnification method based on an event camera, comprising the following steps:

[0050] A spectrometer, an event camera, and an RGB camera are provided, and the output end of the spectrometer is connected to the event camera and the RGB camera respectively, so as to form a dual-camera imaging system;

[0051] Event signals are collected through an event camera, and image signals are collected through an RGB camera;

[0052] like Figure 2 and Figure 3As shown, the collected event signal and image signal are processed by an encoder, an amplifier, and a decoder respectively. The encoder is used to model the features of the event signal and the image signal, and is provided with a second-order recursive propagation module and a channel attention fusion module. The second-order recursive propagation module is used to form a second-order recursive propagation network with the event signal or image signal at adjacent moments, thereby performing feature modeling; the channel attention fusion module is used to fuse the features of the event signal and the image signal;

[0053] The amplifier is used to amplify the motion amplitude detected by the features output by the encoder; the decoder is used to model the features of the amplifier back into the image space to obtain the final motion amplified image.

[0054] Specifically, the encoder includes an image signal encoding branch and an event signal encoding branch. The image signal encoding branch is used to model the image signal to obtain image features, which include texture features and shape features.

[0055] The second-order recursive propagation module and the channel attention fusion module are both set in the event signal encoding branch to model the event signal and fuse image features to construct motion features;

[0056] That is, the event signal of the input event signal encoding branch is processed in sequence by the second-order recursive propagation module and the channel attention fusion module to construct the motion feature.

[0057] The amplifier amplifies the shape features based on the motion features, and then combines them with the texture features and inputs them into the decoder.

[0058] like Figure 1 As shown in Figure 1, the dual-camera imaging system consists of an event camera and an RGB camera. To ensure alignment of the fields of view of the two sensors, a beam splitter is placed between the event camera and the RGB camera. Both cameras are geometrically calibrated and time-synchronized.

[0059] By collecting event-dense event signals from an event camera and spatially dense image signals from an RGB camera, this solution can economically amplify high-frequency motion.

[0060] like Figure 2 As shown, this is the second-order recursive propagation module proposed by this method.

[0061] The second-order recurrent propagation network (RPN) is a deep learning model primarily used to process sequence data, especially data with complex temporal dependencies. Compared to traditional recurrent neural networks (RNNs), the second-order RPN is better able to capture long-term dependencies in sequences because it considers not only the relationship between the current state and the previous state, but also the relationship between the two previous states.

[0062] The basic structure of a second-order recurrent propagation network is to add a memory unit to the traditional RNN unit. This memory unit stores the state information of the previous two moments. In this way, when updating the current state, the network considers not only the information of the current moment and the previous moment, but also the information of the previous two moments. This enables the network to better capture and understand long-term dependencies in the sequence.

[0063] Although recurrent neural network architectures are renowned for their efficiency, they may not be suitable for our scenario, as the task requires simulating long-range temporal dependencies, requiring the insertion of a large number of frames (up to 80). Therefore, this method proposes a second-order recursive propagation module, each of which takes as input the event signal or image signal at the current moment, as well as the output of the second-order recursive propagation module at the previous two moments. By adding skip connections to the temporal dimension of the network, the network's ability to model long-range temporal information is effectively enhanced, improving the effect of motion amplification.

[0064] In practical applications, it's often desirable to amplify motion at specific frequencies to reduce noise interference. As a preferred implementation, this solution proposes applying a temporal filter to motion features during the inference phase. Specifically, the amplifier also incorporates a temporal filter, which performs a pixel-by-pixel Fourier transform on each frame's motion features along the temporal dimension. This filter then uses a bandpass filter to extract the desired motion frequencies and applies an inverse Fourier transform to return them to the amplified time domain features. This temporal filter effectively reduces temporal noise in the event camera, further enhancing the regularity of motion.

[0065] This solution improves the network's ability to model long-range features by designing a second-order recursive propagation module and proposes the use of a temporal filter to reduce the impact of noise in event signals on small movements. The overall algorithm can effectively recover and amplify high-frequency motion information.

[0066] The proposed method has been validated through multiple experiments, including simulation data and real-world data. Examples of real-world data include 256Hz and 512Hz tuning forks, guitar string vibrations, fan vibrations, muscle tremors, and vibrations caused by impact. These experiments demonstrate that this method can effectively amplify high-frequency motion and accurately measure its frequency.

[0067] Figure 4-Figure 6 The algorithmic improvement of this method over other methods in high-frequency motion amplification is demonstrated in the paper. By using the time-free filter, the traditional recursive propagation method and this method to amplify the high-frequency motion of a 256 Hz tuning fork, it can be seen that this method can amplify high-frequency motion, well show the shape of the object's motion, and reflect the actual motion frequency.

[0068] The above is an introduction to a method embodiment. The following further illustrates the solution of the present invention through an apparatus embodiment.

[0069] This embodiment also relates to a motion amplification device based on an event camera, comprising:

[0070] The output of the optical splitter is connected to the event camera and the RGB camera respectively;

[0071] An event camera, used to collect event signals;

[0072] RGB camera, used to collect image signals;

[0073] The encoder is used to model the features of event signals and image signals and is equipped with a second-order recursive propagation module and a channel attention fusion module. The second-order recursive propagation module is used to form a second-order recursive propagation network with event signals or image signals at adjacent moments to perform feature modeling; the channel attention fusion module is used to fuse the features of event signals and image signals;

[0074] An amplifier for amplifying the motion amplitude detected by the features output by the encoder;

[0075] The decoder is used to model the features of the upscaler back to the image space to obtain the final motion upscaled image.

[0076] Optionally, the encoder includes an image signal encoding branch and an event signal encoding branch, the image signal encoding branch is used to model the image signal to obtain image features, the image features including texture features and shape features;

[0077] The second-order recursive propagation module and the channel attention fusion module are both set in the event signal encoding branch to model the event signal and fuse image features to construct motion features;

[0078] The amplifier amplifies the shape features based on the motion features, and then combines them with the texture features and inputs them into the decoder.

[0079] Optionally, the amplifier is further provided with a time filter, which is used to perform pixel-by-pixel Fourier transform on the motion features of each frame along the time dimension, extract the required motion frequency using a bandpass filter, and apply an inverse Fourier transform to return it to the time domain features of the amplifier.

[0080] It should be noted that the specific content and beneficial effects of the device of the present application can be found in the above method embodiments and will not be repeated here.

[0081] This embodiment further provides a computer-readable storage medium, on which a computer program is stored. The computer program is used by a processor to execute the above-mentioned event camera-based motion magnification method.

[0082] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. Such program code can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0083] In the context of the present invention, machine-readable medium can be a tangible medium that can contain or store a program for use with an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. More specific examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0084] The above describes in detail the preferred embodiments of the present invention. It should be understood that those skilled in the art can make numerous modifications and variations based on the concepts of the present invention without inventive effort. Therefore, any technical solutions that can be derived by those skilled in the art through logical analysis, reasoning, or limited experimentation based on the concepts of the present invention and the prior art should be within the scope of protection defined by the claims.

Claims

1. A motion magnification method based on event camera, characterized in that: The following steps are involved: A spectrometer, an event camera, and an RGB camera are provided, and the output end of the spectrometer is connected to the event camera and the RGB camera respectively, so as to form a dual-camera imaging system; Event signals are collected through an event camera, and image signals are collected through an RGB camera; The collected event signals and image signals are processed through an encoder, an amplifier, and a decoder, respectively. The encoder is used to model the features of the event signals and image signals and is equipped with a second-order recursive propagation module and a channel attention fusion module. The second-order recursive propagation module is used to form a second-order recursive propagation network with the event signals or image signals at adjacent moments to perform feature modeling; the channel attention fusion module is used to fuse the features of the event signals and image signals; The amplifier is used to amplify the motion amplitude detected by the features output by the encoder; the decoder is used to model the features of the amplifier back into the image space to obtain a final motion amplified image; The encoder includes an image signal encoding branch and an event signal encoding branch, wherein the image signal encoding branch is used to model the image signal to obtain image features, which include texture features and shape features; The second-order recursive propagation module and the channel attention fusion module are both set in the event signal encoding branch to model the event signal and fuse image features to construct motion features; The amplifier amplifies the shape feature based on the motion feature, and then combines it with the texture feature and inputs it into the decoder.

2. The motion magnification method based on event camera according to claim 1, characterized in that: The amplifier is also provided with a time filter, which is used to perform pixel-by-pixel Fourier transform on the motion features of each frame along the time dimension, use a bandpass filter to extract the required motion frequency, and apply inverse Fourier transform to return it to the time domain features of the amplifier.

3. The motion magnification method based on event camera according to claim 1, characterized in that: The event signal input into the event signal encoding branch is processed in sequence by a second-order recursive propagation module and a channel attention fusion module to construct motion features.

4. The motion magnification method based on event camera according to claim 1, characterized in that: The input of each of the second-order recursive propagation modules includes the event signal or image signal at the current moment and the output of the second-order recursive propagation module at the previous two moments.

5. The motion magnification method based on event camera according to claim 1, characterized in that: The event camera and RGB camera are both geometrically calibrated and time-synchronized.

6. A motion magnification device based on an event camera, characterized in that: include: The output of the optical splitter is connected to the event camera and the RGB camera respectively; An event camera, used to collect event signals; RGB camera, used to collect image signals; An encoder for modeling the features of event signals and image signals, and comprising a second-order recursive propagation module and a channel attention fusion module. The second-order recursive propagation module is used to form a second-order recursive propagation network with event signals or image signals at adjacent moments to perform feature modeling; the channel attention fusion module is used to fuse the features of event signals and image signals; An amplifier for amplifying the motion amplitude detected by the features output by the encoder; The decoder is used to model the features of the amplifier back to the image space to obtain the final motion-amplified image; The encoder includes an image signal encoding branch and an event signal encoding branch, wherein the image signal encoding branch is used to model the image signal to obtain image features, which include texture features and shape features; The second-order recursive propagation module and the channel attention fusion module are both set in the event signal encoding branch to model the event signal and fuse image features to construct motion features; The amplifier amplifies the shape feature based on the motion feature, and then combines it with the texture feature and inputs it into the decoder.

7. The motion amplification device based on an event camera according to claim 6, characterized in that: The amplifier is also provided with a time filter, which is used to perform pixel-by-pixel Fourier transform on the motion features of each frame along the time dimension, use a bandpass filter to extract the required motion frequency, and apply inverse Fourier transform to return it to the time domain features of the amplifier.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and the computer program is used by a processor to execute the method according to any one of claims 1 to 5.

Images