A method for magnifying minute motion in image sequences based on event stream reconstruction

Abstract

This invention discloses a method for magnifying minute motion in reconstructed image sequences based on event streams, solving the problem of blurred minute motion details in poorly reconstructed images, belonging to the field of computer vision. The method includes: converting the original event stream data of the target in the task into a grayscale image sequence; performing a Fourier transform to obtain a frequency domain sub-band set; introducing a complex controllable pyramid algorithm to obtain a feature sub-band set with multi-scale and multi-directional joint characteristics, separating phase component sets and amplitude component sets; performing S-transform time-frequency filtering on the separated phase component set to obtain a phase component set in the time-frequency domain, and obtaining an enhanced phase component set based on a linear amplification factor; fusing the enhanced phase component set and the separated amplitude component set sub-band by sub-band to obtain a reconstructed frequency domain sub-band set, and outputting a magnified minute motion image sequence after Fourier inverse transformation to the time domain. This invention achieves accurate extraction of motion information from poorly reconstructed images.

Description

Technical Field

[0001] This invention belongs to the field of video motion magnification technology in computer vision, and relates to a method for magnifying minute motions in image sequences based on event stream reconstruction. Background Technology

[0002] Video motion magnification technology originates from the need in computer vision for sub-pixel-level motion resolution. Its purpose is to enhance the perception of minute motion changes that are difficult for the human eye to directly perceive through algorithms. However, traditional cameras, due to inherent limitations, struggle to accurately capture moving targets in complex scenes with high temporal resolution and high dynamic range. Against this backdrop, the biologically inspired sensing mechanism employed by event cameras has injected new vitality into the development of motion magnification technology. Event cameras possess asynchronous sampling characteristics, allowing them to overcome the limitations of traditional sensor temporal resolution and effectively capture high-speed, minute movements with microsecond-level temporal accuracy. Methods for implementing video motion magnification based on event cameras include:

[0003] Method 1 involves accumulating event data to generate a binary image, i.e., an event frame image, and then using video motion magnification to enhance the display of motion in the event stream. However, Method 1 presents the technical challenge of extracting and enhancing information from the binary image.

[0004] For example, application number: 202410073894.0, publication number: CN 118037770 A, invention title: A motion magnification method, device and storage medium based on event camera; this application has poor magnification effect when dealing with the problems of low signal-to-noise ratio and weak motion characteristics.

[0005] To address the issues with Method 1, Method 2 is proposed. First, the event stream is converted into a grayscale image sequence using image reconstruction methods. Then, existing mature video motion amplification algorithms are adaptively improved to ultimately achieve subtle motion enhancement for the reconstructed image sequence. Method 2 remains feasible and widely applicable even under complex lighting conditions such as exposure and low light.

[0006] For example, in the field of industrial manufacturing and inspection, traditional optical sensors have difficulty detecting the minute vibrations and displacements of precision parts. Method 2 can accurately capture these movements, helping to promptly identify potential defects and improve product quality and production efficiency. In biomedical research, the minute movements of cells are weak and difficult to observe with the human eye and traditional imaging equipment. Method 2 can effectively amplify and display cell movements, providing strong support for research. In security monitoring scenarios, especially under low-light conditions, Method 2 can accurately capture minute changes in the movement of people or objects, enabling timely detection of potential security threats.

[0007] However, in practice, Method 2 suffers from inherent problems such as low signal-to-noise ratio and weak motion features in the images reconstructed from event streams. During motion magnification, the details of minute movements are difficult to clearly present, easily leading to motion blur, misjudgments, and other adverse effects. This significantly impacts the accuracy and reliability of the method in practical applications. For example, in fields with extremely high requirements for the precision of minute movements, such as medical image analysis and industrial precision inspection, it may lead to erroneous diagnostic results or detection errors. Summary of the Invention

[0008] To address the technical problems of low signal-to-noise ratio and weak motion features in event stream reconstructed images, which make it difficult to clearly present details of minute motions during motion magnification, leading to motion blur and misjudgment, this invention proposes a method for minute motion magnification based on event stream reconstructed image sequences. This method constructs an effective noise suppression and feature enhancement mechanism and proposes an S-transform time-frequency filtering method. By utilizing the energy distribution characteristics of the signal in the time-frequency domain, the influence of noise is greatly suppressed, while maintaining accurate phase characterization capabilities, enabling the analysis of non-stationary motion in event stream reconstructed image sequences. Traditional sensors become completely ineffective at frame rates ≤2fps. Through experimental verification, this invention can achieve accurate extraction of motion information under extreme imaging conditions with an illumination of 1.2 lux.

[0009] The objective of this invention is specifically achieved through the following technical solutions:

[0010] This invention discloses a method for magnifying minute motion in image sequences based on event stream reconstruction, the method comprising:

[0011] Step 1: Convert the original event stream data of the target in the task into a grayscale image sequence, and perform Fourier transform on the grayscale image sequence to obtain a frequency domain subband set containing the amplitude component set and the phase component set;

[0012] Step 2: The complex controllable pyramid algorithm is introduced to decompose the frequency domain subband set to obtain a feature subband set with multi-scale and multi-directional joint characteristics, and which separates the phase component set and the amplitude component set.

[0013] Step 3: Perform S-transform time-frequency filtering on the phase component set separated from the feature sub-band set to obtain the phase component set in the time-frequency domain. Motion enhancement is achieved by calculating the phase difference between the previous and next frames in the phase component set in the time-frequency domain and multiplying it by a linear amplification factor, resulting in the enhanced phase component set.

[0014] Step 4: The enhanced phase component set and the amplitude component set separated from the corresponding feature sub-band set are fused one by one to obtain the reconstructed frequency domain sub-band set. The spatial resolution of the reconstructed frequency domain sub-band set is restored to be consistent with the grayscale image sequence through iterative reconstruction using high-pass and low-pass filters, thus obtaining the reconstructed frequency domain sub-band set with restored spatial resolution.

[0015] Step 5: After converting the reconstructed frequency domain subband set that restores spatial resolution to the time domain using inverse Fourier transform, a sequence of magnified images of minute motions is generated and output.

[0016] In step one, the method for performing a Fourier transform on the grayscale image sequence to obtain the frequency domain subband set containing the amplitude component set and the phase component set is as follows:

[0017] ;

[0018] In the formula, It is a set of frequency domain subbands, including frequency domain complex signals. For frequency domain coordinates, Let n be the discrete-time index of the current frame, and n be the frame index in the grayscale image sequence. To perform a Fourier transform on a grayscale image sequence, For Fourier transform, It is a grayscale image sequence. For spatial coordinates, It is a natural constant. For complex units, This represents the number of pixels in the grayscale image in the horizontal direction. is the number of pixels in the grayscale image in the vertical direction; where,

[0019] ;

[0020] In the formula, For the set of amplitude components, It is the set of phase components.

[0021] In step two, the method of introducing the complex-operable pyramid algorithm to decompose the frequency domain subband set to obtain a characteristic subband set with multi-scale and multi-directional joint characteristics, and which separates the phase component set and the amplitude component set, includes:

[0022] Step S1: Initially decompose the frequency domain subband set using the high-pass and low-pass filters in the complex controllable pyramid algorithm to obtain the high-frequency component set and the low-frequency component set.

[0023] Step S2: Apply a bandpass filter bank with direction selectivity to the low-frequency component set at four directional angles of 0°, 45°, 90° and 135° to generate a characteristic subband set with direction selectivity. At the same time, the low-frequency component set is decomposed by a second-level low-pass filter with a cutoff frequency lower than that of the low-pass filter to obtain a second-level low-frequency component set.

[0024] Step S3: Repeat steps S1 and S2 to perform the decomposition operation until the preset number of scale layers is reached, then stop the decomposition operation to form a feature sub-band set with multi-scale and multi-directional joint characteristics, and which separates the phase component set and the amplitude component set.

[0025] In step S1, the initial decomposition formula is:

[0026] ;

[0027] In the formula, The first one obtained by processing the high-pass filter The set of high-frequency components at each frame sampling time. The first result obtained by processing a low-pass filter The set of low-frequency components at each frame sampling time. The number of scale layers, , To preset the number of layers, here .

[0028] In step S2, the method for generating the set of feature subbands with directional selectivity is as follows:

[0029] ;

[0030] In the formula, For having a direction angle The set of characteristic subbands, It is a bandpass filter bank with direction selectivity. Let be the direction angle, where ;

[0031] The calculation method for the second-order low-frequency component set is as follows:

[0032] ;

[0033] In the formula, It is a set of second-order low-frequency components. It is a two-stage low-pass filter.

[0034] In step S3, the characteristic sub-band set that has multi-scale and multi-directional joint characteristics and separates the phase component set and amplitude component set is as follows:

[0035] ;

[0036] In the formula, For the first Layer scale, The set of characteristic subbands in the direction, For the separated first Layer scale, The set of amplitude components in the direction; For the separated first Layer scale, The set of phase components in the direction.

[0037] Step three involves performing S-transform time-frequency filtering on the set of phase components separated from the feature sub-band set to obtain the set of phase components in the time-frequency pass domain. This includes:

[0038] An S-transform is applied to the set of phase components separated from the feature subband set to establish a two-dimensional time-frequency distribution matrix of the signal. A time-frequency filtering operator is then used to preserve the set of phase components within the time-frequency domain of the two-dimensional time-frequency distribution matrix. The two-dimensional time-frequency distribution matrix of the signal is:

[0039] ;

[0040] In the formula, Let f be the two-dimensional time-frequency distribution matrix of the signal, where f is the frequency; This represents the total number of frames in the video sequence. For the separated first Layer scale, Direction, number The set of phase components at each frame sampling time. For the discrete-time index of the historical frame, representing the first... The time point of the frame, This is a time index used to determine the position of the Gaussian window on the time axis. It is a Gaussian window function. This refers to the time delay between the formation of historical frames and the current frame.

[0041] The set of phase components in the time-frequency domain of the two-dimensional time-frequency distribution matrix is ​​as follows:

[0042] ;

[0043] In the formula, It is the set of phase components in the time-frequency pass domain of the two-dimensional time-frequency distribution matrix. For S-transform, Represents the inverse S-transform. It is a time-frequency filtering operator. R is the time-frequency domain.

[0044] In step three, the method for calculating the enhanced phase component set is as follows:

[0045] ;

[0046] In the formula, For the enhanced phase component set, The second time-frequency distribution matrix is ​​the first time-frequency distribution matrix in the time-frequency domain. The set of phase components at each frame sampling time. For the first Discrete-time index of frames, This is the linear amplification factor.

[0047] In step four, the method for fusing the enhanced phase component set and the amplitude component set separated from the corresponding characteristic sub-band set sub-band by sub-band to obtain the reconstructed frequency domain sub-band set is as follows:

[0048] ;

[0049] In the formula, To reconstruct the set of frequency domain subbands, including the reconstructed frequency domain complex signals;

[0050] The set of reconstructed frequency domain subbands for restoring spatial resolution is as follows:

[0051] ;

[0052] In the formula, A set of reconstructed frequency domain subbands that restore spatial resolution to a grayscale image sequence; The iteration obtained by processing the low-pass filter is up to the 1st The first layer scale The set of low-frequency components at the frame sampling time, where, correspond direction, correspond direction, correspond direction, correspond The direction.

[0053] In step five, the method for generating and outputting a sequence of magnified images of minute motions after transforming the reconstructed frequency domain subband set that restores spatial resolution to the time domain using inverse Fourier transform is as follows:

[0054] ;

[0055] In the formula, A sequence of magnified images of minute movements. This is the inverse Fourier transform.

[0056] The beneficial effects of this invention are:

[0057] 1. This invention introduces a complex manipulable pyramid algorithm to decompose the frequency domain subband set, obtaining a feature subband set with multi-scale and multi-directional joint characteristics, and separating the phase component set and the amplitude component set; compared with the Laplace pyramid, which can only obtain multi-scale features, the complex manipulable pyramid, while preserving scale features, realizes directional dimension feature decomposition through a manipulable directional filter bank, thus forming a scale-directional feature subband set with directional selectivity.

[0058] 2. This invention constructs an effective noise suppression and feature enhancement mechanism, and proposes an S-transform time-frequency filtering method. By utilizing the energy distribution characteristics of the signal in the time-frequency domain, the influence of noise is suppressed to a great extent, the noise is optimized, and the accurate phase characterization capability is maintained. This enables the analysis of non-stationary motion in the image sequence reconstructed from the event stream, significantly improving the image clarity and reducing the influence of artifacts.

[0059] 3. By using S-transform time-frequency filtering, noise effects are effectively suppressed, and accurate phase characterization is maintained. After analyzing non-stationary motion, the phase difference between the previous and next frames is calculated and multiplied by a linear amplification factor to achieve motion enhancement.

[0060] 4. Under extreme imaging conditions with an illumination of 1.2 lux, and under adverse observation conditions such as low light and strong light, the present invention can achieve accurate extraction of motion information.

[0061] 5. This invention solves the technical problems of low signal-to-noise ratio and weak motion features in event stream reconstructed images, which make it difficult to clearly present the details of minute movements during motion magnification, resulting in motion blur, misjudgment, and other technical issues. It can accurately extract motion information from poor reconstructed images. Attached Figure Description

[0062] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments.

[0063] Figure 1 This is a schematic diagram of a satellite model provided in this invention.

[0064] Figure 2 This is a schematic diagram illustrating the result of amplifying minute motions using the PVMM method, as provided in this invention example.

[0065] Figure 3 yes Figure 2 A magnified view of the part marked ①.

[0066] Figure 4 yes Figure 2 A magnified view of the part labeled ②.

[0067] Figure 5 This is a schematic diagram illustrating the result of using a method for magnifying minute motion based on an event stream-based reconstruction of an image sequence to perform minute motion magnification, as provided in this invention example.

[0068] Figure 6 yes Figure 5 A magnified view of the part marked ①.

[0069] Figure 7 yes Figure 5 A magnified view of the part labeled ②.

[0070] Figure 8 This is a schematic diagram of the illuminance measurement results using the Teens TA8121 digital illuminance meter in an experimental environment. Detailed Implementation

[0071] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0072] This invention provides a method for magnifying minute motion in image sequences based on event stream reconstruction, the method comprising:

[0073] Step 1: Convert the original event stream data of the target in the task into a grayscale image sequence, and perform Fourier transform on the grayscale image sequence to obtain a frequency domain subband set containing the amplitude component set and the phase component set;

[0074] Step 2: The complex controllable pyramid algorithm is introduced to decompose the frequency domain subband set to obtain a feature subband set with multi-scale and multi-directional joint characteristics, and which separates the phase component set and the amplitude component set.

[0075] Step 3: Perform S-transform time-frequency filtering on the phase component set separated from the feature sub-band set to obtain the phase component set in the time-frequency domain. Motion enhancement is achieved by calculating the phase difference between the previous and next frames in the phase component set in the time-frequency domain and multiplying it by a linear amplification factor, resulting in the enhanced phase component set.

[0076] Step 4: The enhanced phase component set and the amplitude component set separated from the corresponding feature sub-band set are fused one by one to obtain the reconstructed frequency domain sub-band set. The spatial resolution of the reconstructed frequency domain sub-band set is restored to be consistent with the grayscale image sequence through iterative reconstruction using high-pass and low-pass filters, thus obtaining the reconstructed frequency domain sub-band set with restored spatial resolution.

[0077] Step 5: After converting the reconstructed frequency domain subband set that restores spatial resolution to the time domain using inverse Fourier transform, a sequence of magnified images of minute motions is generated and output.

[0078] In step one, the method of converting the original event stream data of the target in the task into a grayscale image sequence can be achieved by using the event stream image reconstruction method.

[0079] For example, the e2vid event stream image reconstruction method can be used to convert the original event stream data of the target in the task into a grayscale image sequence. This step transforms the target's raw event stream data into a visible light image sequence.

[0080] In step one, the method for performing a Fourier transform on the grayscale image sequence to obtain the frequency domain subband set containing the amplitude component set and the phase component set is as follows:

[0081] ;

[0082] In the formula, It is a set of frequency domain subbands, including frequency domain complex signals. For frequency domain coordinates, Let n be the discrete-time index of the current frame, and n be the frame index in the grayscale image sequence. To perform a Fourier transform on a grayscale image sequence, For Fourier transform, It is a grayscale image sequence. For spatial coordinates, is a natural constant, with a value of approximately 2.71828. For complex units, , This represents the number of pixels in the grayscale image in the horizontal direction. is the number of pixels in the grayscale image in the vertical direction; where,

[0083] ;

[0084] In the formula, For the set of amplitude components, It is the set of phase components.

[0085] In step two, the method of introducing the complex-operable pyramid algorithm to decompose the frequency domain subband set to obtain a characteristic subband set with multi-scale and multi-directional joint characteristics, and which separates the phase component set and the amplitude component set, includes:

[0086] Step S1: Initially decompose the frequency domain subband set using the high-pass and low-pass filters in the complex controllable pyramid algorithm to obtain the high-frequency component set and the low-frequency component set.

[0087] Step S2: Apply a bandpass filter bank with direction selectivity to the low-frequency component set at four directional angles of 0°, 45°, 90° and 135° to generate a characteristic subband set with direction selectivity. At the same time, the low-frequency component set is decomposed by a second-level low-pass filter with a cutoff frequency lower than that of the low-pass filter to obtain a second-level low-frequency component set.

[0088] Step S3: Repeat steps S1 and S2 to perform the decomposition operation until the preset number of scale layers is reached, then stop the decomposition operation to form a feature sub-band set with multi-scale and multi-directional joint characteristics, and which separates the phase component set and the amplitude component set.

[0089] In step S1, the initial decomposition formula is:

[0090] ;

[0091] In the formula, The first one obtained by processing the high-pass filter The set of high-frequency components at each frame sampling time. The first result obtained by processing a low-pass filter The set of low-frequency components at each frame sampling time. The number of scale layers, , To preset the number of layers, here .

[0092] In step S2, the method for generating the set of feature subbands with directional selectivity is as follows:

[0093] ;

[0094] In the formula, For having a direction angle The set of characteristic subbands, It is a bandpass filter bank with direction selectivity. Let be the direction angle, where ;

[0095] The calculation method for the second-order low-frequency component set is as follows:

[0096] ;

[0097] In the formula, It is a set of second-order low-frequency components. It is a two-stage low-pass filter.

[0098] In step S3, the characteristic sub-band set that has multi-scale and multi-directional joint characteristics and separates the phase component set and amplitude component set is as follows:

[0099] ;

[0100] In the formula, For the first Layer scale, The set of characteristic subbands in the direction, For the separated first Layer scale, The set of amplitude components in the direction; For the separated first Layer scale, The set of phase components in the direction.

[0101] Step three involves performing S-transform time-frequency filtering on the set of phase components separated from the feature sub-band set to obtain the set of phase components in the time-frequency pass domain. This includes:

[0102] An S-transform is applied to the set of phase components separated from the feature subband set to establish a two-dimensional time-frequency distribution matrix of the signal. A time-frequency filtering operator is then used to preserve the set of phase components within the time-frequency domain of the two-dimensional time-frequency distribution matrix. The two-dimensional time-frequency distribution matrix of the signal is:

[0103] ;

[0104] In the formula, Let f be the two-dimensional time-frequency distribution matrix of the signal, where f is the frequency; This represents the total number of frames in the video sequence. For the separated first Layer scale, Direction, number The set of phase components at each frame sampling time. For the discrete-time index of the historical frame, representing the first... The time point of the frame, This is a time index used to determine the position of the Gaussian window on the time axis. It is a Gaussian window function. This refers to the time delay between the formation of historical frames and the current frame.

[0105] The set of phase components in the time-frequency domain of the two-dimensional time-frequency distribution matrix is ​​as follows:

[0106] ;

[0107] In the formula, It is the set of phase components in the time-frequency pass domain of the two-dimensional time-frequency distribution matrix. For S-transform, Represents the inverse S-transform. It is a time-frequency filtering operator. R is the time-frequency domain.

[0108] In step three, the method for calculating the enhanced phase component set is as follows:

[0109] ;

[0110] In the formula, For the enhanced phase component set, The second time-frequency distribution matrix is ​​the first time-frequency distribution matrix in the time-frequency domain. The set of phase components at each frame sampling time. For the first Discrete-time index of frames, This is the linear amplification factor.

[0111] In step four, the method for fusing the enhanced phase component set and the amplitude component set separated from the corresponding characteristic sub-band set sub-band by sub-band to obtain the reconstructed frequency domain sub-band set is as follows:

[0112] ;

[0113] In the formula, To reconstruct the set of frequency domain subbands, including the reconstructed frequency domain complex signals;

[0114] The set of reconstructed frequency domain subbands for restoring spatial resolution is as follows:

[0115] ;

[0116] In the formula, A set of reconstructed frequency domain subbands that restore spatial resolution to a grayscale image sequence; The iteration obtained by processing the low-pass filter is up to the 1st The first layer scale The set of low-frequency components at the frame sampling time, where, correspond direction, correspond direction, correspond direction, correspond The direction.

[0117] In step five, the method for generating and outputting a sequence of magnified images of minute motions after transforming the reconstructed frequency domain subband set that restores spatial resolution to the time domain using inverse Fourier transform is as follows:

[0118] ;

[0119] In the formula, A sequence of magnified images of minute movements. This is the inverse Fourier transform.

[0120] To verify the effectiveness of the technical solution of the present invention, a specific example is provided for illustration:

[0121] The objectives in the task are as follows Figure 1 As shown; in a darkroom environment, the sample was collected as follows Figure 1 The original event stream data of the satellite model shown is converted into a grayscale image sequence using the e2vid event stream image reconstruction method. The e2vid event stream image reconstruction method is the method described in the 2019 article "High Speed ​​and High Dynamic Range Video with an Event Camera" in the IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINEINTELLIGENCE journal.

[0122] This example compares the amplification effect of the existing Phase-Based Video Motion Magnification (PVMM) method with the method disclosed in this invention. The PVMM method uses Fourier series expansion to decompose the signal, representing the target motion as sinusoidal components of different frequencies, thereby realizing the extraction of the motion of interest.

[0123] The results of amplifying minute motions using the PVMM method and their magnified local images are shown below. Figure 2 , Figure 3 and Figure 4 As shown; the results of micro-motion magnification using the event-stream-based image sequence reconstruction method disclosed in this invention, and its magnified local images are shown below. Figure 5 , Figure 6 and Figure 7 As shown.

[0124] In the motion magnification comparison of event stream reconstructed images, the method disclosed in this invention significantly separates noise in the reconstructed image due to the S-transform time-frequency filtering, and only amplifies the frequency of interest, thereby significantly suppressing noise, significantly improving image clarity, and reducing the influence of artifacts.

[0125] During the experimental data acquisition process, this invention used a Teens TA8121 digital illuminance meter to measure illuminance under experimental conditions. The illuminance measurement results are as follows: Figure 8As shown, the results indicate that when traditional sensors become completely ineffective at frame rates ≤ 2fps, the method disclosed in this invention can amplify and accurately extract motion information under extreme imaging conditions with an illumination of 1.2 lux. This invention has significant application potential for target motion detection in low-light conditions in space environments.

[0126] The beneficial effects of the embodiments of the present invention are:

[0127] 1. This invention introduces a complex manipulable pyramid algorithm to decompose the frequency domain subband set, obtaining a feature subband set with multi-scale and multi-directional joint characteristics, and separating the phase component set and the amplitude component set; compared with the Laplace pyramid, which can only obtain multi-scale features, the complex manipulable pyramid, while preserving scale features, realizes directional dimension feature decomposition through a manipulable directional filter bank, thus forming a scale-directional feature subband set with directional selectivity.

[0128] 2. This invention constructs an effective noise suppression and feature enhancement mechanism, and proposes an S-transform time-frequency filtering method. By utilizing the energy distribution characteristics of the signal in the time-frequency domain, the influence of noise is suppressed to a great extent, the noise is optimized, and the accurate phase characterization capability is maintained. This enables the analysis of non-stationary motion in the image sequence reconstructed from the event stream, significantly improving the image clarity and reducing the influence of artifacts.

[0129] 3. By using S-transform time-frequency filtering, noise effects are effectively suppressed, and accurate phase characterization is maintained. After analyzing non-stationary motion, the phase difference between the previous and next frames is calculated and multiplied by a linear amplification factor to achieve motion enhancement.

[0130] 4. Under extreme imaging conditions with an illumination of 1.2 lux, and under adverse observation conditions such as low light and strong light, the present invention can achieve accurate extraction of motion information.

[0131] 5. This invention solves the technical problems of low signal-to-noise ratio and weak motion features in event stream reconstructed images, which make it difficult to clearly present the details of minute movements during motion magnification, resulting in motion blur, misjudgment, and other technical issues. It can accurately extract motion information from poor reconstructed images.

[0132] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for magnifying minute motion in image sequences based on event stream reconstruction, characterized in that, The method includes: Step 1: Convert the original event stream data of the target in the task into a grayscale image sequence, and perform Fourier transform on the grayscale image sequence to obtain a frequency domain subband set containing the amplitude component set and the phase component set; Step two involves introducing the complex-operable pyramid algorithm to decompose the frequency domain subband set, obtaining a feature subband set with multi-scale and multi-directional joint characteristics, and separating the phase component set and the amplitude component set. The method includes: Step S1: Initially decompose the frequency domain subband set using the high-pass and low-pass filters in the complex controllable pyramid algorithm to obtain the high-frequency component set and the low-frequency component set. Step S2: Apply a bandpass filter bank with direction selectivity to the low-frequency component set at four directional angles of 0°, 45°, 90° and 135° to generate a characteristic subband set with direction selectivity. At the same time, the low-frequency component set is decomposed by a second-level low-pass filter with a cutoff frequency lower than that of the low-pass filter to obtain a second-level low-frequency component set. Step S3: Repeat steps S1 and S2 to perform the decomposition operation until the preset number of scale layers is reached, then stop the decomposition operation to form a feature sub-band set with multi-scale and multi-directional joint characteristics, and which separates the phase component set and the amplitude component set. Step 3: Perform S-transform time-frequency filtering on the phase component set separated from the feature sub-band set to obtain the phase component set in the time-frequency domain. Motion enhancement is achieved by calculating the phase difference between the previous and next frames in the phase component set in the time-frequency domain and multiplying it by a linear amplification factor, resulting in the enhanced phase component set. Step 4: The enhanced phase component set and the amplitude component set separated from the corresponding feature sub-band set are fused one by one to obtain the reconstructed frequency domain sub-band set. The spatial resolution of the reconstructed frequency domain sub-band set is restored to be consistent with the grayscale image sequence through iterative reconstruction using high-pass and low-pass filters, thus obtaining the reconstructed frequency domain sub-band set with restored spatial resolution. Step 5: After converting the reconstructed frequency domain subband set that restores spatial resolution to the time domain using inverse Fourier transform, a sequence of magnified images of minute motions is generated and output.

2. The method as described in claim 1, characterized in that, In step one, the method for performing a Fourier transform on the grayscale image sequence to obtain the frequency domain subband set containing the amplitude component set and the phase component set is as follows: ； In the formula, It is a set of frequency domain subbands, including frequency domain complex signals. For frequency domain coordinates, Let n be the discrete-time index of the current frame, and n be the frame index in the grayscale image sequence. To perform a Fourier transform on a grayscale image sequence, For Fourier transform, It is a grayscale image sequence. For spatial coordinates, It is a natural constant. For complex units, This represents the number of pixels in the grayscale image in the horizontal direction. is the number of pixels in the grayscale image in the vertical direction; where, ； In the formula, For the set of amplitude components, It is the set of phase components.

3. The method as described in claim 1 or 2, characterized in that, In step S1, the initial decomposition formula is: ； In the formula, The first one obtained by processing the high-pass filter The set of high-frequency components at each frame sampling time. The first result obtained by processing a low-pass filter The set of low-frequency components at each frame sampling time. The number of scale layers, , To preset the number of layers, here .

4. The method as described in claim 3, characterized in that, In step S2, the method for generating the set of feature subbands with directional selectivity is as follows: ； In the formula, For having a direction angle The set of characteristic subbands, It is a bandpass filter bank with direction selectivity. Let be the direction angle, where ; The calculation method for the second-order low-frequency component set is as follows: ； In the formula, It is a set of second-order low-frequency components. It is a two-stage low-pass filter.

5. The method as described in claim 4, characterized in that, In step S3, the characteristic sub-band set that has multi-scale and multi-directional joint characteristics and separates the phase component set and amplitude component set is as follows: ； In the formula, For the first Layer scale, The set of characteristic subbands in the direction, For the separated first Layer scale, The set of amplitude components in the direction; For the separated first Layer scale, The set of phase components in the direction.

6. The method as described in claim 5, characterized in that, Step three involves performing S-transform time-frequency filtering on the set of phase components separated from the feature sub-band set to obtain the set of phase components in the time-frequency pass domain. This includes: An S-transform is applied to the set of phase components separated from the feature subband set to establish a two-dimensional time-frequency distribution matrix of the signal. A time-frequency filtering operator is then used to preserve the set of phase components within the time-frequency domain of the two-dimensional time-frequency distribution matrix. The two-dimensional time-frequency distribution matrix of the signal is: ； In the formula, Let f be the two-dimensional time-frequency distribution matrix of the signal, where f is the frequency; This represents the total number of frames in the video sequence. For the separated first Layer scale, Direction, number The set of phase components at each frame sampling time. For the discrete-time index of the historical frame, representing the first... The time point of the frame, This is a time index used to determine the position of the Gaussian window on the time axis. It is a Gaussian window function. This refers to the time delay between the formation of historical frames and the current frame. The set of phase components in the time-frequency domain of the two-dimensional time-frequency distribution matrix is ​​as follows: ； In the formula, It is the set of phase components in the time-frequency pass domain of the two-dimensional time-frequency distribution matrix. For S-transform, Represents the inverse S-transform. It is a time-frequency filtering operator. R is the time-frequency domain.

7. The method as described in claim 6, characterized in that, In step three, the method for calculating the enhanced phase component set is as follows: ； In the formula, For the enhanced phase component set, The second time-frequency distribution matrix is ​​the first time-frequency distribution matrix in the time-frequency domain. The set of phase components at each frame sampling time. For the first Discrete-time index of frames, This is the linear amplification factor.

8. The method as described in claim 7, characterized in that, In step four, the method for fusing the enhanced phase component set and the amplitude component set separated from the corresponding characteristic sub-band set sub-band by sub-band to obtain the reconstructed frequency domain sub-band set is as follows: ； In the formula, To reconstruct the set of frequency domain subbands, including the reconstructed frequency domain complex signals; The set of reconstructed frequency domain subbands for restoring spatial resolution is as follows: ； In the formula, A set of reconstructed frequency domain subbands that restore spatial resolution to a grayscale image sequence; The iteration obtained by processing the low-pass filter is up to the th The first layer scale The set of low-frequency components at the frame sampling time, where, correspond direction, correspond direction, correspond direction, correspond The direction.

9. The method as described in claim 8, characterized in that, In step five, the method for generating and outputting a sequence of magnified images of minute motions after transforming the reconstructed frequency domain subband set that restores spatial resolution to the time domain using inverse Fourier transform is as follows: ； In the formula, A sequence of magnified images of minute movements. This is the inverse Fourier transform.

Images