An adaptive micro-motion highlighting system

Abstract

The present invention discloses a method for adaptively dividing Fourier complex space and performing micro-motion highlighting. First, the collected video is subjected to non-negative matrix decomposition denoising and visual equalization preprocessing; then, the different micro-motion components in the video are subjected to variational modal decomposition, and the video is split into sub-modal videos containing only a single motion component according to a predetermined number of decompositions, and then micro-motion highlighting processing is performed respectively; finally, the micro-motion highlighting processing module filters the video frames of each sub-modal video at different scales, extracts the micro-motion frequency band of interest, performs phase amplification and inverse scale transformation, and superimposes and reconstructs the same frame after multiple modal inverse transformations to obtain a micro-motion highlighted video, thereby achieving the effect of adaptive micro-motion highlighting of the video, and providing a new tool for the problem of highlighting multiple micro-motion components in various videos.

Description

Technical Field

[0001] The present invention relates to the field of video processing and in particular to a method for highlighting minute movements. Background Art

[0002] In everyday life, there are countless tiny movements captured by cameras. Although these pixel-level movements are nearly invisible to the human eye, they often reveal important properties and intrinsic characteristics of the object. For example, the periodic microvascular bulges in the carotid artery are closely related to the pulsation of blood supply to the human brain and can reflect vital signs. The tiny fluctuations caused by the breathing of organisms at the scene of a natural disaster provide hope for immediate rescue. The tiny vibrations of bridges and buildings under load greatly facilitate non-contact analysis of their structural dynamic properties. Furthermore, the tiny movements of microorganisms, human cells, and tissues in the microscopic world provide a new analytical perspective for uncovering the mysteries of life.

[0003] Currently, the most direct way to observe these subtle movements is to summarize the target's motion patterns through long-term comparative observation. However, this method is not conducive to timely obtaining effective information from subtle movements, and its accuracy cannot be guaranteed. Therefore, it is necessary to use video processing technology to automatically and timely analyze these subtle movements. In the field of video processing, there has been some previous work on subtle movement highlighting, but it still faces a bottleneck: the phase-based subtle movement highlighting technology proposed by previous researchers (N Wadhwa et al. 2013) is effective for videos containing specific motion components, but is more limited for videos containing multiple subtle movement components. Naturally captured videos often contain more than one motion component, and these components are reflected as different frequencies and directions in the Fourier complex space of each video frame. Therefore, a video processing method that can adaptively highlight multiple subtle movement components is urgently needed. Summary of the Invention

[0004] To adaptively highlight the different subtle motion components present in naturally captured videos, the present invention, based on non-negative matrix factorization and variational mode decomposition, decomposes the pre-processed original video into multiple sub-modal videos containing a single subtle motion component, then highlights the motion component in each sub-modal video, and then fuses each subtle motion-highlighted sub-video to produce a subtle motion-highlighted video. The technical solution of the present invention is as follows:

[0005] An adaptive small motion highlighting system, such as Figure 1 and Figure 2 As shown, it consists of the following modules:

[0006] 1) Video acquisition module, collecting original video data V1;

[0007] 2) The preprocessing module performs three preprocessing steps on the original video data V1: non-negative matrix factorization denoising, video filtering, and normalization, and passes the preprocessed video data V2 to the variational mode decomposition module;

[0008] 3) Variational modal decomposition module, which performs variational modal decomposition on each frame of image V2 to obtain k (k = 1, 2...N, N is a positive integer) sub-modal videos V 3k , and V 3k Send to the micro-motion highlighting module;

[0009] 4) Micro-motion highlighting module, for each V 3k The micro-motion highlighting is performed separately, and then each micro-motion highlighting sub-video is fused to obtain the micro-motion highlighting video V4.

[0010] Preferably, in order to satisfy the system's ability to uniformly process different types of natural videos, a non-negative matrix decomposition method is used to denoise the video, that is, the video signal is converted into a two-dimensional space-time matrix M1, the number of rows of M1 is: W×L, the number of columns of M1 is F, where W is the width of the video, L is the height of the video, and F is the number of frames of the video; M1 is non-negative matrix decomposition, decomposed into two two-dimensional matrices, one of which is M2 with a size of (W×L)×Q, and the other is M3 with a size of Q×F, and all element values ​​in M2 and M3 are non-negative real numbers, where Q is a priori hyperparameter; the two decomposed matrices M2 and M3 are multiplied to obtain the denoised space-time matrix M4, and M4 is transposed to obtain a three-dimensional matrix M5 of W×L×F, M5 is the denoised video; the equalization processing method is histogram equalization.

[0011] Preferably, in order to separate different action components in the video and perform variational mode decomposition, the variational mode decomposition module is composed of a video splitting unit, a variational iteration unit, and a video reconstruction unit; the video splitting unit splits the video data V2 into image frames P m (m=1,2,…,M; M is the number of video frames); the variational iterative unit converts the image frame P m Perform Hilbert transform to obtain the analytical signal R m , R m According to the assumed center frequency ω of each mode k Perform spectrum shifting to obtain the corresponding baseband signal Then establish the objective function based on the minimum bandwidth and principle The optimization operator adopts the alternating direction multiplier algorithm, and finally the objective function is iteratively optimized to solve the problem, thereby splitting the original signal of the image frame into k (k = 1, 2...N; N is a positive integer, K is the prior information) sub-modal signals P with different center frequencies mk, the video reconstruction unit will have the same k value P mk Reconstructed into submodal videos V in time sequence 3k .

[0012] Preferably, in order to highlight the action of the video containing different micro-action components after decomposition, the micro-action highlighting module is composed of a scale transformation group, a filter, a phase amplifier, and an inverse scale transformation group; the scale transformation group transforms the video data V 3k The video data are converted into video data of different scales, and the scale transformation method is one of wavelet transform, Gaussian pyramid decomposition, and steerable direction pyramid transform; the signal filter receives the video data of different scales, and based on a preset bandpass range, performs signal filtering on the video data of different scales and sends it to the phase amplifier; the phase amplifier receives the signal filtering result, multiplicatively amplifies the phase thereof and sends it to the inverse scale transformation group; the inverse scale transformation group receives the phase amplification result of different scales, and performs inverse scale reconstruction on the phase amplification result to obtain the video data V4 with slight movement highlighting.

[0013] The present invention has the following beneficial effects: the sub-modes in the Fourier space are adaptively divided through non-negative matrix decomposition and variational mode decomposition, so that the existing various small motion components can be effectively adaptively highlighted. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 This is a general processing flow chart of the system of the present invention for highlighting small movements in a video;

[0015] Figure 2 This is a detailed processing flow chart of the system of the present invention for highlighting small movements in a video;

[0016] Figure 3 is the first frame image of the simulation video in the embodiment of the present invention and the corresponding position of each component in the Fourier complex space;

[0017] Figure 4 It is a comparison diagram of space-time slices after the simulation video is pre-processed in the embodiment of the present invention, without applying any processing, highlighting by the previous method, and highlighting by the method of the present invention. DETAILED DESCRIPTION

[0018] The present invention is further described below by way of examples to facilitate a better understanding of the technical solution of the present invention, but the present invention is not limited thereto. The video data and other resources used in the following examples, unless otherwise specified, can be obtained from various conventional video acquisition devices.

[0019] 1. Build a simulation video

[0020] In order to better illustrate the effect of the system in adaptively highlighting different component actions compared with the gold standard, a simulated video is incorporated into this embodiment to replace the original video data V1 obtained by the video acquisition module. Figure 3 The first frame of the simulation video and the corresponding positions of its components in Fourier complex space are shown. Due to the symmetry of Fourier space, we can ignore Figure 3 The left half of the right figure specifies Figure 3 In the right image, the dark gray part of the center area of ​​the rectangle is mode one, the dark gray part in the upper right corner is mode two, and the dark gray part in the lower right corner is mode three, for a total of three modes. Therefore, the number of subsequent prior modes is set to 3, that is, there are three different motion components. In order to maintain generality, the video also contains three different small motion components: the white ellipse in the upper left corner of the video and the gray rectangle below (both mode one) are the overall reciprocating vertical vibration, the wide striped ellipse on the left (mode two) is the overall reciprocating horizontal vibration, and the thin striped ellipse on the right (mode three) is the overall reciprocating vertical vibration. Because the direction and intensity of the grayscale changes at the edges of the three components are different, the direction and frequency in the Fourier complex space after Fourier transform of each frame are different. The frequency of all vibrations is 3Hz, and the vibration amplitude changes by 1 pixel along each direction. The total number of video frames is 30 frames, and the sampling frequency is 30Hz. A change of one pixel is relatively small relative to the 256×256 aspect ratio, making it difficult for the naked eye to detect such subtle changes in video. Furthermore, compared to actual video, simulated video exhibits known and quantitatively small changes. Therefore, it is more suitable as a gold standard for validating the effects of this patented invention than actual recorded video.

[0021] 2. Video Preprocessing

[0022] In this example, simulated video is used to replace the original video data V1 acquired by the acquisition device. Non-negative matrix factorization denoising, video filtering, and video normalization are performed sequentially. The video width is 256 pixels, the height is 256 pixels, and the frame rate is 30. First, the video signal is converted into a two-dimensional spatiotemporal matrix M1 with 256×256 rows and 30 columns. Non-negative matrix factorization is then performed on M1, decomposing it into two two-dimensional matrices: one matrix M2 is 256×256×5, and the other matrix M3 is 5×30. All elements in M2 and M3 are non-negative real numbers, where 5 is a manually selected hyperparameter. The two decomposed matrices M2 and M3 are multiplied to obtain the denoised spatiotemporal matrix M4. M4 is then transposed to obtain a three-dimensional matrix M5 of 256×256×30, which is the denoised video. Finally, by selecting histogram equalization for visual normalization, the preprocessed video V2 is obtained.

[0023] 3. Variational Mode Decomposition

[0024] Furthermore, the pre-processed video is subjected to adaptive variational modal decomposition to obtain sub-modal videos containing micro-motion components of different frequencies and directions. First, the video splitting group splits V2 into 30 image frames P according to the number of video frames used in this embodiment. m (m=1,2,..,30). Observing the spectrum of the first frame of the simulation video, we find that there are 3 center frequencies, so the prior modal k value is set to 3. The variational iteration unit transforms the image frame P m Perform Hilbert transform to obtain the analytical signal R m , R m According to the assumed center frequency ω of each mode k Perform spectrum shifting to obtain the corresponding baseband signal Then establish the objective function based on the minimum bandwidth and principle The optimization operator adopts the alternating direction multiplier algorithm, and finally the objective function is iteratively optimized to solve the problem, thereby splitting the original signal of the image frame into three sub-modal frames P with different center frequencies. m1 、P m2 、P m3 (m=1,2,..,30). The boundary condition for stopping the iterative optimization is that the similarity between the sum of the sub-modal frame signals and the original signal is greater than the preset value δ (δ=1-10 -5 ). The similarity is calculated using the perceptual hash algorithm. Finally, the video reconstruction group merges and reconstructs each frame belonging to the same sub-modality to obtain three sub-videos V containing different frequency and direction components. 31 、V 32 、V 33 .

[0025] 4. Small movements stand out

[0026] After variational mode decomposition, the video V containing different motion components 31 、V 32 、V 33The images are processed for micro-motion highlighting respectively, and the processing steps include scale transformation, filtering, phase amplification and inverse scale transformation. The purpose of rescaling is to analyze image frames containing a single motion component at different scales, thereby decomposing each image frame into different frequency bands for processing and providing more accurate highlighting results. The rescaling method can be one of wavelet transform, Gaussian pyramid decomposition, and steerable directional pyramid transform. In this embodiment, steerable directional pyramid transform is used to decompose the image frames at multiple scales and multiple directions simultaneously, with the number of transform layers selected to be 1 / 2 of the maximum number of decomposition layers. The filter bank then receives the decomposition results of the image frames at different scales and extracts the components of interest according to a preset bandpass range of 0.1 to 5 Hz. The phase amplifier group extracts the phase information contained in each spatial scale image in each submodal video and each frame of the image pyramid. The phase information difference at the same scale between adjacent frames is the micro-motion occurring between two frames at that scale for a certain motion component. The phase difference is multiplicatively amplified by 10 times to obtain the phase after the micro-motion occurring between frames is highlighted, and the phase is sent to the inverse rescaling group for video reconstruction. The inverse rescaling group replaces the original phase information at each scale of each frame with the highlighted phase information. The steerable directional pyramid reconstruction is used to obtain the phase information of each frame after the motion is highlighted. m1d 、P m2d 、P m3d (m=1,2,..,30), the three components of each frame are superimposed to obtain the action-highlighted frame P md (m=1,2,..,30); Finally, P md (m=1,2,..,30) are reconstructed into the micro-motion highlighting video data V4 in chronological order.

[0027] Figure 4 Compared with the previous methods, the method of the present invention obtains a small movement highlighting effect. By taking three vertical single-column pixels a, b, c and one horizontal single-column pixel d in the video through dotted lines and unfolding them along the time axis and superimposing them into a picture respectively, the time-space slice diagrams of different parts can be obtained respectively. The horizontal axis of the time-space slice diagram corresponds to the time of the video, and the vertical axis corresponds to a column of pixels obtained by the dotted line. Since the directions of the four dotted lines are not completely consistent with the directions of the small movements contained therein, different highlighting effects will be observed on the four dotted lines due to different small movements.

[0028] The expected highlighting effect, the textual description of the highlighting effect obtained by the previous method and the present method are shown in the following table.

[0029]

[0030]

[0031] By comparison, it can be seen that the previous methods highlight a relatively single component and are not adaptive to the direction. Therefore, there are situations in which the highlighted direction is wrong and the highlighted content is omitted in the highlighted results. The method of the present invention highlights the three main action components, which is closest to the expected effect.

[0032] This embodiment obtains a simulation video containing different micro-motion components, and the components are expressed in Fourier complex space as follows: Figure 3 As shown, according to the processing method of the present invention, the micro-movements of different components in the video are adaptively decomposed and highlighted to obtain Figure 4 The results shown. Figure 3 is the first frame image of the simulated video in the embodiment of the present invention and the corresponding position of each component in the Fourier complex space, indicating that different components in the adopted video have different frequencies and directions in the Fourier complex space; Figure 4 This is a comparison of the spatiotemporal slices of the pre-processed video in an embodiment of the present invention, when the video is not processed, when the previous method is applied, and when the method of the present invention is applied before and after highlighting. It shows that the present invention can more effectively and adaptively highlight the multiple small motion components that exist simultaneously in the video compared with the previous method.

Claims

1. An adaptive micro-motion highlighting system, characterized in that: It consists of the following modules: Video acquisition module, collecting original video data V1; The preprocessing module performs denoising and equalization processing on the original video data V1 to obtain the preprocessed video data V2, and then passes V2 to the variational mode decomposition module; wherein the denoising method is non-negative matrix decomposition; The variational modal decomposition module performs variational modal decomposition on each frame of V2 to obtain k sub-modal videos V 3k , where k = 1, 2…N, N is a positive integer, and then V 3k The variational modal decomposition module is composed of a video splitting unit, a variational iteration unit, and a video reconstruction unit; the video splitting unit splits the video data V2 into image frames P m , where m = 1,2,…,M; M is the number of video frames; the variational iteration unit converts the image frame P m Perform Hilbert transform to get the analytical signal R m , R m According to the assumed center frequency ω of each mode k Perform spectrum shifting to obtain the corresponding baseband signal , and then establish the objective function based on the principle of minimum bandwidth and , the optimization operator adopts the alternating direction multiplier algorithm, and finally the objective function is iteratively optimized to solve the image frame in V2, thereby splitting the image frame in V2 into k sub-modal signals P with different center frequencies mk , where k = 1, 2...N; N is a positive integer, k is a priori information; the video reconstruction unit converts P mk Reconstructed into sub-modal videos V in time sequence 3k ; The micro-movement highlights the module, for each V 3k The micro-motion highlighting module is composed of a scale transformation group, a bandpass filter, a phase amplifier, and an inverse scale transformation group. The scale transformation group transforms the video data V 3k Converted into video data of multiple different scales, the scale transformation method is one of wavelet transform, Gaussian pyramid decomposition, and steerable directional pyramid transform; the bandpass filter receives video data of different scales, and based on a preset bandpass range, performs bandpass filtering on the video data of different scales and sends it to the phase amplifier; the phase amplifier receives the signal filtering result, multiplicatively amplifies its phase and sends it to the inverse scale transformation group; the inverse scale transformation group receives the phase amplification results of different scales, and performs inverse scale reconstruction on them to obtain the small motion highlighting video data V4.

2. The system according to claim 1, characterized in that The preprocessing module performs non-negative matrix decomposition denoising, video filtering and standardization on the original video data V1, and passes the processed video data V2 to the variational mode decomposition module; wherein, the video is denoised by a non-negative matrix decomposition method, that is, the video signal is converted into a two-dimensional space-time matrix M1, the number of rows of M1 is: W×L, the number of columns of M1 is F, where W is the width of the video, L is the height of the video, and F is the number of frames of the video; M1 is decomposed into two two-dimensional matrices by non-negative matrix decomposition, one of which is M2 with a size of (W×L)×Q, and the other is M3 with a size of Q×F, and all element values ​​in M2 and M3 are non-negative real numbers, where Q is a manually selected hyperparameter; the two decomposed matrices M2 and M3 are multiplied to obtain the denoised space-time matrix M4, and M4 is transposed to obtain a three-dimensional matrix M5 of W×L×F, and M5 is the denoised video; the equalization processing method is histogram equalization.

Images