A method for synthesizing one-dimensional motion curves in color video motion amplification

Abstract

A method for synthesizing one-dimensional motion curves in color video motion amplification is characterized by comprising the following steps: (1) reading in a section of color video data and performing preprocessing, wherein the preprocessed color video is denoted as b, the number of frames is N, and the frame rate is fs Hz; (2) extracting certain features of the R color image, the G color image, and the B color image of each frame of the color image respectively, and then generating three characteristic time series curves corresponding to the three colors from the color video respectively, denoted as r(n), g(n), b(n), n=0, 1, ..., N-1; (3) performing Fourier transform DFT on the three time series respectively, denoted as R(k), G(k), B(k), k=0, 1, ..., N-1; (4) selecting the spectrum segments of interest of R(k), G(k) and B(k) and their background spectrum segments according to prior knowledge; (5) assuming that the synthesis formula of the one-dimensional time series curve to be synthesized is rgb(n)=C r *r(n)+C g *g(n)+C b *b(n), the corresponding spectrum synthesis formula is RGB(k)=C r *R(k)+C g *G(k)+C b *B(k), then the combination coefficient C r , C g and C b It is obtained by solving a maximization optimization problem.

Description

Technical Field

[0001] The present application relates to a method for synthesizing a one-dimensional motion curve in color video motion magnification.

[0002] Humans are living organisms, with heartbeats, breathing, pulsating blood vessels, and even changes in surface color. Machinery and equipment often have motors rotating, which in turn causes surface vibrations. Monitoring their motion can be used to determine whether these moving individuals or equipment are functioning normally. Video monitoring offers the advantage of being non-contact, compared to wearable and contact monitoring. Some movements are more noticeable and easily observed in video, while other movements are subtle and difficult or unclear to detect in video, requiring software amplification. Motion amplification is a hot topic in video monitoring.

[0003] Video motion amplification often requires converting a moving image sequence into a one-dimensional motion curve. This curve is a time series with time on the horizontal axis and a characteristic value calculated for each frame on the vertical axis, reflecting the motion characteristics of the monitored object from a different perspective. The benefits of converting a moving image sequence into a one-dimensional motion curve include significantly reducing the amount of data, facilitating spectral and waveform analysis, and facilitating the estimation of parameters such as the subject's heart rate, respiratory rate, or device rotational speed. In some cases, the ultimate goal of video motion amplification is to obtain such a one-dimensional motion curve for motion parameter analysis. In other cases, the ultimate goal of video motion amplification is visual enhancement, requiring the display of the amplified video for human observation. This intermediate step—converting the moving image sequence into a one-dimensional motion curve—is also necessary to facilitate analysis and determination of filtering parameters for pixel-by-pixel temporal filtering.

[0004] Color video involves three primary colors, R, G and B, and three image sequences, R, G and B. When it is converted into a one-dimensional motion curve, if the three image sequences of R, G and B are first converted into one-dimensional motion curves respectively, there is still a problem of how to further combine the three one-dimensional motion curves into one one-dimensional motion curve. Background Art

[0005] A typical video motion amplification system includes a camera, a video transmission line, a computer, etc. The camera captures the video, and the computer reads the video and performs motion amplification processing.

[0006] The principle of video motion amplification processing, taking linear amplification as an example, can be described as follows: (1) If there is no motion component in the video, then the image sequence of each frame is a constant sequence, which does not need to be amplified; (2) If there is a motion component, then its image sequence is an image sequence that changes with time. Accordingly, a certain color value or brightness value of each pixel constitutes a one-dimensional time series, which has both DC and AC components. The DC component represents the unchanging static component in the sequence and does not need to be amplified, and the AC component represents the moving and changing part in the sequence; (3) Only part of the AC component may be the AC component of interest - which needs to be amplified; (4) Appropriate bandpass filtering is performed on each pixel color or brightness time series to extract the motion component of interest and amplify it, and then the corresponding value of the original sequence is added to achieve the amplification and enhancement of the motion component of interest in the video.

[0007] As mentioned previously, in some cases, the ultimate goal of video motion amplification is to obtain a one-dimensional motion curve, such as a pulse or heartbeat curve. To achieve this, the original color video often undergoes a series of preprocessing steps, including low-pass filtering and re-decimation, pixel-by-pixel temporal filtering, and amplification, before being converted into a one-dimensional curve. In some cases, while the ultimate goal is visual enhancement through motion amplification, the intermediate steps before temporal filtering and amplification involve preprocessing such as spatial filtering and image segmentation. The preprocessed color video is then converted into a one-dimensional motion curve for analysis in an attempt to determine the parameters of the temporal bandpass filter. We may denote the preprocessed color video in any of the above scenarios as b.

[0008] Converting the preprocessed color video b into a one-dimensional motion curve essentially creates a time series of color video image features. These image features can be the average R color value per frame, the average G color value per frame, or the average B color value per frame. Accordingly, a one-dimensional time series can be constructed: the R mean time series, the G mean time series, and the B mean time series, resulting in three basic one-dimensional time series curves. Finally, a one-dimensional motion curve must be synthesized.

[0009] The average brightness curve of a color video image is a synthesis of more than three basic time series curves, but it is a fixed linear synthesis with the synthesis formula of Y=0.2989*R+0.5870*G+0.1140*B. However, it may not be the optimal synthesis and may not be able to achieve the highest signal-to-noise ratio for the one-dimensional motion curve and its spectrum. Summary of the Invention

[0010] Purpose of the Invention

[0011] A synthesis method of one-dimensional motion curves in color video motion magnification is proposed to achieve optimal synthesis and generate one-dimensional motion curves and their spectrum with the highest signal-to-noise ratio.

[0012] Technical Solution

[0013] A method for synthesizing one-dimensional motion curves in color video motion amplification is characterized by comprising the following steps: (1) reading in a section of color video data and performing preprocessing, wherein the preprocessed color video is denoted as b, the number of frames is N, and the frame rate is fsHz; (2) extracting certain features of the R color image, the G color image, and the B color image of each frame of the color image, respectively, and then generating three characteristic time series curves corresponding to the three colors from the color video, denoted as r(n), g(n), and b(n), where n=0, 1, ..., N-1; and (3) performing Fourier transform (DFT) on the three time series, denoted as Let R(k), G(k), B(k), k=0, 1, ..., N-1; (4) select the spectrum segments of interest R1(k), G1(k), B1(k), k=M1, M1+1, ..., N1, M1, N1∈[1, N / 2] and its background spectrum segments, R0(k), G0(k), B0(k), k=M0, M0+1, ..., N0, M0, N0∈[1, N / 2] according to prior knowledge; (5) suppose the synthesis formula of the one-dimensional time series curve to be synthesized is rgb(n)=C r *r(n)+C g *g(n)+C b *b(n), n=0, 1, ..., N-1, the corresponding spectrum synthesis formula is RGB(k)=C r *R(k)+C g *G(k)+C b *B(k), k=0,1,...,N-1, then the combination coefficient C r , C g and C b By solving the following maximization optimization problem:

[0014]

[0015] According to the above-mentioned method for synthesizing one-dimensional motion curves in color video motion magnification, step (2) extracts certain features of the R color image, G color image, and B color image of each frame of the color image respectively, characterized in that the mean of the R color value, the mean of the G color value, and the mean of the B color value of each frame of the color image can be used as features, thereby obtaining the R mean time series r(n), n=0, 1, ..., N-1, the G mean time series g(n), n=0, 1, ..., N-1, and the B mean time series b(n), n=0, 1, ..., N-1.

[0016] According to the above-mentioned method for synthesizing one-dimensional motion curves in color video motion amplification, the step (5) determines the combination coefficient C r , C g and C b The optimization problem is characterized in that the norm of the measurement vector size ||·|| in formula (1) can be the 1-norm of the orientation quantity for both the numerator and the denominator.

[0017] According to the above-mentioned method for synthesizing one-dimensional motion curves in color video motion amplification, the step (5) determines the combination coefficient C r , C g and C b The optimization problem is characterized in that the norm of the measurement vector size ||·|| in formula (1) can take the ∞ norm for the numerator and the 1 norm for the denominator.

[0018] Beneficial effects

[0019] The beneficial effects can be seen through experiments.

[0020] Figure 2 The video shown is a schematic diagram of the video to be magnified. It is a video of a baby sleeping with inconspicuous breathing and heartbeat movements that are difficult to observe. The goal is to try to convert its color image sequence into a one-dimensional motion curve in order to analyze the heart rate or breathing rate, etc.

[0021] Figure 3 Shown is the sequence and spectrum of the R color mean of each frame image, which is the waveform and spectrum of the one-dimensional motion curve of the R color mean. The upper figure is the sequence waveform diagram, the horizontal axis is time, the unit is s, and the vertical axis is the change of the red color average value. The lower figure is its amplitude-spectrum diagram, that is, the Fourier transform modulus of the R color mean sequence, the horizontal axis is frequency, the unit is Hz, and the vertical axis is the size of each frequency component. Figure 3 The mid-frequency spectrum shows that there is a spectrum peak at 0.0434906 at 0.5Hz, but it is not the highest.

[0022] Figure 4 Shown are the sequence and spectrum of the G color mean value of each frame of the image, which is the waveform and spectrum of the one-dimensional motion curve of the G color mean value. The upper figure is the sequence waveform diagram, with the horizontal axis being time in seconds and the vertical axis being the change in the green color mean value. The lower figure is its amplitude-spectrum diagram, that is, the Fourier transform modulus of the G color mean value sequence, with the horizontal axis being frequency in Hz and the vertical axis being the size of each frequency component. Figure 4 The mid-frequency spectrum shows that there is a spectrum peak at 0.0406859 at 0.5Hz, but it is not the highest.

[0023] Figure 5Shown are the sequence and spectrum of the B color mean value of each frame of the image, which is the waveform and spectrum of the one-dimensional motion curve of the B color mean value. The upper figure is the sequence waveform diagram, with the horizontal axis being time in seconds and the vertical axis being the change in the blue color mean value. The lower figure is its amplitude-spectrum diagram, that is, the Fourier transform modulus of the B color mean value sequence, with the horizontal axis being frequency in Hz and the vertical axis being the size of each frequency component. Figure 5 The mid-frequency spectrum shows that there is a spectrum peak at 0.0331584 at 0.5Hz, but it is not the highest.

[0024] Figure 6 The figure shows the sequence and spectrum of the mean brightness values ​​for each frame. This is the waveform and spectrum of a one-dimensional motion curve for the mean brightness value. The top figure shows the sequence waveform, with time (in seconds) on the horizontal axis and the change in the mean brightness value (in seconds) on the vertical axis. The bottom figure shows the amplitude-spectrum plot, which is the Fourier transform modulus of the mean brightness value sequence. The horizontal axis shows frequency (in Hz) and the vertical axis shows the magnitude of each frequency component. The brightness curve is a combination of the basic motion curves for the three primary colors R, G, and B. The formula for this combination is: brightness Y = 0.2989*R + 0.5870*G + 0.1140*B. Figure 6 The mid-frequency spectrum shows that there is a spectrum peak at 0.0403389 at 0.5Hz, but it is not the highest.

[0025] Figure 7 The figure shows the sequence and spectrum of the optimized combination values ​​for each frame according to the present invention. This is the waveform and spectrum of the one-dimensional motion curve of the optimized combination values. The top figure is a waveform diagram of the sequence, with time (in seconds) on the horizontal axis and the change in the optimized combination value (in seconds) on the vertical axis. The bottom figure is an amplitude-spectrum diagram, i.e., the modulus of the Fourier transform of the optimized combination value time series, with frequency (in Hz) on the horizontal axis and the magnitude of each frequency component on the vertical axis. The optimized combination formula is: rgb(n) = 0*r(n) + 0.66017*g(n) + 0.33983*b(n). Figure 7 The mid-frequency spectrum shows that there is a spectrum peak at 0.0379732 at 0.5 Hz, which is the highest spectrum peak nearby.

[0026] Obviously, relative to Figures 3 to 6 ,in accordance with Figure 7 It is easier to determine the breathing frequency as 0.5 Hz by combining prior knowledge, because the spectrum peak above it is the highest in the vicinity, that is, it has the highest signal-to-noise ratio. Figure 8 After determining the breathing frequency, the breathing-related curve determined by time filtering can greatly improve the signal-to-noise ratio. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 , the present application provides steps of a method for synthesizing a one-dimensional motion curve in color video motion amplification.

[0028] Figure 2, schematic diagram of a video image magnified in an embodiment test. The original image is a color image with three color components R, G and B.

[0029] Figure 3 The sequence and spectrum of the R color mean values ​​for each frame of the preprocessed color video in this embodiment are shown. The top figure is a waveform diagram of the sequence, with time on the horizontal axis (in seconds) and the change in the red color mean value on the vertical axis. The bottom figure is an amplitude-spectrogram, i.e., the Fourier transform modulus of the R color mean value sequence, with frequency on the horizontal axis (in Hz) and the magnitude of each frequency component on the vertical axis.

[0030] Figure 4 The sequence and spectrum of the G color mean values ​​for each frame of the preprocessed color video in this embodiment are shown. The top figure is a waveform diagram of the sequence, with time on the horizontal axis (in seconds) and the change in the red color mean value on the vertical axis. The bottom figure is an amplitude-spectrogram, i.e., the Fourier transform modulus of the G color mean value sequence, with frequency on the horizontal axis (in Hz) and the magnitude of each frequency component on the vertical axis.

[0031] Figure 5 The sequence and spectrum of the B color mean values ​​for each frame of the preprocessed color video in this embodiment are shown. The top figure is a waveform diagram of the sequence, with time on the horizontal axis (in seconds) and the change in the red color mean value on the vertical axis. The bottom figure is an amplitude-spectrogram, i.e., the Fourier transform modulus of the B color mean value sequence, with frequency on the horizontal axis (in Hz) and the magnitude of each frequency component on the vertical axis.

[0032] Figure 6 The figure above shows the mean value sequence and spectrum of each frame of the preprocessed color video. The upper figure shows the sequence waveform, with time (in seconds) on the horizontal axis and the change in brightness value on the vertical axis. The lower figure shows the amplitude-spectrogram, which is the Fourier transform modulus of the brightness sequence. The horizontal axis shows frequency (in Hz) and the vertical axis shows the magnitude of each frequency component.

[0033] Figure 7 The following figure shows the one-dimensional sequence and spectrum of the preprocessed color video after optimization and combination according to the present invention. The top figure is a waveform diagram of the sequence, with the horizontal axis representing time in seconds and the vertical axis representing the change in the three-color combination value. The bottom figure is an amplitude-spectrogram, i.e., the Fourier transform modulus of the combined sequence, with the horizontal axis representing frequency in Hz and the vertical axis representing the magnitude of each frequency component.

[0034] Figure 8 , for the embodiment Figure 7 The one-dimensional time series curves shown in this case illustrate the sequence and spectrum of the filtering results based on the configured parameters. The top figure is a waveform plot of the sequence, with time (in seconds) on the horizontal axis and the magnitude of the filtering results (in seconds) on the vertical axis. The bottom figure is an amplitude-spectrum plot, which is the Fourier transform modulus of the filtering results sequence, with frequency (in Hz) on the horizontal axis and the magnitude of each frequency component on the vertical axis. DETAILED DESCRIPTION

[0035] Example. A method for synthesizing one-dimensional motion curves in color video motion amplification of the present application is implemented based on Matlab programming. The specific implementation steps are as follows:

[0036] (1) Read in a piece of color video data and preprocess it. The color video after preprocessing is denoted as b. Figure 2 As shown, the size is 544x960x3x301, and the frame rate fs = 30 Hz; during preprocessing, the rows and columns of the read video image are decimated at a rate of 4, and a 4x4 order average filter is performed before decimation. The size after decimation is 136x240x3x300, which is the size of b. Among them, only the first 300 frames (even number) are taken, that is, N = 300, to facilitate the subsequent Fourier transform DFT.

[0037] (2) Certain features of the R color image, G color image, and B color image of each frame of color image are extracted respectively, and three feature time series curves corresponding to the three colors are generated by the color video, which are recorded as r(n), n=0, 1, ..., N-1, g(n), n=0, 1, ..., N-1, b(n), n=0, 1, ..., N-1. Here, for the preprocessed color video b, the R color image feature value of each frame of color image can be taken as the average value of the R color values ​​of all its pixels, the G color image feature value of each frame of color image can be taken as the average value of the G color values ​​of all its pixels, and the B color image feature value of each frame of color image can be taken as the average value of the B color values ​​of all its pixels. In this way, three one-dimensional time series curves are generated - R mean time series, G mean time series, and B mean time series. For the specific processed video, the three time series curves obtained in this way are respectively shown in FIG. Figures 3 to 5 In the upper part of the waveform graph, the horizontal axis is time in seconds, and the vertical axis is the average color value.

[0038] (3) Perform Fourier transform DFT on the three time series, denoted as R(k), k = 0, 1, ..., N-1, G(k), k = 0, 1, ..., N-1, B(k), k = 0, 1, ..., N-1. Before DFT, the DC components of the three series are removed as usual. For the specific video to be processed, the amplitude spectrum obtained by Fourier transform DFT is shown in Figure 2. Figures 3 to 5 In the spectrum in the lower middle part, the horizontal axis is frequency in Hz, and the vertical axis is the component size. Figures 3 to 5 The amplitude spectrum diagrams only show the main frequency band of interest, 0-5Hz.

[0039] (4) According to prior knowledge, the spectrum segments of interest R1(k), G1(k), B1(k), k = M1, M1+1, ..., N1, M1, N1∈[1, N / 2] and its background spectrum segments R0(k), G0(k), B0(k), k = M0, M0+1, ..., N0, M0, N0∈[1, N / 2] of R(k), G(k) and B(k) are selected. In this example, the respiratory component is extracted as an example. According to prior knowledge, the respiratory frequency of newborns is between 0.3 Hz and 0.8 Hz. This is determined as the frequency segment of interest, and the corresponding M1 = 0.3 / (f s / N)+1=0.3*300 / 30+1=4, similarly N1=0.8 / (f s / N)+1=9. As for the background spectrum segment, M0=1, N0=5 / (f s / N)+1=5*300 / 30+1=51.

[0040] (5) Assume that the synthesis formula of the one-dimensional time series curve to be synthesized is rgb(n)=C r *r(n)+C g *g(n)+C b *b(n), n=0, 1, ..., N-1, the corresponding spectrum synthesis formula is RGB(k)=C r *R(k)+C g *G(k)+C b *B(k), k=0,1,...,N-1, then the combination coefficient C r , C g and C b By solving the following maximization optimization problem:

[0041]

[0042] Among them, for this example, M1 = 4, N1 = 9, M0 = 1, N0 = 51. After solving the optimization problem of this example, the combination coefficient C is obtained r =0, C g =0.66017, C b =0.33983, the synthesized one-dimensional time series curve and its DFT amplitude spectrum are shown in Figure 7 shown.

Claims

1. A method for synthesizing one-dimensional motion curves in color video motion amplification, characterized in that: The method comprises the following steps: (1) reading in a segment of color video data and performing preprocessing, wherein the preprocessed color video is denoted as b, the number of frames is N, and the frame rate is fsHz; (2) extracting certain features of the R color image, the G color image, and the B color image of each frame of the color image respectively, and then generating three feature time series curves corresponding to the three colors from the color video respectively, denoted as r(n), g(n), b(n), n=0, 1, ..., N-1; (3) performing Fourier transform DFT on the three time series respectively, denoted as R(k), G(k), B (k), k=0,1,…,N-1; (4) Select the spectrum segments of interest R1(k), G1(k), B1(k), k=M1,M1+1,…,N1,M1,N1∈[1,N / 2] and its background spectrum segments, R0(k), G0(k), B0(k), k=M0,M0+1,…,N0,M0,N0∈[1,N / 2] of R(k), G(k) and B(k) according to prior knowledge; (5) Assume that the synthesis formula of the one-dimensional time series curve to be synthesized is rgb(n)=C r *r(n)+C g *g(n)+C b *b(n), n=0, 1, ..., N-1, the corresponding spectrum synthesis formula is RGB(k)=C r *R(k)+C g *G(k)+C b *B(k), k=0, 1, ..., N-1, then the combination coefficient C r , C g and C b By solving the following maximization optimization problem:

2. According to the method for synthesizing one-dimensional motion curves in color video motion amplification according to claim 1, step (2) extracts certain features of the R color image, the G color image, and the B color image of each frame of the color image, characterized in that: The mean R color value, the mean G color value, and the mean B color value of each frame of the color image can be used as features, so as to obtain the R mean time series r(n), n=0, 1, ..., N-1, the G mean time series g(n), n=0, 1, ..., N-1, and the B mean time series b(n), n=0, 1, ..., N-1.

3. The method for synthesizing one-dimensional motion curves in color video motion amplification according to claim 1, wherein the step (5) determines the combination coefficient C r , C g and C b The optimization problem is characterized in that the norm of the measurement vector size ||·|| in formula (1) can be the 1-norm of the orientation quantity for both the numerator and the denominator.

4. The method for synthesizing one-dimensional motion curves in color video motion amplification according to claim 1, wherein the step (5) determines the combination coefficient C r , C g and C b The optimization problem is characterized in that the norm of the measurement vector size ||·|| in formula (1) can take the ∞ norm for the numerator and the 1 norm for the denominator.

Images