A video motion zooming software system for patrol

Abstract

A kind of video motion amplification software system for inspection, it is characterized in that, it includes following calculation steps, (1) read in a short video data, (2) to each frame image is carried out spatial low-pass filtering and is reduced sampling, record result as b, (3) select the attention area of b, record as b1, (4) to b1, select the average value of the feature of each frame image the color or color combination of pixel, waveform and frequency spectrum analysis are carried out to time sequence of feature value average value, combine priori knowledge, try and finally determine the type, cut-off frequency and order parameter of time band-pass filter, (5) according to the determined parameter, time band-pass filtering is carried out to video b pixel by pixel, the result is recorded as video c, (6) video c is enlarged according to the configured amplification parameter G, and add original video b, then constitute the final enlarged video, record as d, there is d=G*c+b;Each step involves process parameter supports man-machine interaction mode setting and adjustment.

Description

Technical Field

[0001] This application relates to a video motion magnification software system for inspection purposes.

[0002] Humans are moving bodies with heartbeats, respiration, pulsating blood vessels, and even changes in skin color; machines and equipment often have rotating motors, which in turn cause vibrations on their surfaces. By monitoring their motion, we can determine whether these moving people or equipment are functioning properly.

[0003] Compared to wearable or contactless monitoring, video surveillance offers the advantage of being non-contact. Some movements are significant and easily observed in video, while others are subtle and difficult to see or discern in video, requiring software amplification. Motion amplification, including in inspection scenarios, is a hot research topic in the field of video surveillance.

[0004] Video monitoring in inspection scenarios, compared to continuous video monitoring in fixed scenarios, does not target a specific object or scene, but rather monitors multiple scenes in rotation or roving. Because the moving objects monitored in time-division scenarios are numerous and varied, with varying frequencies and intensities of movement, as well as varying levels of background noise and interference, this presents challenges for the design of video motion amplification systems. Background Technology

[0005] A typical complete video motion magnification system for inspection includes cameras, video transmission lines, and computers. The cameras capture a video segment of each inspected object, transmit it to the computer or cloud for storage, and the computer reads the short video segment, performs motion magnification, and plays it back for observation. Since the characteristics of a video motion magnification system mainly lie in the motion magnification processing software on the computer, in a narrow sense, a video motion magnification system, including those used in inspection scenarios, is essentially a short video motion magnification processing software system.

[0006] The principle of video motion amplification processing can be described as follows by taking linear amplification as an example: (1) if there is no motion component in the video, the image sequence of each frame is a constant sequence, which does not need to be amplified; (2) if there is a motion component, the image sequence is a time-varying image sequence, and the color value or brightness value of each pixel constitutes a one-dimensional time sequence, which has a direct current component and an alternating current component. The direct current component represents the static component in the sequence, which does not need to be amplified, and the alternating current component represents the motion change part in the sequence; (3) only part of the alternating current component may be the alternating current component of interest that needs to be amplified; (4) the time sequence of each color is appropriately band-pass filtered to extract the motion component of interest and amplified, and then the corresponding value of the original sequence is added, so that the visual enhancement of the motion component of interest in the video can be realized.

[0007] The basic steps of linear motion amplification visual enhancement are: spatial filtering, time filtering, amplification and presentation. The current spatial filtering is usually pyramid sub-band decomposition for each frame image, and multiple sub-band image sequences are obtained after decomposition; then, time filtering is performed on each sub-band image sequence to obtain the motion component of interest; finally, the motion component of interest of each sub-band obtained by time filtering is amplified and added to the original sub-band image sequence, and the amplified and enhanced sub-band image sequences are recombined to obtain the visual enhanced amplified video and present it.

[0008] The principle and steps of motion amplification processing based on phase or involving transform domain are slightly different from those of linear amplification. Motion amplification processing based on phase or involving transform domain involves transformation, which consumes a lot of time and has relatively weak real-time performance. The present application belongs to the scheme under the linear motion amplification framework.

[0009] In video motion amplification, a short video is read into a computer, and the resolution of the read video image is large or small, which can be up to 1920x1080 or even higher, or as low as 320x64 or even lower, and the software should be able to support it. Due to the large time consumption in the time filtering in the following step, it is necessary to perform spatial low-pass filtering and re-sampling to reduce the image resolution to an appropriate resolution to reduce the calculation amount of time filtering. However, the existing pyramid sub-band decomposition involves spatial low-pass filtering and re-sampling, but the total data amount of all sub-band videos is not less than that before pyramid processing, and the low-pass filtering introduced in the low-pass filtering and corresponding recombination has greatly increased the calculation amount and execution time.

[0010] The intensity of the motion component to be amplified is different in different parts of the field of view. For example, the flow of blood is stronger in the exposed skin of the human body in the video image, while there is almost no signal in the image area covered by clothes. Therefore, when analyzing the motion component based on video, it is necessary to select the relevant area in the video image according to prior knowledge for analysis, which can be called the focus area. Selecting the focus area is a work of target detection. Existing automatic image target detection mostly uses supervised deep learning to determine the target image area, but it involves a certain amount of calculation, especially the need for prior learning, and it is difficult to detect diversified monitoring objects in the inspection background, and it is difficult to establish a learning sample library.

[0011] In video motion amplification, time filtering is a core step, which extracts the motion component of interest and then realizes amplification. Time filtering often uses band-pass filtering, because the direct current component represents the component that does not "move", and the signal-to-noise ratio of the high frequency part is often low. How to determine the upper and lower cutoff frequency and the order of the band-pass filter, on the one hand, the user of the software system considers based on the background knowledge and prior knowledge of the motion component of the specific monitoring object, and on the other hand, based on the waveform spectrum analysis of the time sequence of the image features. The image features can be the average R color value, or the average G color value, or the average B color value, or the average brightness value of the three color combination, or the average value of other linear combinations of three colors, etc. The significance of the motion component of interest in its time sequence is different. Therefore, for diversified monitoring objects in the inspection background, the selection of image features and the determination of time band-pass filter parameters of video amplification software should support the debugging of human-computer interaction, and support the decision-making based on the prior knowledge of specific monitoring objects and spectrum analysis. From the existing literature, there is no motion amplification software interface arranged with amplitude spectrum to support spectrum analysis, and there is no motion amplification software interface to support image feature selection and free selection of filter parameters.

[0012] In summary, video monitoring in the inspection scene, relative to continuous video monitoring in the fixed scene, is not aimed at specific objects and scenes, but is aimed at N multiple scenes for rotation or patrol monitoring, which objectively requires the video motion amplification software system to have wide applicability and universality, and to support all process parameters affecting the amplification result to be set and adjusted and optimized in the software interface in the way of human-computer interaction. The existing solution of video motion amplification software system is far from this objective requirement. At the same time, video motion amplification, as a kind of video processing, has a huge computing load. Even if the real-time requirement of the inspection system is not so absolute, the processing time delay generated by the existing system solution is still criticized by users, and the motion amplification processing process should be optimized and some calculation processes should be simplified if necessary. SUMMARY

[0013] Invention purposes

[0014] The technical scheme of the video motion amplification software system for patrol is proposed, so that the space filtering, attention area selection, time domain band-pass filtering and other processes involved in motion amplification are optimized, on the one hand, to adapt to the objective requirements of diversified monitoring objects in the patrol scene, and on the other hand, to reduce the calculation amount of the processing process and the time delay caused by motion amplification as much as possible.

[0015] Technical scheme

[0016] A video motion amplification software system for patrol, characterized in that it comprises the following calculation steps: (1) reading in a short video data, denoted as a, (2) performing spatial low-pass filtering and down-sampling on each frame of image of the short video a, and denoting the obtained output video as b, (3) selecting the attention area of the image of the video b, and denoting the output attention area image sequence or video as b1, (4) selecting a certain color in the R, G, B three primary colors of the attention area video b1, or selecting a certain linear combination of the R, G, B three primary colors as a feature, calculating the average value of the value of the feature of all pixels in each frame of image, and performing waveform and spectrum analysis on the time sequence composed of the average values of the feature values of all frames of image, combining the prior knowledge of the specific monitoring object, trying and finally determining the filter type, cutoff frequency and order parameters of the time domain band-pass filter, (5) performing time domain band-pass filtering on the video b pixel by pixel according to the determined parameters of the time domain band-pass filter, and denoting the band-pass filtering result video as video c, (6) amplifying the video c according to the configured amplification parameter G, and adding the original video b, to constitute the final video amplification result, i.e. the target video, denoted as d, i.e. d=G*c+b; at the same time, the process parameters involved in the above steps support the setting and adjustment optimization of the interface in a man-machine interactive manner.

[0017] According to the video motion amplification software system for patrol described above, the video b is obtained by performing spatial low-pass filtering and down-sampling on each frame of image of the short video a in the second step, characterized in that the re-sampling multiples of the image rows and columns are configured through man-machine interaction first, and then the average filtering and down-sampling are performed according to the configured re-sampling multiples, and it is assumed that the re-sampling multiples of the rows and columns are both M, then the color values of all pixels of the image block of MxM pixels are averaged as the color value of one pixel after down-sampling.

[0018] According to the above-described video motion magnification software system for inspection, in step (3), the region of interest of the image of video b is selected, and the output region of interest image sequence or video is recorded as b1. The feature is that a frame of video b is displayed on the human-machine interface as a representative. Based on the graphical input method, the upper left corner and the lower right corner of the rectangular image region to be focused are clicked respectively. In this way, the region of interest of the representative frame image is selected, and thus the region of interest image sequence or video b1 is determined.

[0019] According to the above-described video motion magnification software system for inspection, in step (4), for the video b1 of the region of interest, a certain color among the three primary colors of R, G, and B pixels, or a certain linear combination of the three primary colors of R, G, and B, is selected as a feature. The average value of the feature value of all pixels in each frame is calculated. Waveform and spectrum analysis are performed on the time series composed of the feature average values ​​of all frames. Combined with the prior knowledge of the specific monitoring object, the filter type, cutoff frequency, and order parameters of the time domain bandpass filter are tried and finally determined. The feature is that any color among R, G, and B can be tried by combining the prior knowledge through the human-machine interface. The brightness of the R, G, and B color combination, or the weighted sum of 1 / 3 of each of the R, G, and B color combinations, is used as a feature. The average feature value of all pixels in each frame of the image is calculated. The initial waveform and spectrum analysis of the one-dimensional time series formed by the average feature values ​​of all frames of the image are performed online. Alternatively, the waveform and spectrum analysis of the time series of average feature values ​​after bandpass filtering can be performed through a human-machine interface under different FIR bandpass filter parameter configurations. The waveform and amplitude-frequency diagram effects of the filtering results under different filter parameter configurations are observed. Finally, the optimized selection of R, G, and B colors or their combinations, as well as the configuration of the cutoff frequency and order parameters of the FIR bandpass filter are obtained.

[0020] According to the above-described video motion magnification software system for inspection, in step (5), time-domain bandpass filtering is performed pixel by pixel on video b, and the video with the bandpass filtering result is recorded as video c. The feature is that the pixel-by-pixel time-domain bandpass filtering is performed pixel by pixel on the time series of each primary color component of the three primary colors R, G, B of the color image using the bandpass filter parameters optimized in step (4).

[0021] According to the above-described video motion magnification software system for inspection, the magnification parameter G in step (6) can be the same or different for the three primary colors R, G, and B, and can be configured interactively in the human-machine interface.

[0022] Beneficial effects

[0023] The biggest feature of the scheme is that the algorithm space filtering link completely gives up the pyramid sub-band decomposition operation in the literature, and instead uses an average low-pass filter. Correspondingly, the subsequent time filtering and amplification reconstruction link no longer operates on the sub-band image sequence, but operates on the result image sequence b of the spatial low-pass filtering, performs a time band-pass filtering on the result video c of the time band-pass filtering, amplifies the result video c, and adds the original video b to obtain the target video. Therefore, the scheme is very different from the traditional linear motion amplification scheme based on sub-band decomposition. It is a complete subversion of the traditional linear motion amplification scheme based on sub-band decomposition. It is based on the applicant's deep insight and understanding of the role of spatial filtering: Spatial filtering is not the core step of motion amplification relative to time filtering. Its main task is to reduce the resolution of the image to an appropriate level, and to minimize the computational load of subsequent other amplification operations without causing significant damage to the visual effect. The correctness of this cognition is verified by the amplification case, as described in the embodiments below.

[0024] The above features of the scheme greatly reduce the computational load of the algorithm. The spatial low-pass filtering and down-sampling of each frame of the short video a obtain the video b. The average filtering and down-sampling are performed according to the configured up-sampling factor. Assuming that the up-sampling factor of the row and column is M, the color values of all pixels in an MxM pixel image block are averaged, and the average result is used as the color value of one pixel after down-sampling. The average filtering operation on such an MxM pixel image block in the present case only involves M 2 real addition operations. For the traditional pyramid operation, if an MxM pixel image block is subjected to FIR Gaussian low-pass filtering with an MxM array as the convolution kernel, even if FFT is used, it involves 3M 2 complex multiplications. Compared with the traditional pyramid operation, the computational load of the space filtering part of the present scheme can be almost ignored.

[0025] The second big feature of the scheme is that based on the attention region video b1, a certain color in the R, G, and B three primary colors of the pixels or a certain linear combination of the R, G, and B three primary colors is selected as a feature, the average value of the value of the feature of all pixels in each frame of image is calculated, the time sequence formed by the average values of the features of all frames of image is subjected to waveform and spectrum analysis, the filter type, cutoff frequency, and order parameters of the time domain band-pass filter are determined in combination with the prior knowledge of the specific monitoring object, and the filter type, cutoff frequency, and order parameters of the time domain band-pass filter are determined in combination with the prior knowledge of the specific monitoring object. The way of using a one-dimensional time sequence to help determine the parameters of the time filter undoubtedly greatly reduces the computational load compared with directly trying time filtering on the video image sequence b.

[0026] The third feature of the application is that all the process parameters affecting the result of motion amplification, including the selection of spatial average low-pass filter order, the selection of the region of interest, the selection of image features for spectral analysis, the configuration of filter parameters, the configuration of amplification factor, etc., are all moved to the interface, and the prior knowledge is tried to be determined through human-computer interaction, so as to maximize the objective requirements of the inspection system for diversified monitoring objects. BRIEF DESCRIPTION OF DRAWINGS

[0027] Figure 1 The computing block diagram of the video motion amplification system of the application.

[0028] Figure 2 The initial interface of the motion amplification system of the embodiment. The figure shows that the "video file" is waiting to be selected.

[0029] Figure 3 The interface diagram after the "video file" is selected and the data is read in. The file path is displayed on the interface, and the resolution of the read-in video is 544x960x3x301, and the frame rate is 30.

[0030] Figure 4 The interface diagram after spatial filtering. The interface shows that the video size has changed: 136x240x3x300.

[0031] Figure 5 The interface diagram of the embodiment after the "region of interest" is selected. The coordinate cursor in the interface figure shows the method of selecting the region of interest.

[0032] Figure 6 The interface diagram after "initial spectrum" is clicked in the embodiment. The interface shows that Gray5, i.e. the combination of R, G and B colors 0.2989*R+0.5870*G+0.1140*B, is selected, and the waveform and amplitude spectrum of the average value time sequence are displayed. The 0.5Hz respiratory component of interest is found.

[0033] Figure 7 The interface diagram of the embodiment after the average value sequence of the selected color feature is filtered by the configured filter. When the lower limit cutoff frequency of the FIR1 filter is configured to be 0.48Hz, the upper limit to be 0.52Hz, and the order to be 170, the result of clicking "filtered spectrum" is obtained.

[0034] Figure 8 The interface diagram of the embodiment after the amplified video is obtained according to the configured filter and gain. The result of clicking "filtering and amplification" is obtained. DETAILED DESCRIPTION

[0035] Embodiment. A video motion amplification software system for inspection is realized based on Matlab, and the system interface is as follows: Figure 2As shown, the left side is mainly display area, including 3 coordinate system; the right side is mainly control area, including "video file", "space filter", 'focus area", "initial spectrum", "filter spectrum", "filter zoom", "change zoom" and "store zoom" and other main menu items. Its calculation steps are as follows:

[0036] (1) read a short video data, recorded as a. Click "video file" to select the zoom in motion video, read into memory and play the original video. The interface on the upper part of the display video file path, image resolution and frame rate and other information. Such as Figure 3 As shown, the resolution of the read video is 544x960x3x301, and the frame rate is 30.

[0037] (2) for each image of short video a space low pass filter and down sampling, record its output video for b. Click "space filter", according to the selected parameters of the original video average filter and down sampling, list box parameters "M1; M2; M3; M4; M5; M6" can be configured by man-machine interaction, corresponding to 1, 2, 3, 4, 5 or 6 times the row and column are extracted; for example, M4 means that the color value of 4x4 pixel small area is averaged to get the color value of 1 pixel after down sampling. The result of down sampling is shown in Figure 4 As shown, the video size has changed: 136x240x3x300. After down sampling, the video image will become blurred due to the reduction of resolution.

[0038] (3) select the image of video b focus area, record its output focus area image sequence or video for b1. Click "focus area" to select the focus area by man-machine interaction, such as Figure 5 As shown, click on the upper left corner of the focus area, and then click on the lower right corner of the focus area, so as to determine a rectangular focus area, see Figure 6 As shown, 106x181x3x300. By default, the default focus area is the entire video image area.

[0039] (4) for focus area video b1, the average value of the selected color component or its combination of time sequence is analyzed by waveform and spectrum, and the cut-off frequency and order parameters of FIR time domain band pass filter are determined combined with prior knowledge. Click "initial spectrum" to calculate the average value of the selected color component value of the focus area of each image, draw the curve of the average value of the image with time (waveform) and its spectrum, such as Figure 6R1 represents the R component of the RGB color image, G2 represents the G component, B3 represents the B component, RGB4 represents the average of R, G and B, and Gray5 represents the gray image. The default range of the horizontal axis of the spectrum is [0, fs / 2], which is [0, 15] when the frame rate is 30 Hz. If the check box is checked, the range of the spectrum is displayed according to the indicated f_Range. Clicking "Filter Spectrum" filters the average value curve (waveform) of the color value of the region of interest of each frame of image according to the set filter, and displays the filtered waveform and spectrum, as shown in Figure 7 FIR1 is a band-pass filter designed by a window function method (the window function is a HAMMING window), and the three parameters [0.48, 0.52, 170] are the lower limit cutoff frequency, the upper limit cutoff frequency and the filter order, respectively. Bidirectional filtering is used.

[0040] (5) According to the determined parameters of the time domain band-pass filter, the video b is subjected to time domain band-pass filtering pixel by pixel, and the band-pass filtering result video is recorded as video c.

[0041] (6) The video c is enlarged according to the configured amplification parameter G, and the original video b is added, thereby constituting the final video amplification result, recorded as d, that is, d = G * c + b. Clicking "Filter and Amplify" filters the time sequence of a certain (or composite) color component value of each pixel according to the current filter setting, multiplies the filtering result by the configured gain, and adds it to the original pixel sequence, thereby realizing the amplification of the motion of the pixel point and playing the amplified video, as shown in Figure 8 Clicking "Change Amplification" multiplies the result sequence of each pixel time filtering according to the latest gain configuration, adds it to the original pixel sequence, realizes new amplification, d = G * c + b, and then plays the amplified result.

[0042] In addition, clicking "Store Amplification" stores the current latest amplified video.

Claims

1. A video motion magnification software system oriented towards inspection, characterized in that, It includes the following calculation steps, (1) read a short video data, recorded as a, (2) for each frame image of short video a space low pass filter and down sampling, record its output video b, (3) select the image of video b area of interest, record its output area of interest image sequence or video b1, (4) for the area of interest video b1, select its pixel R, G, B three color or select its R, G, B three linear combination of color as a feature, calculate the average value of the value of each frame image all pixels of the feature, the time sequence of the average value of the feature value of all frames is analyzed, combined with the prior knowledge of the specific monitoring object, try and finally determine the filter type, cutoff frequency and order parameters of the time domain band pass filter, (5) according to the parameters of the time domain band pass filter, the video b is pixel by pixel time domain band pass filter, the band pass filter result video is recorded as video c, (6) video c is amplified according to the configured amplification parameter G, and the original video b is added, which constitutes the final video amplification result, that is, the target video, recorded as d, that is, d=G*c+b; at the same time, the process parameters involved in the above steps support man-machine interaction mode to set and adjust and optimize in the interface.

2. The video motion magnification software system for patrol according to claim 1, wherein the step (2) is characterized by performing spatial low-pass filtering and down-sampling on each frame of the short video a to obtain a video b. First, the row and column re-sampling multiples are configured through man-machine interaction, and the average filtering and down sampling are performed according to the configured re-sampling multiples. Assuming that the row and column re-sampling multiples are both M, the color values of all pixels in an MxM pixel image block are averaged to obtain the color value of one pixel after down sampling.

3. The video motion magnification software system for patrol according to claim 1, wherein the step (3) selects the region of interest of the image of the video b, and records the output sequence of the image of the region of interest or the video b1. The man-machine interface displays a frame of image of video b as a representative, and the left upper corner and the right lower corner of the rectangular image area to be focused are clicked based on the graphical input mode, so as to select the focus area of the representative image of the frame, and further determine the focus area image sequence or video b1.

4. The video motion amplification software system according to claim 1, wherein in step (4), the average value of the selected feature of all pixels in each frame of the video b1 is calculated, and the time series of the average values of the selected feature of all frames of the video b1 is analyzed to determine the type, cutoff frequency and order of the time domain band-pass filter, characterized in that, The man-machine interface can be used to try to select any color in R, G and B, or the brightness of the combination of R, G and B, or the 1 / 3 weighted sum of R, G and B as a feature, calculate the average value of the feature value of all pixels in each frame image, and perform initial waveform and spectrum analysis on the one-dimensional time sequence composed of the average value of the feature value of all frames. The waveform and spectrum analysis results of the time sequence after band pass filtering of the feature value average value under different FIR band pass filter parameter configurations can also be observed, and the waveform and amplitude-frequency effect of the filtering results under different filter parameter configurations can be observed, so as to finally obtain the optimized selection of R, G and B colors or their combinations and the configuration of the cutoff frequency and order parameters of the FIR band pass filter.

5. The video motion amplification software system according to claim 1, wherein the step (5) is to perform a time domain band-pass filtering on the video b pixel by pixel, and the band-pass filtered result is denoted as video c, and wherein the video c is characterized in that, The pixel by pixel time domain band pass filtering is performed on each color component of the R, G and B three color components of the color image by using the optimized band pass filter parameters of step (4).

6. The video motion magnification software system for patrol according to claim 1, wherein, The amplification parameters G of step (6) can be the same or different for R, G and B, which can be configured interactively in the man-machine interface.

Images