Vibration detecting device and vibration detecting program

Abstract

To provide a vibration detecting device capable of highly accurately detecting vibration in an optical axis direction of photographing from a video image obtained through photographing, and a vibration detecting program.SOLUTION: A vibration detecting device 10 comprises: an acquisition unit 11A which acquires a video image obtained through photographing by a photographing device 20; a derivation unit 11B which derives a two-dimensional space power spectrum of the video image by spatial frequency analysis or logarithmic transformation of a spatial wave number axis; and a detection unit 11C which detects a physical quantity on vibration in an optical axis direction by the photographing device 20 of the video image on the basis of the two-dimensional space power spectrum.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a vibration detection device and a vibration detection program. [Background technology]

[0002] In vibration detection technology that detects minute vibrations of a subject using software-based video processing of video data obtained by video shooting, etc., the out-of-plane vibration components of a planar vibrating body are converted into vibrations in the optical axis direction during shooting, making it difficult to directly capture the displacement on a two-dimensional plane such as a captured image.

[0003] The following techniques can be applied to detect such vibration components in the optical axis direction.

[0004] Patent Document 1 discloses a technique for detecting out-of-plane vibrations (vibrations in the optical axis direction) using the pitch of a repeating pattern such as regular stripes or a grid, by an algorithm called the sampling moiré method.

[0005] Moreover, Patent Document 2 discloses a technique for detecting out-of-plane vibrations using an in-plane displacement vector field on an imaging plane.

[0006] In addition, Non-Patent Documents 1 and 2 disclose a technology for achieving highly accurate inter-image matching by using a phase-only correlation function obtained by inverse Fourier transform of a spatial phase spectrum obtained by a two-dimensional spatial Fourier transform based on an algorithm called phase-only correlation method, and a peak model without function fitting.

[0007] In the technology disclosed in Non-Patent Document 1 and Non-Patent Document 2, the target of the two-dimensional spatial Fourier transform is data obtained by log-polar transforming the two-dimensional spatial power spectrum (or two-dimensional spatial amplitude spectrum) of the original image, so that the scaling component and the rotational component can be detected without being affected by the translation component of the original image. Therefore, in this technology, the scaling component can be effectively used to detect out-of-plane vibration. Furthermore, when phase information is used, robustness to noise is important, but noise countermeasures are implemented by low-pass filter processing in the spatial frequency domain. [Prior art documents] [Patent documents]

[0008] [Patent Document 1] Patent No. 6534147 [Patent Document 2] Patent No. 6954368 [Non-patent literature]

[0009] [Non-Patent Document 1] "Improvement of Sub-Pixel Image Matching Based on Phase-Only Correlation", Kiyoshi Nagashima, Takafumi Aoki, Tatsuo Higuchi, Koji Kobayashi, 218th Research Meeting of the Tohoku Branch of the Society of Instrument and Control Engineers (2004) [Non-Patent Document 2] "High-precision machine vision based on phase-only correlation: Aiming for image sensing technology that goes beyond the pixel resolution barrier," Takafumi Aoki, Koichi Ito, Takuma Shibahara, and Kiyoshi Nagashima, Fundamentals Review (2007) Summary of the Invention [Problem to be solved by the invention]

[0010] However, the technique disclosed in Patent Document 1 has a problem in that, due to its characteristics, accurate detection is not possible when a regularly repeated pattern does not exist.

[0011] In addition, in the technology disclosed in Patent Document 2, the displacement vector is a composite vector of a displacement component caused by in-plane vibration and a displacement component observed as an in-plane component due to out-of-plane vibration. Therefore, even if it is theoretically possible to distinguish between in-plane and out-of-plane, there is a problem in that there is a limit to the detection accuracy when image noise and the like are taken into consideration.

[0012] Furthermore, in the technology disclosed in Non-Patent Document 1 and Non-Patent Document 2, if the application is to match captured natural images (original images), the spatial complex spectrum obtained by the two-dimensional spatial Fourier transform has information concentrated on the low-frequency side, so the effect of the low-pass filter is considered to be large. However, in the application to detect the scaling components, this technology processes the two-dimensional spatial power spectrum of the original image as described above. Therefore, the result of the two-dimensional Fourier transform on this two-dimensional spatial power spectrum is the result of performing the Fourier transform twice on the original image, so there is a problem in that information is not necessarily concentrated on the low-frequency side.

[0013] An object of the present disclosure is to provide a vibration detection device and a vibration detection program capable of detecting vibration in the optical axis direction of imaging with high accuracy from a moving image obtained by imaging. [Means for solving the problem]

[0014] The vibration detection device of the present invention described in claim 1 comprises an acquisition unit that acquires a moving image obtained by photographing with an imaging device, a derivation unit that derives a two-dimensional spatial power spectrum of the moving image by spatial frequency analysis and logarithmic transformation of the spatial wavenumber axis, and a detection unit that detects a physical quantity related to vibration of the moving image in the optical axis direction caused by the imaging device based on the two-dimensional spatial power spectrum.

[0015] According to the vibration detection device of the present invention as described in claim 1, a moving image obtained by shooting with an imaging device is acquired, a two-dimensional spatial power spectrum of the moving image is derived by spatial frequency analysis and logarithmic transformation of the spatial wave number axis, and a physical quantity related to vibration in the optical axis direction caused by the imaging device for the moving image is detected based on the two-dimensional spatial power spectrum, thereby making it possible to detect vibration in the optical axis direction of the shooting with high accuracy from the moving image obtained by shooting.

[0016] The vibration detection device of the present invention as described in claim 2 is the vibration detection device as described in claim 1, wherein the derivation unit derives a one-dimensional spatial power spectrum in any spatial direction from the two-dimensional spatial power spectrum for each frame, derives a cross-correlation function between the one-dimensional spatial power spectrum of a reference frame and the one-dimensional spatial power spectrum of another frame for each frame, derives peak fluctuation by fitting the peak of the cross-correlation function for each frame with a continuous function, derives a time spectrum from the peak fluctuation, and the detection unit uses the time spectrum to detect a physical quantity related to the vibration.

[0017] According to the vibration detection device of the present invention as set forth in claim 2, a one-dimensional spatial power spectrum in any spatial direction is derived for each frame from the two-dimensional spatial power spectrum, a cross-correlation function between the one-dimensional spatial power spectrum of the reference frame and the one-dimensional spatial power spectrum of another frame is derived for each frame, a peak fluctuation is derived by fitting the peak of the cross-correlation function for each frame with a continuous function, a time spectrum is derived from the peak fluctuation, and a physical quantity related to the vibration is detected using the time spectrum, thereby making it possible to detect the physical quantity related to the vibration with higher accuracy.

[0018] The vibration detection device of the present invention as described in claim 3 is a vibration detection device as described in claim 1 or claim 2, wherein the detection unit detects both the vibration frequency and amplitude of the vibration, or only the vibration frequency, as a physical quantity related to the vibration.

[0019] According to the vibration detection device of the present invention as described in claim 3, at least one of the vibration frequency and amplitude of the vibration can be detected by detecting both the vibration frequency and amplitude of the vibration, or only the vibration frequency, as physical quantities related to the vibration.

[0020] A vibration detection device according to the present invention as set forth in claim 4 is the vibration detection device as set forth in claim 1 or claim 2, wherein the moving image is at least one of black and white and color moving images.

[0021] According to the vibration detection device of the present invention as described in claim 4, by making the moving images into at least one of black and white and color moving images, it is possible to detect vibration in the optical axis direction of shooting from at least one of black and white and color moving images with high accuracy.

[0022] The vibration detection program of the present invention as described in claim 5 causes a computer to execute a process of acquiring a moving image obtained by photographing with an imaging device, deriving a two-dimensional spatial power spectrum of the moving image by spatial frequency analysis and logarithmic transformation of the spatial wave number axis, and detecting a physical quantity related to vibration of the moving image in the optical axis direction caused by the imaging device based on the two-dimensional spatial power spectrum.

[0023] According to the vibration detection program of the present invention as set forth in claim 5, a moving image obtained by shooting with an imaging device is acquired, a two-dimensional spatial power spectrum of the moving image is derived by spatial frequency analysis and logarithmic transformation of the spatial wave number axis, and a physical quantity related to vibration in the optical axis direction caused by the imaging device for the moving image is detected based on the two-dimensional spatial power spectrum, thereby making it possible to detect vibration in the optical axis direction of the shooting with high accuracy from the moving image obtained by shooting. Effect of the Invention

[0024] As described above, according to the present invention, it is possible to detect vibrations in the optical axis direction of photography with high accuracy from moving images obtained by photography. [Brief description of the drawings]

[0025] [Figure 1] 1 is a block diagram showing an example of a hardware configuration of a vibration detection device according to an embodiment. [Diagram 2] 1 is a block diagram illustrating an example of a functional configuration of a vibration detection device according to an embodiment. [Diagram 3] FIG. 2 is a schematic diagram showing an example of a configuration of a video database according to the embodiment; [Figure 4] 10 is a flowchart illustrating an example of a vibration detection process according to the embodiment. [Diagram 5] 11A to 11C are diagrams showing simulation image frames used for verifying the vibration detection process according to the embodiment. [Figure 6] 11A and 11B are diagrams for explaining vibration in the optical axis direction assumed in a simulation according to the embodiment. [Figure 7] 13 is a graph showing an example of a time waveform (both amplitude values ​​2×10 −2%) of Δωpeak(t) calculated for simulation moving image information in verification of vibration detection processing according to the embodiment. [Figure 8] 13 is a graph showing an example of a time waveform (both amplitude values ​​1.4×10 −4%) of Δωpeak(t) calculated for simulation moving image information in verification of vibration detection processing according to the embodiment. [Figure 9] FIG. 11 is a front view showing a checkerboard used as an actual moving image in verification of the vibration detection process according to the embodiment. [Figure 10] 11A and 11B are conceptual diagrams illustrating an example of a shooting situation in verification of a vibration detection process according to the embodiment using actual moving images. [Figure 11] 13 is a graph showing an example of a time amplitude spectrum (uncompressed / both amplitude values ​​0.5 mm) of Δωpeak(t) calculated for actual moving image information in verification of vibration detection processing according to the embodiment. [Figure 12] 13 is a graph showing an example of a time amplitude spectrum (H.264 / both amplitude values ​​1.0 mm) of Δωpeak(t) calculated for actual moving image information in verification of vibration detection processing according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0026] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. In this embodiment, the present invention will be described as being applied to a vibration detection device that processes video images of a building in a state of slight motion under the influence of wind vibration or ground vibration.

[0027] First, a configuration of a vibration detection device 10 according to this embodiment will be described with reference to Figures 1 and 2. Examples of the vibration detection device 10 include information processing devices such as a personal computer and a server computer.

[0028] 1, a vibration detection device 10 according to this embodiment includes a CPU (Central Processing Unit) 11, a memory 12 as a temporary storage area, a non-volatile storage unit 13, an input unit 14 such as a keyboard and a mouse, a display unit 15 such as a liquid crystal display, a medium reading and writing device (R / W) 16, and a communication interface (I / F) unit 18. The CPU 11, the memory 12, the storage unit 13, the input unit 14, the display unit 15, the medium reading and writing device 16, and the communication interface (I / F) unit 18 are connected to each other via a bus B1. The medium reading and writing device 16 reads information written in a recording medium 17 and writes information to the recording medium 17.

[0029] The storage unit 13 is realized by an HDD (Hard Disk Drive), an SSD (Solid State Drive), a flash memory, or the like. A vibration detection program 13A is stored in the storage unit 13 as a storage medium. The vibration detection program 13A is stored in the storage unit 13 by setting a recording medium 17, on which the vibration detection program 13A is written, in the medium reading and writing device 16, and the medium reading and writing device 16 reading the vibration detection program 13A from the recording medium 17. The CPU 11 reads the vibration detection program 13A from the storage unit 13, expands it in the memory 12, and sequentially executes the processes of the vibration detection program 13A. In addition, various databases such as a video database 13B are stored in the storage unit 13.

[0030] In the vibration detection device 10 according to this embodiment, an imaging device 20 that captures moving images is connected to a communication I / F unit 18. The imaging device 20 is used to capture images that include an object that is the subject of vibration detection.

[0031] The photographing method by the photographing device 20 may be any of aerial photographing, moving photographing by a person on the ground, fixed photographing using a tripod or fixed member, etc. However, fixed photographing is most preferable in order to avoid a photographing method that generates displacement in the optical axis direction during photographing. In addition, in this embodiment, a photographing device that photographs monochrome (black and white) images is applied as the photographing device 20.

[0032] Next, a functional configuration of the vibration detection device 10 according to this embodiment will be described with reference to Fig. 2. As shown in Fig. 2, the vibration detection device 10 includes an acquisition unit 11A, a derivation unit 11B, and a detection unit 11C. The derivation unit 11B includes a first derivation unit 11B1, a second derivation unit 11B2, a third derivation unit 11B3, and a fourth derivation unit 11B4. The CPU 11 of the vibration detection device 10 executes the vibration detection program 13A to function as the acquisition unit 11A, the derivation unit 11B, and the detection unit 11C.

[0033] The acquisition unit 11A according to this embodiment acquires a moving image obtained by photographing with the photographing device 20. In this embodiment, a building is applied as the object, but is not limited thereto. For example, natural objects such as structures other than buildings such as bridges and towers, mountains, trees, etc., biological information such as pulsation, facility equipment such as air conditioning ducts and transformers, surface members such as ceilings, floors, and walls, or a combination of a plurality of types of these may be applied as the object.

[0034] Moreover, the derivation unit 11B according to the present embodiment derives a two-dimensional spatial power spectrum of the moving image by spatial frequency analysis and logarithmic transformation of the spatial wave number axis. Then, the detection unit 11C according to the present embodiment detects a physical quantity related to the vibration of the moving image in the optical axis direction by the imaging device 20 based on the two-dimensional spatial power spectrum. In the present embodiment, the vibration frequency of the vibration in the optical axis direction by the imaging device 20 is applied as the physical quantity, but this is not limited thereto. For example, instead of the vibration frequency, the amplitude of the vibration in the optical axis direction by the imaging device 20 may be applied as the physical quantity, or both the amplitude and the vibration frequency may be applied as the physical quantity.

[0035] Here, the first derivation unit 11B1 according to the present embodiment derives a one-dimensional spatial power spectrum in an arbitrary spatial direction from the two-dimensional spatial power spectrum for each frame. The second derivation unit 11B2 according to the present embodiment derives a cross-correlation function between the one-dimensional spatial power spectrum of the reference frame and the one-dimensional spatial power spectrum of another frame for each frame. The third derivation unit 11B3 according to the present embodiment derives a peak fluctuation by fitting the peak of the cross-correlation function for each frame with a continuous function. Furthermore, the fourth derivation unit 11B4 according to the present embodiment derives a time spectrum from the peak fluctuation. Then, the detection unit 11C according to the present embodiment detects a physical quantity related to the vibration using the time spectrum.

[0036] Next, the video database 13B according to this embodiment will be described with reference to Fig. 3. As shown in Fig. 3, the video database 13B according to this embodiment stores video information obtained by capturing videos by the imaging device 20 for each video ID (Identification) assigned in advance. In this manner, in this embodiment, video information is previously acquired from the imaging device 20 and registered in the video database 13B, but this is not limiting. For example, the imaging device 20 may continuously capture video, and video information obtained from the imaging device 20 when vibration of a predetermined level or higher occurs may be used online in real time or non-real time. However, the online processing does not need to be real time processing (a processing speed that can keep up with 1x speed playback of captured data).

[0037] Next, the operation of the vibration detection device 10 according to this embodiment will be described with reference to Fig. 4. Fig. 4 is a flowchart showing an example of the vibration detection process according to this embodiment. When a command input for starting execution of the vibration detection program 13A is performed by the user via the input unit 14, the CPU 11 of the vibration detection device 10 executes the vibration detection program 13A, thereby executing the vibration detection process shown in Fig. 4. Note that, in order to avoid confusion, a case will be described here in which the video database 13B has already been constructed and the video information to be processed is designated by the user.

[0038] In step 200 of FIG. 4, the CPU 11 acquires the video information I(x, y, t) specified by the user by reading it from the video database 13B.

[0039] In step 202, the CPU 11 calculates still images I at time t=0 and time t=k-th frame (k=1, 2, . . . , N-1: N is the number of frames of the moving image) of the moving image information I(x, y, t). t Performs the two-dimensional spatial Fourier transform of (x,y).

[0040] In step 204, the CPU 11 obtains a two-dimensional spatial power spectrum P obtained by logarithmically transforming both axes of spatial wave numbers Ωx and Ωy obtained by the two-dimensional spatial Fourier transform. t (ω x ,ω y ) is required.

[0041] That is, the CPU 11 calculates a two-dimensional spatial complex spectrum for all frames by a spatial frequency analysis method such as two-dimensional FFT (fast Fourier transform) processing, and obtains a two-dimensional spatial power spectrum. At this time, both spatial wave number axes of the two-dimensional spatial power spectrum are subjected to logarithmic transformation processing. By obtaining the power spectrum, it is possible to ignore even if vibration components in the in-plane direction perpendicular to the shooting optical axis are mixed in the moving image information to be processed. This utilizes the fact that a parallel movement in the real space domain (within the image plane) retains 100% of the information in the phase spectrum in the Fourier domain and does not affect the power spectrum. Furthermore, by logarithmizing the spatial wave number axis, the expansion and contraction of the real space domain is replaced with a parallel movement in the Fourier domain, so that the expansion and contraction information can be obtained regardless of the spatial wave number included in the moving image information to be processed.

[0042] Alternatively, a method for obtaining the two-dimensional spatial power spectrum may be applied that is different from the method for obtaining the two-dimensional spatial power spectrum, such as a Fourier transform of an autocorrelation function.

[0043] In step 206, the CPU 11 calculates the two-dimensional spatial power spectrum P t (ω x ,ω y ) one-dimensional spatial power spectrum P of the spectrum waveform in any direction (angle: -180°<θ≦+180°) t,θ (ω) is extracted for each frame.

[0044] That is, the CPU 11 extracts a one-dimensional spatial power spectrum in any one direction of the logarithmic two-dimensional spatial power spectrum for all frames. In this case, it is desirable to select a characteristic axis from spatial information contained in the moving image information to be processed. For example, the x-axis direction is selected for a subject that has a strong tendency for vertical stripes (periodicity along the x-axis of the image), and the y-axis direction is selected for a subject that has a strong tendency for horizontal stripes (periodicity along the y-axis of the image). In addition, since one line can be easily extracted for a total of four axes, the x-axis, y-axis, and two mutually orthogonal diagonal axes of the moving image information to be processed, the extraction axes may be limited to these four axes.

[0045] Alternatively, the two-dimensional spatial power spectrum may be converted into polar coordinates to separate it into the spatial wave number ω axis and the argument angle θ axis, and the desired θ component may be extracted. In either case, since the component with the largest power value is likely to be distributed along the directional component with the strongest periodicity in the moving image information to be processed, it is also possible to automatically detect directionality by combining threshold processing of the power value with straight line detection, etc. Detecting directionality makes it possible to extract information with a high S / N ratio whose power spectrum distribution is less likely to change due to image noise or vibration.

[0046] In step 208, the CPU 11 calculates the one-dimensional spatial power spectrum P t=0,θ (ω) and one-dimensional spatial power spectrum P t=k,θ (ω) and cross-correlation function C t=k Calculate (Δω).

[0047] That is, the CPU 11 uses the following equation (1) to obtain a cross-correlation function of one-dimensional spatial power spectra in a reference frame (approximately an initial frame) and an arbitrary frame of the moving image to be processed.

[0048]

number

[0049] Here, C t(Δω) is the cross-correlation function in the t frame, and Δω is the shift in the spatial wave number. Furthermore, P 0,θ (ω) and P k,θ (ω) is the one-dimensional spatial power spectrum in the reference frame and any t=k frame.

[0050] In step 210, the CPU 11 calculates the cross-correlation function C t=k Quadratic function fitting is performed in the range (Δω1 to Δω2) of the spatial wave number shift amount Δω in the vicinity of where (Δω) has a peak value. Here, the CPU 11 performs continuous function fitting to obtain the cross-correlation function C t (Δω) is the peak shift amount Δω peak (t) is calculated regardless of the interval width of the spatial wavenumber shift amount Δω.

[0051] That is, the vibrations targeted in this embodiment are minute vibrations of the order of subpixels (information of one pixel or less) to a few pixels on the image, so it is assumed that the spatial structure of the pixel region to be analyzed will hardly change (within a slight expansion or contraction). Therefore, the waveforms of the one-dimensional spatial power spectra are roughly the same, so the cross-correlation function C t (Δω) is assumed to have a peak value near Δω=0, and it is considered that fluctuations occur in a range smaller than the step size of Δω in the sampled discrete data. Therefore, the cross-correlation function C t By fitting (Δω) with a convex continuous function such as a quadratic function, minute peak fluctuations that are not dependent on the step size of Δω can be captured. This process is performed for all frames. Note that a linear function with an absolute value may be applied instead of the quadratic function described above.

[0052] Shift amount Δω peak Since the vibration waveform of (t) represents the vibration of the moving image information I(x, y, t), in step 212, the CPU 11 calculates the shift amount Δω peak The main peak frequency of the time power spectrum (time amplitude spectrum) obtained by subjecting (t) to a time Fourier transform is determined as the vibration frequency in the optical axis direction of the moving image information I(x, y, t), after which the vibration detection process is terminated.

[0053] That is, the shift amount Δω of the spatial wave number at which the cross-correlation function peaks in all frames of the moving image to be processed peak The frequency of the micro-vibration component in the optical axis direction contained in the moving image to be processed is specified by calculating the time spectrum of (t) using one-dimensional FFT processing, etc., and detecting the peak frequency using peak detection processing, etc. Note that if the amplitude of the vibration in the optical axis direction is also required, the time spectrum can be calculated as a time amplitude spectrum and converted into the physical quantity of the amplitude based on the shooting distance.

[0054] The inventors of the present invention verified the validity of the above vibration detection process using simulated moving images and actual moving images. The verification will be described below with reference to Figs.

[0055] First, the verification using a simulation video will be described.

[0056] The image used in this simulation (hereinafter referred to as the "simulation image") is 500 x 500 pixels as shown in Figure 5, and has a sine wave waveform with a wavelength of 4 pixels in the x-axis direction, and a Hanning window is applied in the x-axis direction to make the brightness value 0 (zero) at both ends of the image. This simulation image is assumed to have a sinusoidal vibration in the optical axis direction with the arrangement relationship shown in Figure 6. In other words, since it is assumed that the vibration occurs in the optical axis direction at both amplitude values ​​Δdpp at a shooting distance d, if the length of the subject on the far side (furthest state) is x, then on the close side (closest state) it becomes x'=x(d-Δdpp) / d due to the similarity relationship. If the range of the dashed dotted line is considered to be the angle of view of the imaging device, these x and x' are both sampled with the same number of pixels (500 pixels) and appear to expand and contract on the captured image, so the vibration in the optical axis direction was reproduced by relatively expanding the simulation image on the close side and relatively contracting it on the far side.

[0057] In this verification experiment, the ratio of the amplitude value Δdpp of the subject to the shooting distance d was changed, and the -2 (%) (hereinafter referred to as "Case 1") and -4The results for two cases are shown, one for which the amplitude is equivalent to Δdpp = 0.2 mm and the other for which the amplitude is equivalent to Δdpp = 1.4 μm, assuming a shooting distance d = 1,000 mm.

[0058] Figure 7 shows the Δω for one period of vibration (expansion / contraction) in the first case. peak The time waveform of (t) and the theoretical value are shown in Fig. 8. The Δω for one period of vibration (expansion / contraction) in the second case is shown in Fig. 8. peak The time waveform and theoretical value of (t) are shown.

[0059] As shown in Fig. 7, in the first case, i.e., when the amplitude is sufficiently large relative to the shooting distance, it can be said that the vibration waveform itself, including the vibration frequency and amplitude, is generally correctly reproduced. On the other hand, as shown in Fig. 8, in the second case, i.e., when the amplitude is small relative to the shooting distance, the amplitude cannot be correctly reproduced and the vibration waveform is not smooth, but it can be said that the vibration frequency is accurately detectable by calculating the time spectrum. Since it has been confirmed that it is difficult to detect the vibration frequency at amplitudes lower than this, this method uses both amplitude values ​​of 1.4 × 10 for the shooting distance. -4 It can be said that the device is capable of detecting the vibration frequency of minute vibrations of about 100%, sampled at 500 pixels.

[0060] The above pixel counts are listed for reference. Although reducing the pixel count too much may result in a decrease in analysis accuracy, it is not realistic to assume an extremely high pixel count (for example, even a 4K video is roughly 4,000 x 2,000 pixels, so anything more than that is unrealistic). Assuming that the subject of analysis is captured using a general Full HD or 4K pixel camera and then cut out, a pixel count of around 500 pixels is considered appropriate.

[0061] Next, verification using actual moving images will be described.

[0062] The actual moving images used here were taken with a USB (Universal Serial Bus) camera placed approximately 2,000 mm away and facing approximately vertically downward, while an A1-sized checkerboard as shown in Figure 9 was being vibrated by a vibrator in an approximately vertical direction as shown in Figure 10 (it was not guaranteed that the optical axis of the camera and the checkerboard surface were strictly orthogonal, and it was not guaranteed that the vibration direction of the vibrator was strictly aligned with the optical axis of the camera). The pixel data transferred from the camera was acquired in YUV422 format, and we verified the output of moving images in uncompressed YUV422 avi format and the output of moving images by converting the data into H.264 compressed mp4 format of YUV420, which is the same as that of general digital cameras. The resolution and frame rate of the captured image were 640 x 480 pixels and 30 fps (actually, the accurate value was 29.83 fps calculated separately experimentally), and the image was taken for 10 seconds.

[0063] For both the uncompressed and H.264 formats, the vibrator applied a voltage of 5 Hz to produce displacements of 4.0, 2.0, 1.0, 0.5, and 0.1 mm amplitude values. Since the checkerboard was attached to the styrene board on which it was printed with double-sided tape, the displacement amplitude value of the vibrator became the amplitude setting value of the checkerboard, which was the subject.

[0064] As preprocessing for vibration analysis, the central 256 x 256 pixels of the captured image (640 x 480 pixels) were extracted and a two-dimensional Hanning window was applied to eliminate the effects of the image edges.

[0065] Figure 11 shows the Δω at both amplitude values ​​of 0.5 mm, which is the minimum amplitude that could be analyzed in a non-compressed format. peak The time amplitude spectrum of (t) is shown in Fig. 12. The minimum amplitude that could be analyzed in H.264 format, Δω peak 11 and 12 show the time amplitude spectrum of (t). As shown in Fig. 11 and Fig. 12, a 5 Hz peak is clearly visible in both cases. For vibration conditions smaller than this, the peak was buried in the noise level of the spectrum and could not be confirmed.

[0066] The vibration detection process according to this embodiment makes it possible to detect components in the direction of the optical axis of the image capture in vibration analysis techniques including micro-vibrations based on video analysis techniques, and to detect out-of-plane vibrations such as those on walls, floors, and ceilings. In addition, by utilizing the detected vibration frequency components, applications to health monitoring of buildings, structures, equipment, and the like can be expected.

[0067] As described above, according to this embodiment, a moving image obtained by shooting with a shooting device is acquired, a two-dimensional spatial power spectrum of the moving image is derived by spatial frequency analysis and logarithmic transformation of the spatial wave number axis, and a physical quantity related to vibration in the optical axis direction of the shooting device for the moving image is detected based on the derived two-dimensional spatial power spectrum. Therefore, vibration in the optical axis direction of the shooting can be detected with high accuracy from the moving image obtained by shooting.

[0068] Moreover, according to this embodiment, a one-dimensional spatial power spectrum in an arbitrary spatial direction is derived for each frame from the two-dimensional spatial power spectrum, a cross-correlation function between the one-dimensional spatial power spectrum of the reference frame and the one-dimensional spatial power spectrum of another frame is derived for each frame, a peak fluctuation is derived by fitting the peak of the cross-correlation function for each frame with a continuous function, a time spectrum is derived from the peak fluctuation, and a physical quantity related to vibration is detected using the time spectrum. Therefore, the physical quantity related to vibration can be detected with higher accuracy.

[0069] Furthermore, according to the present embodiment, both the vibration frequency and the amplitude of the vibration, or only the vibration frequency, are detected as the physical quantity related to the vibration. Therefore, at least one of the vibration frequency and the amplitude of the vibration can be detected.

[0070] Furthermore, according to this embodiment, the moving image is a black and white moving image, so that vibrations in the direction of the optical axis of photography can be detected with high accuracy from the black and white moving image.

[0071] In the above embodiment, the case where a photographing device that photographs black and white images is applied as the photographing device 20 has been described, but the present invention is not limited thereto. For example, a photographing device that photographs color images may be applied as the photographing device 20. In this case, however, it is necessary to convert the color moving image obtained by photographing into a black and white moving image before deriving the two-dimensional spatial power spectrum, or to derive and apply a physical quantity related to vibration in the optical axis direction of the photographing device for each of R (red), G (green), and B (blue). In the latter case, the average value of the physical quantities derived for each of RGB is taken, or one of them is selectively applied.

[0072] In the above embodiment, for example, the hardware structure of the processing unit that executes each process of the acquisition unit 11A, the derivation unit 11B (first derivation unit 11B1 to fourth derivation unit 11B4), and the detection unit 11C may be various processors as shown below. As described above, the various processors include a CPU, which is a general-purpose processor that executes software (programs) and functions as a processing unit, as well as a programmable logic device (PLD), which is a processor whose circuit configuration can be changed after manufacture such as an FPGA (Field-Programmable Gate Array), a dedicated electric circuit, which is a processor having a circuit configuration designed exclusively for executing a specific process such as an ASIC (Application Specific Integrated Circuit), etc.

[0073] The processing unit may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA).The processing unit may also be configured with a single processor.

[0074] As an example of configuring the processing unit with one processor, first, there is a form in which one processor is configured with a combination of one or more CPUs and software, as represented by computers such as client and server, and this processor functions as the processing unit. Second, there is a form in which a processor is used that realizes the functions of the entire system including the processing unit with one IC (Integrated Circuit) chip, as represented by System On Chip (SoC), etc. In this way, the processing unit is configured using one or more of the above various processors as a hardware structure.

[0075] Furthermore, more specifically, the hardware structure of these various processors can be an electric circuit that combines circuit elements such as semiconductor elements. [Explanation of symbols]

[0076] 10 Vibration detection device 11 CPU 11A Acquisition Department 11B Derivation part 11B1 1st derivation part 11B2 Second derived part 11B3 Third derivation part 11B4 4th derivation part 11C Detector 12. Memory 13 Storage section 13A Vibration Detection Program 13B Video Database 14 Input section 15 Display 16 Media Read / Write Device 17 Recording media 18 Communication I / F section 20 Imaging Equipment

Claims

1. an acquisition unit that acquires a moving image obtained by shooting using the shooting device; a derivation unit that derives a two-dimensional spatial power spectrum of the moving image by spatial frequency analysis and logarithmic transformation of a spatial wavenumber axis; a detection unit that detects a physical quantity related to vibration of the moving image in the optical axis direction caused by the imaging device based on the two-dimensional spatial power spectrum; A vibration detection device comprising:

2. The lead-out portion is deriving a one-dimensional spatial power spectrum in an arbitrary spatial direction for each frame from the two-dimensional spatial power spectrum; Deriving a cross-correlation function between the one-dimensional spatial power spectrum of a reference frame and the one-dimensional spatial power spectrum of another frame for each frame; Deriving a variation of the peak by fitting a continuous function to the peak of the cross-correlation function for each frame; Deriving a time spectrum from the fluctuation of the peak; The detection unit is Detecting a physical quantity related to the vibration using the time spectrum. The vibration detection device according to claim 1 .

3. The detection unit detects both a vibration frequency and an amplitude of the vibration, or only the vibration frequency, as a physical quantity related to the vibration. The vibration detection device according to claim 1 or 2.

4. The moving image is at least one of black and white and color moving images. The vibration detection device according to claim 1 or 2.

5. Acquire a moving image obtained by photographing with a photographing device; deriving a two-dimensional spatial power spectrum of the video by spatial frequency analysis and logarithmic transformation of a spatial wavenumber axis; detecting a physical quantity related to vibration of the moving image in the optical axis direction caused by the imaging device based on the two-dimensional spatial power spectrum; A vibration detection program that causes a computer to execute the processing.

Images