Visual vibration measurement method, system and device based on multi-order derivative enhanced phase motion estimation and storage medium

Abstract

The invention provides a visual vibration measurement method and system based on multi-order derivative enhanced phase motion estimation, equipment and a storage medium, and belongs to the technical field of vibration measurement. According to the method, the multi-order derivative of the intensity image is calculated to construct the complex analysis signal containing the motion phase information, and accurate data support is provided for subsequently obtaining the displacement signal from the phase difference; obtaining a scale factor between the phase difference and the displacement through the intensity signal and the multi-order derivative; and obtaining an accurate motion estimation result according to the phase difference and the scale factor. According to the method, the accuracy and reliability of phase motion estimation can be further improved, and measurement errors caused by parameter setting and approximate estimation are avoided.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of vibration measurement, and particularly relates to a visual vibration measurement method and system based on multi-order derivative enhanced phase motion estimation, a device and a storage medium. BACKGROUND

[0002] Vibration is the basic form of structural dynamic characteristics. In engineering practices such as civil engineering and mechanical systems, vibration measurement is not only a key link of structural health monitoring, but also provides important data support for equipment state evaluation, fault diagnosis and vibration control. Currently, contact sensors such as accelerometers and displacement sensors are mainly used for vibration measurement. However, the contact measurement method has problems such as additional mass effect, sparse spatial sampling, high installation and maintenance requirements, and is difficult to meet the demand of full-field and high spatial resolution vibration measurement.

[0003] With the development of imaging devices and image processing technology, visual vibration measurement methods have attracted much attention. Compared with traditional contact measurement, visual vibration measurement methods have the advantages of non-invasiveness, easy operation and high spatial measurement density. Existing visual vibration measurement methods mainly include digital image correlation method, target tracking method and optical flow method. Among them, the digital image correlation method and the target tracking method often need surface treatment such as spraying speckle pattern or arranging marker points, which limits their application in large structures or hard-to-reach environments. The optical flow method stands out because it does not require surface treatment. The phase-based optical flow method extracts spatial displacement from frequency domain phase change based on Fourier displacement theorem, and has stronger noise resistance and accuracy than intensity-based optical flow method, so it is widely used in vibration measurement. However, the existing phase-based motion estimation methods generally have the problem that key parameters such as filter size and phase difference-displacement scaling factor need to be manually adjusted or approximated by experience, which introduces large measurement errors. These problems reduce the accuracy of visual vibration measurement and restrict the popularization and development of the technology in practical applications.

[0004] Chinese invention patent "Low-frequency vibration measurement method based on machine vision", with publication number CN 109883533 B. This patent uses a key point detection algorithm to detect the edge information of the more prominent objects in the image sequence, takes a region containing multiple obvious corner points as a template, and sequentially calculates the template matching degree in the subsequent image sequence to obtain the matching matrix of the template region. The best matching position is searched for by using the curve fitting method for the matching matrix, and the extraction of the amplitude and frequency of vibration is completed. However, the template matching algorithm depends on good imaging quality, and a noisy measurement environment can also cause the method to fail, and the method has high computational complexity.

[0005] A Chinese invention patent, "Structure vibration video measurement method and system based on spatial time direction division", with publication number CN115452125B. This patent first uses the space-frequency conversion method to process the time sequence change image sequence to obtain the first frequency domain spatial complex feature sequence; uses the spatial time direction division to process the adjacent frames of the first frequency domain spatial complex feature sequence to obtain the second frequency domain spatial complex feature sequence; uses the four-quadrant inverse tangent operation to operate the second frequency domain spatial complex feature sequence to obtain the adjacent frame phase change rich in relative displacement change field information; calculates the measured object conversion proportion factor, and finally obtains the vibration displacement signal. However, this method uses the traditional image convolution-based space-frequency conversion method to obtain complex features and phase information, which not only has large calculation amount and low measurement efficiency, but also needs to manually set and adjust the size of the image convolution kernel, and the wrong convolution kernel size will also affect the accuracy of the measurement result.

[0006] A Chinese invention patent, "Structure vibration video measurement method and system based on derivative phase optical flow method", with publication number CN115841504A. This patent first uses the camera to collect the vibration process of the measured structure at a frame rate that meets the Nyquist sampling rate to obtain a time sequence change image sequence rich in vibration information, which is also used as the real part sequence; uses a two-dimensional first derivative operator to do two-dimensional convolution with each image sequence frame by frame to obtain the imaginary part sequence of the analytic signal; uses a complex combination operator to combine the real part sequence and the imaginary part sequence one by one to obtain an analytic sequence rich in phase information; uses the four-quadrant inverse tangent operation to operate the above analytic sequence to obtain the adjacent frame phase change rich in relative displacement change field information, and finally obtains the vibration displacement signal. However, this method only uses the first derivative of the intensity image to construct the analytic signal, and there is an unknown parameter in the process of constructing the analytic signal, which also participates in the calculation from the phase difference to the vibration displacement signal. In actual operation, this unknown parameter needs to be approximately estimated and manually set, and inappropriate parameter setting will introduce vibration displacement measurement error. SUMMARY

[0007] The purpose of the present application is to solve the problem that the measurement accuracy of the existing phase-based visual vibration measurement method is limited due to the manual selection or empirical approximation of key parameters. A visual vibration measurement method and system based on multi-order derivative enhanced phase motion estimation are proposed. The method uses intensity images and their multi-order spatial derivatives to construct complex analytic signals autonomously, and directly calculates the proportion factor between phase difference and displacement, which is beneficial to improve the accuracy and reliability of visual vibration measurement.

[0008] The present application provides a visual vibration measurement method based on multi-order derivative enhanced phase motion estimation, comprising:

[0009] Step 1: Obtain the vibration video through the video acquisition device;

[0010] Step 2: Calculate the first derivative, second derivative and third derivative of the vibration video intensity image along the vibration direction frame by frame;

[0011] Step 3: Construct a complex analytic signal based on the derivatives along the vibration direction, perform an arctangent transformation on the constructed complex analytic signal, extract the phase and calculate the phase difference;

[0012] Step 4: Calculate the phase difference-displacement scaling factor using the intensity image and the derivative signals, and extract the displacement signal from the phase difference according to the scaling factor.

[0013] Further, in step 1, the frame rate of the video acquisition satisfies the Nyquist sampling theorem.

[0014] Further, in step 2, the derivative calculation method includes a classical gradient operator method based on finite difference, a Laplace operator method and an optimization method based on gradient.

[0015] Further, in step 2, taking x-direction vibration as an example, it is specifically:

[0016] Step 2.1: Process the collected image using a single image scale model, select the intensity image I(x,y,t0) at time t0, where (x,y) is the spatial coordinate;

[0017]

[0018] Step 2.2: For intensity image features with vibration only along the x-direction, use first derivative, second derivative and third derivative operations;

[0019]

[0020] Further, the complex analytic signal is:

[0021]

[0022] Where i is the imaginary unit;

[0023] Use the arctangent transformation operation to analyze the sequence and obtain the relative displacement change field information;

[0024]

[0025] Step 33, for the video frame at time t0+Δt, the vibration δ x occurs in the x-direction, obtaining the complex analytic signal at time t0+Δt and the corresponding phase;

[0026]

[0027] Where δ xa displacement signal in the x direction at time t0 and t0+Δt;

[0028] Step 34, calculating the phase difference related to the vibration signal;

[0029]

[0030] Further, the phase difference-displacement scaling factor is:

[0031]

[0032] According to the scaling factor, the displacement signal δ is extracted from the phase difference:

[0033]

[0034] The application also provides a visual vibration measurement system based on multi-order derivative enhanced phase motion estimation, comprising:

[0035] The acquisition module: uses a frame rate satisfying the Nyquist sampling theorem to shoot the vibration state video of the measured object;

[0036] The derivative calculation module: used to calculate the first, second and third derivatives of the intensity image features of each frame of the vibration video;

[0037] The phase difference calculation module: used to construct a complex analytic signal and extract the phase therefrom, and further calculate the phase difference signal before and after displacement;

[0038] The displacement extraction module: used to calculate the phase difference-displacement scaling factor and extract the displacement signal from the phase difference, and complete the vibration measurement.

[0039] The application also provides a computer readable storage medium having a computer program / instruction stored thereon, which, when executed by a processor, realizes the steps of the above-mentioned visual vibration measurement method based on multi-order derivative enhanced phase motion estimation.

[0040] The application also provides an electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor realizes the steps of the visual vibration measurement method based on multi-order derivative enhanced phase motion estimation when executing the program.

[0041] The application has the following beneficial effects:

[0042] The method improves the accuracy of the visual vibration measurement method, constructs a complex analytical signal by using the multi-order spatial derivatives of the intensity image, directly determines the accurate proportion factor between the phase difference and the displacement, and avoids the system error caused by manual adjustment or experience approximation parameters in the traditional method. Meanwhile, the method enhances the measurement efficiency and universality of the visual vibration measurement method, breaks away from the dependence on manual parameter setting by virtue of the self-calibration capability, does not need the user to manually adjust the key parameters, forms a universal measurement framework suitable for different scenes, retains the low computational complexity advantage of the derivative operation, and significantly improves the measurement efficiency and reliability. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 A flowchart of a visual vibration measurement method based on multi-order derivative enhanced phase motion estimation according to an embodiment of the present application;

[0044] Figure 2 A visual vibration measurement experimental scene diagram based on multi-order derivative enhanced phase motion estimation according to an embodiment of the present application;

[0045] Figure 3 A visual vibration measurement result based on multi-order derivative enhanced phase motion estimation according to an embodiment of the present application. DETAILED DESCRIPTION

[0046] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative labor fall within the protection scope of the present application.

[0047] The present application discloses a visual vibration measurement system based on multi-order derivative enhanced phase motion estimation, comprising:

[0048] The acquisition module: uses a camera to shoot the vibration state video of the measured object at a frame rate that meets the Nyquist sampling theorem;

[0049] The derivative calculation module: used for calculating the first-order, second-order and third-order derivatives of each frame of the intensity image of the vibration video;

[0050] The phase difference calculation module: used for constructing a complex analytical signal and extracting the phase therefrom, and then calculating the phase difference signal before and after displacement;

[0051] The displacement extraction module: used for calculating the phase difference-displacement proportion factor and extracting the displacement signal from the phase difference, and completing the vibration measurement.

[0052] The present application also discloses a visual vibration measurement method based on multi-order derivative enhanced phase motion estimation, which specifically comprises:

[0053] Step 1, collect a continuous video sequence containing the target vibration information, each frame of the video sequence can be represented as an intensity matrix I(x, y, t) related to pixel position (x, y) and time t, where the vibration signal δ is encoded in the intensity variation. During video acquisition, the relative position between the imaging device and the target remains stable (no additional relative motion except the target itself vibration), and the video frame rate meets the Nyquist sampling law;

[0054] Step 2, for each frame of the intensity image in the vibration video obtained in step 1, based on the principle of phase flow and the theoretical derivation of motion estimation, a simplified single-image scale model is used for processing. Assuming that the vibration only occurs along the x direction (this processing method can be extended to other vibration directions through derivative operation in the corresponding direction), three consecutive derivative operations are performed along the x direction for each frame of intensity image, and the first derivative I (1) , the second derivative I (2) and the first derivative I (3) of the intensity image are obtained.

[0055] Step 3, based on the first derivative, the second derivative and the third derivative obtained in step 2, a complex analytic signal is constructed. The inverse tangent transformation is performed on the constructed complex analytic signal to extract the phase at a specific time; similarly, the derivative operation of step 2 and the complex analytic signal construction process of this step are repeated for the intensity image at the time after a small time increment, and the phase at this time is extracted, and the phase difference between the phase signals corresponding to the intensity images before and after the small time increment is calculated.

[0056] Step 4, combining the intensity image in step 1, the multiple-order derivatives obtained in step 2 and the phase difference derivation process in step 3, the deterministic proportion factor k between the phase difference and the displacement is obtained through theoretical derivation. Based on the phase difference calculated in step 3 and the phase difference-displacement proportion factor obtained in step 4, the vibration displacement signal δ of the target can be accurately extracted from the phase difference. For the scene where the vibration direction is along the y direction or other directions, the above steps 2-4 can be repeated by performing multiple-order derivative operations on the intensity image along the corresponding direction to realize accurate measurement of the vibration displacement in any direction.

[0057] In the above step 1, the video acquisition device includes a CCD industrial camera, a stable light source, a camera tripod and a computer.

[0058] In the above step 2, the derivative calculation method includes a classical gradient operator method based on finite difference, a Laplace operator method and an optimization method based on gradient, etc.

[0059] In the above step 2, the intensity image I(x, y, t0) at time t0 is selected, where (x, y) is the spatial coordinate;

[0060] When the vibration occurs in the x direction, the first derivative, the second derivative, and the third derivative operations are used for the intensity image features with vibration only in the x direction;

[0061]

[0062] In step 3, the phase difference is specifically calculated as:

[0063] Step 31, for the video frame at time t0, the first derivative signal I (1) , the second derivative signal I (2) , and the third derivative signal I (3) are obtained by deriving the original intensity image I. where i is the imaginary unit.

[0064] Step 32, the phase information is extracted from the complex analytic signal C(x, y, t0) using the arctangent transformation arg.

[0065] Step 33, for the video frame at time t0+Δt, assuming that the vibration δ occurs in the x direction, the operations in steps 41 and 42 are repeated to obtain the complex analytic signal C(x, y, t0+Δt) and the corresponding phase φ(x, y, t0+Δt) at time t0+Δt.

[0066] Step 34, the phase difference is obtained by performing a difference operation on the phase at time t0+Δt and the phase at time t0:

[0067] In step 4, the phase difference-displacement scaling factor is calculated using the intensity image and the derivative signals. According to the phase difference and the scaling factor, the displacement signal is extracted from the phase difference.

[0068] In step 2, when the vibration occurs in the y direction, the first derivative, the second derivative, and the third derivative operations are used for the intensity image features with vibration only in the y direction.

[0069]

[0070] Embodiment 1

[0071] A visual vibration measurement method based on multi-order derivative enhanced phase motion estimation, as shown in Figure 1 , includes:

[0072] Step 1, use a high-speed camera to obtain the vibration state video generated by an air compressor during operation. The experimental hardware device layout is as shown inFigure 2 As shown, LED light sources provide sufficient illumination for the experimental scene. A high-speed camera and laser displacement sensor are activated, and after the air compressor stabilizes and vibrates, vibration video and a reference signal from the air compressor are simultaneously acquired. In the video sequence captured by the high-speed camera, each frame is an intensity matrix I(x,y,t) related to the pixel position (x,y) and time t, with the vibration signal encoded in the intensity changes. The displacement signal acquired by the laser displacement sensor serves as the reference data for subsequent verification of the accuracy of the method of this invention.

[0073] Step 2: Process the vibration video collected in Step 1 frame by frame, and perform continuous third-order derivative operations along the vibration direction (the vibration direction in this embodiment is the horizontal direction) to obtain the first, second and third derivatives of the air compressor vibration video intensity image;

[0074] Step 3: Construct a complex analytic signal C(x,y,t0) based on the third derivative obtained in Step 2. Perform an arctangent transform on the complex analytic signal to extract the phases φ(x,y,t0) and φ(x,y,t0+Δt) at adjacent times t0 and t0+Δt (Δt is the camera frame interval), and then calculate the phase difference Δφ between the two times.

[0075] Step 4: Calculate the phase difference-displacement scaling factor using the intensity image and derivative signals of each order. Based on the scaling factor, extract the displacement signal from the phase difference to complete the quantitative conversion from phase difference to actual displacement.

[0076] like Figure 2 As shown, the visual vibration measurement method based on multi-derivative enhanced phase motion estimation provided by this invention was used to measure the vibration of an air compressor in a laboratory environment experiment, with the measurement results of a laser displacement sensor as a reference; Figure 3 As shown, the vibration displacement signal and spectrum signal of the air compressor obtained by the visual vibration measurement method based on multi-derivative enhanced phase motion estimation provided by the present invention are highly consistent with the measurement results of the laser displacement sensor, verifying the effectiveness and accuracy of the vibration measurement of the present method.

[0077] In some other preferred embodiments of the present invention, a computer-readable storage medium is also provided, on which a computer program / instruction is stored, wherein when the computer program is executed by a processor, it implements the steps of the visual vibration measurement method based on multi-order derivative enhanced phase motion estimation described in any of the above embodiments.

[0078] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, the computer program can include the processes of the above-mentioned visual vibration measurement method based on multi-order derivative enhanced phase motion estimation, which will not be repeated here.

[0079] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or N embodiments or examples. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the present specification and the features of the different embodiments or examples without contradiction.

[0080] In addition, the terms "first", "second" are only for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "N" is at least two, for example, two, three, etc., unless otherwise specifically limited.

[0081] Any process or method descriptions in flow charts or otherwise described herein can be understood as representing code modules, segments, or portions of code that include one or more executable instructions for implementing the specified logic functions (or steps) and / or can be implemented by one or more hardware or software components, either of which can be utilized to perform the processes. It should also be understood that each process or method described in the specification can be embodied in the form of a computer readable medium storing the relevant code modules or portions of code.

[0082] The logic and / or steps represented in flow diagrams or otherwise described herein, for example, can be considered as a sequence of instructions to implement logic functions, and can be embodied in any computer-readable medium for use by an instruction execution system, apparatus, or device, such as a computer-based system, processor- containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. In the context of this specification, a "computer-readable medium" can be any means that can contain, store, communicate, propagate or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The computer-readable medium can be a machine-readable storage device (e.g., magnetic, optical or other) a machine-readable storage diskette (e.g., floppy, flexible or other), a machine-readable storage card (e.g., ROM, EEPROM, flash memory or other), a machine- readable storage tape (e.g., magnetic, optical or other), a machine-readable storage medium (e.g., a portable electronic device, a computer diskette, a computer memory, a programmable logic device, an application-specific integrated circuit, a programmable logic controller, a digital signal processor, a microprocessor, a microprocessor array or other), or a machine- readable interface device (e.g., a wired or wireless interface device). The computer-readable medium can also be, or be included in, a computer program product apparatus that can be employed to program, store, or locate the instructions for use by or in connection with the instruction execution system, apparatus, or device. The terms "machine-readable medium" and "computer-readable medium" include any medium that is capable of storing, communicating, propagating or transporting a program for use by or in connection with an instruction execution system, apparatus, or device. The computer-readable medium can also be paper or other suitable medium upon which the program is printed, as the program can be electronically captured, via, for example, optical scanning of the paper or other medium, then compiled, interpreted, or otherwise processed in a suitable manner, if necessary, and then stored in a computer memory.

[0083] It should be understood that aspects of the application can be implemented in hardware, software, firmware or combinations thereof. In the above embodiments, the N steps or methods can be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. As such, if implemented in hardware, and in another embodiment, any of the following technologies, known in the art, or combinations thereof, can be used: discrete logic circuitry having logic gates for implementing logic functions on data signals, application specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field programmable gate arrays (FPGA), and the like.

[0084] Those of ordinary skill in the art can understand that all or part of the steps carried out by the above-mentioned embodiment methods can be completed by programs instructing relevant hardware, and the programs can be stored in a computer-readable storage medium. When the programs are executed, they include one of the steps of the method embodiments or a combination thereof.

[0085] In addition, each function unit in each embodiment of the present application can be integrated in one processing module, or each unit can be physically present separately, or two or more units can be integrated in one module. The integrated module can be realized in the form of hardware or in the form of a software function module. When the integrated module is realized in the form of a software function module and sold or used as an independent product, it can also be stored in a computer readable storage medium.

[0086] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it should be understood that the above embodiments are exemplary and should not be construed as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above embodiments within the scope of the present application.

Claims

1. A visual vibration measurement method based on phase motion estimation enhanced by multi-order derivatives, characterized in that, include: Step 1: Acquire vibration video using a video capture device; Step 2: Calculate the first, second, and third derivatives of the vibration video intensity image along the direction of vibration frame by frame; Step 3: Construct a complex analytic signal based on the derivatives of each order along the direction of vibration, perform an arctangent transformation on the constructed complex analytic signal, extract the phase, and calculate the phase difference; Step 4: Calculate the phase difference-displacement scaling factor using the intensity image and derivative signals of each order, and extract the displacement signal from the phase difference based on the scaling factor.

2. The visual vibration measurement method based on multi-order derivative enhanced phase motion estimation according to claim 1, characterized in that, In step 1, the frame rate of the video capture satisfies the Nyquist sampling theorem.

3. The visual vibration measurement method based on multi-order derivative enhanced phase motion estimation according to claim 1, characterized in that, In step 2, the derivative calculation methods include the classical gradient operator method based on finite difference, the Laplace operator method, and the gradient-based optimization method.

4. The visual vibration measurement method based on multi-order derivative enhanced phase motion estimation according to claim 1, characterized in that, In step 2, taking vibration in the x-direction as an example, the specific steps are as follows: Step 2.1: Process the acquired images using a single-image scale model, and select the intensity image I(x,y,t0) at time t0, where (x,y) are spatial coordinates; Step 2.2: For the intensity image features of vibration only along the x-direction, use the first, second, and third derivatives for calculation; 5. The visual vibration measurement method based on multi-order derivative enhanced phase motion estimation according to claim 4, characterized in that, The complex analytical signal is: Where i is the imaginary unit; The sequence is analyzed using the arctangent transform operation to obtain information rich in relative displacement change field. Step 33: For the video frame at time t0+Δt, the vibration δ x Occurring in the x-direction, we obtain the complex analytic signal and its corresponding phase at time t0+Δt; Where, δ x The displacement signals in the x-direction at times t0 and t0+Δt; Step 34: Calculate the phase difference related to the vibration signal; 6. The visual vibration measurement method based on multi-order derivative enhanced phase motion estimation according to claim 5, characterized in that, The phase difference-displacement scaling factor is: Based on the scaling factor, the displacement signal δ is extracted from the phase difference:

7. A visual vibration measurement system based on multi-order derivative enhanced phase motion estimation, performing the visual vibration measurement method as described in claims 1-6, characterized in that, include: Acquisition module: Captures video of the vibration state of the object under test using a frame rate that satisfies the Nyquist sampling theorem; Derivative calculation module: used to calculate the first, second, and third derivatives of the intensity image features of each frame of the vibration video; Phase difference calculation module: used to construct complex analytic signals, extract phase from them, and then calculate the phase difference signal before and after displacement; Displacement extraction module: used to calculate the phase difference-displacement scaling factor and extract the displacement signal from the phase difference to complete the vibration measurement.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method described in any one of claims 1 to 6.

Images