A microwave vibration camera system integrating stereo vision and microwave technology and its visualization method

Abstract

This invention provides a microwave vibration camera system and visualization method that integrates stereo vision and microwave technology. The system includes: a microwave transceiver and a stereo camera fixedly connected; the stereo camera outputs camera data of the target to a camera data processing module to calculate the three-dimensional coordinates of each pixel in the camera coordinate system; a coordinate transformation module converts the three-dimensional coordinates into microwave distance-angle thermographic coordinates; the microwave data processing module processes the baseband signal obtained from the microwave transceiver to perform distance-angle thermographic imaging of the target within the field of view; and a displacement visualization module extracts the displacement based on the microwave distance-angle thermographic coordinates corresponding to each pixel, and maps it onto the visual image using methods including color encoding to obtain the visualized real-time vibration state within the field of view. This invention provides a novel method for measuring point identification and vibration source localization in microwave vibration detection from the perspective of vibration visualization, improving the application potential of microwave full-field vibration measurement technology in complex scenarios.

Description

Technical Field

[0001] This invention relates to the field of full-field vibration visualization technology that integrates stereo vision and microwave fusion, specifically, to a microwave vibration camera system and visualization method that integrates stereo vision and microwave fusion. More particularly, it relates to a full-field vibration visualization method and microwave vibration camera system that integrates stereo vision and microwave fusion. Background Technology

[0002] By capturing global vibration displacement, velocity, or acceleration information of the target object, whole-field vibration measurement technology can provide key information such as deformation, stress distribution, and structural performance of the tested object. Therefore, this technology is widely used in fields such as health monitoring of large structures, analysis of mechanical properties of materials, and fault diagnosis of mechanical equipment, and has become a key testing method in engineering and scientific research.

[0003] Microwave sensing technology, with its multi-scale, large detection range, high sensitivity, and all-weather capabilities, offers a new solution for monitoring the full-field vibration of large structures and equipment. Building upon microwave single-point vibration measurement and long-distance multi-point vibration measurement technologies, significant progress has been made in microwave full-field vibration measurement technology—utilizing a distance-angle joint-dimensional nonlinear demodulation algorithm, microwave transceivers can perform distance-angle imaging of all measurement points globally. Furthermore, by employing an interferometric phase evolution tracking algorithm, vibration displacement information along the line-of-sight can be obtained for each measurement point.

[0004] However, due to the low resolution of distance-angle imaging in microwave transceivers, identifying all measurement points and locating vibration sources in the entire field becomes a challenging problem in practical measurements. Although some studies have attempted to align and calibrate visual images and microwave data, these studies often use three-dimensional point clouds for microwave imaging or assume that all targets are in the same plane. This differs significantly from the distance-angle thermal imaging method used in microwave full-field vibration measurements and fails to meet the accuracy requirements for measurement point identification and vibration source location in microwave full-field vibration measurements, thus limiting the application of microwave full-field vibration measurement technology in complex scenarios.

[0005] Therefore, there is an urgent need for an intuitive and accurate microwave vibration measurement point identification and vibration source localization technology and method. Summary of the Invention

[0006] To address the shortcomings of existing technologies, the purpose of this invention is to provide a microwave vibration camera system that integrates stereo vision and microwave technology, as well as its visualization method.

[0007] A microwave vibration camera system integrating stereo vision and microwave fusion according to the present invention includes: a microwave transceiver, a stereo camera, a microwave data processing module, a camera data processing module, a coordinate transformation module, and a displacement visualization module;

[0008] The microwave transceiver and the stereo camera are fixedly connected. The stereo camera outputs the camera data of the target to the camera data processing module to calculate the three-dimensional coordinates of each pixel in the camera coordinate system.

[0009] The coordinate transformation module converts the three-dimensional coordinates into microwave distance-angle thermal map coordinates; the microwave data processing module processes the baseband signal obtained by the microwave transceiver and performs distance-angle thermal map imaging on the target within the field of view.

[0010] The displacement visualization module extracts the displacement based on the microwave distance-angle thermogram coordinates corresponding to each pixel, and maps it onto the visual image through methods including color encoding to obtain the real-time vibration state visualized within the field of view.

[0011] Preferably, the stereo camera uses stereo vision forms including binocular cameras, structured light cameras, and TOF cameras to output camera data of the target;

[0012] The camera data includes parallax or depth information;

[0013] The camera data processing methods include triangulation, time difference calculation, structured light, and deep learning.

[0014] Preferably, the microwave transceiver is used to transmit and receive linear frequency modulated continuous wave microwave signals, output multi-channel baseband signals, and is used for subsequent distance-angle thermal mapping of targets within the field of view, as well as for measuring the vibration displacement of targets within the field of view along the line of sight based on the distance-angle index of each target within the field of view.

[0015] Preferably, the coordinate transformation module is used to calculate the coordinate transformation matrix between the camera coordinate system and the antenna coordinate system, and to transform and map each pixel coordinate to obtain the corresponding microwave distance-angle thermal map coordinates.

[0016] Preferably, the displacement visualization module is used to extract the line-of-sight displacement corresponding to the distance-angle index of each pixel, and after color encoding, it is superimposed on the original RGB image to obtain a real-time displacement amplitude heatmap visualized within the field of view.

[0017] A visualization method for a microwave vibration camera system that fuses stereo vision and microwaves, according to the present invention, includes:

[0018] Step S1: Calibrate the coordinate transformation matrices of the camera coordinate system and the antenna coordinate system, as well as the intrinsic parameter matrix, extrinsic parameter matrix, and distortion coefficients of the stereo camera;

[0019] Step S2: Convert the coordinates of each pixel to obtain the corresponding microwave distance-angle thermal map coordinates;

[0020] Step S3: Extract displacement information along the microwave line of sight and output the displacement visualization.

[0021] Preferably, the intrinsic parameter matrix, extrinsic parameter matrix, and distortion coefficients of the calibrated camera are calibrated using methods including the checkerboard calibration method, the Zhang Zhengyou calibration method, and open-source toolkits.

[0022] The coordinate transformation matrix for calibrating the camera coordinate system and the antenna coordinate system includes denoting the coordinates of each reference target in the camera coordinate system and the antenna coordinate system as follows:

[0023]

[0024] Among them, P cam Let P be the three-dimensional coordinates of the current target in the camera coordinate system. cam One column represents one target; P radar Let P be the three-dimensional coordinates of the current target in the antenna coordinate system. radar Each column represents a target; n≥3;

[0025] Let the average coordinates of all current reference targets in the camera coordinate system be... The average coordinates in the antenna coordinate system are Right now:

[0026]

[0027] Constructing a matrix in Indicates P cam The i-th column, Indicates P radar The i-th column. Perform singular value decomposition on matrix M:

[0028] M=UΣV T

[0029] The rotation matrix from the camera coordinate system to the antenna coordinate system is:

[0030] R = VU T

[0031] The translation matrix is:

[0032]

[0033] Preferably, step S2 includes:

[0034] Step S2.1: Take a pixel in the pixel plane of one of the RGB cameras of the stereo camera, and mark the pixel coordinates of the pixel as (u i ,v i), calculate the three-dimensional coordinates (xc_i, yc_i, zc_i) of the pixel in the camera coordinate system;

[0035] Step S2.2: Then, using the rotation matrix R and translation matrix T obtained in step S1, the three-dimensional coordinates (x, y, y) of the pixel in the antenna coordinate system are obtained. r_i ,y r_i ,z r_i ):

[0036]

[0037] Step S2.3: Determine the three-dimensional coordinates (x, y, z) of the target or measurement point in the antenna coordinate system. r_i ,y r_i ,z r_i Transformed into a microwave distance-angle thermal map:

[0038]

[0039] Among them, R i θ represents the distance value corresponding to this pixel in the microwave distance-angle thermal map. i This represents the angle value corresponding to the pixel in the microwave distance-angle heatmap.

[0040] Repeat steps S2.1 to S2.3 until all pixels of the current RGB camera are traversed, and the distance and angle values ​​of all pixels in the microwave distance-angle heatmap are obtained.

[0041] Preferably, the three-dimensional coordinates of the pixel in the camera coordinate system are calculated using methods including triangulation, time difference calculation, structured light, and deep learning.

[0042] Preferably, step S3 includes:

[0043] Step S3.1: Based on the microwave distance-angle heatmap coordinates corresponding to each pixel obtained in step S2, calculate the distance index and angle index corresponding to the pixel.

[0044] Step S3.2: Extract the displacement value of each pixel at the current moment using the microwave full-field displacement measurement algorithm;

[0045] Step S3.3: At the current moment, all pixels are superimposed onto the original RGB image according to the displacement value using a method including color encoding to visualize the vibration displacement;

[0046] Steps S3.1 to S3.2 are repeated at subsequent times until data acquisition stops, thus completing the visualization of vibration displacement at all times.

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] This invention overcomes the problem that the distance-angle thermal imaging method used in existing microwave full-field vibration measurement technology is difficult to complete the identification of measurement points and the localization of vibration sources in complex scenarios. It proposes a novel concept of a microwave vibration camera, which integrates the vibration measurement information of the microwave transceiver and the spatial positioning information of the stereo camera, realizes the visualization of the vibration of the target within the field of view, expands the application scope of microwave full-field vibration measurement technology, and provides a feasible implementation plan for the identification of measurement points and the localization of vibration sources in microwave full-field vibration measurement in complex scenarios. Attached Figure Description

[0049] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0050] Figure 1 This is a schematic diagram of the full-field vibration visualization method that integrates stereo vision and microwave fusion according to the present invention.

[0051] Figure 2 This is a schematic diagram of the calibration scenario for the coordinate transformation matrix between the camera coordinate system and the antenna coordinate system in this invention;

[0052] Figure 3 This is a schematic diagram of the coordinate mapping process from the pixel coordinate system to the microwave distance-angle heatmap in this invention;

[0053] Figure 4 This is a block diagram of the microwave vibration camera system in this invention;

[0054] Figure 5 This is a schematic diagram of the experimental scenario for Experiment 1 of the present invention;

[0055] Figure 6 This is a schematic diagram illustrating the experimental test results of Experiment 1 of the present invention;

[0056] Figure 7 This is a schematic diagram illustrating the experimental scenario and test results of Experiment 2 of the present invention;

[0057] Figure 8 This is a schematic diagram of the experimental scenario and test results for Experiment 3 of the present invention. Detailed Implementation

[0058] The present invention will be described in detail below with reference to specific embodiments. The following examples will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those skilled in the art, several changes and improvements can be made without departing from the scope of the present invention. These all fall within the scope of protection of the present invention.

[0059] This invention provides a novel method for identifying measurement points and locating vibration sources in microwave vibration detection from the perspective of vibration visualization, thereby enhancing the application potential of microwave full-field vibration measurement technology in complex scenarios.

[0060] According to the present invention, a microwave vibration camera system integrating stereo vision and microwave is provided, such as... Figure 4 As shown, the system includes: a microwave transceiver, a stereo camera, a microwave data processing module, a camera data processing module, a coordinate transformation module, and a displacement visualization module. The microwave transceiver and stereo camera are fixedly connected. The stereo camera outputs camera data of the target to the camera data processing module, which calculates the three-dimensional coordinates of each pixel in the camera coordinate system. The coordinate transformation module converts the three-dimensional coordinates into microwave distance-angle heatmap coordinates. The microwave data processing module processes the baseband signal obtained from the microwave transceiver to perform distance-angle heatmap imaging of the target within the field of view. The displacement visualization module extracts the displacement based on the microwave distance-angle heatmap coordinates corresponding to each pixel, colors it, and superimposes it onto the visual image to obtain a real-time displacement amplitude heatmap visualized within the field of view.

[0061] The microwave transceiver is used to transmit and receive linear frequency modulated continuous wave microwave signals, and outputs multi-channel baseband signals for subsequent range-angle heatmap localization of targets within the field of view, and to measure the vibration displacement of targets in the line-of-sight direction based on the range-angle index of each target within the field of view. The stereo camera uses stereo vision methods including binocular cameras, structured light cameras, and TOF cameras to output camera data of the targets. The camera data includes parallax or depth information. The camera data processing methods include triangulation, time-of-sight calculation, structured light, and deep learning methods. The coordinate transformation module is used to calculate the coordinate transformation matrix between the camera coordinate system and the antenna coordinate system, and to transform and map the coordinates of each pixel to obtain the corresponding microwave range-angle heatmap coordinates. The displacement visualization module is used to extract the line-of-sight displacement corresponding to the range-angle index of each pixel, and after color encoding, superimpose it onto the original RGB image to obtain a real-time displacement amplitude heatmap visualized within the field of view.

[0062] The present invention provides a visualization method for a microwave vibration camera system that integrates stereo vision and microwave fusion, such as... Figure 1 As shown, it includes:

[0063] First, the coordinate transformation matrices of the camera and antenna coordinate systems, as well as the intrinsic and extrinsic parameter matrices and distortion coefficients of the stereo camera, are calibrated. Then, each pixel in the visual image is traversed, and its corresponding distance-angle coordinates in the microwave thermal map are obtained one by one. Finally, the vibration displacement information of each pixel in the microwave line-of-sight direction at the current moment is obtained using a microwave full-field vibration displacement measurement method, and this information is color-coded and superimposed onto the original visual image to visualize the real-time amplitude at the current moment. The steps for the next moment are repeated. Specifically, this includes the following:

[0064] Step S1: Calibrate the coordinate transformation matrices of the camera coordinate system and the antenna coordinate system, as well as the intrinsic parameter matrix, extrinsic parameter matrix, and distortion coefficients of the stereo camera. Step S1 includes:

[0065] Step S1.1: Calibrate the camera's intrinsic parameter matrix, extrinsic parameter matrix, and distortion coefficients. Use methods including the checkerboard calibration method, Zhang Zhengyou calibration method, and open-source toolkits to calibrate the stereo camera's intrinsic parameter matrix, extrinsic parameter matrix, and distortion coefficients.

[0066] Step S1.2: Calibrate the coordinate transformation matrices of the camera coordinate system and the antenna coordinate system. For example... Figure 2 As shown, a calibration scene is set up, and three or more targets within the field of view that can be jointly perceived by the microwave transceiver and the stereo camera are selected as reference targets, for example... Figure 2 A corner reflector that neutralizes the obvious color difference in the ground. The camera coordinate system and antenna coordinate system are as follows: Figure 3 As shown, the coordinates of each reference target in the camera coordinate system and the antenna coordinate system are denoted as follows:

[0067]

[0068] Among them, P cam Let P be the three-dimensional coordinates of the current target in the camera coordinate system. cam One column represents one target; P radar Let P be the three-dimensional coordinates of the current target in the antenna coordinate system. radar Each column represents a target; n≥3, where n represents the number of targets.

[0069] Let the average coordinates of all current reference targets in the camera coordinate system be... The average coordinates in the antenna coordinate system are Right now:

[0070]

[0071] Constructing a matrix in Indicates P cam The i-th column, Indicates Pradar The i-th column. Perform singular value decomposition on matrix M:

[0072] M=UΣV T

[0073] The rotation matrix from the camera coordinate system to the antenna coordinate system is:

[0074] R = VU T

[0075] The translation matrix is:

[0076]

[0077] Step S2: Convert the coordinates of each pixel to obtain the corresponding microwave distance-angle thermal map coordinates. For example... Figure 3 As shown, step S2 includes:

[0078] Step S2.1: Take a pixel in the pixel plane of one of the RGB cameras of the stereo camera and mark its pixel coordinates as (u i ,v i The three-dimensional coordinates (x, y) of the pixel in the camera coordinate system are calculated using methods including triangulation, time-difference calculation, structured light, and deep learning. c_i ,y c_i ,z c_i ):

[0079] Step S2.2: Then, using the rotation matrix R and translation matrix T obtained in step S1.2, obtain its three-dimensional coordinates (x, y, y) in the antenna coordinate system. r_i ,y r_i ,z r_i ):

[0080]

[0081] Step S2.3: Determine the three-dimensional coordinates (x, y, z) of the target or measurement point in the antenna coordinate system. r_i ,y r_i ,z r_i Transformed into a microwave distance-angle thermal map:

[0082]

[0083] Among them, R i θ represents the distance value corresponding to this pixel in the microwave distance-angle thermal map. i This represents the angle value corresponding to the pixel in the microwave distance-angle thermal map.

[0084] Repeat steps S2.1 to S2.3 until all pixels of the RGB camera are traversed, and the distance and angle values ​​of all pixels in the microwave distance-angle heat map are obtained.

[0085] Step S3: Extract displacement information along the microwave line of sight and visualize the displacement. Based on the microwave distance-angle heatmap coordinates of each pixel obtained in Step S2, calculate its corresponding distance and angle indices. Then, extract the displacement value of each pixel at the current moment using a microwave full-field displacement measurement algorithm. At the current moment, color-code all pixels according to their displacement values ​​and overlay them onto the original RGB image to visualize the vibration displacement.

[0086] The process of step S3 is repeated at subsequent moments until data acquisition stops, thus completing the visualization of vibration displacement at all moments.

[0087] Furthermore, by utilizing the microwave vibration camera system proposed in this invention and executing the above-mentioned full-field vibration visualization method that combines stereo vision and microwave fusion, the implementation results in three different experimental scenarios are described in detail below:

[0088] Example 1

[0089] First, the coordinate transformation matrices of the camera and antenna coordinate systems, as well as the intrinsic and extrinsic parameter matrices and distortion coefficients of the stereo camera, were calibrated. The microwave vibration camera, with its coordinate transformation matrices calibrated, was then mounted in front of an optical platform. Four corner reflectors were placed on the optical platform as targets. Targets 1 and 2 were directly fixed to the optical platform and remained stationary throughout the experiment. Targets 3 and 4 were placed on two separate sliding platforms, moving along... Figure 5 The black arrows indicate the direction of the reciprocating motion, with peak values ​​of 10mm and 5mm respectively.

[0090] Then, the pixel coordinates are transformed to obtain the corresponding microwave distance-angle thermal map coordinates. Displacement information along the microwave line-of-sight is extracted and the displacement is visualized and output. For example... Figure 6 As shown in (a), the displacement time series of the four targets along the line of sight measured by the microwave vibration camera shows that, due to the angle between the line of sight and the displacement directions of the two sliding platforms, Figure 6 The peak displacement values ​​of targets 3 and 4 in (a) are slightly smaller than the slide table setting values. Figure 6 (b) to (d) show the imaging results of the microwave vibration camera at three typical moments. The displacement amplitude measured by the microwave vibration camera was normalized, and each pixel was assigned a different color based on the ratio of the current displacement value to the maximum displacement amplitude. It can be seen that at the three typical moments, the normalized displacement values ​​corresponding to the pixel areas of targets 3 and 4 match the displacement values ​​at the corresponding moments very well.

[0091] Example 2

[0092] like Figure 7 As shown, Experiment 2 used a microwave vibration camera on a rooftop to visualize the vibration of a waste gas purification device. First, the coordinate transformation matrices of the camera and antenna coordinate systems, as well as the intrinsic and extrinsic parameter matrices and distortion coefficients of the stereo camera, were calibrated. Then, the coordinates of each pixel were transformed to obtain the corresponding microwave distance-angle thermal map coordinates. Finally, the displacement information along the microwave line-of-sight was extracted. For example... Figure 7 As shown in (a), a microwave vibration camera is facing the exhaust pipe of an exhaust gas purification device. Next to the exhaust pipe, a centrifugal fan is running continuously to extract the purified exhaust gas. The fan generates a lot of noise and causes its casing and exhaust pipe to vibrate.

[0093] Then, the pixel coordinates are transformed to obtain the corresponding microwave distance-angle thermal map coordinates. The microwave transceiver samples at a sampling frequency of 320Hz. Figure 7 (b) shows the imaging results of the microwave vibration camera at frame 76, visually demonstrating the real-time vibration of various parts of the device. Simultaneously, an accelerometer was fixed to the casing of the centrifugal fan, sampling at a sampling rate of 10000Hz. Figure 7 (d) shows the spectrum of the accelerometer sampling sequence.

[0094] Finally, the displacement information along the microwave line of sight is extracted and the displacement is visualized and output. Figure 7 (c) is the spectrum of the displacement time series obtained by a microwave vibration camera monitoring the displacement of a measuring point near the accelerometer on the casing of a centrifugal fan. (Comparison) Figure 7 (c) and Figure 7 (d) These two spectrograms show that the vibration frequency measurement results of the microwave vibration camera are in good agreement with those of the accelerometer.

[0095] Example 3

[0096] like Figure 8 As shown, Experiment 3 visualized the vibrations of a car engine during startup and acceleration using a microwave vibration camera. First, the coordinate transformation matrices of the camera and antenna coordinate systems, as well as the intrinsic and extrinsic parameter matrices and distortion coefficients of the stereo camera, were calibrated. Then, the coordinates of each pixel were transformed to obtain the corresponding microwave distance-angle thermal map coordinates. Finally, the displacement information along the microwave line-of-sight was extracted, and the displacement was visualized and output. The experimental results are as follows:

[0097] like Figure 8As shown in (a), a microwave vibration camera was mounted on a tripod in front of the car engine, and an accelerometer was fixed on the engine cover. Their sampling frequencies remained at 320Hz and 10000Hz, respectively. During simultaneous monitoring by the microwave vibration camera and accelerometer, the car engine was started, and shortly afterward, the accelerator pedal was pressed, causing the engine speed to increase. Figure 8 (b) is the vibration visualization result at the moment of engine start-up, which intuitively shows that the part with larger vibration displacement occurs near the engine casing. Figure 8 (c) and Figure 8 (d) shows the time-frequency plots of the measurement results from the microwave vibration camera and the accelerometer (after downsampling). It can be seen that the frequency change trends of the two are basically consistent, and both can correspond well to the behavior of engine starting and speed increase.

[0098] This invention aims to solve the problem that existing microwave full-field vibration measurement methods cannot accurately and intuitively identify measurement points and locate vibration sources. It uses stereo vision technology to identify microwave distance-angle thermal imaging measurement points and maps the microwave full-field vibration measurement results onto a visual image through methods including color coding, thereby obtaining a visualized real-time vibration state within the field of view.

[0099] Those skilled in the art will appreciate that, in addition to implementing the system and its various devices, modules, and units provided by the present invention in purely computer-readable program code, it is entirely possible to implement the same functions of the system and its various devices, modules, and units provided by the present invention in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system and its various devices, modules, and units provided by the present invention can be considered a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered as structures within the hardware component; the devices, modules, and units for implementing various functions can also be considered as both software modules implementing the method and structures within the hardware component.

[0100] The above describes specific embodiments of the present invention. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art may make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. The embodiments of this application and the features in the embodiments may be combined with each other in any manner unless there is a conflict.

Claims

1. A microwave vibration camera system integrating stereo vision and microwave technology, characterized in that, include: Microwave transceiver, stereo camera, microwave data processing module, camera data processing module, coordinate transformation module, and displacement visualization module; The microwave transceiver is used to transmit and receive linear frequency modulated continuous wave microwave signals and output multi-channel baseband signals. The microwave transceiver and the stereo camera are fixedly connected. The stereo camera outputs the camera data of the target to the camera data processing module to calculate the three-dimensional coordinates of each pixel in the camera coordinate system. The coordinate transformation module converts the three-dimensional coordinates into microwave distance-angle heatmap coordinates, specifically including: first, calibrating the coordinate transformation matrix between the camera coordinate system and the antenna coordinate system; then, taking a pixel point on the pixel plane of one of the RGB cameras of the stereo camera and calculating the three-dimensional coordinates of the pixel point in the camera coordinate system; next, using the calibrated coordinate transformation matrix between the camera coordinate system and the antenna coordinate system, obtaining the three-dimensional coordinates of the pixel point in the antenna coordinate system; then, transforming the three-dimensional coordinates of the target or measurement point in the antenna coordinate system into the microwave distance-angle heatmap; finally, traversing all pixels of the current RGB camera to obtain the distance and angle values ​​corresponding to all pixels in the microwave distance-angle heatmap; The microwave data processing module processes the baseband signal obtained by the microwave transceiver and performs range-angle thermal imaging of the target within the field of view. The displacement visualization module will calculate the distance index and angle index corresponding to each pixel based on the microwave distance-angle heatmap coordinates of each pixel, and then extract the displacement of each pixel at the current moment. At the current moment, all pixels are mapped onto the original visual image according to the displacement value through a method including color encoding to obtain the real-time vibration state visualized within the field of view. This process is repeated at each subsequent moment until data acquisition stops, thus completing the visualization of vibration displacement at all moments.

2. The microwave vibration camera system integrating stereo vision and microwave fusion according to claim 1, characterized in that, The stereo camera uses stereo vision forms including binocular cameras, structured light cameras, and TOF cameras to output camera data of the target. The camera data includes parallax or depth information; The camera data processing methods include triangulation, time difference calculation, structured light, and deep learning.

3. The microwave vibration camera system for stereo vision and microwave fusion according to claim 1, characterized in that, The microwave transceiver is used to transmit and receive linear frequency modulated continuous wave microwave signals and output multi-channel baseband signals for subsequent range-angle thermal mapping of targets within the field of view, and to measure the vibration displacement of targets within the field of view along the line of sight based on the range-angle index of each target within the field of view.

4. The microwave vibration camera system for stereo vision and microwave fusion according to claim 1, characterized in that, The coordinate transformation module is used to calculate the coordinate transformation matrix between the camera coordinate system and the antenna coordinate system, and to transform and map the coordinates of each pixel to obtain the corresponding microwave distance-angle thermal map coordinates.

5. The microwave vibration camera system for stereo vision and microwave fusion according to claim 1, characterized in that, The displacement visualization module is used to extract the line-of-sight displacement corresponding to the distance-angle index of each pixel, and map it onto the visual image using methods including color encoding to obtain the displacement within the field of view. Target Visualized real-time vibration status.

6. A visualization method for a microwave vibration camera system that integrates stereo vision and microwave fusion, characterized in that, include: Step S1: Calibrate the coordinate transformation matrices of the camera coordinate system and the antenna coordinate system, as well as the intrinsic parameter matrix, extrinsic parameter matrix, and distortion coefficients of the stereo camera; Step S2: Transform the coordinates of each pixel to obtain the corresponding microwave distance-angle heatmap coordinates. Specifically, this includes: taking a pixel point on the pixel plane of one of the RGB cameras of the stereo camera and calculating the three-dimensional coordinates of the pixel point in the camera coordinate system; then using the coordinate transformation matrix of the calibrated camera coordinate system and antenna coordinate system, obtaining the three-dimensional coordinates of the pixel point in the antenna coordinate system; then transforming the three-dimensional coordinates of the target or measurement point in the antenna coordinate system into the microwave distance-angle heatmap; finally, traversing all the pixels of the current RGB camera to obtain the distance and angle values ​​corresponding to all pixels in the microwave distance-angle heatmap. Step S3: Extract displacement information along the microwave line of sight and output the displacement visualization. Step S3 includes: Step S3.1: Based on the microwave distance-angle heatmap coordinates corresponding to each pixel obtained in step S2, calculate the distance index and angle index corresponding to the pixel. Step S3.2: Extract the displacement value of each pixel at the current moment using the microwave full-field displacement measurement algorithm; Step S3.3: At the current moment, map all pixels onto the original visual image according to the displacement value using a method including color encoding to obtain the real-time vibration state of the target visualization within the field of view; Steps S3.1 to S3.2 are repeated at subsequent times until data acquisition stops, thus completing the visualization of vibration displacement at all times.

7. The visualization method for a microwave vibration camera system integrating stereo vision and microwave fusion according to claim 6, characterized in that, The intrinsic parameter matrix, extrinsic parameter matrix, and distortion coefficients of the calibrated camera are calibrated using methods including the checkerboard calibration method, the Zhang Zhengyou calibration method, and open-source toolkits. The coordinate transformation matrix for calibrating the camera coordinate system and the antenna coordinate system includes denoting the coordinates of each reference target in the camera coordinate system and the antenna coordinate system as follows: ， ， in, The three-dimensional coordinates of the current target in the camera coordinate system. One column represents one target; The three-dimensional coordinates of the current target in the antenna coordinate system. One column represents one target; ; Let the average coordinates of all current reference targets in the camera coordinate system be... The average coordinates in the antenna coordinate system are ,Right now: ， ， Constructing a matrix ,in express The i List, express The i Columns, matrices Perform singular value decomposition: The rotation matrix from the camera coordinate system to the antenna coordinate system is: The translation matrix is: 。 8. The visualization method for a microwave vibration camera system integrating stereo vision and microwave fusion according to claim 6, characterized in that, In step S2, the coordinate transformation matrix of the calibrated camera coordinate system and antenna coordinate system is used to obtain the three-dimensional coordinates of the pixel in the antenna coordinate system, as shown in the following formula: In the formula, This represents the three-dimensional coordinates of a pixel in the antenna coordinate system. This represents the three-dimensional coordinates of a pixel in the camera coordinate system. Represents the rotation matrix. Represents the translation matrix; The three-dimensional coordinates of the target or measurement point in the antenna coordinate system are transformed into the microwave distance-angle thermal map, as follows: in, This represents the distance value corresponding to the pixel in the microwave distance-angle heatmap. This represents the angle value corresponding to the pixel in the microwave distance-angle thermal map.

9. The visualization method for a microwave vibration camera system integrating stereo vision and microwave fusion according to claim 8, characterized in that, The three-dimensional coordinates of the pixel in the camera coordinate system are calculated using methods including triangulation, time difference calculation, structured light, and deep learning.

Images