Coaxial radar and image displacement sensor three-dimensional deformation measurement system and method

Abstract

The invention discloses a coaxial radar and image displacement sensor three-dimensional deformation measurement system and method, and the method comprises the steps: transmitting and receiving a radar signal through a radar module, obtaining the distance and radial deformation information of a target structure, and transmitting the obtained deformation data to a data processing module; the image displacement sensor module and the radar are coaxially installed, image data of a target structure are collected in real time, and the image data are transmitted to the data processing module; the deformation data and the image data are analyzed and processed by using a data processing module, and fused three-dimensional deformation data are calculated through translation and rotation algorithms; and acquiring data in the data processing module through a display and interaction module, and visually displaying the fused three-dimensional deformation data. The space error between the radar and the image displacement sensor is effectively reduced, and the synchronism and accuracy of data are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of deformation measurement, and particularly to a three-dimensional deformation measurement system and method for a coaxial radar and an image displacement sensor. Background Art

[0002] In the safety monitoring of engineering structures, accurately obtaining the three-dimensional deformation information of the structure is of great significance for evaluating the health status of the structure and predicting potential risks. Traditional deformation measurement methods have many limitations. For example, although contact measurement methods have high accuracy, they are complex to install, can cause certain damage to the measured structure, and it is difficult to achieve large-area and real-time monitoring; among non-contact measurement methods, although single radar measurement has the advantages of all-day and all-weather, it can only measure the displacement change in the radial direction. In addition, in some scenarios, it is easily affected by multipath, resulting in problems such as missed detection, false detection, and ambiguity of the actual monitoring position in monitoring; while image displacement sensors can provide intuitive images and accurate position information, however, they are seriously interfered by environmental factors such as light and weather, affecting the measurement accuracy.

[0003] Currently, there is little research on the fusion of radar and image displacement sensors for three-dimensional deformation measurement, and most of the existing fusion methods are just simple data superposition, with limited fusion degree, unable to effectively solve problems such as large data differences and difficult matching between the two sensors, and it is difficult to meet the requirements of high-precision and high-reliability three-dimensional deformation measurement. Summary of the Invention

[0004] The purpose of the present invention is to solve the above problems, and a three-dimensional deformation measurement system and method for a coaxial radar and an image displacement sensor are designed.

[0005] Furthermore, to achieve the above object, the technical solution of the present invention is as follows. In the above-mentioned three-dimensional deformation measurement system for a coaxial radar and an image displacement sensor, the three-dimensional deformation measurement system includes the following modules:

[0006] A radar module, which is used to transmit and receive radar signals, obtain the distance and radial direction deformation information of the target structure, and transmit the obtained deformation data to the data processing module;

[0007] An image displacement sensor module, which is used to be coaxially installed with the radar, collect the image data of the target structure in real time, and transmit the image data to the data processing module;

[0008] A data processing module, which is used to analyze and process the deformation data and image data, obtain the displacement change between the target point and the host, and calculate the fused three-dimensional deformation data through translation and rotation algorithms;

[0009] A display and interaction module, which is used to obtain the data in the data processing module and visually display the fused three-dimensional deformation data.

[0010] Further, in the above three-dimensional deformation measurement system of a coaxial radar and an image displacement sensor, the radar module includes the following units:

[0011] A multimodal configuration unit, which is used to transmit and receive radar signals by using a dual-band radar, and generate tensor data including distance, azimuth, altitude, and speed;

[0012] A dynamic waveform optimization unit, which is used to switch between the FMCW and UWB modes according to the target structure material, where the FMCW mode is used for smooth surface, and the UWB pulse mode is used for rough surface penetration detection;

[0013] An interference processing unit, which is used to separate the structural vibration signal and environmental noise based on the JPDA (Joint Probability Data Association) algorithm to obtain deformation data.

[0014] Further, in the above three-dimensional deformation measurement system of a coaxial radar and an image displacement sensor, the image displacement sensor module includes the following units:

[0015] A visual perception unit, which is used to imitate the structure of an insect compound eye, configure 6 groups of 5 million-pixel global shutter cameras with a circular distribution, and calculate the global displacement field through the optical flow method;

[0016] A heterogeneous data synchronization unit, which is used to generate synchronous trigger pulses through an FPGA, and when the radar scanning frequency does not match the camera frame rate, use cubic spline interpolation for alignment;

[0017] A preprocessing unit, which is used to deploy a Jetson Nano edge computing unit, and run an improved YOLOv8 model in real time to segment the target structure image to obtain image data.

[0018] Further, in the above three-dimensional deformation measurement system of a coaxial radar and an image displacement sensor, the data processing module includes the following units:

[0019] A signal processing unit, which is used to process the signals transmitted from the radar module, generate radar pulse compression data, and obtain the displacement change in the radial direction between the target point and the host through differential interferometry;

[0020] An image analysis unit, which is used to analyze the images collected by the image displacement sensor module, extract the feature points of the target structure, and calculate the displacement changes in the vertical and horizontal directions of the feature points;

[0021] A data processing unit, which is used to convert the system coordinate data to the measured object coordinate system through translation and rotation algorithms, and obtain the three-dimensional deformation data of the measured object.

[0022] Further, in a three-dimensional deformation measurement method using a coaxial radar and an image displacement sensor, the three-dimensional deformation measurement method includes the following steps:

[0023] Use the radar module and the image displacement sensor module to simultaneously collect data from the target structure and obtain data of different modalities;

[0024] Use the signal processing unit to preprocess the radar signal, and the image analysis unit to denoise and extract features from the image data;

[0025] Taking the centroid feature point of the measured object itself as the origin, and establishing a rectangular coordinate system according to the characteristics of the target in the coordinate axis directions, to obtain the three-dimensional deformation data of the measured object;

[0026] Output the processed three-dimensional deformation data in the form of graphics and reports to the display and interaction module for display.

[0027] Further, in the above three-dimensional deformation measurement method using a coaxial radar and an image displacement sensor, the step of taking the centroid feature point of the measured object itself as the origin, establishing a rectangular coordinate system according to the characteristics of the target in the coordinate axis directions, and obtaining the three-dimensional deformation data of the measured object includes:

[0028] Define a rectangular coordinate system (x, y, z), with the mechanical axis center of the system as the coordinate origin, the radar and optoelectronic pointing directions as the y-axis, and the direction perpendicular to the radar and optoelectronic axis planes upward as the z-axis, and determine the x-axis according to the right-hand screw rule.

[0029] Further, in the above three-dimensional deformation measurement method using a coaxial radar and an image displacement sensor, the step of taking the centroid feature point of the measured object itself as the origin, establishing a rectangular coordinate system according to the characteristics of the target in the coordinate axis directions, and obtaining the three-dimensional deformation data of the measured object includes:

[0030] Obtain the translation vector and rotation matrix of the radar relative to the target through measurement. Let the position of the system in the coordinate system of the measured object be t = [t x , t y , t2] T , and the rotation matrix of the system coordinate system relative to the coordinate system of the measured object be R;

[0031] Let P r be the coordinates of a point in the system coordinate system, and p k be the coordinates of the point in the coordinate system of the measured object, then the conversion formula is:

[0032] p t = Rp r + t

[0033] Among them, the calculation of the rotation matrix R is obtained by combining three basic rotation matrices. The basic rotation matrices include R for rotating by an angle α around the x-axis s(α), R rotated by an angle β about the y-axis y (β) and R rotated by an angle γ about the z-axis z (γ);

[0034]

[0035] Then the total rotation matrix R is obtained by first rotating about the x-axis, then about the y-axis, and finally about the z-axis: R = R2(γ)R y (β)R x (α).

[0036] Furthermore, in the above three-dimensional deformation measurement method of a coaxial radar and an image displacement sensor, a rectangular coordinate system is established with the centroid feature point of the measured object itself as the origin, and the coordinate axis directions are determined according to the characteristics of the target to obtain the three-dimensional deformation data of the measured object, including:

[0037] The rotation angles α, β, γ and the translation vector t are solved by measuring multiple groups of corresponding values. Suppose n groups of points p in the radar coordinate system are measured r,i and the corresponding points p in the target coordinate system i,i (i = 1, 2,..., n), then there is:

[0038] p i,i = Rp r,i + t

[0039] Expanding the equation, for each pair of points (p i,i , p i,i ) an equation can be obtained. Suppose p r,i = [x r,i , y r,i , z r,i T , p t,i = [x t,i , y i,i , z t,i T , t = [t x , t y , t s T , and R is a 3×3 rotation matrix:

[0040]

[0041] Expanding it into three linear equations:

[0042] x t,j = r 11 x r,i + r 12 y r,i + r 13 z r,i ​​​+t x

[0043] y i,i = r 21 x i,i + r 22 y i,i + r 23 z i,i + t y

[0044] z i,i = r 31 x i,j + r 33 y i,j + r 33 z i,i + t z

[0045] where x r,i , y r,i , z r,i represent the coordinate values of the point on the x, y, and z axes of the radar coordinate system respectively, and x t,i , y i,i , z t,i are the coordinate values of the point on the x, y, and z axes of the target coordinate system respectively; t x , t y , t s represent the translation amounts in the x, y, and z axis directions from the radar coordinate system to the target coordinate system respectively;

[0046] When there are n pairs of points, construct an overdetermined system of equations with 3n equations and solve for R and t using the least squares method.

[0047] Furthermore, in the above three-dimensional deformation measurement method of a coaxial radar and an image displacement sensor, taking the centroid feature point of the measured object itself as the origin and establishing a rectangular coordinate system according to the characteristics of the target in the coordinate axis directions to obtain the three-dimensional deformation data of the measured object includes:

[0048] Determine the values of α, β, and γ according to R = R3(γ)R θ (β)R x (α) as follows:

[0049]

[0050] Extract the rotation angles from the rotation matrix R through the following formula:

[0051] β = -arcsin(R 31 )

[0052] α = arctan2(R 23 / cosβ, R 33 / cosβ)

[0053] γ = arctan2(R 21 / cosβ, R 11 / cosβ

[0054] wherein, R ij represents the element in the i-th row and j-th column of the rotation matrix R, and R 31 represents the element in the 3rd row and 1st column of the rotation matrix R. arcsin represents the arcsine function, which is used to find the corresponding angle according to the sine value; arctan2 represents the two-variable arctangent function.

[0055] Its beneficial effects are as follows: 1. Through the coaxial design, the spatial error between the radar and the image displacement sensor is effectively reduced, and the synchronization and accuracy of the data are improved. 2. The deep learning algorithm is used for data fusion, fully mining the complementary information of the data of the two sensors, improving the perception accuracy, and reducing the probability of missed detection and false detection. 3. The system has good environmental adaptability and can achieve high-precision three-dimensional deformation measurement of the target structure in a variety of complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] By reading the following detailed description of the preferred embodiments, various other advantages and benefits will become clear to those of ordinary skill in the art. The drawings are only for the purpose of showing the preferred embodiments and are not considered to be a limitation of the present invention.

[0057] Figure 1 It is a schematic diagram of the first embodiment of a coaxial radar and image displacement sensor three-dimensional deformation measurement system in an embodiment of the present invention;

[0058] Figure 2 It is a schematic diagram of the second embodiment of a coaxial radar and image displacement sensor three-dimensional deformation measurement system in an embodiment of the present invention;

[0059] Figure 3 It is a schematic diagram of the first embodiment of a coaxial radar and image displacement sensor three-dimensional deformation measurement method in an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0060] In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0061] Those skilled in the art can understand that, unless specifically stated otherwise, the singular forms "a", "an", "the", and "said" used herein may also include the plural forms. It should be further understood that the term "comprising" used in the specification of the present invention means the presence of the described features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or their groups.

[0062] The present invention will be specifically described below with reference to the accompanying drawings. As Figure 1 shown, a three-dimensional deformation measurement system for a coaxial radar and an image displacement sensor, the three-dimensional deformation measurement system includes the following modules:

[0063] A radar module, configured to transmit and receive radar signals, obtain distance and radial direction deformation information of a target structure, and transmit the obtained deformation data to a data processing module;

[0064] Specifically, in this embodiment, it includes a multi-modal configuration unit, configured to use a dual-band radar to transmit and receive radar signals, and generate tensor data including distance, azimuth angle, altitude, and speed; a dynamic waveform optimization unit, configured to switch between FMCW and UWB modes according to the material of the target structure, where the FMCW mode is used for smooth surface, and the UWB pulse mode is used for rough surface penetration detection; an interference processing unit, configured to separate the structure vibration signal and environmental noise based on the JPDA (Joint Probability Data Association) algorithm to obtain deformation data.

[0065] An image displacement sensor module, configured to be coaxially installed with the radar, collect image data of the target structure in real time, and transmit the image data to the data processing module;

[0066] Specifically, in this embodiment, it includes a visual perception unit, configured to imitate the compound eye structure of insects, configure 6 groups of 5 million pixel global shutter cameras with a circular distribution, and calculate the global displacement field through the optical flow method; a heterogeneous data synchronization unit, configured to generate a synchronous trigger pulse through an FPGA, and when the radar scanning frequency does not match the camera frame rate, use cubic spline interpolation for alignment; a preprocessing unit, configured to deploy a Jetson Nano edge computing unit, and run an improved YOLOv8 model in real time to segment the target structure image to obtain image data.

[0067] Specifically, the improved YOLOv8 model is to introduce the CBAM (Convolutional Block Attention Module) attention mechanism into the YOLOv8 model; first, define the code of the CBAM module, and insert the CBAM module at an appropriate position in the feature extraction network of YOLOv8 (such as CSPDarknet). Add the CBAM module after each C3 module.

[0068] A data processing module, which is used to analyze and process the deformation data and image data to obtain the displacement change between the target point and the host, and calculate the fused three-dimensional deformation data through translation and rotation algorithms.

[0069] Specifically, in this embodiment, it includes a signal processing unit, which is used to process the signals transmitted by the radar module to generate radar pulse compression data, and obtain the displacement change in the radial direction between the target point and the host through differential interferometry; an image analysis unit, which is used to analyze the images collected by the image displacement sensor module, extract the feature points of the target structure, and calculate the displacement changes in the vertical and horizontal directions of the feature points; a data processing unit, which is used to convert the system coordinate system data to the measured object coordinate system through translation and rotation algorithms to obtain the three-dimensional deformation data of the measured object.

[0070] A display and interaction module, which is used to obtain the data in the data processing module and visually display the fused three-dimensional deformation data.

[0071] Visually display the fused three-dimensional deformation data, provide intuitive monitoring results for users, and support users to perform interactive operations such as parameter setting and data query.

[0072] Specifically, the measurement method using this system is as follows:

[0073] Data acquisition: The radar module and the image displacement sensor module simultaneously collect data on the target structure to obtain data in different modalities.

[0074] Data preprocessing: The signal processing unit preprocesses the radar signals, and the image analysis unit performs processing such as noise reduction and feature extraction on the image data.

[0075] Coordinate system conversion: First, define a system rectangular coordinate system (x, y, z), with the system mechanical axis center as the coordinate origin, the radar and optoelectronic pointing as the y-axis, and the direction perpendicular to the radar and optoelectronic axis plane upward as the z-axis. Determine the x-axis according to the right-hand screw rule. The measured object coordinate system is established according to actual needs, such as using a certain feature point (such as the centroid) of the measured object itself as the origin, and the coordinate axis directions are determined according to the characteristics or application requirements of the target. After establishing the coordinate system, the conversion process is as follows:

[0076] 1. Determine the position and attitude relationship between the system coordinate system and the measured object coordinate system

[0077] Through measurement, clarify the position (translation vector) and attitude (rotation matrix) of the radar relative to the target. Assume that the position of the system in the measured object coordinate system is t = [t x , t y , t2] T . The rotation matrix of the system coordinate system relative to the measured object coordinate system is R.

[0078] 2. Coordinate Transformation Formula

[0079] Let P r be the coordinates of a point in the system coordinate system, and p k be the coordinates of this point in the measured object coordinate system. Then the transformation formula is

[0080] p t = Rp r + t

[0081] where the calculation of the rotation matrix R can be obtained by combining three basic rotation matrices. The basic rotation matrices include R s (α) that rotates by an angle α around the x-axis, R y (β) that rotates by an angle β around the y-axis, and R z (γ) that rotates by an angle γ around the z-axis.

[0082]

[0083] Then the total rotation matrix R can be obtained through combinations in different orders. For example, first rotate around the x-axis, then around the y-axis, and finally around the z-axis:

[0084] R = R2(γ)R y (β)R x (α)

[0085] In practical applications, the rotation angles α, β, γ and the translation vector t can be solved by measuring multiple sets of corresponding values. Suppose n sets of points p r,i in the radar coordinate system and the corresponding points p i,i in the target coordinate system (i = 1, 2,..., n) are measured. Then there is:

[0086] p i,i = Rp r,i + t

[0087] Expand the above equation. For each pair of points (p i,i , p i,i ), an equation can be obtained. Assume p r,i = [x r,i , y r,i , z r,i T , p t,i = [x t,i , y i,i , z t,i T , t = [t x , t y , t s T , and R is a 3×3 rotation matrix:​​​

[0088]

[0089] Expand it into three linear equations:

[0090] x t,j = r 11 x r,i + r 12 y r,i + r 13 z r,i + t x

[0091] y i,i = r 21 x i,i + r 22 y i,i + r 23 z i,i + t y

[0092] z i,i = r 31 x i,j + r 33 y i,j + r 33 z i,i + t z

[0093] When there are n pairs of points, an overdetermined system of equations containing 3n equations can be constructed. To solve for R and t, the least squares method is used.

[0094] If we want to determine the values of α, β, γ, according to R = R3(γ)R θ (β)R x (α), we have:

[0095]

[0096] Furthermore, the rotation angles are extracted from the rotation matrix R through the following formulas:

[0097] β = -arcsin(R 31 )

[0098] α = arctan2(R 23 / cosβ, R 33 / cosβ)

[0099] γ = arctan2(R 21 / cosβ, R 11 / cosβ)

[0100] Result Output and Analysis: The display and interaction module outputs the processed three-dimensional deformation data in the form of graphs, reports, etc., and users can perform data analysis and processing according to their needs.

[0101] The beneficial effects of the present invention are as follows: The radar module emits and receives radar signals to obtain the distance and radial deformation information of the target structure, and transmits the obtained deformation data to the data processing module; the image displacement sensor module is coaxially installed with the radar to collect the image data of the target structure in real time and transmit the image data to the data processing module; the data processing module analyzes and processes the deformation data and the image data to obtain the displacement change between the target point and the host, and calculates the fused three-dimensional deformation data through translation and rotation algorithms; the display and interaction module obtains the data in the data processing module and visualizes the fused three-dimensional deformation data. 1. It effectively reduces the spatial error between the radar and the image displacement sensor, and improves the synchronization and accuracy of the data. 2. The deep learning algorithm is used for data fusion, fully mining the complementary information of the two sensor data, improving the perception accuracy and reducing the probability of missed detection. 3. The system has good environmental adaptability and can achieve high-precision three-dimensional deformation measurement of the target structure in a variety of complex environments.

[0102] Please refer to Figure 2 , in a coaxial radar and image displacement sensor three-dimensional deformation measurement system, the image displacement sensor module further includes the following units:

[0103] The visual perception unit is used to imitate the structure of the compound eyes of insects, configure 6 groups of 5 million pixel global shutter cameras with annular distribution, and calculate the global displacement field through the optical flow method;

[0104] The heterogeneous data synchronization unit is used to generate synchronous trigger pulses through the FPGA, and when the radar scanning frequency does not match the camera frame rate, cubic spline interpolation is used for alignment;

[0105] The preprocessing unit is used to deploy the Jetson Nano edge computing unit to run the improved YOLOv8 model to segment the target structure image in real time and obtain the image data.

[0106] Please refer to Figure 3 , in a coaxial radar and image displacement sensor three-dimensional deformation measurement method, the three-dimensional deformation measurement method includes the following steps:

[0107] Step 301: Use the radar module and the image displacement sensor module to collect data from the target structure simultaneously to obtain data in different modalities;

[0108] Specifically, in this embodiment,

[0109] Multi-modal radar configuration:

[0110] Adopt a dual - band radar (24GHz + 77GHz) to balance short - range (0.5 - 30m) and medium - range (30 - 200m) deformation monitoring, and avoid signal interference through frequency - division multiplexing technology37.

[0111] Introduce 4D millimeter - wave radar technology to generate tensor data (4D point - cloud matrix) containing distance, azimuth angle, altitude, and speed, improving the resolution of the three - dimensional deformation field.

[0112] Dynamic waveform optimization: Switch between FMCW (Frequency - Modulated Continuous Wave) and UWB (Ultra - Wideband) modes according to the target structure material (such as concrete / steel structure): The FMCW mode is used for smooth surfaces (error ±1mm); the UWB pulse mode is used for rough - surface penetration detection (error ±3mm).

[0113] Intelligent anti - interference processing: Deploy multi - target tracking algorithms (such as JPDA - Joint Probability Data Association) to separate structural vibration signals from environmental noise; adopt polarization filtering technology to suppress the multipath effect caused by rain and fog weather

[0114] Step 302: Use the signal processing unit to pre - process the radar signal, and the image analysis unit to denoise and extract features from the image data;

[0115] Specifically, in this embodiment,

[0116] Bionic vision perception architecture,

[0117] Imitate the compound - eye structure of insects, configure 6 groups of 5 - megapixel global - shutter cameras with annular distribution to achieve 360° coverage (field - of - view angle of a single group is 60°), calculate the global displacement field through the optical flow method; integrate a laser speckle projector to generate artificial feature points on low - texture surfaces (such as the inner wall of a tunnel).

[0118] Heterogeneous data synchronization mechanism

[0119] Hardware - level synchronization: Adopt the IEEE 1588 Precision Time Protocol (PTP), generate synchronous trigger pulses through FPGA, and the time deviation < 1μs68;

[0120] Dynamic frame - rate adaptation: When the radar scanning frequency (20Hz) does not match the camera frame rate (60Hz), use cubic spline interpolation to achieve sub - pixel - level data alignment.

[0121] Embedded pre - processing acceleration

[0122] Deploy a Jetson Nano edge - computing unit at the camera end to run the improved YOLOv8 model in real - time, segment the key parts of the structure (such as the hinge point of a bridge), and only transmit the data of the ROI area, reducing the bandwidth by 70%.

[0123] Step 303: Taking the centroid feature point of the object under test as the origin, establish a rectangular coordinate system according to the characteristics of the target with the coordinate axis directions, and obtain the three-dimensional deformation data of the object under test;

[0124] Specifically, in this embodiment, a rectangular coordinate system (x, y, z) is defined. Taking the system mechanical axis center as the coordinate origin, the radar and optoelectronic pointing directions as the y-axis, and the direction perpendicular to the radar and optoelectronic axes upward as the z-axis, the x-axis is determined according to the right-hand screw rule.

[0125] By measuring, obtain the translation vector and rotation matrix of the radar relative to the target. Let the position of the system in the coordinate system of the object under test be t = [t x , t y , t2] T , and the rotation matrix of the system coordinate system relative to the coordinate system of the object under test be R;

[0126] Let P r be the coordinate of a point in the system coordinate system, and p k be the coordinate of the point in the coordinate system of the object under test. Then the conversion formula is:

[0127] p t = Rp r + t

[0128] where the rotation matrix R is calculated by combining three basic rotation matrices. The basic rotation matrices include R s (α) for rotating by an angle α around the x-axis, R y (β) for rotating by an angle β around the y-axis, and R z (γ) for rotating by an angle γ around the z-axis;

[0129]

[0130] Then the total rotation matrix R is obtained by first rotating around the x-axis, then around the y-axis, and finally around the z-axis: R = R2(γ)R y (β)R x (α).

[0131] By measuring multiple sets of corresponding values to solve for the rotation angles α, β, γ and the translation vector t. Suppose n sets of points p r,i in the radar coordinate system and the corresponding points p i,i in the target coordinate system are measured (i = 1, 2,..., n), then there is:

[0132] p i,i = Rp r,i + t

[0133] Expand the equation. For each pair of points (p i,i , p i,i ) an equation can be obtained. Suppose p r,i= [x r,i , y r,i , z r,i T , p t,i = [x t,i , y i,i , z t,i T , t = [t x , t y , t s T , R is a 3×3 rotation matrix:

[0134]

[0135] Expand it into three linear equations:

[0136] x t,j = r 11 x r,i + r 12 y r,i + r 13 z r,i + t x

[0137] y i,i = r 21 x i,i + r 22 y i,i + r 23 z i,i + t y

[0138] z i,i = r 31 x j,j + r 33 y i,j + r 33 z i,i + t z

[0139] Where, x r,i , y r,i , z r,i respectively represent the coordinate values of the point on the x, y, z axes of the radar coordinate system, and x t,i , y i,i , z t,i are the coordinate values of the point on the x, y, z axes of the target coordinate system respectively; t x , t y , t s respectively represent the translation amounts in the x, y, z axis directions from the radar coordinate system to the target coordinate system;

[0140] ​​​When there are n groups of point pairs, an overdetermined system of equations containing 3n equations is constructed, and R and t are solved using the least squares method.

[0141] According to R = R3(γ)R θ (β)R x (α) to determine the values of α, β, and γ, there are:

[0142]

[0143] Extract the rotation angle from the rotation matrix R through the following formula:

[0144] β = -arcsin(R 31 )

[0145] α = arctan2(R 23 / cosβ, R 33 / cosβ)

[0146] γ = arctan2(R 21 / cosβ, R 11 / cosβ)

[0147] Among them, R ij represents the element in the i-th row and j-th column of the rotation matrix R, R 31 represents the element in the 3rd row and 1st column of the rotation matrix R, arcsin represents the arcsine function, which is used to find the corresponding angle according to the sine value; arctan2 represents the two-variable arctangent function.

[0148] Step 304: Output the processed three-dimensional deformation data to the display and interaction module in the form of graphics and reports, and display it.

[0149] Specifically, in this embodiment, the digital twin visualization system constructs a BIM + point cloud fusion model, supporting LOD (Level of Detail) rendering: a macroscopic deformation heat map (accuracy ±5 cm) is displayed outside 50 m; a bolt-level microscopic deformation vector field (accuracy ±0.1 mm) is displayed within 5 m; 35 an integrated HoloLens 2 MR device is used to achieve spatial projection of deformation data. For intelligent diagnosis and early warning, a deformation mode knowledge graph is established to associate typical diseases); an LSTM-Attention prediction model is deployed to give an early warning of the critical deformation state 30 minutes in advance.

[0150] The basic principles, main features and advantages of the present invention have been shown and described above. Those skilled in the art should understand that the present invention is not limited by the above embodiments. The above embodiments and the descriptions in the specification are only preferred examples of the present invention and are not used to limit the present invention. Without departing from the spirit and scope of the present invention, the present invention will have various changes and improvements, and these changes and improvements fall within the scope of the present invention claimed. The scope of the present invention claimed is defined by the appended claims and their equivalents.

Claims

1. A coaxial radar and image displacement sensor three-dimensional deformation measurement system, characterized in that The three-dimensional deformation measurement system includes the following modules: The radar module is used to transmit and receive radar signals, obtain the distance and radial deformation information of the target structure, and transmit the obtained deformation data to the data processing module; The image displacement sensor module is coaxially installed with the radar, and is used to collect the image data of the target structure in real time and transmit the image data to the data processing module; The data processing module is used to analyze and process the deformation data and image data, obtain the displacement change between the target point and the host, and calculate the fused three-dimensional deformation data through translation and rotation algorithms; The display and interaction module is used to obtain the data in the data processing module and visually display the fused three-dimensional deformation data.

2. The three-dimensional deformation measurement system of a coaxial radar and an image displacement sensor according to claim 1, wherein The radar module includes the following units: The multi-modal configuration unit is used to transmit and receive radar signals using a dual-band radar, and generate tensor data including distance, azimuth, altitude, and speed; The dynamic waveform optimization unit is used to switch between the FMCW and UWB modes according to the material of the target structure. The FMCW mode is used for smoothing the surface, and the UWB pulse mode is used for rough surface penetration detection; The interference processing unit is used to separate the structure vibration signal and environmental noise based on the JPDA (Joint Probability Data Association) algorithm to obtain deformation data.

3. A coaxial radar and image displacement sensor three-dimensional deformation measurement system according to claim 1, characterized in that, The image displacement sensor module includes the following units: The visual perception unit is used to imitate the compound eye structure of insects, configure 6 groups of 5 million pixel global shutter cameras with a circular distribution, and calculate the global displacement field through the optical flow method; The heterogeneous data synchronization unit is used to generate synchronous trigger pulses through the FPGA. When the radar scanning frequency does not match the camera frame rate, cubic spline interpolation is used for alignment; The preprocessing unit is used to deploy the Jetson Nano edge computing unit to run the improved YOLOv8 model in real time to segment the target structure image and obtain image data.

4. A coaxial radar and image displacement sensor three-dimensional deformation measurement system according to claim 1, characterized in that, The data processing module includes the following units: The signal processing unit is used to process the signals transmitted by the radar module, generate radar pulse compression data, and obtain the displacement change in the radial direction between the target point and the host through differential interferometry; The image analysis unit is used to analyze the images collected by the image displacement sensor module, extract the feature points of the target structure, and calculate the displacement changes in the vertical and horizontal directions of the feature points; The data processing unit is used to convert the system coordinate system data to the measured object coordinate system through translation and rotation algorithms to obtain the three-dimensional deformation data of the measured object.

5. A three-dimensional deformation measurement method for a coaxial radar and an image displacement sensor, characterized in that, The three-dimensional deformation measurement method includes the following steps: Use the radar module and the image displacement sensor module to simultaneously collect data on the target structure to obtain data of different modalities; Use the signal processing unit to preprocess the radar signals, and the image analysis unit to perform noise reduction and feature extraction on the image data; Taking the centroid feature point of the measured object itself as the origin, establish a rectangular coordinate system according to the characteristics of the target in the coordinate axis direction, and obtain the three-dimensional deformation data of the measured object; Output the processed three-dimensional deformation data to the display and interaction module in the form of graphics and reports, and display them.

6. The three-dimensional deformation measurement method of a coaxial radar and an image displacement sensor according to claim 5, characterized in that, Taking the centroid feature point of the object under test as the origin, a rectangular coordinate system is established with the coordinate axis directions determined according to the characteristics of the target, and three-dimensional deformation data of the object under test is obtained, including: Define a rectangular coordinate system (x, y, z), with the mechanical axis center of the system as the coordinate origin, the radar and optoelectronic pointing direction as the y-axis, and the direction perpendicular to the radar and optoelectronic axis plane upward as the z-axis. The x-axis is determined according to the right-hand screw rule.

7. A three-dimensional deformation measurement method for a coaxial radar and an image displacement sensor according to claim 6, characterized in that Taking the centroid feature point of the object under test as the origin, a rectangular coordinate system is established with the coordinate axis directions determined according to the characteristics of the target, and three-dimensional deformation data of the object under test is obtained, including: Obtain the translation vector and rotation matrix of the radar relative to the target through measurement. Assume the position of the system in the measured object coordinate system is t = [t x , t y , t2] T , and the rotation matrix of the system coordinate system relative to the measured object coordinate system is R; Let P r be the coordinates of a point in the system coordinate system, and p k be the coordinates of the point in the coordinate system of the object under test. Then the conversion formula is as follows: p t = Rp r + t Among them, the calculation of the rotation matrix R is obtained by combining three basic rotation matrices. The basic rotation matrices include R s (α) that rotates by an angle α around the x-axis, R y (β) that rotates by an angle β around the y-axis, and R z (γ) that rotates by an angle γ around the z-axis; Then the total rotation matrix R is obtained by first rotating about the x-axis, then about the y-axis, and finally about the z-axis: R = R2(γ)R y (β)R x (α).

8. The three-dimensional deformation measurement method of a coaxial radar and an image displacement sensor according to claim 7, characterized in that, Taking the centroid feature point of the object under test as the origin, a rectangular coordinate system is established with the coordinate axis directions determined according to the characteristics of the target, and three-dimensional deformation data of the object under test is obtained, including: The rotation angles α, β, γ and the translation vector t are solved by measuring multiple sets of corresponding values. Suppose n sets of points p in the radar coordinate system are measured r,i and the corresponding points p in the target coordinate system i,i (i = 1, 2, …, n), then there are: p i,i = Rp r,i + t Expand the equation. For each pair of points (p i,i , p i,i ), an equation can be obtained. Assume p r,i = [x r,i , y r,i , z r,i T , p t,i = [x t,i , y i,i , z t,i T , t = [t x , t y , t s T , and R is a 3×3 rotation matrix:​​​ Expand it into three linear equations: x t,j = r 11 x r,i + r 12 y r,i + r 13 z r,i + t x y i,i = r 21 x i,i + r 22 y i,i + r 23 z i,i + t y z i,i = r 31 x i,j + r 33 y i,j + r 33 z i,i + t z where x r,i , y r,i , z r,i represent the coordinate values of the point on the x, y, and z axes of the radar coordinate system respectively, and x t,i , y i,i , z t,i are the coordinate values of the point on the x, y, and z axes of the target coordinate system respectively; t x , t y , t s represent the translation amounts in the x, y, and z axis directions from the radar coordinate system to the target coordinate system respectively; When there are n groups of point pairs, construct an overdetermined system of equations containing 3n equations, and use the least squares method to solve for R and t.

9. A three-dimensional deformation measurement method for a coaxial radar and an image displacement sensor according to claim 8, characterized in that Taking the centroid feature point of the object under test as the origin, a rectangular coordinate system is established with the coordinate axis directions determined according to the characteristics of the target, and three-dimensional deformation data of the object under test is obtained, including: According to \(R = R_3(\gamma)R\) θ (\beta)R x (\alpha) Determine the values of \(\alpha\), \(\beta\), and \(\gamma\), and we have: Extract the rotation angle from the rotation matrix R through the following formula: β = -arcsin(R 31 ) α = arctan2(R 23 / cosβ, R 33 / cosβ) γ = arctan2(R 21 / cosβ, R 11 / cosβ) where R ij represents the element at the i-th row and j-th column in the rotation matrix R, and R 31 represents the element at the 3rd row and 1st column in the rotation matrix R. arcsin represents the arcsine function, which is used to find the corresponding angle according to the sine value; arctan2 represents the two-argument arctangent function.

Images