Deformation monitoring device and method for linear structures

Abstract

This application relates to a deformation monitoring device and method for a linear structure, comprising: a first monitoring line located on the central axis or one side of the linear structure to be monitored; three-dimensional markers horizontally placed at monitoring points along the first monitoring line; and bidirectional horizontal-view cameras horizontally positioned at observation points along the first monitoring line. The bidirectional horizontal-view cameras capture images of the three-dimensional markers at adjacent monitoring points in both the front and rear directions to obtain the displacement of each three-dimensional marker and camera in the X and Z axes. A second monitoring line located on the opposite side of the linear structure to be monitored, with bidirectional tilt-view cameras horizontally placed at each observation point. The bidirectional tilt-view cameras capture images of the three-dimensional markers at adjacent monitoring points along the first monitoring line to obtain the displacement of each three-dimensional marker and camera in the Y axis direction. Thus, by using multi-view images of three-dimensional markers, the dimensionality of existing visual point-based deformation measurements is expanded, improving the accuracy of three-dimensional displacement measurement of linear structures.

Description

Technical Field

[0001] This application relates to the field of three-dimensional displacement measurement technology, and in particular to a deformation monitoring device and method for linear structures. Background Technology

[0002] Vision-based displacement measurement transforms the image coordinates of artificial (or natural) markers placed at the measurement point into corresponding actual object displacements. It features easy installation, low cost, and high precision. With the continuous development of computer vision technology and image acquisition equipment, this method is widely used in various practical engineering projects such as bridges, tunnels, and pipelines. Vision-based displacement measurement mainly includes camera calibration, marker detection and tracking, and pixel displacement to object displacement conversion. Most engineering projects, such as bridge displacement monitoring points, do not have natural marker features and usually require the installation of artificial markers. Artificial markers, as the tracking targets for visual displacement measurement, are a key factor affecting the accuracy of displacement measurement. Artificial markers generally utilize the image features and texture structure of the structural surface as markers, with planar strips and regular patterns being the most common.

[0003] Due to the limitations of the visual displacement measurement principle, the current method can only obtain the two-dimensional displacement deformation of the measuring point marker in the plane, and cannot obtain the displacement in the direction of the camera's line of sight. However, in actual safety monitoring projects such as bridges and tunnels, the three-dimensional displacement deformation of the measuring point is an essential monitoring quantity and is an important supporting data for relevant departments to assess engineering structural damage, collapse and other major disasters.

[0004] Therefore, there is a need for a widely applicable three-dimensional displacement deformation measurement marker and method that breaks through the two-dimensional limitations of existing planar markers, capable of obtaining the three-dimensional displacement deformation value of the measurement point and ensuring the accuracy of the three-dimensional displacement measurement of the point. Summary of the Invention

[0005] This application provides a deformation monitoring device and method for linear structures to solve the problem that currently only the two-dimensional displacement deformation of the measuring point marker in the plane can be obtained, and the displacement deformation value of the measuring point marker in the line of sight direction cannot be obtained in real time.

[0006] The first aspect of this application provides a deformation monitoring device for a linear structure, comprising: a first monitoring line and a second monitoring line, wherein the first monitoring line is located on the central axis or one side of the linear structure to be monitored, the first monitoring line includes multiple monitoring points and multiple first observation points, each monitoring point is provided with at least one monitoring point, and a three-dimensional marker is horizontally placed on each monitoring point, the monitoring points are evenly distributed at equal intervals along the linear direction of the linear structure to be monitored or at the turning points of the linear structure to be monitored, each first observation point is set in the middle of adjacent monitoring points, and a bidirectional horizontal viewing angle camera is horizontally arranged on each first observation point, each bidirectional horizontal viewing angle camera is used to capture images of the three-dimensional markers on adjacent monitoring points in the front and rear directions, so as to obtain the displacement of each three-dimensional marker and the camera in the X and Z axis directions based on the images of the three-dimensional markers on adjacent monitoring points in the front and rear directions;

[0007] The second monitoring line is located on the opposite side of the linear structure to be monitored. The second monitoring line includes multiple second observation points, the number of which is equal to the number of multiple first observation points. The multiple second observation points and the multiple first observation points are set in a one-to-one correspondence. A bidirectional tilting camera is placed horizontally on each second observation point. The bidirectional tilting cameras are all at a preset angle and face the monitoring point direction of the first monitoring line. Each bidirectional tilting camera is used to capture images of three-dimensional markers on adjacent monitoring points of the first monitoring line based on the preset angle, so as to obtain the displacement of each three-dimensional marker and the camera in the Y-axis direction based on the images of the three-dimensional markers on adjacent monitoring points of the first monitoring line.

[0008] Optionally, the three-dimensional markers are all cubes or cuboids.

[0009] Optionally, the six planes of the cube or cuboid have identical patterns, wherein each plane consists of a black square ring, a white square ring, a black square and four circles, the four circles being symmetrically distributed, and the center of each circle being the displacement monitoring and tracking point of the three-dimensional marker. The distribution of the four circles is used to monitor the rotation state of each three-dimensional marker.

[0010] Optionally, the six planes of the cube or cuboid are made of aerospace-grade aluminum plate.

[0011] A second aspect of this application provides a deformation monitoring method for a linear structure, employing the aforementioned deformation monitoring device for a linear structure. The method includes the following steps: acquiring images of a three-dimensional marker at the current moment and images of the three-dimensional marker at the previous moment, as well as the current and previous coordinates of each three-dimensional marker, captured by multiple bidirectional horizontal-view cameras and multiple bidirectional tilt-view cameras; and constructing the displacement of each three-dimensional marker relative to its corresponding bidirectional horizontal-view camera in the X-axis and Z-axis directions based on the images of the three-dimensional marker at the current moment and the images of the three-dimensional marker at the previous moment captured by the multiple bidirectional horizontal-view cameras, and the current and previous coordinates of each three-dimensional marker. The first equation relating the quantity to the pixel change is used; based on the images of the 3D markers at the current moment and the images of the 3D markers at the previous moment captured by the multiple bidirectional tilt-view cameras, and the current and previous coordinates of each 3D marker, a second equation relating the displacement of each 3D marker and the corresponding bidirectional tilt-view camera in the Y-axis direction to the pixel change is constructed; combining the first and second equations, the displacement of each 3D marker in the X-axis, Y-axis, and Z-axis directions is obtained using least squares adjustment, so as to obtain the 3D deformation monitoring results of the linear structure based on the displacement of each 3D marker in the X-axis, Y-axis, and Z-axis directions.

[0012] Optionally, the step of constructing a first relational equation between the displacement and pixel change of each 3D marker and its corresponding bidirectional horizontal viewpoint camera in the X-axis and Z-axis directions based on the images of the 3D marker at the current moment and the images of the 3D marker at the previous moment captured by the plurality of bidirectional horizontal viewpoint cameras includes: determining the current moment bounding box and the previous moment bounding box of each 3D marker based on the images of the 3D marker at the current moment and the images of the 3D marker at the previous moment captured by the plurality of bidirectional horizontal viewpoint cameras; determining the displacement of each 3D marker and its corresponding bidirectional horizontal viewpoint camera based on the current moment bounding box and the previous moment bounding box of each 3D marker; determining the pixel change of each 3D marker based on the current moment bounding box and the previous moment bounding box of each 3D marker; and obtaining the first relational equation based on a preset imaging geometry relationship and the displacement and pixel change of each 3D marker and its corresponding bidirectional horizontal viewpoint camera.

[0013] Optionally, the step of constructing a second relational equation between the displacement and pixel change of each 3D marker and its corresponding bidirectional tilt-view camera in the Y-axis direction based on the images of the 3D marker at the current moment and the images of the 3D marker at the previous moment captured by the plurality of bidirectional tilt-view cameras includes: determining the current moment bounding box and the previous moment bounding box of each 3D marker based on the images of the 3D marker at the current moment and the images of the 3D marker at the previous moment captured by the plurality of bidirectional tilt-view cameras; determining the displacement of each 3D marker and its corresponding bidirectional tilt-view camera based on the current moment bounding box and the previous moment bounding box of each 3D marker; determining the pixel change of each 3D marker based on the current moment bounding box and the previous moment bounding box of each 3D marker; and obtaining the second relational equation based on a preset imaging geometry relationship and the displacement and pixel change of each 3D marker and its corresponding bidirectional tilt-view camera.

[0014] Optionally, the first relational equation is:

[0015]

[0016]

[0017] Among them, Z m Let Z be the Z-axis coordinate of the object square variable of the m-th 3D marker at time t. N Let f1 be the Z-axis coordinate of the object square variable of the Nth camera at time t, and let f1 be the focal length of the camera. Let be the change in camera pitch angle at time t. Y is the change in the camera's horizontal deflection angle at time t. N Let Y be the Y-axis coordinate of the object square variable of the Nth camera at time t. m Let L be the Y-axis coordinate of the square variable of the m-th 3D marker at time t. Nm0 Let X be the initial distance between camera N and 3D marker m along the Y-axis. m Let X be the X-axis coordinate of the object square variable of the m-th 3D marker at time t. N Let ΔZ be the X-axis coordinate of the object square variable of the Nth camera at time t. Nmt Let ΔX be the vertical pixel displacement of the 3D marker m in the image acquired by camera N at time t. Nmt Let be the horizontal pixel displacement of the 3D marker m in the image acquired by camera N at time t.

[0018] Optionally, the second relational equation is:

[0019]

[0020] Where DS is the distance between the 3D marker m and the camera N in the X-axis direction, α is the angle between the camera's principal axis and the Y-axis, β is the angle between the 3D marker imaging ray Ax and the principal axis at the initial moment, and γ is the angle between the 3D marker imaging ray A”x' and the principal axis at time t.

[0021] Optionally, determining the current time bounding box and the previous time bounding box of each three-dimensional marker includes: based on a preset YOLOv5 model, detecting the images of the three-dimensional markers at the current time and the images of the three-dimensional markers at the previous time captured by the plurality of bidirectional horizontal view cameras and the plurality of bidirectional tilt view cameras, respectively, to obtain the current time bounding box and the previous time bounding box of each three-dimensional marker.

[0022] In the above embodiments, multiple monitoring points and multiple first observation points are set up along the first monitoring line. At least one monitoring point is arranged at each monitoring point, and a three-dimensional marker is horizontally placed at each monitoring point. The monitoring points are evenly spaced along the linear direction of the structure to be monitored or placed at the turning points of the structure. Each first observation point is located between adjacent monitoring points, and a bidirectional horizontal-view camera is horizontally set at each first observation point. Each bidirectional horizontal-view camera is used to capture images of the three-dimensional markers at adjacent monitoring points in both the front and rear directions, so as to obtain the three-dimensional markers at each monitoring point based on the images of the three-dimensional markers at adjacent monitoring points in both the front and rear directions. The displacement of the marker and camera in the X and Z axes is determined by setting multiple second observation points along the second monitoring line. The number of second observation points is equal to the number of first observation points, and each second observation point corresponds to one of the first observation points. A bidirectional tilting camera is horizontally placed at each second observation point, with each camera at a preset angle and facing the monitoring point direction of the first monitoring line. Each camera captures images of the 3D marker at adjacent monitoring points along the first monitoring line based on these preset angles. The displacement of each 3D marker and camera in the Y axis direction is then obtained from these images. This solves the problem that current methods only obtain two-dimensional displacement deformation in the planar direction of the marker and cannot obtain the displacement deformation value along the visual axis in real time. By designing a method for acquiring images of 3D markers and multiple viewing angles, the dimensionality of existing visual point deformation measurements is expanded, while simultaneously improving the accuracy of 3D displacement measurements of linear structures.

[0023] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0024] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0025] Figure 1 This is a schematic diagram of a deformation monitoring device for a linear structure according to an embodiment of this application;

[0026] Figure 2 This is a schematic diagram of a three-dimensional marker according to an embodiment of this application;

[0027] Figure 3 This is a schematic diagram illustrating the principle of calculating the deformation of a three-dimensional marker in the X and Z axis directions according to an embodiment of this application;

[0028] Figure 4 This is a schematic diagram illustrating the principle of calculating the displacement of a three-dimensional marker in the Y-axis direction according to an embodiment of this application;

[0029] Figure 5 This is a schematic diagram of a deformation monitoring device for a linear structure according to an embodiment of this application;

[0030] Figure 6 This is an imaging schematic diagram of a three-dimensional marker according to an embodiment of this application;

[0031] Figure 7 This is a schematic diagram illustrating the extraction of a three-dimensional marker rectangle and the location of its center point according to an embodiment of this application.

[0032] Figure 8 This is a flowchart of a deformation monitoring method for a linear structure according to an embodiment of this application. Detailed Implementation

[0033] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0034] The deformation monitoring device and method for a linear structure according to embodiments of this application are described below with reference to the accompanying drawings. Addressing the problem mentioned in the background art that currently only two-dimensional displacement deformation of the measuring point marker in the plane can be obtained, and the displacement deformation value of the measuring point marker in the line of sight direction cannot be obtained in real time, this application provides a deformation monitoring device for a linear structure. In this device, multiple monitoring points and multiple first observation points are set up along a first monitoring line. At least one monitoring point is arranged at each monitoring point, and a three-dimensional marker is horizontally placed on each monitoring point. The monitoring points are evenly distributed at equal intervals along the linear direction of the linear structure to be monitored or at the turning points of the linear structure to be monitored. Each first observation point is set in the middle of adjacent monitoring points, and a bidirectional horizontal viewing angle camera is horizontally arranged on each first observation point. Each bidirectional horizontal viewing angle camera is used to capture three-dimensional images of adjacent monitoring points in both the front and rear directions. The image of the marker is used to obtain the displacement of each 3D marker and camera in the X and Z axes based on the images of the 3D markers at adjacent monitoring points in both the front and rear directions. Multiple second observation points are set up along the second monitoring line, with the number of second observation points equal to the number of first observation points, and each second observation point corresponds to one of the first observation points. A bidirectional tilting camera is placed horizontally at each second observation point, with both cameras at a preset angle and facing the monitoring point direction of the first monitoring line. Each bidirectional tilting camera captures images of the 3D markers at adjacent monitoring points along the first monitoring line based on the preset angle. The displacement of each 3D marker and camera in the Y axis direction is obtained based on these images. This solves the problem that current methods can only obtain the planar two-dimensional displacement deformation of the marker and cannot obtain the displacement deformation value along the visual axis in real time. By designing a method for acquiring 3D marker and multi-viewpoint marker images, the dimensionality of existing visual point deformation measurement is expanded, while simultaneously improving the accuracy of 3D displacement measurement of linear structures.

[0035] Specifically, Figure 1 This is a schematic diagram of a deformation monitoring device for a linear structure provided in an embodiment of this application.

[0036] like Figure 1As shown, the deformation monitoring device for the linear structure includes: a first monitoring line and a second monitoring line. The first monitoring line is located on the central axis or one side of the linear structure to be monitored. The first monitoring line includes multiple monitoring points and multiple first observation points. Each monitoring point has at least one monitoring point, and a three-dimensional marker is placed horizontally at each monitoring point. The monitoring points are evenly spaced along the linear direction of the linear structure to be monitored or at the turning points of the linear structure to be monitored. Each first observation point is located between adjacent monitoring points, and a bidirectional horizontal viewing angle camera is horizontally installed at each first observation point. Each bidirectional horizontal viewing angle camera is used to capture images of the three-dimensional markers at adjacent monitoring points in both the front and rear directions, so as to determine the deformation of the three-dimensional markers at adjacent monitoring points in both the front and rear directions. The images of the three-dimensional markers are used to obtain the displacement of each three-dimensional marker and camera in the X and Z axes. The second monitoring line is located on the opposite side of the linear structure to be monitored. The second monitoring line includes multiple second observation points, the number of which is equal to the number of first observation points. The multiple second observation points and the multiple first observation points are set in a one-to-one correspondence. A bidirectional tilting camera is placed horizontally on each second observation point. The bidirectional tilting cameras are all at a preset angle and face the monitoring point direction of the first monitoring line. Each bidirectional tilting camera is used to capture images of the three-dimensional markers on the adjacent monitoring points of the first monitoring line based on the preset angle, so as to obtain the displacement of each three-dimensional marker and camera in the Y axis direction based on the images of the three-dimensional markers on the adjacent monitoring points of the first monitoring line.

[0037] Optionally, in some embodiments, the three-dimensional markers are cubes or cuboids.

[0038] Optionally, in some embodiments, the patterns of the six planes of the cube or cuboid are all the same, wherein each plane consists of a black square ring, a white square ring, a black square and four circles, the four circles are symmetrically distributed, the center of each circle is the displacement monitoring tracking point of the three-dimensional marker, and the distribution of the four circles is used to monitor the rotation state of each three-dimensional marker.

[0039] Optionally, in some embodiments, the six planes of the cube or cuboid are made of aerospace-grade aluminum sheet.

[0040] This application provides a three-dimensional marker based on visual displacement deformation measurement to address the shortcomings of existing technologies. The marker is a three-dimensional marker that is transformed from a planar object. The three-dimensional marker has a cuboid (or cube) structure. A two-way camera can simultaneously acquire images of the three-dimensional marker from six perspectives: up, down, front, back, left, and right. The pixel transformation information of the detected three-dimensional marker is then converted into actual displacement values, and the displacement deformation values ​​of the three-dimensional marker in the three-axis directions are obtained in real time.

[0041] The composition of three-dimensional markers is as follows: Figure 2As shown, the length, width, and height of the three-dimensional marker are 15cm, and the patterns on the six faces are identical. The centroid is the origin, and the three axes of the cube are the X, Y, and Z axes. Each plane consists of a black square ring, a white square ring, a black square, and four circles. The width of the black square ring is 0.5cm, the width of the white square ring is 2cm, the size of the black square is 10cm, the diameter of the small circle is 3cm, and the diameter of the large circle is 4cm.

[0042] The six planes of the 3D marker are made of aviation-grade aluminum plate, which is non-reflective and opaque. The black and white contrast facilitates the detection of the edges of large and small circles and the location of their centers. The centers serve as the displacement monitoring and tracking points for the 3D marker. The lines connecting the four centers of each plane of the 3D marker can form a square. Multiple centers satisfy the target resolution calculation conditions for determining the conversion from pixel displacement to object displacement using the displacement calibration method (the 3D marker undergoes lateral or vertical displacement in the imaging system; the target resolution is obtained by calculating the proportional coefficient between the pixel displacement value of the center moment and the known physical displacement value of the center moment when the marker in the image undergoes displacement). When the 3D marker rotates during imaging, the four circles, of different sizes and symmetrically distributed, can detect the rotation state of the 3D marker.

[0043] It should be understood that for settlement monitoring of linear structures such as bridges and tunnels, markers are placed along their alignment and inflection points to monitor deformation. Specific placement schemes are as follows: Figure 1 As shown, three-dimensional markers are placed at displacement monitoring points (or fixed points), which are typically located at both ends of the linear structure to be monitored (e.g., points A and B). The bidirectional horizontal-view camera of the first monitoring line is placed horizontally, with its line of sight perpendicular to the plane formed by the X and Z axes of the three-dimensional marker. The bidirectional horizontal-view camera consists of two CCDs and can perform bidirectional imaging, acquiring images of the three-dimensional marker in the forward and backward directions. The bidirectional tilt-view camera of the second monitoring line is placed horizontally, with its line of sight horizontal and obliquely intersecting the plane formed by the Y and Z axes of the three-dimensional marker at a 60° angle. The bidirectional tilt-view camera consists of two CCDs at a certain angle (120°), corresponding to the camera with the same number as the camera in the first monitoring line, and is used to acquire images of the three-dimensional marker in the forward and backward directions corresponding to the camera in the first monitoring line. Based on the principle of visual displacement deformation measurement, the displacement of monitoring points in the X and Z axes can be obtained through the first monitoring line, and the displacement of monitoring points in the Y and Z axes can be obtained through the second monitoring line of the same specification. By combining the monitoring results of displacement deformation of the two lines, the three-dimensional deformation monitoring results of the linear structure to be monitored in the X, Y and Z axes can be obtained.

[0044] The specific steps for deformation monitoring of linear structures are as follows:

[0045] Step 1: Extraction and localization of 3D markers

[0046] Using images of 3D landmarks at the current time and the previous time, as well as the current and previous coordinates of each 3D landmark, acquired by multiple bidirectional horizontal and bidirectional tilt cameras, the YOLOv5 deep learning model is used to detect the images of the 3D landmarks at the current time and the previous time, respectively, to obtain the current and previous bounding boxes of each 3D landmark.

[0047] For each rectangular bounding box of a 3D marker, the image is first preprocessed by grayscale conversion, median filtering, and Otsu binarization. Then, the Canny operator is used to extract edges and fit ellipses to edge points, finally obtaining the centers of four ellipses.

[0048] Step 2: 3D marker tracking and pixel displacement

[0049] Displacement monitoring analyzes the changes in the position coordinates of three-dimensional markers at different times, thus requiring tracking the positions of markers in adjacent frames within the image. This application employs a three-dimensional marker bounding box proximity tracking method. For any three-dimensional marker in the current frame, the method searches for the three-dimensional marker in adjacent frames that is closest to the bounding box of that target three-dimensional marker; this is the tracked three-dimensional marker with the same name. The difference in pixel coordinates of the three-dimensional markers at different times represents the pixel displacement.

[0050] Step 3: Calculate the target resolution

[0051] The proportional calibration method is used, which is the ratio of the distance L from the 3D marker to the camera to the focal length f:

[0052]

[0053] Step 4: Deformation calculation model to obtain displacement in the three axes.

[0054] Specifically, based on the images of the 3D markers at the current moment and the images of the 3D markers at the previous moment captured by multiple bidirectional horizontal view cameras, as well as the current and previous coordinates of each 3D marker, a first relational equation is constructed between the displacement of each 3D marker and the corresponding bidirectional horizontal view camera in the X-axis and Z-axis directions and the pixel change. Based on the images of the 3D markers at the current moment and the images of the 3D markers at the previous moment captured by multiple bidirectional tilt view cameras, as well as the current and previous coordinates of each 3D marker, a second relational equation is constructed between the displacement of each 3D marker and the corresponding bidirectional tilt view camera in the Y-axis direction and the pixel change.

[0055] The first monitoring line can calculate the displacement of the three-dimensional marker and the camera in the X and Z axes according to formulas (2) and (3), and the principle is as follows: Figure 3 As shown.

[0056] Specifically, by using the current and previous coordinates of each 3D marker, the object square variables (Xm, Ym, Zm) of the 3D marker m at time t are obtained, ΔZ. Nmt For the three-dimensional marker m at time t t The vertical pixel displacement ΔX in the image is acquired by camera N. Nmt Let L be the horizontal pixel displacement of the 3D marker m in the image acquired by camera N at time t. Nm L is the distance between camera N and 3D marker m along the Y-axis. Nm f1 and f1 are derived from calibration, taking into account the deformation of camera N at time t (X N ,Y N Z N ) and pitch angle changes Given the relationship, the displacement of the three-dimensional marker m in the Z-axis direction is:

[0057]

[0058] Similarly, considering the deformation (X) of camera N at time t... N ,Y N Z N ) and horizontal deflection angle change Given the relationship, the displacement of the three-dimensional marker m in the direction perpendicular to the X-axis is:

[0059]

[0060] Specifically, such as Figure 1 As shown, multiple cameras N (N = 1, 2, ..., K) and multiple three-dimensional markers m (i = 1, 2, ..., M) are deployed on the first monitoring line and the second detection line, respectively. Assume that the number of cameras is K and the number of three-dimensional markers is M. Consider the distance L in the Y-axis direction between the three-dimensional marker m and camera N after deformation at time t. Nm At the same time, L will change. Nm =L Nm0 +(Y m -Y N ), L Nm0 Let N be the initial distance between the camera N and the 3D marker m along the Y-axis. Equations (2) and (3) can be expressed as equations (4) and (5), thus obtaining the first equation relating the displacement of each 3D marker to the corresponding bidirectional horizontal view camera in the X-axis and Z-axis directions to the pixel change:

[0061]

[0062] The second monitoring line calculates the displacement AO of the three-dimensional marker in the Y-axis direction using equation (6), the principle of which is as follows: Figure 4 As shown.

[0063]

[0064] exist Figure 4 In the diagram, S is the camera projection center, D is the projection point of S on the Y-axis, A is the initial position of the 3D marker, A” is the position of the marker at time t, A' is the intersection of SA” and the Y-axis, O is the projection point of A” on the Y-axis, d is the distance of the 3D marker from the Y-axis at time t, AO is the displacement of the 3D marker on the Y-axis at time t, DS is the initial distance of the 3D marker from the camera in the X direction, and f2 is the camera focal length. DS and f2 can be obtained through calibration.

[0065] In equation (6), x and x′ are the horizontal coordinates of the three-dimensional marker at the initial time and time t, respectively; x0 is the horizontal coordinate of the camera's main axis on the image; α is the angle between the camera's main axis and the Y-axis (α = 30°); β is the angle between the three-dimensional marker's imaging ray Ax and the main axis at the initial time; and γ is the angle between the marker's imaging ray A”x’ and the main axis at time t.

[0066] Further transforming formula (6) into coordinate form, we obtain the second relational equation between the displacement of each three-dimensional marker and the corresponding bidirectional tilt camera in the Y-axis direction and the pixel change.

[0067]

[0068] Finally, by combining the first and second relational equations, the displacement of each three-dimensional marker in the X-axis, Y-axis and Z-axis directions is obtained by using least squares adjustment equations (4), (5) and (7), so as to obtain the three-dimensional deformation monitoring results of the linear structure based on the displacement of each three-dimensional marker in the X-axis, Y-axis and Z-axis directions.

[0069] To enable those skilled in the art to further understand the deformation monitoring method for linear structures according to the embodiments of this application, the following detailed description is provided in conjunction with specific embodiments. Figure 5 As shown.

[0070] Specifically, four cameras are deployed on the first and second monitoring lines respectively. Assume that there are four groups of cameras, A1, A2, B1, B2, C1, C2 and D1, D2, all placed horizontally. The deformation values ​​of each group of cameras are equal at time t. Fifteen three-dimensional markers are deployed on the first monitoring line, all placed horizontally.

[0071] The two-way horizontal-view camera on the first monitoring line can acquire images of six three-dimensional markers located in front of and behind it. For example, the two-way horizontal-view camera A1 can acquire images of markers m1-m6 with a focal length of f1 = 15mm. Simultaneously, the two-way tilt-view camera on the second monitoring line can acquire images of six three-dimensional markers located in front of it. For example, the two-way tilt-view camera A2 can acquire images of three-dimensional markers m1-m6 with a focal length of f2 = 30mm. Three-dimensional markers m1 and m15 are stationary points.

[0072] The distance L from the three-dimensional marker to the camera along the Y-axis required in formulas (4), (5), and (7) Nm As shown in Table 1, the distance DS in the X-axis direction is shown in Table 2, with the unit being meters.

[0073] Table 1

[0074]

[0075] Table 2

[0076]

[0077] As can be seen from the above, the three-dimensional markers m4, m5, and m14 have displacements in the X, Y, and Z axes relative to the initial time. The three-dimensional markers m8 and m12 have displacements in the X axis direction, the three-dimensional marker m9 has displacements in the Z axis direction, the three-dimensional marker m11 has displacements in the Y axis direction, the three-dimensional marker m10 has displacements in both the Y and Z axes directions, cameras A1, A2, C1, and C2 have displacements in the X and Y axes directions, and cameras B1, B2, D1, and D2 have displacements in the Z axis direction.

[0078] Imaging of three-dimensional markers, such as Figure 6 As shown, in Figure 6 In the diagram, "A, B, C, D" represent the camera station locations. The second digit, "1" and "2," indicate the initial image and the image after distortion, respectively. "f" indicates forward-looking, and "b" indicates backward-looking. Images with the suffix "-1" were obtained from the first monitoring line, and images with the suffix "-2" were obtained from the second monitoring line. Following steps 1 and 2, markers can be located, such as... Figure 7 As shown, the pixel displacements of the three-dimensional marker in the X and Y axes at the initial time and time t are obtained.

[0079] After obtaining the point coordinates and pixel displacement of the 3D marker image, the distance from the 3D marker to the camera in the X, Y, and Z axes, and the camera focal length, the 3D displacement between the 3D marker and the camera is obtained by least squares adjustment of the first and second relational equations, thus obtaining the 3D displacement monitoring results of the linear structure, as shown in Table 3, with the unit being mm.

[0080] Table 3

[0081]

[0082] As can be seen from Table 3, the three-dimensional marker based on visual displacement deformation measurement designed in the embodiments of this application can measure the three-dimensional displacement of linear structures. The root mean square error (RMSE) of displacement in the X-axis, Y-axis and Z-axis directions is 0.44 mm, 0.74 mm and 0.48 mm, respectively.

[0083] According to the deformation monitoring device for a linear structure proposed in this application, multiple monitoring points and multiple first observation points are set up along a first monitoring line. At each monitoring point, at least one monitoring point is arranged, and a three-dimensional marker is horizontally placed on each monitoring point. The monitoring points are evenly spaced along the linear direction of the linear structure to be monitored or are placed at the turning points of the linear structure. Each first observation point is located between adjacent monitoring points, and a bidirectional horizontal viewing angle camera is horizontally set on each first observation point. Each bidirectional horizontal viewing angle camera is used to capture images of the three-dimensional markers on adjacent monitoring points in both the front and rear directions, so as to determine the deformation based on the three-dimensional markers on adjacent monitoring points in both the front and rear directions. The imagery captures the displacement of each 3D marker and camera in the X and Z axes. Multiple second observation points are set up along the second monitoring line, with the number of second observation points equal to the number of first observation points, and each second observation point corresponds to one of the first observation points. A bidirectional tilting camera is horizontally placed at each second observation point, with both cameras at a preset angle and facing the monitoring point direction of the first monitoring line. Each bidirectional tilting camera captures images of the 3D markers at adjacent monitoring points along the first monitoring line based on these preset angles. The displacement of each 3D marker and camera in the Y axis direction is then obtained from these images. This solves the problem that current methods can only obtain the planar two-dimensional displacement deformation of the measuring point markers and cannot obtain the displacement deformation value along the visual axis in real time. By designing a method for acquiring 3D marker images and multi-view measuring point marker images, the dimensionality of existing visual point deformation measurement is expanded, while simultaneously improving the accuracy of 3D displacement measurement of linear structures.

[0084] Next, with reference to the accompanying drawings, a deformation monitoring method for linear structures proposed according to embodiments of this application is described.

[0085] Figure 8 This is a schematic diagram of a deformation monitoring method for a linear structure according to an embodiment of this application.

[0086] like Figure 8 As shown, the deformation monitoring method for this linear structure includes the following steps:

[0087] In step S801, the images of the three-dimensional markers at the current moment and the images of the three-dimensional markers at the previous moment, as well as the current and previous coordinates of each three-dimensional marker, are acquired from multiple bidirectional horizontal view cameras and multiple bidirectional tilt view cameras.

[0088] In step S802, based on the images of the three-dimensional markers at the current moment captured by multiple bidirectional horizontal view cameras, the images of the three-dimensional markers at the previous moment, the current moment coordinates and the previous moment coordinates of each three-dimensional marker, a first relational equation is constructed between the displacement of each three-dimensional marker and the corresponding bidirectional horizontal view camera in the X-axis and Z-axis directions and the pixel change.

[0089] In step S803, based on the images of the three-dimensional markers at the current moment captured by multiple bidirectional tilt-view cameras, the images of the three-dimensional markers at the previous moment, the current moment coordinates and the previous moment coordinates of each three-dimensional marker, a second relational equation is constructed between the displacement of each three-dimensional marker and the corresponding bidirectional tilt-view camera in the Y-axis direction and the pixel change.

[0090] In step S804, the first relational equation and the second relational equation are combined, and the displacement of each three-dimensional marker in the X-axis, Y-axis and Z-axis directions is obtained by least squares adjustment, so as to obtain the three-dimensional deformation monitoring results of the linear structure based on the displacement of each three-dimensional marker in the X-axis, Y-axis and Z-axis directions.

[0091] Optionally, in some embodiments, based on the images of the 3D marker at the current moment captured by multiple bidirectional horizontal view cameras, the image of the 3D marker at the previous moment, and the current and previous coordinates of each 3D marker, a first relational equation is constructed between the displacement and pixel change of each 3D marker and the corresponding bidirectional horizontal view camera in the X-axis and Z-axis directions. This includes: determining the current and previous bounding boxes of each 3D marker based on the images of the 3D marker at the current moment captured by multiple bidirectional horizontal view cameras and the image of the 3D marker at the previous moment; determining the displacement of each 3D marker and the bidirectional horizontal view camera based on the current and previous coordinates of each 3D marker; determining the pixel change of each 3D marker based on the current and previous bounding boxes of each 3D marker; and obtaining the first relational equation based on a preset imaging geometry relationship and the displacement and pixel change of each 3D marker and the corresponding bidirectional horizontal view camera.

[0092] Optionally, in some embodiments, based on the images of the 3D marker at the current moment captured by multiple bidirectional tilt-view cameras, the image of the 3D marker at the previous moment, and the current and previous coordinates of each 3D marker, a second relational equation is constructed between the displacement of each 3D marker and the corresponding bidirectional tilt-view camera in the Y-axis direction and the pixel change. This includes: determining the current and previous bounding boxes of each 3D marker based on the images of the 3D marker at the current moment captured by multiple bidirectional tilt-view cameras, and determining the displacement of each 3D marker relative to the bidirectional tilt-view camera based on the current and previous coordinates of each 3D marker; determining the pixel change of each 3D marker based on the current and previous bounding boxes of each 3D marker; and obtaining the second relational equation based on a preset imaging geometry relationship, according to the displacement of each 3D marker relative to the corresponding bidirectional tilt-view camera and the pixel change of each 3D marker.

[0093] Optionally, in some embodiments, the first relational equation is:

[0094]

[0095]

[0096] Among them, Z m Let Z be the Z-axis coordinate of the object square variable of the m-th 3D marker at time t. N Let f1 be the Z-axis coordinate of the object square variable of the Nth camera at time t, and let f1 be the focal length of the camera. Let be the change in the camera's pitch angle at time t. Y is the change in the camera's horizontal deflection angle at time t. N Let Y be the Y-axis coordinate of the object square variable of the Nth camera at time t. m Let L be the Y-axis coordinate of the square variable of the m-th 3D marker at time t. Nm0 Let X be the initial distance between camera N and 3D marker m along the Y-axis. m Let X be the X-axis coordinate of the object square variable of the m-th 3D marker at time t. N Let ΔZ be the X-axis coordinate of the object square variable of the Nth camera at time t. Nmt Let ΔX be the vertical pixel displacement of the 3D marker m in the image acquired by camera N at time t. Nmt Let be the horizontal pixel displacement of the 3D marker m in the image acquired by camera N at time t.

[0097] Optionally, in some embodiments, the second relational equation is:

[0098]

[0099] Where DS is the distance between the 3D marker m and the camera N in the X-axis direction, α is the angle between the camera's principal axis and the Y-axis, β is the angle between the 3D marker imaging ray Ax and the principal axis at the initial moment, and γ is the angle between the 3D marker imaging ray A”x' and the principal axis at time t.

[0100] Optionally, in some embodiments, determining the current time bounding box and the previous time bounding box of each 3D marker includes: based on a preset YOLOv5 model, detecting the images of the 3D marker at the current time and the images of the 3D marker at the previous time captured by multiple bidirectional horizontal view cameras and multiple bidirectional tilt view cameras, respectively, to obtain the current time bounding box and the previous time bounding box of each 3D marker.

[0101] It should be noted that the foregoing explanation of the deformation monitoring device embodiment for linear structures also applies to the deformation monitoring method for linear structures in this embodiment, and will not be repeated here.

[0102] According to the deformation monitoring method for linear structures proposed in this application, images of the three-dimensional markers at the current moment and the previous moment, as well as the current and previous coordinates of each three-dimensional marker, are acquired from multiple bidirectional horizontal view cameras and multiple bidirectional tilt view cameras. Based on the images of the three-dimensional markers at the current moment and the previous moment, and the current and previous coordinates of each three-dimensional marker, a first relationship is constructed between the displacement and pixel change of each three-dimensional marker and the corresponding bidirectional horizontal view camera in the X-axis and Z-axis directions. The equations, based on images of the 3D markers captured by multiple bidirectional tilt-view cameras at the current moment, images of the 3D markers at the previous moment, and the current and previous coordinates of each 3D marker, construct a second relational equation between the displacement and pixel change of each 3D marker and its corresponding bidirectional tilt-view camera in the Y-axis direction. Combining the first and second relational equations, least squares adjustment is used to obtain the displacement of each 3D marker in the X, Y, and Z axes. This allows for the determination of the 3D deformation monitoring results of the linear structure based on the displacement of each 3D marker in these directions. This solves the problem that current methods can only obtain the planar two-dimensional displacement deformation of the measuring point markers and cannot obtain the displacement deformation value along the visual axis in real time. By designing 3D markers and multi-view measuring point marker image acquisition methods, the dimensions of existing visual point deformation measurement are expanded, while simultaneously improving the accuracy of 3D displacement measurement of linear structures.

[0103] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0104] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0105] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

Claims

1. A deformation monitoring device for a linear structure, characterized in that, include: The first monitoring line and the second monitoring line, among which... The first monitoring line is located on the central axis or one side of the linear structure to be monitored. The first monitoring line includes multiple monitoring points and multiple first observation points. At least one monitoring point is arranged at each monitoring point. A three-dimensional marker is placed horizontally at each monitoring point. The monitoring points are evenly distributed at equal intervals along the linear direction of the linear structure to be monitored or at the turning points of the linear structure to be monitored. Each first observation point is set in the middle of adjacent monitoring points, and a bidirectional horizontal viewing angle camera is horizontally arranged at each first observation point. Each bidirectional horizontal viewing angle camera is used to capture images of the three-dimensional markers on adjacent monitoring points in the front and rear directions, so as to obtain the displacement of each three-dimensional marker and the camera in the X and Z axis directions based on the images of the three-dimensional markers on adjacent monitoring points in the front and rear directions. The second monitoring line is located on the opposite side of the linear structure to be monitored. The second monitoring line includes multiple second observation points, the number of which is equal to the number of multiple first observation points. The multiple second observation points and the multiple first observation points are set in a one-to-one correspondence. A bidirectional tilting camera is placed horizontally on each second observation point. The bidirectional tilting cameras are all at a preset angle and face the monitoring point direction of the first monitoring line. Each bidirectional tilting camera is used to capture images of three-dimensional markers on adjacent monitoring points of the first monitoring line based on the preset angle, so as to obtain the displacement of each three-dimensional marker and the camera in the Y-axis direction based on the images of the three-dimensional markers on adjacent monitoring points of the first monitoring line.

2. The deformation monitoring device for a linear structure according to claim 1, characterized in that, All three-dimensional markers are cubes or cuboids.

3. The deformation monitoring device for a linear structure according to claim 2, characterized in that, The patterns on all six planes of the cube or cuboid are identical, wherein, Each plane consists of a black square ring, a white square ring, a black square, and four circles. The four circles are symmetrically distributed, and the center of each circle is the displacement monitoring and tracking point of the three-dimensional marker. The distribution of the four circles is used to monitor the rotation state of each three-dimensional marker.

4. The deformation monitoring device for a linear structure according to claim 2, characterized in that, The six planes of the cube or cuboid are made of aviation aluminum plate.

5. A method for monitoring the deformation of a linear structure, characterized in that, The deformation monitoring device for a linear structure as described in any one of claims 1-4, wherein the method comprises the following steps: Acquire images of the 3D landmarks at the current moment and the 3D landmarks at the previous moment, captured by multiple bidirectional horizontal view cameras and multiple bidirectional tilt view cameras; acquire the current moment coordinates and the previous moment coordinates of each 3D landmark. Based on the images of the three-dimensional markers at the current moment and the images of the three-dimensional markers at the previous moment captured by the multiple bidirectional horizontal perspective cameras, as well as the current and previous coordinates of each three-dimensional marker, a first relational equation is constructed between the displacement and pixel change of each three-dimensional marker and the corresponding bidirectional horizontal perspective camera in the X-axis and Z-axis directions. Based on the images of the three-dimensional markers at the current moment and the images of the three-dimensional markers at the previous moment captured by the multiple bidirectional tilt-view cameras, as well as the current and previous coordinates of each three-dimensional marker, a second relationship equation is constructed between the displacement and pixel change of each three-dimensional marker and the corresponding bidirectional tilt-view camera in the Y-axis direction. By combining the first and second relational equations, the displacement of each three-dimensional marker in the X-axis, Y-axis and Z-axis directions is obtained using least squares adjustment, so as to obtain the three-dimensional deformation monitoring results of the linear structure based on the displacement of each three-dimensional marker in the X-axis, Y-axis and Z-axis directions.

6. The method according to claim 5, characterized in that, The first relationship equation for the displacement and pixel change of each 3D marker relative to its corresponding bidirectional horizontal view camera in the X and Z axes is constructed based on the images of the 3D marker at the current moment captured by the plurality of bidirectional horizontal view cameras, the images of the 3D marker at the previous moment, the current moment coordinates and the previous moment coordinates of each 3D marker, including: Based on the images of the three-dimensional markers at the current moment and the images of the three-dimensional markers at the previous moment captured by the multiple bidirectional horizontal perspective cameras, the current moment bounding box and the previous moment bounding box of each three-dimensional marker are determined, and the displacement of each three-dimensional marker relative to the bidirectional horizontal perspective camera is determined based on the current moment coordinates and the previous moment coordinates of each three-dimensional marker. The pixel change of each three-dimensional marker is determined based on the current time frame and the previous time frame of each three-dimensional marker. Based on the preset imaging geometry, the first relational equation is obtained according to the displacement of each three-dimensional marker and the corresponding bidirectional horizontal view camera and the pixel change of each three-dimensional marker.

7. The method according to claim 5, characterized in that, The second relationship equation for the displacement and pixel change of each 3D marker in the Y-axis direction between the marker and the corresponding bidirectional tilt-view camera is constructed based on the images of the 3D marker at the current moment, the images of the 3D marker at the previous moment, and the current and previous coordinates of each 3D marker captured by the plurality of bidirectional tilt-view cameras. This includes: Based on the images of the three-dimensional markers at the current moment and the images of the three-dimensional markers at the previous moment captured by the multiple bidirectional tilt-view cameras, the current moment bounding box and the previous moment bounding box of each three-dimensional marker are determined, and the displacement of each three-dimensional marker relative to the bidirectional tilt-view cameras is determined based on the current moment coordinates and the previous moment coordinates of each three-dimensional marker. The pixel change of each three-dimensional marker is determined based on the current time frame and the previous time frame of each three-dimensional marker. Based on the preset imaging geometry, the second relational equation is obtained according to the displacement of each three-dimensional marker and the corresponding bidirectional tilting camera and the pixel change of each three-dimensional marker.

8. The method according to claim 6, characterized in that, The first relational equation is: Among them, Z m Let Z be the Z-axis coordinate of the object square variable of the m-th 3D marker at time t. N Let f1 be the Z-axis coordinate of the object square variable of the Nth camera at time t, and let f1 be the focal length of the camera. Let be the change in camera pitch angle at time t. Y is the change in the camera's horizontal deflection angle at time t. N Let Y be the Y-axis coordinate of the object square variable of the Nth camera at time t. m Let L be the Y-axis coordinate of the square variable of the m-th 3D marker at time t. Nm0 Let X be the initial distance between camera N and 3D marker m along the Y-axis. m Let X be the X-axis coordinate of the object square variable of the m-th 3D marker at time t. N Let ΔZ be the X-axis coordinate of the object square variable of the Nth camera at time t. Nmt Let ΔX be the vertical pixel displacement of the 3D marker m in the image acquired by camera N at time t. Nmt Let be the horizontal pixel displacement of the 3D marker m in the image acquired by camera N at time t.

9. The method according to claim 7, characterized in that, The second relational equation is: Where DS is the distance between the 3D marker m and the camera N in the X-axis direction, α is the angle between the camera's principal axis and the Y-axis, β is the angle between the 3D marker imaging ray Ax and the principal axis at the initial moment, and γ is the angle between the 3D marker imaging ray A”x' and the principal axis at time t.

10. The method according to claim 7, characterized in that, Determining the current time bounding box and the previous time bounding box of each three-dimensional marker includes: Based on the preset YOLOv5 model, the images of the three-dimensional markers at the current moment and the images of the three-dimensional markers at the previous moment, captured by the multiple bidirectional horizontal view cameras and the multiple bidirectional tilt view cameras, are detected to obtain the current moment bounding box and the previous moment bounding box of each three-dimensional marker.

Images