A large structure hoisting process monitoring method and system based on a panoramic camera and laser scanning

Abstract

The application discloses a large structure hoisting process monitoring method and system based on a panoramic camera and laser scanning, and belongs to the technical field of structure health monitoring and engineering surveying. In view of the problems of the existing large structure displacement measurement method, such as dependence on a total station or a multi-camera system, low efficiency, limited coverage range, and difficulty in adapting to dynamic changes, the application adopts a panoramic camera and a three-dimensional laser scanner to obtain images and high-precision point clouds, and realizes the unification of image and point cloud coordinates through joint calibration. An improved TPH-YOLOv5 model is used to automatically identify the node area in the panoramic image, and the node area is mapped to the point cloud. Then, a density peak clustering algorithm is used to judge the node overlap condition, and a minimum distance optimization algorithm is used to accurately calculate the three-dimensional coordinates of the node, so that high-precision monitoring of the full-field multi-point dynamic displacement is finally realized.

Description

Technical Field

[0001] This invention belongs to the field of structural health monitoring and engineering measurement technology, specifically relating to a method and system for monitoring the hoisting process of large structures based on panoramic cameras and laser scanning. Background Technology

[0002] In the construction and operation of large structures, accurate measurement of displacement changes is crucial for ensuring engineering safety. Traditional measurement methods largely rely on total stations, which, while highly accurate, suffer from low efficiency, limited coverage, and difficulty adapting to dynamic changes. In recent years, 3D laser scanning technology has been widely used due to its high precision and non-contact measurement advantages, but challenges remain in large-scale data processing and target recognition. Panoramic cameras can provide wide-range, real-time image information, but images alone are insufficient for high-precision 3D displacement measurement. Current technologies lack an automated dynamic displacement monitoring solution that integrates rapid imaging from panoramic cameras with high-precision point cloud data from laser scanning. Summary of the Invention

[0003] To address the shortcomings of existing technologies, this invention provides a method and system for monitoring the hoisting process of large structures based on panoramic cameras and laser scanning.

[0004] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0005] In a first aspect, the present invention provides a method for monitoring the hoisting process of large structures based on panoramic cameras and laser scanning, comprising the following steps: S1: A panoramic image sequence of the cable net structure is acquired using a panoramic camera, and a point cloud is obtained using a 3D laser scanner; the panoramic image sequence and the point cloud are temporally aligned and paired data is formed; the in-camera distortion parameters are obtained and the panoramic images are geometrically corrected. S2: Establish calibration constraints by combining the characteristics of the cable net structure and the characteristics of the field environment; obtain corresponding observations of the same node in the panoramic image and the point cloud respectively, and solve the external parameter transformation between the camera and the laser scanner to establish the mapping relationship from the point cloud to the pixel coordinates of the panoramic image; S3: Input the calibrated panoramic image into the deep learning node detection network; output the region of interest (ROI) of the node; S4: Project the point cloud onto the panoramic image plane based on extrinsic parameter transformation and camera intrinsic parameters; extract the set of projection points falling within the ROI range, and back-associate the projection point set with the 3D space to form a node point cloud; output the node point cloud corresponding to each ROI; S5: Determine the number of nodes in the point cloud corresponding to the ROI, and separate the point cloud in the case of multiple nodes; perform center optimization on the separated point cloud to obtain the three-dimensional coordinates of the nodes and form a set of three-dimensional coordinates of the nodes. S6: Repeat S1 to S5 at different times and establish the temporal association of the same node; compare the three-dimensional coordinates of the same node at different times, calculate the displacement, and output the dynamic displacement monitoring results of the node.

[0006] Optionally, in step S1: The camera intrinsic parameters form a camera intrinsic parameter matrix. K Its formula is: ; in, and These are the equivalent focal lengths in pixels along the horizontal and vertical directions of the image, respectively. and These are the horizontal and vertical pixel coordinates of the camera principal point in the image coordinate system; 0 and 1 are fixed elements of the intrinsic parameter matrix.

[0007] Optionally, in step S2, the solution of the extrinsic parameter transformation between the camera and the laser scanner specifically includes: The extrinsic parameter transformation is represented using a homogeneous matrix: ; in, For the coordinate system of the laser scanner l To the camera coordinate system c The 4×4 homogeneous transformation matrix; This is a 3×3 rotation matrix that rotates the laser scanner coordinate system to the camera coordinate system. For the corresponding 3×1 translation vector; subscript c and l These represent the camera coordinate system and the laser scanner coordinate system, respectively; the last row of the matrix [0, 0, 0, 1] is used for homogeneous coordinate representation; The point cloud coordinates to camera coordinates satisfy: ; in, These represent the coordinates of the point cloud in three directions; , and The three-dimensional coordinates of the point in the camera coordinate system; The camera coordinates to pixel coordinates satisfy: ; in, u and v These are the horizontal and vertical pixel coordinates of the projection point in the image, respectively; K This is the camera intrinsic parameter matrix.

[0008] Optionally, in step S2, the mapping relationship from the point cloud to the pixel coordinates of the panoramic image is equivalently represented as: ; ; in, For the first i Each node obtains its predicted pixel coordinates based on the current calibration parameters; Π(·) For perspective normalization projection operators; I It is a 3×3 identity matrix. Used to extract 3D camera coordinates from homogeneous coordinates; This is the homogeneous transformation matrix from the laser scanner coordinate system to the camera coordinate system; For the first i Homogeneous 3D coordinates of each node; , and These are its three coordinate components.

[0009] Optionally, in step S2, acquiring the corresponding observations of the same node in both the panoramic image and the point cloud specifically involves: At least four corresponding observations of the same cable net structure feature node are obtained from both the image and the point cloud, forming an image observation set. and point cloud observation set ,in:

[0010] in, For the first i Two-dimensional pixel coordinate vectors of nodes; and The first i The horizontal and vertical pixel coordinates of each node; i For node sequence number, i =1, 2, 3, 4; superscript T Indicates the transpose of a matrix or vector;

[0011] in, For the first i The three-dimensional coordinate vectors of each node in the laser scanner coordinate system; 、 and These are its three coordinate components; i Node number; superscript T This represents the transpose of a vector.

[0012] Optionally, in step S2, the solution of the extrinsic parameter transformation between the camera and the laser scanner specifically includes: Based on the initial values ​​of the extrinsic parameters, structural constraints and environmental constraints are introduced for joint optimization, resulting in an optimized extrinsic parameter matrix that satisfies the following equation: ; in, This is the optimal extrinsic transformation matrix obtained after joint optimization; SE(3) Represents a three-dimensional rigid body transformation group; ρ (·) is a robust loss function used to reduce the impact of outlier matching points; , and These are the weighting coefficients for each constraint term; The constraint terms for the node point cloud to satisfy the main geometric features of the cable net structure include at least one of the cable net principal plane, local fitting surface and circumferential principal curve; For structural dimensional constraints, the constraints include at least one of node spacing, node radial distance, and design dimension range; This is an environmental direction constraint, which includes at least one of the following: gravity direction, ground normal, and equipment installation attitude reference direction.

[0013] Optionally, in step S3, the deep learning node detection network adopts the TPH-YOLOv5 model and introduces the Transformer attention mechanism in the prediction head.

[0014] Optionally, in step S5, the step of determining the number of nodes in the point cloud corresponding to the ROI and performing point cloud separation in the case of multiple nodes specifically includes: First, define the local density:

[0015] in, For local density; χ (·) is a truncation kernel function or indicator function; Calculate the distance between points in the point cloud; To cut off the distance;

[0016] in, x for ; The relative distance and the discriminant satisfy the following:

[0017] in, δ i For the first i The minimum distance between a point and all points whose local density is higher than its own. d ij For the firsti The point and the first j The distance between points; I s m is a high-density candidate point index set constructed for the current point; i n represents the minimum value within the set; Calculate the first i Cluster center discriminant of each point :

[0018] Finally, Sort the nodes in descending order and use a threshold to determine the number of nodes within the ROI region:

[0019] in, For the current RO I The number of nodes determined internally; and They are respectively The maximum and second largest values ​​after sorting in descending order; For preset or adaptive discrimination thresholds; others i se indicates a situation where the aforementioned conditions are not met; when When the value is greater than or equal to 2, the point cloud is split to obtain multiple node point clouds.

[0020] Optionally, in step S5, the center optimization solution for the separated node point cloud specifically includes: First, the separated node point cloud is projected in two dimensions and fitted to obtain multiple linear constraints:

[0021] in, , and For the first k The coefficients of the straight lines; x and y Let be the coordinates of any point within the two-dimensional projection plane; k The line number; n The total number of fitted lines participating in the center optimization; Next, construct the objective function:

[0022] in, For candidate center points to n The objective function is the sum of squared distances between the fitted lines; and Here are the two-dimensional center coordinates of the node to be determined; Then by solving and Obtain the center point ; The vectorized equivalent form of the objective function for:

[0023] in, .

[0024] Secondly, the present invention provides a monitoring system for the hoisting process of large structures based on panoramic cameras and laser scanning, characterized in that it includes: A panoramic camera used to acquire panoramic image sequences of cable net structures; A 3D laser scanner is used to acquire 3D point clouds of cable net structures. The joint calibration module is used to establish calibration constraints by combining the characteristics of the cable net structure with the characteristics of the field environment; it acquires corresponding observations of the same node in the panoramic image and the point cloud respectively, and solves the extrinsic parameter transformation between the camera and the laser scanner to establish the mapping relationship from the point cloud to the pixel coordinates of the panoramic image. The image processing and small object detection module is used to temporally align panoramic image sequences and point clouds to form paired data; acquire in-camera distortion parameters and perform geometric correction on panoramic images; input the corrected panoramic images into a deep learning node detection network; and output the region of interest (ROI) of the nodes. The point cloud mapping and node extraction module is used to project the point cloud onto the panoramic image plane based on extrinsic parameter transformation and camera intrinsic parameters; extract the set of projection points falling within the ROI range, and back-associate the set of projection points back into the three-dimensional space to form a node point cloud; output the node point cloud corresponding to each ROI; The node classification and center optimization module is used to determine the number of nodes in the point cloud corresponding to the ROI and to separate the point cloud in the case of multiple nodes; the center optimization solution is performed on the separated point cloud to obtain the three-dimensional coordinates of the nodes and form a set of three-dimensional coordinates of the nodes. The displacement calculation and output module is used to establish the temporal correlation of the same node; compare the three-dimensional coordinates of the same node at different times, calculate the displacement, and output the dynamic displacement monitoring results of the node.

[0025] The beneficial effects of this invention are: This invention, by integrating panoramic imaging and laser scanning, enables non-contact, full-field automated monitoring of multiple nodes in large structures, reducing point-by-point measurements and the need for multiple camera deployments, thereby increasing the monitoring range and operational efficiency. Through joint calibration, small target detection, and image-point cloud mapping, this invention improves the stability and reliability of node identification and 3D localization in complex environments. Furthermore, by employing density peak clustering and center optimization, this invention achieves overlapping node separation and multi-node displacement calculation, enhancing displacement monitoring capabilities under complex node conditions and providing support for construction safety control. Attached Figure Description

[0026] Figure 1 This is a detailed flowchart of the present invention; Figure 2 This is a schematic diagram of panoramic image segmentation according to the present invention; Figure 3 This is a flowchart illustrating the joint calibration of the panoramic camera and laser scanner of the present invention. Figure 4 This is a schematic diagram illustrating the mapping of image nodes to point clouds according to the present invention; Figure 5 This is a schematic diagram of the density peak clustering node classification results of the present invention; Figure 6 This is a schematic diagram of the node center optimization algorithm of the present invention; Figure 7 This is a comparative diagram of the experimental setup for this invention; Figure 8 A schematic diagram of point cloud projection onto an image according to the present invention. Detailed Implementation

[0027] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0028] Example 1 This embodiment provides a method for monitoring the hoisting process of a large-scale space frame structure based on panoramic cameras and laser scanning. The specific process is as follows: Figure 1 As shown.

[0029] 1) Data Input like Figure 7 As shown, this embodiment uses a panoramic camera to acquire a full-field image sequence of the structure's dynamic process; a 3D laser scanner acquires the structure's 3D point cloud. The panoramic camera is a dual-lens panoramic camera, capable of outputting high-resolution panoramic images and performing distortion correction on the input node detection and point cloud projection images. Images and point clouds can be synchronized via timestamps, triggered synchronization, or phased alignment to form associated data pairs. For example... Figure 8 As shown, this embodiment supports projecting point clouds onto an image plane and filtering them according to the image ROI, providing a foundation for subsequent node point cloud extraction and 3D localization.

[0030] 2) Camera-laser joint calibration like Figure 3 As shown, this embodiment unifies the coordinate systems of the panoramic camera and the 3D laser scanner through the correspondence of structural nodes, and establishes a mapping relationship from point cloud to pixel.

[0031] The extrinsic homogeneous transformation is expressed as:

[0032] in, For the coordinate system of the laser scanner l To the camera coordinate system c The 4×4 homogeneous transformation matrix; This is a 3×3 rotation matrix that rotates the laser scanner coordinate system to the camera coordinate system. For the corresponding 3×1 translation vector; subscript c and l These represent the camera coordinate system and the laser scanner coordinate system, respectively; the last row of the matrix [0, 0, 0, 1] is used for homogeneous coordinate representation.

[0033] Point cloud Transformation to the camera coordinate system satisfies:

[0034] in, These represent the coordinates of the point cloud in three directions; , and Let be the three-dimensional coordinates of the point in the camera coordinate system.

[0035] The camera intrinsic parameter matrix is:

[0036] in, K This is the camera intrinsic parameter matrix; and These are the equivalent focal lengths in pixels along the horizontal and vertical directions of the image, respectively. and These are the horizontal and vertical pixel coordinates of the camera principal point in the image coordinate system, respectively; 0 and 1 in the matrix are fixed elements of the intrinsic parameter matrix.

[0037] The pixel projection relationship is as follows:

[0038] in, u and v These are the horizontal and vertical pixel coordinates of the projection point in the image, respectively. u , v ,1] These are homogeneous pixel coordinates; , and The three-dimensional coordinates of the point in the camera coordinate system; K This is the camera intrinsic parameter matrix.

[0039] Select at least four structural nodes with the same name from the image and point cloud to establish a corresponding set:

[0040] in, U This is a set of image observations for four groups of structural nodes with the same name. For the first i Two-dimensional pixel coordinate vectors of nodes; and The first i The horizontal and vertical pixel coordinates of each node; i For node sequence number, i =1, 2, 3, 4; superscript T This indicates the transpose of a matrix or vector.

[0041]

[0042] in, This is a set of 3D nodes of the laser point cloud corresponding to the image observation; For the first i The three-dimensional coordinate vectors of each node in the laser scanner coordinate system; , and These are its three coordinate components; i Node number; superscript T This represents the transpose of a vector.

[0043] The projected prediction points are:

[0044] in, For the first i Each node obtains its predicted pixel coordinates based on the current calibration parameters; Π(·) The perspective normalized projection operator is obtained by dividing the camera coordinates by the depth to get the two-dimensional pixel coordinates. K This is the camera intrinsic parameter matrix; I It is a 3×3 identity matrix, [ I [0] is used to extract 3D camera coordinates from homogeneous coordinates; This is the homogeneous transformation matrix from the laser scanner coordinate system to the camera coordinate system; For the first i The homogeneous three-dimensional coordinates of each node.

[0045]

[0046] Solving for the initial values ​​of the extrinsic parameters by minimizing the reprojection error:

[0047] in, and These are the initial estimates of the rotation matrix and the translation vector, respectively. argmin This indicates finding the parameters that minimize the objective function; For the first i The measured pixel coordinates of each node. These are the corresponding predicted pixel coordinates; Σ This represents the sum of errors for the correspondence between the four sets of nodes.

[0048] In addition, a robust loss function is introduced and combined with structural geometry and environmental reference direction constraints to refine the extrinsic parameters:

[0049] in, This is the optimal extrinsic transformation matrix obtained after joint optimization; SE(3) Represents a three-dimensional rigid body transformation group; ρ (·) A robust loss function used to reduce the impact of outlier matching points; , and These are the weighting coefficients for each constraint term; The constraint term is used to satisfy the main geometric features of the cable net structure for the node point cloud. The main geometric features include at least one of the cable net principal plane, local fitted surface and circumferential principal curve. The structural dimension constraint term includes at least one of node spacing, node radial distance, and design dimension range, and is used to suppress scale drift and mismatch in the external parameter solution. This is an environmental direction constraint term, which includes at least one of the gravity direction, ground normal, and equipment installation attitude reference direction, and is used to improve the stability of joint calibration.

[0050] Then through and K Point clouds can be projected onto the image plane, providing a mapping basis for subsequent ROI selection and node point cloud extraction, such as... Figure 4 As shown.

[0051] 3) Image processing and node ROI detection like Figure 2 As shown, this embodiment performs necessary projection segmentation and geometric correction on the panoramic image to obtain a multi-directional view or an equivalent distortion-free view for node detection. The image is input into the node small target detection network, and the output is the region of interest (ROI) of each node, such as... Figure 4As shown, the detection network uses TPH-YOLOv5, and a Transformer attention mechanism is introduced into the prediction head to enhance the recognition ability of small targets in the cable net nodes.

[0052] ROI is represented as a pixel rectangle:

[0053] in, For the first i The rectangular region of interest for each node in the image; and These are the minimum and maximum values ​​of the horizontal pixel coordinates of the region, respectively. and These are the minimum and maximum values ​​of the vertical pixel coordinates, respectively.

[0054] 4) ROI—Point Cloud Mapping and Node Point Cloud Extraction like Figure 4 and Figure 8 As shown, based on and K Projecting the point cloud onto the pixel plane yields each point in the point cloud. Predicted pixel coordinates For each The set of projection points falling into the ROI is filtered and back-linked to the 3D point cloud to form a node point cloud:

[0055] in, For the first i A set of node point clouds obtained by reverse extraction of each ROI; For the first laser scanner coordinate system j Point cloud points.

[0056] In addition, it is possible to Density threshold filtering or neighborhood consistency filtering can be performed to suppress noise points at the ROI boundary and improve the purity of the node point cloud.

[0057] 5) Node aliasing detection and cluster separation like Figure 5 As shown, when a single ROI contains multiple node point cloud projections (node ​​aliasing), this embodiment uses density peak clustering to determine the number of nodes and complete the separation.

[0058] First, calculate the distance between the points in the point cloud. Select cutoff distance And define the local density:

[0059] in, χ (·) is a truncation kernel function or indicator function.

[0060]

[0061] in, χ ( x ) is a binary indicator function; x for .

[0062] Define relative distance and discriminant:

[0063] Where, δ i For the first i The minimum distance between a point and all points whose local density is higher than its own. d ij For the first i The point and the first j The distance between points; I s For the current point i Constructed high-density candidate point index set; m i n represents the minimum value within the set.

[0064]

[0065] in, For local density higher than the first i The set of indices of all points; j Candidate point number; and The first j The point and the first i Local density at each point.

[0066]

[0067] in, For the first i Cluster center discriminant of each point. The larger the value, the higher the local density and the greater the relative distance, making it more likely to become a cluster center.

[0068] according to Sort the nodes in descending order and use a threshold to determine the number of nodes:

[0069] in, This represents the number of nodes determined within the current ROI. and They are respectively The maximum and second largest values ​​after sorting in descending order; For preset or adaptive discrimination thresholds. When If it is true, it is determined to be 1 node; otherwise, it is determined to be 2 nodes. i "se" indicates a situation where the aforementioned conditions are not met.

[0070] when When the value is (or larger), clustering is performed on the point cloud to obtain multiple node point cloud clusters.

[0071] 6) Optimized positioning of node center like Figure 6 As shown, by projecting each node point cloud cluster onto a two-dimensional plane, n linear constraints are obtained through fitting:

[0072] in, , and For the first k The coefficients of the straight lines; x and y Let be the coordinates of any point within the two-dimensional projection plane; k The line number; n The total number of fitted lines participating in the center optimization.

[0073] Construct the objective function and find its extreme value to obtain the center point:

[0074] in, For candidate center points to n The objective function is the sum of squared distances between the fitted lines; and Let be the two-dimensional center coordinates of the node to be determined.

[0075]

[0076]

[0077] Its vectorized equivalent form is:

[0078]

[0079] After obtaining the two-dimensional center, the three-dimensional center coordinates of the node are recovered by combining the three-dimensional information of the point cloud cluster. P i(t) This forms a set of three-dimensional coordinates of the nodes for displacement calculation.

[0080] 7) Displacement calculation and result output like Figure 7As shown, the three-dimensional center coordinates of nodes with the same name at different times are... P i(t) Compare the results, calculate the displacement vectors, and output the displacement results:

[0081] The system outputs the values ​​of each node. X, Y, Z It can generate displacement sequences and displacement-time curves in different directions, and export monitoring reports or data files for dynamic displacement monitoring and analysis during structural construction and operation.

[0082] In summary, this invention provides an automatic dynamic displacement monitoring method for grid or cable net structures based on panoramic cameras and 3D laser scanning. First, a panoramic image sequence is acquired to obtain point clouds. Then, the two images are aligned in time or stage, and calibration constraints are established using structural and environmental features. The extrinsic rotation matrix and translation vector between the camera and laser scanner are solved using at least four structural nodes with the same name in the image and point cloud, establishing a point cloud-to-pixel coordinate mapping. Next, a deep learning model is used to detect nodes with regions of interest (ROIs) in the image, and the point cloud is projected onto the image plane. Points falling into ROIs are selected and back-associated to obtain the node point cloud. Then, the node point cloud is subjected to node number discrimination and clustering separation, and the 3D center of the nodes is calculated through minimum distance optimization. The displacement of the nodes is obtained by comparing the 3D coordinates at different times and outputting the results. This invention achieves stable mapping between the image and point cloud in a unified coordinate system through a joint calibration mechanism of "structural-environmental feature constraints," thereby realizing 3D node localization, node aliasing discrimination and center optimization, and automatic calculation and output of multi-node displacement.

[0083] Example 2 This embodiment provides a monitoring system for the hoisting process of a large-scale space frame structure based on panoramic cameras and laser scanning, including: The system includes a panoramic camera, a 3D laser scanner, a joint calibration module, an image processing and node small target detection module, a point cloud mapping and node extraction module, a node classification and center optimization module, and a displacement calculation and output module.

[0084] The system comprises: a panoramic camera for acquiring panoramic image sequences of a wire mesh or cable net structure; a 3D laser scanner for acquiring 3D point clouds of the wire mesh or cable net structure; a joint calibration module for establishing constraints by combining structural and environmental features, and aligning the coordinate systems and data of the panoramic camera and the 3D laser scanner; an image processing and small target detection module for temporally aligning the panoramic image sequence and point cloud to form paired data; acquiring camera intrinsic distortion parameters and performing geometric correction on the panoramic image; inputting the corrected panoramic image into a deep learning node detection network to automatically identify node ROI regions from the panoramic image based on deep learning; a point cloud mapping and node extraction module for mapping ROIs to the point cloud under extrinsic parameter mapping relationships and extracting node point clouds; a node classification and center optimization module for determining the number of nodes based on density peak clustering and separating the node point clouds, and calculating the precise 3D coordinates of the nodes using a minimum distance center optimization algorithm; and a displacement calculation and output module for comparing the 3D coordinates of nodes at different times and outputting the displacement results.

[0085] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0086] The above are merely preferred embodiments of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should be considered within the scope of protection of the present invention.

Claims

1. A method for monitoring the hoisting process of large structures based on panoramic cameras and laser scanning, characterized in that, Includes the following steps: S1: A panoramic image sequence of the cable net structure is acquired using a panoramic camera, and a point cloud is obtained using a 3D laser scanner; the panoramic image sequence and the point cloud are temporally aligned and paired data is formed; the in-camera distortion parameters are obtained and the panoramic images are geometrically corrected. S2: Establish calibration constraints by combining the characteristics of the cable net structure and the characteristics of the field environment; obtain corresponding observations of the same node in the panoramic image and the point cloud respectively, and solve the external parameter transformation between the camera and the laser scanner to establish the mapping relationship from the point cloud to the pixel coordinates of the panoramic image; S3: Input the calibrated panoramic image into the deep learning node detection network; Output the region of interest (ROI) of the node; S4: Project the point cloud onto the panoramic image plane based on extrinsic parameter transformation and camera intrinsic parameters; extract the set of projection points falling within the ROI range, and back-associate the projection point set with the 3D space to form a node point cloud; output the node point cloud corresponding to each ROI; S5: Determine the number of nodes in the point cloud corresponding to the ROI, and separate the point cloud in the case of multiple nodes; perform center optimization on the separated point cloud to obtain the three-dimensional coordinates of the nodes and form a set of three-dimensional coordinates of the nodes. S6: Repeat S1 to S5 at different times and establish the temporal association of the same node; compare the three-dimensional coordinates of the same node at different times, calculate the displacement, and output the dynamic displacement monitoring results of the node.

2. The method for monitoring the hoisting process of large structures based on panoramic cameras and laser scanning according to claim 1, characterized in that, In step S1: The camera intrinsic parameters form a camera intrinsic parameter matrix. K Its formula is: ； in, and These are the equivalent focal lengths in pixels along the horizontal and vertical directions of the image, respectively. and These are the horizontal and vertical pixel coordinates of the camera principal point in the image coordinate system; 0 and 1 are fixed elements of the intrinsic parameter matrix.

3. The method for monitoring the hoisting process of large structures based on panoramic cameras and laser scanning according to claim 1, characterized in that, In step S2, the specific steps of solving the extrinsic parameter transformation between the camera and the laser scanner include: The extrinsic parameter transformation is represented using a homogeneous matrix: ； in, For the coordinate system of the laser scanner l To the camera coordinate system c The 4×4 homogeneous transformation matrix; This is a 3×3 rotation matrix that rotates the laser scanner coordinate system to the camera coordinate system. For the corresponding 3×1 translation vector; subscript c and l These represent the camera coordinate system and the laser scanner coordinate system, respectively; the last row of the matrix [0, 0, 0, 1] is used for homogeneous coordinate representation; The point cloud coordinates to camera coordinates satisfy: ； in, These represent the coordinates of the point cloud in three directions; , and The three-dimensional coordinates of the point in the camera coordinate system; The camera coordinates to pixel coordinates satisfy: ； in, u and v These are the horizontal and vertical pixel coordinates of the projection point in the image, respectively; K This is the camera intrinsic parameter matrix.

4. The method for monitoring the hoisting process of large structures based on panoramic cameras and laser scanning according to claim 3, characterized in that, In step S2, the mapping relationship from point cloud to panoramic image pixel coordinates is equivalently represented as: ； ； in, For the first i Each node obtains its predicted pixel coordinates based on the current calibration parameters; Π(·) For perspective normalization projection operators; I It is a 3×3 identity matrix. Used to extract 3D camera coordinates from homogeneous coordinates; This is the homogeneous transformation matrix from the laser scanner coordinate system to the camera coordinate system; For the first i Homogeneous 3D coordinates of each node; , and These are its three coordinate components.

5. The method for monitoring the hoisting process of large structures based on panoramic cameras and laser scanning according to claim 4, characterized in that, In step S2, acquiring the corresponding observations of the same node in both the panoramic image and the point cloud specifically involves: At least four corresponding observations of the same cable net structure feature node are obtained from both the image and the point cloud, forming an image observation set. and point cloud observation set ,in: in, For the first i Two-dimensional pixel coordinate vectors of nodes; and The first i The horizontal and vertical pixel coordinates of each node; i For node sequence number, i =1, 2, 3, 4; superscript T Indicates the transpose of a matrix or vector; in, For the first i The three-dimensional coordinate vectors of each node in the laser scanner coordinate system; 、 and These are its three coordinate components; i Node number; superscript T This represents the transpose of a vector.

6. The method for monitoring the hoisting process of large structures based on panoramic cameras and laser scanning according to claim 5, characterized in that, In step S2, the specific steps of solving the extrinsic parameter transformation between the camera and the laser scanner include: Based on the initial values ​​of the extrinsic parameters, structural constraints and environmental constraints are introduced for joint optimization, resulting in an optimized extrinsic parameter matrix that satisfies the following equation: ； in, This is the optimal extrinsic transformation matrix obtained after joint optimization; SE(3) Represents a three-dimensional rigid body transformation group; ρ (·) is a robust loss function used to reduce the impact of outlier matching points; , and These are the weighting coefficients for each constraint term; The constraint terms for the node point cloud to satisfy the main geometric features of the cable net structure include at least one of the cable net principal plane, local fitting surface and circumferential principal curve; For structural dimensional constraints, the constraints include at least one of node spacing, node radial distance, and design dimension range; This is an environmental direction constraint, which includes at least one of the following: gravity direction, ground normal, and equipment installation attitude reference direction.

7. The method for monitoring the hoisting process of large structures based on panoramic cameras and laser scanning according to claim 1, characterized in that, In step S3, the deep learning node detection network adopts the TPH-YOLOv5 model and introduces the Transformer attention mechanism in the prediction head.

8. The method for monitoring the hoisting process of large structures based on panoramic cameras and laser scanning according to claim 1, characterized in that, In step S5, the step of determining the number of nodes in the point cloud corresponding to the ROI and performing point cloud separation in the case of multiple nodes specifically includes: First, define the local density: in, For local density; χ (·) is a truncation kernel function or indicator function; Calculate the distance between points in the point cloud; To cut off the distance; in, x for ; The relative distance and the discriminant satisfy the following: in, δ i For the first i The minimum distance between a point and all points whose local density is higher than its own. d ij For the first i The point and the first j The distance between points; I s m is a high-density candidate point index set constructed for the current point; i n represents the minimum value within the set; Calculate the first i Cluster center discriminant of each point : Finally, Sort the nodes in descending order and use a threshold to determine the number of nodes within the ROI region: in, For the current RO I The number of nodes determined internally; and They are respectively The maximum and second largest values ​​after sorting in descending order; For preset or adaptive discrimination thresholds; others i se indicates a situation where the aforementioned conditions are not met; when When the value is greater than or equal to 2, the point cloud is split to obtain multiple node point clouds.

9. A method for monitoring the hoisting process of large structures based on panoramic cameras and laser scanning according to claim 1, characterized in that, In step S5, the specific steps of performing center optimization on the separated node point cloud include: First, the separated node point cloud is projected in two dimensions and fitted to obtain multiple linear constraints: in, , and For the first k The coefficients of the straight lines; x and y Let be the coordinates of any point within the two-dimensional projection plane; k The line number; n The total number of fitted lines participating in the center optimization; Next, construct the objective function: in, For candidate center points to n The objective function is the sum of squared distances between the fitted lines; and Here are the two-dimensional center coordinates of the node to be determined; Then by solving and Obtain the center point ; The vectorized equivalent form of the objective function for: in, .

10. A monitoring system for the hoisting process of large structures based on panoramic cameras and laser scanning, characterized in that, include: A panoramic camera used to acquire panoramic image sequences of cable net structures; A 3D laser scanner is used to acquire 3D point clouds of cable net structures. The joint calibration module is used to establish calibration constraints by combining the characteristics of the cable net structure with the characteristics of the field environment; it acquires corresponding observations of the same node in the panoramic image and the point cloud respectively, and solves the extrinsic parameter transformation between the camera and the laser scanner to establish the mapping relationship from the point cloud to the pixel coordinates of the panoramic image. The image processing and small object detection module is used to temporally align panoramic image sequences and point clouds to form paired data; acquire in-camera distortion parameters and perform geometric correction on panoramic images; input the corrected panoramic images into a deep learning node detection network; and output the region of interest (ROI) of the nodes. The point cloud mapping and node extraction module is used to project the point cloud onto the panoramic image plane based on extrinsic parameter transformation and camera intrinsic parameters; extract the set of projection points falling within the ROI range, and back-associate the set of projection points back into the three-dimensional space to form a node point cloud; output the node point cloud corresponding to each ROI; The node classification and center optimization module is used to determine the number of nodes in the point cloud corresponding to the ROI and to separate the point cloud in the case of multiple nodes; the center optimization solution is performed on the separated point cloud to obtain the three-dimensional coordinates of the nodes and form a set of three-dimensional coordinates of the nodes. The displacement calculation and output module is used to establish the temporal correlation of the same node; compare the three-dimensional coordinates of the same node at different times, calculate the displacement, and output the dynamic displacement monitoring results of the node.

Images