Roadway deformation intelligent monitoring and evaluation system

Abstract

The invention relates to a roadway deformation intelligent monitoring and evaluation system, and belongs to the technical field of roadway deformation monitoring. Comprises: a data acquisition and processing module for denoising original point cloud data and image data to obtain point cloud data; the coarse registration module is used for performing coarse registration on the point cloud data to obtain coarse registration point cloud data; the feature extraction module is used for acquiring point cloud geometric features of each effective point and image features of each effective projection point corresponding to the point cloud geometric features, and fusing the point cloud geometric features to obtain fusion features of each effective point; the displacement field acquisition module is used for generating a final displacement field of each point in the coarse registration point cloud data; the slicing module is used for performing sliding slicing on the standard point cloud data to obtain a plurality of continuous section point clouds; the label configuration module is used for dividing a section point cloud area and configuring key area labels; the risk index calculation module is used for calculating the risk index of each section point cloud; and the deformation evaluation module is used for determining a roadway deformation evaluation result of each section point cloud. And the comprehensiveness and the accuracy of roadway deformation analysis are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of roadway deformation monitoring, and in particular to a roadway deformation intelligent monitoring and evaluation system. BACKGROUND

[0002] In underground operation scenes such as coal mining, the roadway is a key channel for underground production transportation and personnel passage, and its structural stability is a core prerequisite for ensuring operation safety and production continuity. With the continuous increase of mining depth, the effect of ground stress is increasingly significant, and the roadway deformation risk continues to rise under the influence of mining, so higher requirements are put forward for the accuracy and reliability of the monitoring technology. At present, the roadway deformation monitoring has gradually shifted to the technical path of three-dimensional laser point cloud and optical image fusion, but in the complex underground environment, there are still many problems to be solved in the existing technology.

[0003] For example: in the multi-source data fusion link, the existing technology cannot guarantee the spatio-temporal consistency of three-dimensional laser point cloud and optical image, the spatial coordinate mapping precision is insufficient, which leads to unreliable multi-source data correlation basis, and affects the accuracy of subsequent roadway deformation analysis. In the deformation registration link, the existing algorithm is limited by the rigid transformation assumption, and cannot adapt to the non-rigid deformation actually existing in the roadway, the registration error is large, and even false deformation results are produced, which is difficult to accurately reflect the real deformation state of the roadway. In the feature extraction link, the feature extraction technology has limitations, and the extracted features cannot completely represent the complex deformation of the roadway, affecting the comprehensiveness and accuracy of the roadway deformation analysis. SUMMARY

[0004] To solve the above technical problems, the present application provides a roadway deformation intelligent monitoring and evaluation system. The technical scheme of the present application is as follows: The roadway deformation intelligent monitoring and evaluation system comprises: A data acquisition and processing module is used to acquire original point cloud data and image data of the roadway, and to denoise the original point cloud data by improving a bilateral filtering algorithm to obtain point cloud data. A coarse registration module is used to construct a mapping relationship between the point cloud data and the image data, and to perform coarse registration on the point cloud data according to reference point cloud data and the mapping relationship to obtain coarse registration point cloud data. A feature extraction module is used to project all points in the coarse registration point cloud data, and to determine a plurality of effective points in the coarse registration point cloud data and a plurality of effective projection points corresponding to the plurality of effective points in an image plane of the image data according to the projection result, to acquire point cloud geometric features of each effective point and image features of each effective projection point corresponding thereto, and to fuse the point cloud geometric features of each effective point and the image features of the effective projection point corresponding thereto to obtain fusion features of each effective point. The displacement field acquisition module is configured to calculate an initial displacement field of each point in the coarse registration point cloud data according to the fusion feature of each effective point, and globally coarsely adjust and locally moderately adjust the initial displacement field of all points according to the reference point cloud data, so as to obtain a final displacement field of each point in the coarse registration point cloud data. The slicing module is configured to generate standard point cloud data according to the final displacement field of each point in the coarse registration point cloud data, and perform sliding slicing on the standard point cloud data according to the heading of the roadway, so as to obtain a plurality of continuous section point clouds, wherein each section point cloud is composed of a plurality of standard points. The label configuration module is configured to divide the standard points in each section point cloud into regions, and configure a key region label for each standard point in each section point cloud. The risk index calculation module is configured to calculate a risk index of each section point cloud according to the key region label of each standard point in each section point cloud. The deformation evaluation module is configured to determine a roadway deformation evaluation result of each section point cloud according to the risk index of each section point cloud.

[0005] Preferably, the data acquisition and processing module comprises: The data acquisition unit is configured to synchronously acquire original point cloud data and image data of the roadway by using a high-precision synchronous trigger laser radar and a camera. The denoising unit is configured to distinguish between noise points and real points in the original point cloud data by using an improved bilateral filtering algorithm, and remove the noise points in the original point cloud data, so as to obtain point cloud data.

[0006] Preferably, the coarse registration module comprises: The mapping relationship construction unit is configured to establish a mapping relationship between the point cloud data and the image data according to the camera information and the laser radar information. The alignment unit is configured to align the point cloud data and the image data according to the mapping relationship, and eliminate error points in the point cloud data and the image data from the point cloud data, so as to obtain first point cloud data. The pose calculation unit is configured to extract a plurality of image key points from the image data, match the plurality of image key points with reference key points in reference image data, and calculate an initial pose transformation matrix of the image data relative to the reference image data based on a matching result. The registration unit is configured to perform nearest point registration on the first point cloud data and the reference point cloud data according to the initial pose transformation matrix, and iteratively adjust the initial pose transformation matrix based on an objective function in the nearest point registration process, so as to obtain an optimal pose transformation matrix, and determine the coarse registration point cloud data according to the optimal pose transformation matrix and the first point cloud data.

[0007] Preferably, the mapping relationship construction unit is configured to establish a mapping relationship between any target point in the point cloud data and any target point in the image data according to the camera information and the laser radar information. with the mapping relationship between the corresponding pixel points in the image data The mapping relationship between the corresponding pixel points in the image data (1); (2); In formula (1), is an external parameter matrix of the camera, denotes a rotation matrix in the external parameter matrix of the camera, denotes a translation vector in the external parameter matrix of the camera; denotes an intrinsic matrix of the camera, denotes a depth value of the target point in the camera coordinate system; In formula (2), denotes a rotation matrix between the camera and the lidar, denotes a translation vector between the camera and the lidar, denotes the coordinates of the point of the target point in the camera coordinate system after coordinate conversion.

[0008] Preferably, the feature extraction module comprises: a projection unit configured to project all points in the coarse registration point cloud data through camera information to obtain a plurality of projection points located in an image plane of the image data; a screening unit configured to screen a plurality of valid projection points from the plurality of projection points according to a mask, and determine a corresponding valid point of each valid projection point in the coarse registration point cloud data according to a projection relationship; an image feature extraction unit configured to extract an image feature of each valid projection point through a feature extraction convolutional neural network; a point cloud geometric feature extraction unit configured to screen a plurality of sampling points from the plurality of valid points through sampling processing, extract a point cloud geometric feature of each sampling point through a point cloud simplification processing algorithm, and finally extend the point cloud geometric features with the same number of sampling points to the same number of valid points through interpolation to obtain a point cloud geometric feature of each valid point; a feature fusion unit configured to concatenate the point cloud geometric feature of each valid point and the image feature of each valid projection point in a channel dimension to obtain an initial fusion feature, and reduce the dimension of each initial fusion feature based on a channel attention mechanism to obtain a fusion feature of each valid point.

[0009] Preferably, the displacement field acquisition module comprises: The initial displacement field acquisition unit is used to map the fusion features of each valid point through a multilayer perceptron to obtain the effective displacement field of each valid point. It performs interpolation processing on all other points in the coarse registration point cloud data except for the valid points to generate other displacement fields for each other point. The effective displacement fields of all valid points and the other displacement fields of all other points are used as the initial displacement field of each point in the coarse registration point cloud data. The global coarse adjustment unit is used to calculate the optimal pose information between the reference point cloud data and the coarse registration point cloud data through the iterative nearest point algorithm, and to superimpose the optimal pose information onto the initial displacement field of each point to obtain the coarse adjustment displacement field of each point in the coarse registration point cloud data. The local adjustment unit is used to calculate the residual between the coarse adjustment displacement field of each point in the coarse registration point cloud data and the average coarse adjustment displacement field of all its neighboring points. If the absolute value of the residual between the coarse adjustment displacement field of any point and the average coarse adjustment displacement field of all its neighboring points is greater than a preset threshold, the coarse adjustment displacement field of that point is superimposed with a preset adjustment amount and the iteration continues until a preset iteration termination condition is met to obtain the final displacement field of that point. The preset adjustment amount is the product of the residual of each point and the preset residual weight. Otherwise, the coarse adjustment displacement field of that point is taken as its final displacement field.

[0010] Preferably, the slicing module includes: The standard point cloud data generation unit is used to superimpose the coordinates of each point in the coarse registration point cloud data with the corresponding final displacement field to obtain each standard point, and combine all standard points to obtain standard point cloud data. The slicing unit is used to slide slice the standard point cloud data along the roadway direction according to the preset slice length and sliding step size to obtain multiple continuous cross-sectional point clouds.

[0011] Preferably, the label configuration module includes: The center determination unit is used to calculate the center coordinates of the center of each cross-section point cloud in the lidar coordinate system according to the direction of the roadway, and to calculate the polar angle of the center coordinates of each cross-section point cloud and each standard point, so as to obtain the polar angle of each standard point in each cross-section point cloud. The label configuration unit is used to configure key region labels for each standard point according to the polar angle of each standard point in each cross-sectional point cloud. The types of key region labels include top plate, bottom plate, left side and right side.

[0012] Preferably, the center determination unit calculates the center coordinates of the center of any cross-sectional point cloud in the lidar coordinate system. With any of its standard points polar angle This can be achieved through formulas (3) and (4): (3); (4); In formula (3), denotes the center of the section point cloud in the coordinate of the X axis, denotes the center of the section point cloud in the coordinate of the Z axis, denotes the number of standard points in the section point cloud, denotes the sum of the coordinates of all standard points in the section point cloud in the X axis, denotes the sum of the coordinates of all standard points in the section point cloud in the Z axis, wherein the origin of the laser radar coordinate system is the emission center of the laser radar, the X axis represents the direction directly above the roadway, the Y axis represents the direction along the roadway, and the Z axis represents the direction perpendicular to the X axis and the Y axis; In formula (4), denotes the standard point in the coordinate of the X axis, denotes the standard point in the coordinate of the Z axis, denotes the two-parameter arctangent function.

[0013] Preferably, the risk indicators include convergence, roof subsidence, and twist angle, and the risk indicator calculation module includes: a convergence calculation unit, configured to obtain standard points in each section point cloud with a key area label of left and right sides, take the coordinate of the leftmost standard point with a key area label of the left side in the X axis as the left side boundary of each section point cloud, take the coordinate of the rightmost standard point with a key area label of the right side in the X axis as the right side boundary of each section point cloud, take the absolute value of the difference between the left side boundary and the right side boundary of each section point cloud as the section width of each section point cloud, and take the absolute value of the difference between the section width of each section point cloud and the section width of the corresponding section point cloud in the reference point cloud data as the convergence of each section point cloud; a roof subsidence calculation unit, configured to take the mean value of the coordinates of all standard points with a key area label of the roof in each section point cloud in the Z axis and the absolute value of the difference between the preset roof height of the corresponding section point cloud in the reference point cloud data as the roof subsidence of each section point cloud; a twist angle calculation unit, configured to perform plane fitting on all standard points in each section point cloud to obtain a plane fitting equation of each section point cloud, obtain a normal vector of the plane fitting equation, and take the included angle between the normal vector and the horizontal plane as the twist angle of each section point cloud.

[0014] All the optional technical solutions described above can be combined in any manner, and the application does not describe the structures after combination in detail.

[0015] By means of the above-mentioned solutions, the application has the following beneficial effects: The original point cloud data and image data of the roadway are synchronously collected by the data acquisition and processing module, and the point cloud data is obtained after denoising the original point cloud data, and the mapping relationship between the point cloud data and the image data is constructed by the coarse registration module, and the point cloud data is coarsely registered by combining the reference point cloud data, to obtain coarse registration point cloud data, by synchronously collecting the advantages of combining the point cloud data and the image data, the space-time consistency problem of the three-dimensional laser point cloud and the optical image is solved, and the quality of the roadway point cloud data is effectively improved by denoising and coarse registration, thereby providing an accurate data basis for subsequent roadway deformation analysis.

[0016] The feature extraction module screens a plurality of effective points and a plurality of effective projection points corresponding to the plurality of effective points in an image plane of the image data from the coarse registration point cloud data, and obtains the point cloud geometric features of each effective point and the image features of each effective projection point corresponding to each effective point, and then fuses to obtain the fusion features of each effective point, so that the combination of the point cloud geometric features and the image features is realized, and accurate and multi-source fusion features are provided for subsequent roadway deformation analysis.

[0017] The displacement field acquisition module calculates an initial displacement field according to the fusion features of each effective point, and globally coarsely adjusts and locally finely adjusts all the initial displacement fields to obtain a final displacement field, which can significantly improve the registration accuracy of the coarse registration point cloud data and the reference point cloud data, and provide a high-quality registration basis for subsequent roadway deformation analysis.

[0018] The slice module generates standard point cloud data according to the final displacement field of each point in the coarse registration point cloud data, and performs sliding slicing on the standard point cloud data according to the trend of the roadway to obtain a plurality of continuous cross-section point clouds, and the label configuration module configures a key area label for each standard point of each cross-section point cloud, and the risk index calculation module calculates a risk index of each cross-section point cloud according to the key area label of each standard point. Since the cross-section point cloud after slicing has both rigid deformation and non-rigid deformation, the present application can accurately analyze the non-rigid deformation of the roadway, solve the problem of large registration error caused by non-rigid deformation of the roadway in the prior art, and improve the comprehensiveness and accuracy of the roadway deformation analysis.

[0019] The above description is only a summary of the technical scheme of the present application, in order to more clearly understand the technical means of the present application, and to implement the content of the description, the following will describe the preferred embodiments of the present application in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1 FIG. 1 is a structural schematic diagram of a roadway deformation intelligent monitoring and evaluation system provided by an embodiment of the present application. DETAILED DESCRIPTION

[0021] The specific embodiments of the present application are described in further detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present application, but are not used to limit the scope of the present application.

[0022] As Figure 1 shown, the embodiment of the present application provides a roadway deformation intelligent monitoring and evaluation system, comprising the following modules: A data acquisition and processing module is configured to acquire original point cloud data and image data of the roadway, and to denoise the original point cloud data by improving a bilateral filtering algorithm to obtain point cloud data. A coarse registration module is configured to construct a mapping relationship between the point cloud data and the image data, and to perform coarse registration on the point cloud data according to reference point cloud data and the mapping relationship to obtain coarse registration point cloud data. A feature extraction module is configured to project all points in the coarse registration point cloud data, and to determine a plurality of effective points in the coarse registration point cloud data and a plurality of effective projection points corresponding to the plurality of effective points in an image plane of the image data according to a projection result, to acquire a point cloud geometric feature of each effective point and an image feature of each effective projection point corresponding to the effective point, and to fuse the point cloud geometric feature of each effective point and the image feature of the effective projection point corresponding to the effective point to obtain a fusion feature of each effective point. A displacement field acquisition module is configured to calculate an initial displacement field of each point in the coarse registration point cloud data according to the fusion feature of each effective point, and to perform global coarse adjustment and local intermediate adjustment on the initial displacement field of all points according to the reference point cloud data to obtain a final displacement field of each point in the coarse registration point cloud data. A slicing module is configured to generate standard point cloud data according to the final displacement field of each point in the coarse registration point cloud data, and to perform sliding slicing on the standard point cloud data according to a roadway trend to obtain a plurality of continuous cross-section point clouds, wherein each cross-section point cloud is composed of a plurality of standard points. A label configuration module is configured to divide regions of the standard points in each cross-section point cloud, and to configure a key region label for each standard point in each cross-section point cloud. A risk index calculation module is configured to calculate a risk index of each cross-section point cloud according to the key region label of each standard point in each cross-section point cloud. A deformation evaluation module is configured to determine a roadway deformation evaluation result of each cross-section point cloud according to the risk index of each cross-section point cloud.

[0023] Specifically, in the data acquisition and processing module, the original point cloud data and the image data of the roadway are synchronously collected by a precise mechanical structure fixed laser radar and a camera, and a high-precision synchronous trigger device is used when collecting the original point cloud data and the image data to ensure the consistency of the two kinds of data in time.

[0024] In the coarse registration module, the mapping relationship refers to the coordinate transformation relationship between the point cloud data and the image data. The coarse registration is to eliminate the pose error between the point cloud data and the reference point cloud data, so as to facilitate subsequent analysis of the deformation of the roadway according to the deformation difference of the point cloud data of the roadway at different times. The reference point cloud data refers to the point cloud data collected by the laser radar before the roadway is mined and obtained after coarse registration. For example, if the laser radar collects first initial point cloud data on the first day of the roadway mining and collects second initial point cloud data on the second day before the roadway is mined, the first initial point cloud data and the second initial point cloud data are coarsely registered, and the first point cloud data after coarse registration is taken as the reference point cloud data. By coarsely registering the first initial point cloud data and the second initial point cloud data, the error caused by the angle change of the laser radar can be eliminated.

[0025] In the feature extraction module, the effective point refers to a point in the coarse registration point cloud data that has a good projection relationship with the image data after screening. The effective projection point is a point corresponding to the effective point projected onto the image plane where the image data is located according to the projection relationship. The projection relationship refers to a geometric correspondence relationship established between the point cloud data and the image data, which is determined by the camera information in the embodiment of the application. The image feature is used to describe the visual features of each effective projection point in the image data, including color, texture, shape, edge and corner, etc. In the embodiment of the application, the image feature is 256-dimensional. The point cloud geometric feature is a numerical feature describing the spatial position, shape and structural characteristics of each effective point in the coarse registration point cloud data, including coordinates, normal vectors, curvatures and densities, etc. In the embodiment of the application, the point cloud geometric feature is 128-dimensional. The fusion feature is a multi-modal comprehensive feature representation of each effective point formed by combining the point cloud geometric feature of each effective point in the coarse registration point cloud data with the image feature of the corresponding effective projection point. In the embodiment of the application, the fusion feature is 128-dimensional.

[0026] In the displacement field acquisition module, the initial displacement field refers to an initial displacement estimation vector of the coarse registration point cloud data relative to the reference point cloud data calculated according to the fusion feature of each effective point. The global coarse adjustment is to adjust the coarse registration point cloud data globally with the reference point cloud data as a reference, so as to eliminate the registration error of the initial displacement field in the global range of the coarse registration point cloud data. The local fine adjustment is to adjust the initial displacement field of each point in the coarse registration point cloud data after the global coarse adjustment and its neighborhood points, so as to obtain the final displacement field of each point. The final displacement field is a displacement vector of the coarse registration point data relative to the reference point cloud data. By adjusting the coordinates of each point in the coarse registration point data according to the final displacement field, the standard point cloud data is obtained.

[0027] In the slicing module, sliding slicing refers to generating multiple continuous two-dimensional slices (i.e., cross-sectional point clouds) in standard point cloud data by sliding a window according to a certain step size and direction. The step size is generally 1m, and the direction is the tunnel direction.

[0028] In the label configuration module, the types of labels for key areas include top plate, bottom plate, left side and right side.

[0029] In the risk indicator calculation module, the risk indicator is an indicator that quantifies the degree of deformation of the roadway structure, including convergence, roof subsidence, and torsion angle.

[0030] In the deformation evaluation module, the roadway deformation evaluation result is the result obtained after comprehensive analysis and evaluation of the roadway deformation. For example, the risk level of the point cloud at the 10th cross section is Level II, the evaluation result is that the deformation has intensified, the observation frequency should be increased to once a day, and the focus should be on checking the integrity of the support.

[0031] In one specific embodiment, the data acquisition and processing module includes: The data acquisition unit is used to simultaneously acquire raw point cloud data and image data of the tunnel using a high-precision synchronously triggered lidar and camera. The denoising unit is used to distinguish between noise and real points in the original point cloud data by improving the bilateral filtering algorithm, and to remove noise from the original point cloud data to obtain the point cloud data.

[0032] Specifically, in the data acquisition unit, the mechanical structure that fixes the lidar and camera for synchronous triggering typically triggers once a day. Spatially, a precise mechanical structure is used to fix the lidar and camera, ensuring their relative positions are stable and providing a good foundation for subsequent spatiotemporal calibration.

[0033] When processing the original point cloud data using an improved bilateral filtering algorithm, the denoising unit first calculates the weight value of the difference in reflection intensity between each point in the original point cloud data and all points in its neighborhood. Specifically, in calculating the weight value of the first point in the original point cloud data... i The weighted value of the difference in reflection intensity between a point and all points in its neighborhood is achieved by formula (5): (5); In formula (5), This represents the first point in the original point cloud data. i The weighted value of the difference in reflection intensity between a point and all points in its neighborhood; This represents the first point in the original point cloud data. i One point, This indicates the first point in the original point cloud data. i Within the neighborhood of point 1 j One point; Indicates the firsti the spatial distance between the point and the j point in its neighborhood; denotes the distance weight kernel, and satisfies ; wherein, denotes the distance coefficient, which is adaptively adjusted according to the density of the original point cloud data, and when the density of the original point cloud data is large, a smaller value is taken, and vice versa a larger value is taken, generally in the range of 0.01-0.1; denotes the intensity weight kernel, which is a weight value assigned to all points in the neighborhood of each point in the original point cloud data according to the difference in reflection intensity of each point in the original point cloud data, and the intensity weight kernel adopts a Gaussian function, and the formula is: , wherein, is the reflection intensity of the i point in the original point cloud data, is the reflection intensity of the i point in the neighborhood of the j point in the original point cloud data, and the reflection intensity is a parameter directly measured and recorded by the laser radar when collecting the original point cloud data, without additional calculation, and is one of the attributes of the original point cloud data; is the preset smoothing coefficient of the i point; the calculation of the normalization factor strictly follows , to ensure that the original point cloud data after filtering will not be shifted as a whole; denotes a set composed of all points in the neighborhood of the i point; denotes In the formula (5), the smaller the difference in reflection intensity of a point and all points in its neighborhood, the greater the weight value of the difference in reflection intensity of the point, so that the real point with a smaller weight value of the difference in reflection intensity of the neighborhood point is better preserved during filtering, while the noise point with a larger weight value of the difference in reflection intensity of the neighborhood point is suppressed.

[0034] Further, the denoising unit divides all points in the original point cloud data into real points and noise points according to a preset filtering weight threshold, specifically, points with a weight value of the difference in reflection intensity greater than the preset filtering weight threshold in the original point cloud data are classified as noise points and removed, wherein the preset filtering weight threshold is a historical experience value for distinguishing real points and noise points according to historical point cloud data.

[0035] In a specific embodiment, the coarse registration module comprises: a mapping relationship construction unit, configured to establish a mapping relationship between the point cloud data and the image data according to the camera information and the laser radar information; The alignment unit is configured to align the point cloud data with the image data according to the mapping relationship, and eliminate error points in the point cloud data from the point cloud data, to obtain first point cloud data. The pose calculation unit is configured to extract a plurality of image key points from the image data, match the plurality of image key points with reference key points in reference image data, and calculate an initial pose transformation matrix of the image data relative to the reference image data based on a matching result. The registration unit is configured to perform nearest point registration of the first point cloud data and the reference point cloud data according to the initial pose transformation matrix, and iteratively update the initial pose transformation matrix based on a target function during the nearest point registration, to obtain an optimal pose transformation matrix, and determine coarse registration point cloud data based on the optimal pose transformation matrix and the first point cloud data.

[0036] Specifically, in the mapping relationship construction unit, the camera information includes an external parameter matrix of the camera, an intrinsic parameter matrix of the camera, and the like. The lidar information includes pose information between the camera and the lidar.

[0037] When the alignment unit aligns the point cloud data with the image data according to the mapping relationship, each point in the point cloud data is first mapped to a mapping point in the image data according to the mapping relationship, each mapping point and a nearest point in the image data form a point pair, and then a pixel error of each point pair is calculated by using a least square method, and a corresponding point of a mapping point with an error greater than 0.5 pixels in the point cloud data is regarded as an error point and deleted.

[0038] In the pose calculation unit, the image key points are points that are significant and easy to identify in the image data, and the image key points are found in the image data by using a SIFT algorithm. When the image key points are found in the image data by using the SIFT algorithm, a multi-scale space extreme value detection method is used, and the method includes the following steps: a Gaussian pyramid is constructed on different scale spaces, each octave (a size unit of the Gaussian pyramid) contains five images of different scales, a difference Gaussian pyramid is obtained by performing a difference operation on the reference image data and the image data, local extreme points are detected on the difference Gaussian pyramid as initial image key points, and finally image key points with scale and rotation invariance in the image data are obtained through positioning, direction assignment and other steps. The reference image data is image data of a roadway before mining, and the reference point cloud data is collected at the same time. Any image key point can be represented as wherein, is a scale of the image key point; is a main direction of the image key point (reflecting a rotation angle of the feature), and represent a horizontal coordinate and a vertical coordinate of the image key point in the camera coordinate system, respectively.

[0039] When calculating the initial pose transformation matrix, the pose calculation unit obtains the transformation matrix between any pair of keypoints and uses this transformation matrix as the initial pose transformation matrix of the image data relative to the reference image data. Specifically, a pair of keypoints is defined as those that are present in both the reference and reference image data and have the highest similarity to their pixel values. The initial pose transformation matrix includes the rotation matrix and translation vector of the image data relative to the reference image data.

[0040] In the registration unit, nearest-point registration is achieved by iteratively minimizing the objective function, which can be expressed as: ;in, This represents the rotation matrix in the initial pose transformation matrix. This represents the translation vector in the initial pose transformation matrix. This represents any resource point in the first cloud data. This represents the reference point corresponding to the resource point in the reference point cloud data. In each iteration, the resource point closest to the resource point is selected to continue the iteration. The iteration process calculates the optimal rotation matrix and translation matrix by minimizing the objective function, and then continues to search for the reference point corresponding to the next resource point in the reference point cloud data until the number of iterations is completed and the optimal pose transformation matrix is ​​obtained. Generally, the number of iterations is set to 20-50 times to lay a good foundation for subsequent fine registration.

[0041] It should be noted that in this embodiment of the invention, the mapping between image data and point cloud data only addresses the correspondence between point cloud data and image data at the same time. That is, it binds point cloud data and image data at the same time. For example, for a pixel in the image data, its corresponding 3D coordinates in the point cloud data can be found. However, this only establishes the association rules between the two types of data and does not involve comparing data from different times. This is the basis for data fusion. Coarse registration, on the other hand, aligns and compares point cloud data from different times, such as aligning and comparing point cloud data from the first day and the second day. The purpose is to calculate the deformation difference between the two times, which is a prerequisite for deformation analysis.

[0042] In one specific embodiment, the mapping relationship construction unit establishes any target point in the point cloud data based on camera information and LiDAR information. Corresponding pixel in image data The mapping relationship between them is achieved through formulas (1) and (2): (1); (2); In formula (1), For the camera's extrinsic parameter matrix, R, which represents a rotation matrix in an external parameter matrix of the camera, t, which represents a translation vector in the external parameter matrix of the camera; K, which represents an intrinsic matrix of the camera, Z, which represents a depth value of the target point in the camera coordinate system; in formula (2), R, which represents a rotation matrix between the camera and the lidar, t, which represents a translation vector between the camera and the lidar, X, which represents a coordinate of a point of the target point in the camera coordinate system after coordinate conversion.

[0043] It should be noted that the intrinsic matrix of the camera needs to be obtained through a strict camera calibration process, and the intrinsic matrix of the camera is composed of intrinsic parameters such as focal length, principal point coordinates; a high-precision calibration board is used in the process of calibrating the intrinsic parameters, and the intrinsic parameters are calibrated more than 10 times repeatedly, and the average value of the intrinsic parameters in all calibration times is taken to reduce the error.

[0044] In one specific embodiment, the feature extraction module comprises: a projection unit configured to project all points in the coarse registration point cloud data through camera information to obtain a plurality of projection points located on an image plane of the image data; a screening unit configured to screen a plurality of valid projection points from the plurality of projection points according to a mask, and determine a corresponding valid point of each valid projection point in the coarse registration point cloud data according to a projection relationship; an image feature extraction unit configured to extract an image feature of each valid projection point through a feature extraction convolutional neural network; a point cloud geometric feature extraction unit configured to screen a plurality of sampling points from the plurality of valid points through sampling processing, extract a point cloud geometric feature of each sampling point through a point cloud simplification processing algorithm, and finally extend the point cloud geometric features with the same number of sampling points to the same number of valid points through interpolation to obtain a point cloud geometric feature of each valid point; a feature fusion unit configured to concatenate the point cloud geometric feature of each valid point and the image feature of each valid projection point in a channel dimension to obtain an initial fusion feature, and reduce the dimension of each initial fusion feature based on a channel attention mechanism to obtain a fusion feature of each valid point.

[0045] Specifically, in the projection unit, for any point in the coarse registration point cloud data, after the point is projected onto the image plane of the image data combined with the camera information, the pixel coordinates of the corresponding projection point are wherein: , ; wherein, and This refers to the camera's focal length (in pixels) on the horizontal and vertical axes, as shown in the camera information. and The principal point coordinates of the image data are inherent parameters in the camera information; this formula represents the projection relationship.

[0046] Projection points include valid and invalid projection points. The filtering unit filters invalid projection points from all projection points based on the mask. Specifically, the filtering unit judges any point... The formula for determining whether a point is an invalid projection point is: ;in, and These are the width and height of the image data, respectively. This condition is to exclude points that are close together. This represents the filtering result obtained by masking. The mask is a binary image of the same size as the image data, which filters all points projected from the coarse registration point cloud data onto the image plane containing the image data using two pixel values. After masking, the number of valid projected points retained is N' (N'≤N, where N is the number of points in the coarse registration point cloud data), where the valid projected points are important structural points of the roadway. The number of valid projected points is the same as the number of valid points. For example, a LiDAR can scan points in image blind spots that a camera cannot capture. The mask can filter and remove these points, leaving important structural points of the roadway and avoiding errors in subsequent cross-modal feature fusion. Suppose that the coarse registration point cloud data of a certain roadway includes 8000 points, but 500 of them correspond to image blind spots. After masking, these 500 points are further removed, and finally 7500 points are taken as valid points.

[0047] In the image feature extraction unit, the feature extraction convolutional neural network uses ResNet34 to extract features from effective projection points. The input of the feature extraction convolutional neural network is image data, and the output is a feature map. The ResNet34 input... Image data (3 for RGB channels) is compressed in stages using residual blocks. The four stages are as follows: Stage 1: The convolutional kernel has a stride of 2, and the output size is... The stride of the convolutional kernel in stage 2 is 2, and the output size is [missing value]. The stride of the stage 3 convolutional kernel is 2, and the output size is... The stride of the convolutional kernel in stage 4 is 2, and the output size is... After four downsampling steps with a step size of 2, the final output size is The feature map. Here and is the size of the image data after down-sampling, through which the data amount can be reduced while the key structure visual features of the image data are preserved. Further, since the size of the feature map is 1 / 16 of the image data, any valid projection point needs to be scaled to the size of the feature map, specifically, the pixel coordinates of the valid projection point U , V are scaled to the size of the feature map, i.e. , Since and may be non-integers, a corresponding feature at the position in the feature map needs to be sampled through a bilinear interpolation method to obtain an image feature of the valid projection point with a dimension of 256. Each valid projection point corresponds to an image feature.

[0048] In the point cloud geometry feature extraction unit, one valid point is randomly selected from the N' valid points as an initial sampling point, then the valid point farthest from the initial sampling point is selected as a second sampling point, then the valid point farthest from the second sampling point is selected as the next sampling point, and so on, until the number of sampling points reaches M (M = N' / 32, for example, when N' = 3200, 100 sampling points are taken, and 32 is a modifiable parameter), to ensure that all sampling points uniformly cover the range of the coarse registration point cloud data. Next, Ball Query grouping is performed, specifically, a spherical neighborhood with a radius of 0.2 m is defined around each sampling point (according to the density setting of the coarse registration point cloud data, to ensure that 16-32 points are contained in each ball), and the valid points of the coarse registration point cloud data in the same spherical neighborhood are divided into a group. Finally, through MLP (Multi-Layer Perceptron) aggregation, the features of the points in each group are extracted and aggregated through a 3-layer MLP, specifically, the point cloud coordinates of the valid points are mapped to 64-dimensional features in the first layer, the dimension is increased to 128 in the second layer, and the maximum value of the 128-dimensional features in the group is obtained through a max-pooling operation in the third layer to obtain the global feature of the sampling point. Finally, since there are only M sampling points, the M features need to be expanded to N' through KNN interpolation (taking 3 nearest sampling points), and finally the point cloud geometry feature of each valid point with a dimension of 128 is output .

[0049] In the feature fusion unit, the point cloud geometry feature of any valid point and the image feature of the corresponding valid projection point are spliced in the channel dimension to strengthen the consistency of the local structure. Specifically, when fusing, the point cloud geometry feature of the valid point with a dimension of 128 is spliced with the image feature with a dimension of 256 into a feature vector with a dimension of 384. Then a one-dimensional convolutional neural network is used to compress the channel number to 128 dimensions to obtain the initial fusion feature with a dimension of 128 . Then, the initial fusion feature is mapped to a query vector by a linear layer (dimension 64). Next, the feature maps output by ResNet34 are flattened into features with dimension , and then mapped to keys (dimension ) and values (dimension ) by linear layers. Then, a mask (consistent with the mask in the above embodiment, used to filter invalid image regions) is introduced, and the correlation weight of each dimension feature is calculated, as follows: ; where represents the attention weight, is an activation function used to normalize the weight to between 0 and 1, is a scaling factor, the purpose of which is to avoid the problem of large weight values causing gradient problems, is the mask. The weighted sum of the values is obtained by the attention weight, and the local fusion feature with dimension 64 is obtained, which can reflect the position of the projection point in the overall space of the tunnel. Then, the initial fusion feature and the local fusion feature are spliced into a feature with dimension 192, and the dimension of the gating weight (the weight value of the gating weight is between 0 and 1) is generated by the MLP (192→128) and the Sigmoid activation function. Finally, the initial fusion feature and the local fusion feature of the valid point are fused by weighting, and the formula is: , where represents a preset fusion weight coefficient, and the fusion feature of the valid point with dimension 128 is finally output . The fusion features of other valid points can be obtained in the same way.

[0050] In one specific embodiment, the displacement field acquisition module comprises: an initial displacement field acquisition unit, configured to map the fusion feature of each valid point by a multi-layer perception machine to obtain the effective displacement field of each valid point, perform interpolation processing on all other points in the coarse registration point cloud data except the valid points to generate other displacement fields of each other point, and take the effective displacement fields of all valid points and the other displacement fields of all other points as the initial displacement field of each point in the coarse registration point cloud data; a global coarse adjustment unit, configured to calculate the optimal pose information between the reference point cloud data and the coarse registration point cloud data by an iterative nearest point algorithm, and superimpose the optimal pose information to the initial displacement field of each point to obtain a coarse adjustment displacement field of each point in the coarse registration point cloud data; The local adjustment unit is used to calculate the residual between the coarse adjustment displacement field of each point in the coarse registration point cloud data and the average coarse adjustment displacement field of all its neighboring points. If the absolute value of the residual between the coarse adjustment displacement field of any point and the average coarse adjustment displacement field of all its neighboring points is greater than a preset threshold, the coarse adjustment displacement field of that point is superimposed with a preset adjustment amount and the iteration continues until a preset iteration termination condition is met to obtain the final displacement field of that point. The preset adjustment amount is the product of the residual of each point and the preset residual weight. Otherwise, the coarse adjustment displacement field of that point is taken as its final displacement field.

[0051] Specifically, in the initial displacement field acquisition unit, the effective displacement field of each effective point is calculated based on the fusion features. Specifically, a bottleneck-type MLP (128→64→32→3) is used to map the fusion features of any effective point, and the bottleneck-type MLP outputs the displacement vector of that effective point. , here This is the initial displacement field of the valid point. For other points in the coarsely registered point cloud data that did not pass the mask screening, the KNN-32 algorithm (i.e., taking the 32 nearest valid points) is used to interpolate each other point, and the other displacement fields of each other point are used as the initial displacement fields.

[0052] When the global coarse adjustment unit calculates the optimal pose information between the reference point cloud data and the coarse registration point cloud data using the iterative nearest point algorithm, it performs ICP calculation for each point pair between the reference point cloud data and the coarse registration point cloud data (any point in the coarse registration point cloud data and its corresponding reference point in the reference point cloud data form a point pair). The formula is as follows: ;in, This represents the Iterative Closest Point Algorithm, used to calculate the optimal rotation matrix and optimal translation vector (optimal pose information) for any point pair between the reference point cloud data and the coarsely registered point cloud data. This represents any point in the coarsely registered point cloud data. This indicates the reference point corresponding to this point in the reference point cloud data. and Let represent the optimal rotation matrix and the optimal translation vector, respectively. Superimposing the optimal rotation matrix and the optimal translation vector onto the initial displacement field at that point yields the coarse-tuned displacement field at that point, which can be expressed as: ,in, This represents the coarse-tuned displacement field at that point. This represents the initial displacement field at that point. It should be noted that since the optimal rotation matrix and optimal translation vector are determined based on the reference point cloud data, the coarse-adjusted displacement field of each point refers to the coarse-adjusted displacement field of each point relative to each corresponding reference point in the reference point cloud data.

[0053] In the local adjustment unit, for a point in the coarse registration point cloud data, the residual of the average coarse adjustment displacement of the point and its 32 neighborhood points is calculated, and the formula is wherein, represents the average coarse adjustment displacement field of the 32 neighborhood points of the point (i.e., the average value of the coarse adjustment displacement fields of the 32 neighborhood points), represents the coarse adjustment displacement field of the point, represents the residual. If (i.e., it is considered that there is a significant deviation), the coarse adjustment displacement field of the point is adjusted by a preset residual weight ( whose value is between 0 and 1), and the adjustment formula is: wherein, represents the adjusted coarse adjustment displacement field of the point. If the absolute value of the residual of the adjusted coarse adjustment displacement field of the point and the average coarse adjustment displacement field of all neighborhood points is greater than a preset threshold, the adjustment is continued by the adjustment formula until a preset iteration stop condition is reached. Generally, the iteration is repeated for 3 times, and the deviation is gradually corrected. The first iteration mainly corrects the deviation of more than 5 mm, and the third iteration corrects the small deviation of 1-2 mm. Finally, the registration error is controlled within 1-3 mm, while the true non-elastic local mutation is preserved.

[0054] In a specific embodiment, the slicing module comprises: a standard point cloud data generation unit configured to superimpose the coordinates of each point in the coarse registration point cloud data on the corresponding final displacement field to obtain each standard point, and combine all the standard points to obtain standard point cloud data; a slicing unit configured to slice the standard point cloud data along the tunnel direction according to a preset slice length and a sliding step to obtain a plurality of continuous cross-section point clouds.

[0055] Specifically, in the slicing unit, the preset slice length is 1 m, and the sliding step is 0.5 m. The data continuity between adjacent cross-section slices is ensured by an overlapping strategy (overlapping rate 50%), which is beneficial to subsequent statistical smoothing processing. Each cross-section point cloud contains all the standard points, the final displacement field of all the standard points, and an identifier of the cross-section point cloud. The identifier is generally a slice serial number 0, 1, 2, ….

[0056] In a specific embodiment, the label configuration module comprises: a center determination unit configured to calculate the center coordinates of the center of each cross-section point cloud in the laser radar coordinate system according to the tunnel direction, and calculate the polar angle of each standard point in each cross-section point cloud based on the center coordinates of the center of each cross-section point cloud to obtain the polar angle of each standard point in each cross-section point cloud; The label configuration unit is configured to configure a key area label for each standard point in each section point cloud according to the polar angle of each standard point, wherein the types of the key area label include a roof, a floor, a left side and a right side.

[0057] Specifically, in the center determination unit, the center coordinate of a certain section point cloud refers to the coordinate of the geometric center of the section point cloud, and the polar angle is the angle of a certain standard point in the section point cloud relative to the center coordinate of the section point cloud.

[0058] In the label configuration unit, based on the polar angle θ When the polar angle of a certain standard point is , the key area label configured for the standard point is the roof; when the polar angle of a certain standard point is , the key area label configured for the standard point is the floor; when the polar angle of a certain standard point is , the key area label configured for the standard point is the left side; and when the polar angle of a certain standard point is , the key area label configured for the standard point is the right side.

[0059] In a specific embodiment, the center determination unit calculates the center coordinate of any section point cloud in the laser radar coordinate system and the polar angle of any standard point in the section point cloud , and the calculation is realized by formula (3) and formula (4): (3); (4); In formula (3), represents the X-axis coordinate of the center of the section point cloud, represents the Z-axis coordinate of the center of the section point cloud, represents the number of standard points in the section point cloud, represents the sum of the X-axis coordinates of all standard points in the section point cloud, represents the sum of the Z-axis coordinates of all standard points in the section point cloud, wherein the origin of the laser radar coordinate system is the emission center of the laser radar, the X-axis represents the direction directly above the roadway, the Y-axis represents the direction along the roadway, and the Z-axis represents the direction perpendicular to the X-axis and the Y-axis; In formula (4), represents the X-axis coordinate of the standard point , represents the Z-axis coordinate of the standard point , represents the two-parameter inverse tangent function, the X-axis represents the roadway directly above, the Y-axis represents the roadway trend, and the Z-axis represents the direction perpendicular to the X-axis and the Y-axis.

[0060] In one specific embodiment, the risk indicators include convergence, roof subsidence, and twist angle, and the risk indicator calculation module includes: The convergence calculation unit is configured to obtain standard points in each section point cloud that are labeled as left and right sides in the key area, take the coordinate of the leftmost standard point labeled as the left side in the key area on the X-axis as the left side boundary of each section point cloud, take the coordinate of the rightmost standard point labeled as the right side in the key area on the X-axis as the right side boundary of each section point cloud, take the absolute value of the difference between the left side boundary and the right side boundary of each section point cloud as the section width of each section point cloud, and take the absolute value of the difference between the section width of each section point cloud and the section width of the corresponding section point cloud in the reference point cloud data as the convergence of each section point cloud. The roof subsidence calculation unit is configured to take the mean value of the coordinates of all standard points labeled as the roof in the key area in each section point cloud on the Z-axis and the absolute value of the difference between the preset roof height of the corresponding section point cloud in the reference point cloud data as the roof subsidence of each section point cloud. The twist angle calculation unit is configured to perform plane fitting on all standard points in each section point cloud to obtain a plane fitting equation of each section point cloud, obtain a normal vector of the plane fitting equation, and take the included angle between the normal vector and the horizontal plane as the twist angle of each section point cloud.

[0061] Specifically, in the convergence calculation unit, the convergence reflects the relative extrusion deformation of the left and right sides of the roadway. When calculating the convergence of any section point cloud, the coordinate extreme values of the standard points labeled as the left and right sides in the key area in the section point cloud on the X-axis are obtained, specifically as follows: ; wherein, represents the coordinate of the dth standard point labeled as the left side in the key area in the section point cloud on the X-axis, and the right side boundary ; ; wherein: represents the coordinate of the kth standard point labeled as the right side in the key area in the section point cloud on the X-axis. Then, the section width of the section point cloud is calculated , and the calculation formula is . Finally, the convergence of the section point cloud is calculated , and the formula is: ; wherein, is the section width of the corresponding section point cloud in the reference point cloud data.

[0062] The roof subsidence amount of the roof subsidence amount calculation unit reflects the vertical deformation of the roof of the roadway. When calculating the roof subsidence amount of a section point cloud, the preset roof height of the corresponding section point cloud in the reference point cloud data is first calculated , and the calculation formula is , wherein represents the sum of the Z-axis coordinates of all reference points corresponding to all standard points in the key area labeled as the roof in the section point cloud in the reference point cloud data, represents the Z-axis coordinate of the reference point corresponding to the mth standard point in the section point cloud in the reference point cloud data, represents the number of reference points included in the section point cloud corresponding to the section point cloud in the reference point cloud data; then, the average of the coordinates of the Z-axis of all standard points in the key area labeled as the roof in the section point cloud is calculated, and the calculation formula is , wherein represents the number of all standard points in the key area labeled as the roof in the section point cloud, represents the Z-axis coordinate of the mth standard point in the section point cloud; and finally, the roof subsidence amount of the section point cloud is calculated , and the calculation formula is . The greater the value of the roof subsidence amount, the more serious the vertical deformation of the roof of the section point cloud.

[0063] In the twist angle calculation unit, the plane fitting equation of any section point cloud can be expressed as ; the normal vector of the plane fitting equation is ; then, the included angle between the normal vector of the section point cloud and the horizontal plane (0, 0, 0) is calculated as the twist angle of the section point cloud.

[0064] In one specific embodiment, the deformation evaluation module determines the roadway deformation evaluation result of each section point cloud according to the risk index of each section point cloud and a preset risk classification threshold table.

[0065] Specifically, the preset risk classification threshold table is an association table that stores the relationship between the risk index and the predetermined risk level and the evaluation result, and the risk level is generally divided into four levels. The deformation evaluation module compares the risk index of each section point cloud with the preset risk classification threshold table, and determines the roadway deformation evaluation result according to the comparison result. The specific content of the preset risk classification threshold table can be referred to Table 1. The risk index, the risk level, the section point cloud identifier and the evaluation result are integrated and output to obtain the roadway deformation evaluation result of each section point cloud.

[0066]

[0067] The above merely describes the preferred embodiments of the present application and is not intended to limit the present application. It should be pointed out that, for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present application, and these improvements and modifications should also be considered as falling within the protection scope of the present application.

Claims

1. A roadway deformation intelligent monitoring and evaluation system, characterized in that, include: The data acquisition and processing module is used to acquire raw point cloud data and image data of the tunnel, and to denoise the raw point cloud data by improving the bilateral filtering algorithm to obtain point cloud data. The coarse registration module is used to construct the mapping relationship between point cloud data and image data, and to perform coarse registration on the point cloud data based on the reference point cloud data and the mapping relationship to obtain coarsely registered point cloud data. The feature extraction module is used to project all points in the coarse registration point cloud data, and determine multiple effective points in the coarse registration point cloud data and multiple effective projection points of the multiple effective points in the image plane where the image data is located based on the projection results. It obtains the point cloud geometric features of each effective point and the image features of each corresponding effective projection point, and fuses the point cloud geometric features of each effective point and the image features of its corresponding effective projection point to obtain the fused features of each effective point. The displacement field acquisition module is used to calculate the initial displacement field of each point in the coarse registration point cloud data based on the fusion features of each valid point, and to perform global coarse adjustment and local intermediate adjustment of the initial displacement field of all points based on the reference point cloud data to obtain the final displacement field of each point in the coarse registration point cloud data. The slicing module is used to generate standard point cloud data based on the final displacement field of each point in the coarse registration point cloud data, and to perform sliding slicing on the standard point cloud data according to the roadway direction to obtain multiple continuous cross-sectional point clouds, wherein each cross-sectional point cloud is composed of multiple standard points. The label configuration module is used to divide the standard points in each cross-sectional point cloud into regions and configure key region labels for each standard point in each cross-sectional point cloud. The risk indicator calculation module is used to calculate the risk indicator of each cross-sectional point cloud based on the key area label of each standard point in each cross-sectional point cloud. The deformation evaluation module is used to determine the roadway deformation evaluation result for each cross-sectional point cloud based on the risk index of each cross-sectional point cloud.

2. The intelligent monitoring and evaluation system for roadway deformation according to claim 1, characterized in that, The data acquisition and processing module includes: The data acquisition unit is used to simultaneously acquire raw point cloud data and image data of the tunnel using a high-precision synchronously triggered lidar and camera. The denoising unit is used to distinguish between noise and real points in the original point cloud data by improving the bilateral filtering algorithm, and to remove noise from the original point cloud data to obtain the point cloud data.

3. The intelligent monitoring and evaluation system for roadway deformation according to claim 1 or 2, characterized in that, The coarse registration module includes: The mapping relationship construction unit is used to establish a mapping relationship between point cloud data and image data based on camera information and LiDAR information; The alignment unit is used to align the point cloud data with the image data according to the mapping relationship, and to eliminate error points in the point cloud data and the image data to obtain the first point cloud data. The pose calculation unit is used to extract multiple image key points from the image data, match the multiple image key points with reference key points in the reference image data, and calculate the initial pose transformation matrix of the image data relative to the reference image data based on the matching results. The registration unit is used to perform nearest-point registration between the first point cloud data and the reference point cloud data based on the initial pose transformation matrix. During the nearest-point registration process, the initial pose transformation matrix is ​​iterated based on the objective function to obtain the optimal pose transformation matrix. The coarsely registered point cloud data is determined based on the optimal pose transformation matrix and the first point cloud data.

4. The intelligent monitoring and evaluation system for roadway deformation according to claim 3, characterized in that, The mapping relationship construction unit establishes any target point in the point cloud data based on camera information and LiDAR information. Corresponding pixel in image data The mapping relationship between them is achieved through formulas (1) and (2): （1）； （2）； In formula (1), For the camera's extrinsic parameter matrix, This represents the rotation matrix in the camera's extrinsic parameter matrix. This represents the translation vector in the camera's extrinsic parameter matrix; This represents the intrinsic parameter matrix of the camera. This represents the depth value of the target point in the camera coordinate system; In formula (2), This represents the rotation matrix between the camera and the lidar. This represents the translation vector between the camera and the lidar. This represents the coordinates of the target point in the camera coordinate system after coordinate transformation.

5. The intelligent monitoring and evaluation system for roadway deformation according to claim 1, characterized in that, The feature extraction module includes: The projection unit is used to project all points in the coarse registration point cloud data using camera information to obtain multiple projection points located on the image plane where the image data is located. The filtering unit is used to filter multiple valid projection points from multiple projection points according to the mask, and to determine the valid point corresponding to each valid projection point in the coarse registration point cloud data according to the projection relationship. The image feature extraction unit is used to extract image features for each effective projection point through a feature extraction convolutional neural network. The point cloud geometric feature extraction unit is used to filter multiple sampling points from multiple valid points through sampling processing, extract the point cloud geometric features of each sampling point through a point cloud simplification processing algorithm, and finally expand the point cloud geometric features with the same number of sampling points to the same number of valid points through interpolation to obtain the point cloud geometric features of each valid point. The feature fusion unit concatenates the point cloud geometric features of each valid point with the image features of each valid projection point in the channel dimension to obtain the initial fused features, and then performs dimensionality reduction on each initial fused feature based on the channel attention mechanism to obtain the fused features of each valid point.

6. The intelligent monitoring and evaluation system for roadway deformation according to claim 1, characterized in that, The displacement field acquisition module includes: The initial displacement field acquisition unit is used to map the fusion features of each valid point through a multilayer perceptron to obtain the effective displacement field of each valid point. It performs interpolation processing on all other points in the coarse registration point cloud data except for the valid points to generate other displacement fields for each other point. The effective displacement fields of all valid points and the other displacement fields of all other points are used as the initial displacement field of each point in the coarse registration point cloud data. The global coarse adjustment unit is used to calculate the optimal pose information between the reference point cloud data and the coarse registration point cloud data through the iterative nearest point algorithm, and to superimpose the optimal pose information onto the initial displacement field of each point to obtain the coarse adjustment displacement field of each point in the coarse registration point cloud data. The local adjustment unit is used to calculate the residual between the coarse adjustment displacement field of each point in the coarse registration point cloud data and the average coarse adjustment displacement field of all its neighboring points. If the absolute value of the residual between the coarse adjustment displacement field of any point and the average coarse adjustment displacement field of all its neighboring points is greater than a preset threshold, the coarse adjustment displacement field of that point is superimposed with a preset adjustment amount and the iteration continues until a preset iteration termination condition is met to obtain the final displacement field of that point. The preset adjustment amount is the product of the residual of each point and the preset residual weight. Otherwise, the coarse adjustment displacement field of that point is taken as its final displacement field.

7. The intelligent monitoring and evaluation system for roadway deformation according to claim 1, characterized in that, The slicing module includes: The standard point cloud data generation unit is used to superimpose the coordinates of each point in the coarse registration point cloud data with the corresponding final displacement field to obtain each standard point, and combine all standard points to obtain standard point cloud data. The slicing unit is used to slide slice the standard point cloud data along the roadway direction according to the preset slice length and sliding step size to obtain multiple continuous cross-sectional point clouds.

8. The intelligent monitoring and evaluation system for roadway deformation according to claim 1, characterized in that, The label configuration module includes: The center determination unit is used to calculate the center coordinates of the center of each cross-section point cloud in the lidar coordinate system according to the direction of the roadway, and to calculate the polar angle of the center coordinates of each cross-section point cloud and each standard point, so as to obtain the polar angle of each standard point in each cross-section point cloud. The label configuration unit is used to configure key region labels for each standard point according to the polar angle of each standard point in each cross-sectional point cloud. The types of key region labels include top plate, bottom plate, left side and right side.

9. The intelligent monitoring and evaluation system for roadway deformation according to claim 8, characterized in that, The center determination unit calculates the center coordinates of the center of any cross-sectional point cloud in the lidar coordinate system. With any of its standard points polar angle This can be achieved through formulas (3) and (4): （3）； （4）； In formula (3), Indicates the center of the point cloud of this cross section. In the X-axis coordinate, Indicates the center of the point cloud of this cross section. In the Z-axis coordinate, This indicates the number of standard points within the point cloud of that cross section. This represents the sum of the X-axis coordinates of all standard points within the point cloud of this cross-section. This represents the sum of the coordinates of all standard points within the point cloud of this cross section on the Z-axis. The origin of the lidar coordinate system is the lidar's transmission center, the X-axis represents the area directly above the tunnel, the Y-axis represents the tunnel's direction, and the Z-axis represents the direction perpendicular to the X and Y axes. In formula (4), Representing standard points In the X-axis coordinate, Representing standard points In the Z-axis coordinate, This represents the two-parameter arctangent function.

10. The intelligent monitoring and evaluation system for roadway deformation according to claim 9, characterized in that, The risk indicators include convergence, roof subsidence, and torsion angle. The risk indicator calculation module includes: The convergence calculation unit is used to obtain the standard points labeled as left and right sides of the key regions in each cross-sectional point cloud. The coordinates of the leftmost standard point labeled as left side of the key region are taken as the left side boundary of each cross-sectional point cloud, and the coordinates of the rightmost standard point labeled as right side of the key region are taken as the right side boundary of each cross-sectional point cloud. The absolute value of the difference between the left side boundary and the right side boundary of each cross-sectional point cloud is taken as the cross-sectional width of each cross-sectional point cloud. The absolute value of the difference between the cross-sectional width of each cross-sectional point cloud and the cross-sectional width of the corresponding cross-sectional point cloud in the reference point cloud data is taken as the convergence amount of each cross-sectional point cloud. The top plate subsidence calculation unit is used to take the absolute value of the difference between the average coordinate of all standard points labeled as top plate in the key area of ​​each cross-section point cloud and the preset top plate height in the corresponding cross-section point cloud data as the top plate subsidence of each cross-section point cloud. The torsion angle calculation unit is used to perform plane fitting on all standard points in each cross-sectional point cloud to obtain the plane fitting equation of each cross-sectional point cloud, and obtain the normal vector of the plane fitting equation. The angle between the normal vector and the horizontal plane is taken as the torsion angle of each cross-sectional point cloud.

Images