Tunnel disease identification model training method and system based on point cloud and image

Abstract

The invention discloses a tunnel disease recognition model training method and system based on a point cloud and an image, and the method comprises the steps: synchronously collecting tunnel point cloud and image data, carrying out the calibration and registration, achieving the spatial alignment, preprocessing the point cloud, generating a gray-scale image and a depth image, collecting continuous images through a line-scan digital camera, generating a spliced image, and carrying out the recognition of tunnel diseases. Partitioning a large-size image after multi-image space-time synchronization; a multi-branch network is constructed, point cloud geometry and image texture features are extracted by using Point Net / 3DCNN and CNN / Transform respectively, and semantic collaborative fusion is realized through an intermediate layer fusion module; parameters are adjusted according to errors through self-adaptive training, manual labeling dependence is reduced in combination with transfer learning and the like, and convergence is accelerated through online iteration, joint loss and self-adaptive weight to improve generalization; the performance of the model is evaluated through field testing and indexes in various tunnel environments, and the structure is optimized according to data to ensure that engineering is feasible and efficient. The model can accurately identify various diseases on different tasks, and can adapt to complex and changeable working conditions in tunnel detection.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of tunnel engineering safety detection, and more particularly relates to a tunnel disease identification model training method and system based on point cloud and image. BACKGROUND

[0002] With the rapid development of China's transportation infrastructure construction, the number and scale of tunnel projects continue to expand. However, tunnels are prone to cracks, water leakage, lining spalling, deformation and other diseases due to changes in geological conditions, environmental erosion, traffic loads and other factors during long-term operation, which seriously threatens the safety of the structure and the efficiency of operation. Therefore, efficient and accurate detection of diseases is crucial to ensure the safe operation of tunnels.

[0003] Current automatic detection of tunnel diseases mainly relies on image and point cloud based technologies. Image-based detection collects tunnel inner wall images through high-resolution cameras and identifies disease characteristics with image processing algorithms, but has obvious defects: the complex lighting in the tunnel affects the image quality, leading to inaccurate disease feature extraction; two-dimensional images lack depth information and cannot describe the three-dimensional morphology and spatial distribution of diseases, making it difficult to identify hidden or three-dimensional diseases; a large number of images need to be collected to ensure accuracy, resulting in large data processing and low detection efficiency.

[0004] Point cloud based detection uses laser scanning to obtain tunnel three-dimensional point cloud data and analyzes disease characteristics through point cloud processing. However, it has limitations: point cloud data lacks color and texture information, making it difficult to identify early-stage cracks, water leakage marks and other diseases related to surface texture; point cloud data is large, uneven in density and noisy, making pre-processing and feature extraction difficult and computationally inefficient; the limited density of laser scanning point cloud results in low accuracy in identifying small diseases.

[0005] Some studies attempt to fuse image and point cloud data, but mostly use post-fusion strategies, i.e. simply stacking or matching after processing the two types of data separately, which has many problems: post-fusion does not fully utilize the complementarity of the two in the feature extraction stage, resulting in insufficient fusion feature expression and inability to effectively improve disease identification accuracy; image and point cloud data are collected using different principles and coordinate systems, making the registration process complex and prone to errors, which directly affects the reliability of the fusion effect; existing fusion models are often trained for specific tunnel environments and disease types, and have poor adaptability to different scenarios and features, with insufficient generalization ability.

[0006] At the same time, existing tunnel disease identification model training highly depends on large-scale labeled data, consuming a lot of manpower, material resources and time. Moreover, the tunnel environment is complex and variable, with significant differences in disease characteristics among different regions and types of tunnels. The model faces the problem of inconsistent data distribution in actual application, leading to decreased detection accuracy.

[0007] In summary, the existing tunnel disease detection technology has deficiencies in data fusion depth, environmental adaptability, feature expression ability, training efficiency and model generalization ability, etc., and it is difficult to meet the actual needs of tunnel safety detection, and it is urgent to develop a tunnel disease identification model and method that can fully integrate the advantages of point cloud and image data and realize intelligent training. SUMMARY

[0008] The present application aims to provide a point cloud and image based tunnel disease identification model intelligent training method, which synchronously collects point cloud and image data and performs high-precision registration, constructs a double-branch feature extraction and deep fusion network, combines adaptive training strategies and online iterative optimization mechanisms, solves the problems of insufficient data fusion, poor environmental adaptability, high training cost and weak generalization ability of existing technologies, realizes high-precision and robust detection of tunnel diseases, reduces the dependence on manual annotation, and improves the real-time performance and reliability of the model in complex environments.

[0009] In view of the above defects or improvement needs of the prior art, as a first aspect of the present application, the present application provides a training method of a tunnel disease identification model based on point cloud and image, comprising:

[0010] S1. Synchronously collect tunnel point cloud and image data to realize spatial alignment through calibrated registration; preprocess the point cloud and generate a gray image and a depth image, and generate a tunnel internal stitching image by collecting continuous images through a line array camera; synchronize the gray image, the depth image and the camera stitching image in time and space; block the large-size image around the disease area;

[0011] S2. Construct a multi-branch network including gray image, depth image and camera stitching image branches, respectively extract point cloud structure geometric features through PointNet / 3DCNN, extract image texture color features through CNN / Transformer architecture, and realize semantic collaborative fusion through an intermediate layer fusion module;

[0012] S3. Automatically adjust the parameters according to the prediction and detection error through an adaptive training method, reduce the dependence on manual annotation by combining methods including transfer learning, data enhancement, active learning, and accelerate the convergence of the model and improve the generalization ability through an online iterative optimization mechanism, a joint loss function and an adaptive weight strategy;

[0013] S4. Evaluate the model identification accuracy, recall rate, real-time performance and robustness through field test, cross-validation, confusion matrix and other indicators in various tunnel environments, and optimize the model structure according to the test data to ensure the engineering feasibility and efficiency.

[0014] Further, the specific method for preprocessing the point cloud and generating a gray image and a depth image in S1 is:

[0015] The single-frame point cloud data frame and encoder data encl are collected through a 3D mobile laser measurement system. The single-frame image size is set and the step length point cloud is selected based on the encoder data encl. The cross-section is extracted by equidistant △y and then filtered to obtain a total frame point cloud. The cross-section is fitted using the least squares method to obtain information including the ellipse center coordinates, major axis, minor axis, horizontal axis, and deflection angle. The tunnel centerline of the step length is smoothed and fitted through the center point of the tunnel cross section, divided into step length grids, and the thickness d is extracted with the tunnel centerline as the center. y The cross section is converted into raster information, and the intensity information of the nearest point is converted into the grayscale value of the corresponding image. The stitched image is corrected by feature point matching and interpolated to generate a grayscale image. By obtaining the point cloud coordinate set P corresponding to each pixel, the optimal value of the point cloud coordinates of T pixels before and after point P is calculated, the distance difference of the current point P is obtained and converted into RGB information, and then a depth map is generated.

[0016] Furthermore, the single-frame point cloud data frame is specifically:

[0017] The data from a laser radar scan includes the x1, y1, z coordinates and intensity information i of each point relative to the laser radar center, as well as the actual three-dimensional coordinate data x, y, z. The Y direction of the coordinate system is the direction of the tunnel centerline, the Z direction is the height direction of the tunnel section, and the X direction is the horizontal direction of the tunnel section. The tunnel section refers to the section perpendicular to the tunnel centerline.

[0018] Furthermore, the specific method of filtering the cross-section point cloud is:

[0019] The single-frame point cloud data frame contains the geometric features of the tunnel cross-section. The tunnel wall conforms to the elliptical feature. The tunnel bottom information is filtered by angle information:

[0020] Assume that the coordinates of any point in a single frame of point cloud data are (x, y, z), and the center point of the tunnel section O (x0, y0, z0) is used as the reference to convert the point cloud coordinates into relative polar coordinates (r, θ, z), where:

[0021]

[0022] θ=arctan2(z-z0,x-x0) θ

[0023] θ is the angle of the point relative to the horizontal axis of the tunnel section;

[0024] Set the bottom track area angle range to θ∈[θ min ,θ max ], the filtering condition is: if Keep the point cloud; otherwise, remove it;

[0025] The tunnel wall contains interference from pipelines and supports, so the pipelines and supports on the tunnel wall are filtered out using the segmented fitting method:

[0026] The tunnel wall angle range θ∈[θ min ,θ max ] is divided into N segments, and the angle interval of each segment is The angle range of segment i is:

[0027] θ i ∈[θ min +(i-1)Δθ,θ min +iΔθ],i=1,2,…,N

[0028] For the i-th point cloud {(x j ,z j )}, and use the least squares method to fit the ellipse equation:

[0029]

[0030] The parameters (A, B, C, D, E, F) are obtained by solving the following optimization problem:

[0031]

[0032] Constraint: B 2 -4AC<0;

[0033] Calculate the point (x j ,z j ) to the distance d from the fitted ellipse j , using the point to quadratic curve distance formula:

[0034]

[0035] Set the distance threshold T = k·σ, σ is the standard deviation of the distance of the point cloud segment, k is the coefficient; the filtering condition is: if d j ≤T, the point cloud is retained; otherwise it is considered as interference and removed.

[0036] Furthermore, the specific process of converting the cross-section point cloud into raster information and then converting the nearest point intensity information into the grayscale value of the corresponding image is as follows:

[0037] Assume the cross-section point cloud data {(x,y,z,x1,y1,z1,i)}, the tunnel cross-section center point O(x0,y0,z0), the image height H, the pixel resolution δ, and the distance ratio k d Intensity extreme value i min ,i max . ;

[0038] Coordinate transformation and raster mapping:

[0039] (Δx, Δz)=(x-x0,z-z0)

[0040] The angle θ of the point relative to the tunnel center is:

[0041] θ=arctan²(Δz,Δx)

[0042] The row coordinates Y and column coordinates X of the grid are determined by the angle θ and the image height H:

[0043]

[0044] Among them, k d is the mapping coefficient between pixels and actual distance, and δ is the actual length represented by a unit pixel;

[0045] The coordinates of the point cloud relative to the center of the lidar are (x1, y1, z1), and its distance to the center of the lidar is:

[0046]

[0047] For all point clouds within the grid (X,Y) Filter the point with the smallest distance:

[0048]

[0049] The intensity value i min Mapping to grayscale value G:

[0050]

[0051] The grayscale value corresponding to the grid (X, Y) is G.

[0052] Furthermore, the specific method of correcting the spliced ​​image by matching feature points and interpolating to generate a grayscale image is:

[0053] The area with a width of ΔH between the edges of the two images is selected as the overlapping part, which is used for subsequent feature point matching and offset calculation;

[0054] The SURF method is used to detect feature points in overlapping areas. The specific process includes: constructing an image pyramid to identify extreme points in different scale spaces; calculating the direction vector of the feature point to ensure rotation invariance; generating a feature descriptor to describe the gradient distribution characteristics of the feature point neighborhood;

[0055] Use FlannBasedMatcher for feature point matching and select reliable matching pairs through the following steps: calculate the Euclidean distance or cosine distance between the feature point descriptors of the current image and the previous image; use Lowe's ratio test to retain matching pairs with the ratio of the nearest neighbor to the next nearest neighbor distance less than t, and eliminate false matches;

[0056] For the matched feature points, a homography matrix H is fitted using the least square method to describe the projection transformation relationship between the two pictures; a translation component in the Y direction is extracted from the homography matrix, denoted as Δy, which reflects the misalignment degree when the pictures are spliced;

[0057] Translation correction picture: according to the offset Δy, a translation transformation is performed on the current picture, and the formula is: (x', y') = (x, y + Δy) where (x, y) is the pixel coordinate before correction, and (x', y') is the coordinate after correction, so as to ensure that the feature points in the overlapping area are aligned;

[0058] Remove the overlapping area and splice: splice the corrected current picture with the previous picture, and use the pixel value of the previous picture in the overlapping area to generate a continuous tunnel wall image;

[0059] Detect missing pixels: detect null points by traversing pixel coordinates;

[0060] Four-neighbor interpolation: for each missing pixel (i, j), take the gray values of its four adjacent valid pixels (i±1, j) and (i, j±1), and interpolate them as follows:

[0061] When all four neighbors are valid, take the arithmetic mean:

[0062]

[0063] When some neighbors are invalid, only take the average of the valid neighbors;

[0064] When all neighbors are invalid, extend to eight neighbors;

[0065] After all missing pixels are interpolated, a gray-scale image that restores the intensity distribution of the tunnel wall surface in 1:1 is obtained, which is used for subsequent disease identification.

[0066] Further, the semantic synergistic fusion through the intermediate layer fusion module is specifically:

[0067] When a multi-modal fusion module is introduced in the intermediate layer to realize semantic synergistic fusion of the two branches, a three-layer processing architecture including feature alignment, cross interaction and semantic enhancement can be constructed;

[0068] First, the spatial transformation network STN is used to align the coordinates of the texture features extracted from the image branch and the geometric features output by the point cloud branch, and an affine transformation matrix is generated using the three-dimensional spatial coordinate information of the point cloud branch to perform operations including rotation and scaling on the image feature map, thereby eliminating the feature misalignment caused by the difference in data acquisition angle;

[0069] Then the cross-attention mechanism is used to realize the information interaction between modalities. The geometric features of the point cloud branch are taken as the query, the texture features of the image branch are taken as the key and the value, the attention weight of the geometric features to each texture feature is calculated, and the geometric features with fused texture information are obtained after weighted aggregation. Meanwhile, the image features are taken as the query, and the point cloud features are taken as the key and the value, to generate the texture features with fused geometric structure. Through bidirectional interaction, the cross-modal feature pair containing the "geometry-texture" correlation is constructed.

[0070] Finally, the feature distillation is performed through the semantic enhancement module. The double-channel feature pyramid structure is used to fuse the low-level fine-grained texture features and the high-level abstract geometric features across scales. The contribution weight of different modal features at each semantic level is automatically learned through the gating mechanism, and finally the fused feature vector containing multi-dimensional semantic information is output.

[0071] Further, the joint loss function is specifically:

[0072] The cross-entropy loss combined with the focus loss is used to construct the classification loss function L cls , which is used for the classification task of tunnel disease types:

[0073]

[0074] where y is the true label, is the predicted probability, α is the class balance coefficient, and γ is the focus parameter;

[0075] The smooth L1 loss combined with the IoU loss is used to construct the regression loss function L reg , which is used for the regression prediction of disease geometric parameters (such as crack length, width, deformation displacement, etc.):

[0076]

[0077] where x i is the true value, is the predicted value, IoU is the intersection over union of the predicted region and the true region, and λ is the weight coefficient;

[0078] The structure consistency loss is constructed to constrain the structure consistency of point cloud and image features, to ensure that the features after multi-modal fusion retain the tunnel geometric structure information:

[0079] Based on the consistency of point cloud normal vectors, the point cloud structure loss L point is calculated by calculating the cosine value of the included angle between the predicted point cloud and the true point cloud:

[0080]

[0081] Adopting perception loss, the feature map is extracted through pre-training VGG network, and the image structure loss is constructed by calculating the high-level semantic feature distance between the predicted image and the real image:

[0082]

[0083] wherein f l is the feature extraction function of the lth layer of the VGG network, C l is the feature dimension of the lth layer of the VGG network, H l is the height of the feature map, and W l is the width of the feature map.

[0084] The total loss function is:

[0085] L total = ω cls L cls + ω reg L reg + ω point L point + ω img L img

[0086] wherein ω cls , ω reg , ω point , ω img are the weights of the classification loss L cls , the regression loss L reg , the point cloud structure loss L point , and the image structure loss L img in the total loss calculation, respectively, and are used to adjust the contribution degree of the corresponding task loss to the total loss.

[0087] As a second aspect of the present application, a training system of a tunnel disease identification model based on point cloud and image is provided, comprising:

[0088] A data acquisition and preprocessing unit is configured to synchronously acquire tunnel point cloud and image data, to realize spatial alignment through calibration and registration, to pre-process the point cloud and generate a gray-scale image and a depth image, to generate a tunnel internal stitching image through continuous image acquisition by a line array camera, to perform multi-image space-time synchronization on the gray-scale image, the depth image and the camera stitching image, and to block large-size images around the disease area.

[0089] A feature extraction and multi-modal fusion unit is configured to construct a multi-branch network including a gray-scale image branch, a depth image branch and a camera stitching image branch, to extract point cloud structure geometric features through PointNet / 3D CNN, to extract image texture color features through CNN / Transformer architecture, and to realize semantic collaborative fusion through an intermediate layer fusion module.

[0090] The intelligent training strategy implementation unit is used for automatically adjusting parameters according to prediction and detection errors by means of an adaptive training method, reducing dependence on manual labeling by means of methods including transfer learning, data enhancement and active learning, and accelerating model convergence and improving generalization ability by means of an online iterative optimization mechanism, a joint loss function and an adaptive weight strategy.

[0091] The model verification and optimization unit is used for evaluating model recognition accuracy, recall rate, real-time performance and robustness by means of field testing, cross-validation and confusion matrix in various tunnel environments, and optimizing the model structure according to test data to ensure engineering feasibility and efficiency.

[0092] As a third aspect of the present application, a computer readable storage medium is also provided, which stores a computer program for executing any step of the above-mentioned training method of the tunnel disease recognition model based on point cloud and image.

[0093] Overall, compared with the prior art, the above technical solutions conceived by the present application can achieve the following beneficial effects:

[0094] 1. The training method of the tunnel disease recognition model based on point cloud and image of the present application introduces a multi-modal fusion module in the intermediate layer to cooperatively fuse the features of the point cloud branch and the image branch in semantics. Specifically, first, a spatial transformation network is used to align the coordinates of the features of the two branches and eliminate misalignment caused by perspective difference; then, cross-attention mechanism is used to realize cross-modal information interaction and construct feature pairs containing "geometry-texture" correlation; finally, a semantic enhancement module is used for cross-scale feature distillation and gate weighting. This technical feature significantly enhances the expression ability of disease features, enabling the model to not only capture surface texture details of diseases such as crack color and direction, but also perceive three-dimensional spatial distribution and structural correlation of diseases such as spatial morphology of lining deformation, thereby improving the recognition accuracy of complex diseases and effectively dealing with the diversity and complexity of diseases in tunnel environment.

[0095] 2. The training method of the tunnel defect recognition model based on point cloud and image of the present invention is achieved by designing a joint loss function including classification loss, regression loss, structural consistency loss, etc. The classification loss adopts the cross entropy loss combined with the focal loss to solve the problem of sample imbalance; the regression loss uses the smooth L1 loss combined with the IoU loss to improve the accuracy of geometric parameter prediction; the structural consistency loss constrains the feature structure from two aspects: the consistency of the point cloud normal vector and the image perception loss. At the same time, combined with the adaptive weight adjustment strategy, the weight is dynamically updated according to the loss value to balance the multi-task conflict. This enables the model to take into account multiple tasks such as defect classification, parameter regression, and point cloud and image structure maintenance during training, effectively improving the performance of the model on different tasks, accurately identifying various types of defects and accurately predicting their related parameters, and adapting to the complex and changeable working conditions in tunnel detection.

[0096] 3. The present invention's method for training a tunnel defect recognition model based on point clouds and images involves matching feature points in overlapping areas of previous and subsequent images, detecting them using the SURF algorithm, matching them using the FlannBasedMatcher, and screening reliable matches using the Lowe's ratio test. The homography matrix is ​​then calculated to extract vertical offsets, and the images are then translationally corrected and spliced ​​together. Possible missing pixels after splicing are then supplemented using methods such as four-neighborhood interpolation. This technical feature effectively corrects misalignment in the spliced ​​images, generating complete and continuous tunnel wall images. This provides accurate, high-quality image data for subsequent defect recognition, ensuring that the model can be trained and predicted based on precise input, and improving the reliability and accuracy of tunnel defect recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0097] Figure 1 This is a flow chart of a method for training a tunnel disease recognition model based on point clouds and images according to an embodiment of the present invention;

[0098] Figure 2 A flowchart for generating grayscale images and depth images according to an embodiment of the present invention;

[0099] Figure 3 This is a result image of projecting a camera image onto a grayscale image according to an embodiment of the present invention;

[0100] Figure 4 This is a training result diagram of an embodiment of the present invention;

[0101] Figure 5 This is a partially enlarged crack sample diagram of an embodiment of the present invention;

[0102] Figure 6 2 is a diagram of system units according to an embodiment of the present invention. DETAILED DESCRIPTION

[0103] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not to limit the present application. In addition, the technical features involved in the various embodiments of the present application described below can be combined with each other as long as they do not conflict with each other.

[0104] Embodiment 1

[0105] Please refer to Figure 1 The present embodiment 1 provides a training method of a tunnel disease identification model based on point cloud and image, comprising

[0106] S1. Synchronously collecting tunnel point cloud and image data to realize spatial alignment through calibrated registration; preprocessing the point cloud and generating a gray scale image and a depth image, and generating a tunnel internal splicing image through a line array camera to collect continuous images; synchronizing the gray scale image, the depth image and the camera splicing image in time and space; dividing large-size images around the disease area;

[0107] S2. Constructing a multi-branch network including a gray scale image branch, a depth image branch and a camera splicing image branch, respectively extracting point cloud structure geometric features through PointNet / 3DCNN, extracting image texture color features through CNN / Transformer architecture, and realizing semantic collaborative fusion through an intermediate layer fusion module;

[0108] S3. Automatically adjusting parameters according to prediction and detection errors through an adaptive training method, reducing the dependence on manual annotation by combining methods including transfer learning, data enhancement and active learning, and accelerating model convergence and improving generalization ability through an online iterative optimization mechanism, a joint loss function and an adaptive weight strategy;

[0109] S4. Evaluating the model recognition accuracy, recall rate, real-time performance and robustness through field tests, cross-validation and confusion matrix in various tunnel environments, and optimizing the model structure according to the test data to ensure engineering feasibility and efficiency.

[0110] In the following content, the present embodiment 1 expands the above steps.

[0111] (1) Data acquisition and preprocessing

[0112] Three-dimensional point cloud data inside the tunnel is collected by using a laser scanning or a structured light sensor, and after noise filtering and filtering processing:

[0113] Gray scale image generation: extracting numerical information reflecting the surface reflection intensity of the point cloud, mapping it into a gray scale image, and describing the surface texture and disease details.

[0114] Depth map generation: according to the distance information of each point in the point cloud, the depth map is generated, which reflects the distance of each region to the sensor, and provides accurate basis for subsequent geometric correction.

[0115] Camera stitching map generation: using a line array camera to collect continuous images, and generating a whole tunnel interior stitching map through geometric calibration, internal and external parameter calibration and image stitching algorithm, to ensure that the stitching map and the point cloud generation map are consistent in space.

[0116] The gray scale image, depth map and camera stitching map are spatially registered and time-synchronized to ensure that they correspond to the same scene area in the same coordinate system, facilitating subsequent multi-modal feature fusion.

[0117] Due to the large size of the image, in a specific embodiment, the image size is 21680*26983, which cannot be trained by the network, so the disease area is segmented and extracted, and the segmented picture is obtained. The stitching map, gray scale image and depth map are sent into the network to realize the fusion of various features and realize multi-modal feature fusion of the image. Figure Three

[0118] Please refer to Figure 2 and Figure 3 In a preferred embodiment, the specific gray scale image and depth map are produced as follows:

[0119] Single-frame point cloud data frame and encoder data encl are collected by a three-dimensional mobile laser measurement system; the size of a single picture is set and the step length point cloud is selected according to the encoder data encl; the cross section is extracted by equal interval △y after the cross section point cloud is passed, and a total of total frames of point cloud are obtained;

[0120] The single-frame point cloud data intersects with the tunnel cross section, and the fitting center is consistent with the tunnel cross section center, so the center point of the tunnel cross section can be obtained by fitting the single-frame point cloud data. The filtered point cloud is least square fitted to an ellipse. The center coordinates, major axis, minor axis, horizontal axis and deflection angle information of the ellipse can be obtained by least square fitting. By fitting total frames of point cloud, the rough tunnel cross section center line can be obtained, and the noise points in the fitting can be filtered out by the method of random sampling consistency. The center line of the total frames of point cloud is least square fitted to a straight line, and the tunnel center line of step length is obtained. Repeat the above operation until the picture width is exceeded.

[0121] ​Further, the length of the grid can be calculated according to the length of the step, and the grid is divided according to the length and width of the grid and the corresponding resolution; the section with a thickness of d is extracted with the tunnel center line as the center, the coordinates of the current section are converted relative to the center point coordinates according to the center point coordinates of the section, the angle information of the current point relative to the tunnel center is calculated, and the X coordinate and Y coordinate of the grid value of the current point can be calculated according to the angle information and the height HEIGHT of the picture. Repeat the above operation until all points are converted. There are j points in each grid, the nearest point to the laser radar is selected as the optimal point, and the intensity information of the current point is converted into the gray value of the corresponding picture. And correct the spliced picture and generate a gray image by interpolation through feature point matching; by obtaining the point cloud coordinate set P corresponding to each pixel, the optimal value of the T pixel point cloud coordinates before and after P point is calculated, the distance difference value of the current P point is obtained and converted into RGB information, and then a depth map is generated.

[0122] In a specific preferred embodiment, the single-frame point cloud data frame specifically comprises:

[0123] The data scanned by the laser radar in one circle includes the x1, y1, z coordinates and intensity information i of each point relative to the center of the laser radar and the real three-dimensional coordinate data x, y, z; the Y direction of the coordinate system is the direction of the tunnel center line, the Z direction is the height direction of the tunnel section, and the X direction is the horizontal direction of the tunnel section. The tunnel section refers to the section perpendicular to the tunnel center line.

[0124] The single-frame point cloud data contains the geometric features of the tunnel section, the tunnel wall conforms to the elliptical feature, and the tunnel bottom is mainly the track information, so it is necessary to filter the information of the tunnel bottom. The tunnel bottom information is filtered according to the angle information, and the tunnel wall contains the interference of pipelines and supports, which can be filtered by the segmented fitting method.

[0125] In a specific preferred embodiment, the specific method for filtering the section point cloud is:

[0126] The single-frame point cloud data frame contains the geometric features of the tunnel section, and the tunnel wall conforms to the elliptical feature. The tunnel bottom information is filtered through the angle information:

[0127] Let the coordinates of any point in the single-frame point cloud data be (x, y, z), and take the center point O(x0, y0, z0) of the tunnel section as the reference to convert the point cloud coordinates into relative polar coordinates (r, θ, z), wherein:

[0128]

[0129] θ = arctan2(z-z0, x-x0) θ

[0130] θ is the included angle of the point relative to the horizontal axis of the tunnel section;

[0131] The bottom rail area angle range is set as θ∈[θ min ,θ max ], and the filtering condition is: if the point cloud is retained; otherwise, it is rejected;

[0132] If the tunnel wall contains interference of pipelines and supports, the pipelines and supports of the tunnel wall are filtered by the method of piecewise fitting:

[0133] The tunnel wall angle range θ∈[θ min ,θ max ] is equally divided into N segments, and the angle interval of each segment is The angle range of the i-th segment is:

[0134] θ i ∈[θ min +(i-1)Δθ,θ min +iΔθ],i=1,2,…,N

[0135] For the i-th segment point cloud {(x j ,z j )}, an ellipse equation is fitted by using the least square method:

[0136]

[0137] Wherein, the parameters (A, B, C, D, E, F) are obtained by solving the following optimization problem:

[0138]

[0139] The constraint condition is: B 2 -4AC<0;

[0140] The distance d j of the point (x j ,z j ) to the fitted ellipse is calculated, and the point-to-quadratic curve distance formula is used:

[0141]

[0142] The distance threshold T=k·σ is set, σ is the standard deviation of the distance of the segment point cloud, and k is the coefficient; the filtering condition is: if d j ≤T, the point cloud is retained; otherwise, it is determined as interference and is rejected.

[0143] In a specific preferred embodiment, the specific process of converting the section point cloud into grid information and then taking the nearest point intensity information into the corresponding picture gray value is:

[0144] ​Let the section point cloud data be {(x, y, z, x1, y1, z1, i)}, the tunnel section center point be O(x0, y0, z0), the picture height be H, the pixel resolution be δ, and the distance ratio be k d Intensity extreme value i min ,i max 。;

[0145] Coordinate conversion and grid mapping:

[0146] (x-x0, z-z0)

[0147] The angle θ of the point relative to the tunnel center is:

[0148] θ = arctan2(Δz, Δx)

[0149] The row coordinate Y and the column coordinate X of the grid are determined by the angle θ and the picture height H:

[0150]

[0151] where k d is the mapping coefficient of the pixel and the actual distance, and δ is the actual length represented by the unit pixel;

[0152] The coordinates of the point cloud relative to the center of the laser radar are (x1, y1, z1), and the distance from the center of the laser radar is:

[0153]

[0154] For all point clouds in the grid (X, Y) Select the point with the smallest distance:

[0155]

[0156] Map the intensity value i min to the gray value G:

[0157]

[0158] The gray value corresponding to the grid (X, Y) is G.

[0159] In a specific preferred embodiment, the specific method of correcting the spliced picture and generating a gray image by feature point matching is:

[0160] Select the area with an edge width of ΔH of the two pictures as the overlapping part, which is used for subsequent feature point matching and offset calculation;

[0161] SURF method is used to detect feature points in the overlapping area, and the specific process includes: constructing an image pyramid, identifying extreme points in different scale spaces; calculating the direction vector of the feature point to ensure rotation invariance; generating a feature descriptor to describe the gradient distribution characteristics of the feature point neighborhood;

[0162] Feature point matching is performed using FlannBasedMatcher, and reliable matching pairs are selected by the following steps: calculating the Euclidean distance or cosine distance between the feature point descriptors of the current image and the previous image; using Lowe's ratio test to retain matching pairs with a nearest neighbor distance ratio less than t, and eliminating false matches;

[0163] For matching successful feature point pairs, a homography matrix H is fitted using the least squares method to describe the projection transformation relationship between the two images; the translation component in the Y direction is extracted from the homography matrix, denoted as Δy, which reflects the degree of misalignment when the image is spliced;

[0164] Translation correction image: according to the offset Δy, the current image is translated, the formula is: (x', y') = (x, y + Δy), where (x, y) is the pixel coordinate before correction, (x', y') is the coordinate after correction, to ensure that the feature points in the overlapping area are aligned;

[0165] Remove overlapping area and splice: splice the corrected current image with the previous image, the overlapping area uses the pixel value of the previous image, and generate a continuous tunnel wall image;

[0166] Detect missing pixels: detect null points by traversing pixel coordinates;

[0167] Four-neighbor interpolation: for each missing pixel (i, j), take the gray values of its four adjacent valid pixels (i±1, j) and (i, j±1), and interpolate as follows:

[0168] When all four neighbors are valid, take the arithmetic mean:

[0169]

[0170] When some of the neighbors are invalid, only take the average of the valid neighbors;

[0171] When all neighbors are invalid, expand to eight neighbors;

[0172] After all missing pixels are interpolated, a gray-scale image is obtained, which restores the intensity distribution of the tunnel wall surface in 1:1, which is used for subsequent disease identification.

[0173] In a preferred embodiment, the point closest to the lidar in each pixel grid is selected as the optimal point P. Based on the current point P, the distance information of the point cloud coordinates of the T pixels before and after it is calculated. The optimal point cloud information is selected as the standard distance Q of point P. The difference between the standard distance Q and the current point P is calculated as the deformation of point P. The above process is repeated until all deformations of the image are calculated. The difference information is matched with the RGB information to obtain the RGB information of the current pixel. The RGB information is converted into image information to obtain a depth map.

[0174] (2) Feature extraction and multimodal fusion

[0175] A multi-branch structure of a multi-input deep neural network is constructed, with each branch responsible for extracting different image features. The grayscale image branch focuses on extracting surface texture and local defect features. Through multi-level processing of convolutional neural networks, detailed information such as the specific direction of cracks, width variations, and grayscale distribution differences in leaking areas is analyzed from grayscale images. The depth image branch focuses on capturing structural changes and deformation information. Using a three-dimensional geometric feature extraction network, it analyzes depth images to obtain geometric features such as the displacement of the lining in space and specific changes in the cross-sectional profile, thereby reflecting the spatial geometry of the tunnel. The camera mosaic image branch is responsible for providing global scene information. It stitches and integrates images from multiple perspectives to form a complete tunnel scene image. Based on this, the global feature extraction network extracts information on the distribution of defects in the overall tunnel environment, including the specific distribution density of defects in the tunnel's circumferential and longitudinal directions.

[0176] In the middle layer, a cross-modal fusion module is used to complement the features extracted from the three branches. The attention mechanism plays a key role in this process, dynamically adjusting the weight of different features in the fusion process by calculating the weights of each branch's features at the channel and spatial levels. For example, when detecting structural deformation, the weight of the depth map features is increased, giving them a higher proportion in the fusion process, thereby more prominently reflecting information related to structural deformation. The multi-scale fusion module fuses the features extracted by the different convolutional layers of each branch, such as shallow edge features of the grayscale image and deep structural features of the depth map, and concatenates them along the channel dimension. This concatenation is then subjected to dimensionality reduction and feature reorganization through specific convolution operations, forming a cross-modal fusion feature vector with a specific dimension. This feature vector preserves multi-dimensional information, covering textural details such as the width of a crack; geometric parameters such as the magnitude of deformation; and spatial location such as the location of the defect. Finally, through specific classification and bounding box regression operations, it is possible to accurately classify the types of defects such as cracks, water leakage, deformation, etc., and accurately locate the location of the defects.

[0177] In a specific preferred embodiment, the semantic collaborative fusion through the intermediate layer fusion module is specifically:

[0178] When the multimodal fusion module is introduced in the intermediate layer to realize the semantic collaborative fusion of the two branches, a three-layer processing architecture including feature alignment, cross interaction and semantic enhancement can be constructed.

[0179] Firstly, the spatial transformation network STN is used to align the texture features extracted by the image branch and the geometric features output by the point cloud branch in coordinates, and an affine transformation matrix is generated using the three-dimensional spatial coordinate information of the point cloud branch to perform operations including rotation and scaling on the image feature map, eliminating the feature misalignment caused by the difference in data acquisition angle.

[0180] Then, the cross attention mechanism is used to realize the information interaction between the two modalities, the geometric features of the point cloud branch are taken as the query, the texture features of the image branch are taken as the key and the value, the attention weight of the geometric features to each texture feature is calculated, and the geometric features with fused texture information are obtained after weighted aggregation; at the same time, the image features are taken as the query, and the point cloud features are taken as the key and the value, the texture features with fused geometric structure are generated, and the cross-modal feature pair containing "geometric-texture" correlation is constructed through bidirectional interaction.

[0181] Finally, the feature distillation is performed through the semantic enhancement module, the double-channel feature pyramid structure is used to fuse the low-level fine-grained texture features and the high-level abstract geometric features across scales, the contribution weight of different modal features at each semantic level is automatically learned through the gating mechanism, for example, the texture feature weight is enhanced when identifying early-stage water disease, and the geometric feature proportion is increased when detecting structural deformation, and finally the fused feature vector containing multi-dimensional semantic information is output. The model can not only capture surface features such as crack color and texture, but also perceive the three-dimensional spatial distribution and structural correlation of the disease, which significantly improves the recognition accuracy of complex diseases.

[0182] (3) Implementation of intelligent training strategy

[0183] A variety of data augmentation methods are used to expand the training samples, including rotating, scaling, translating and adding noise disturbance to the image, etc., so as to increase the diversity of the training data and improve the generalization ability of the model. At the same time, the public dataset or the pre-trained model of similar tasks is used for transfer learning, the knowledge of the pre-trained model is transferred to the current tunnel disease identification task, the difficulty of the initial training of the model is reduced, and the convergence speed of the model is accelerated.

[0184] To address the feature differences of different modal data, a joint loss function is designed, which includes classification loss, regression loss, and structural consistency loss, etc. The classification loss is used to measure the accuracy of the model in classifying disease types, the regression loss is used to evaluate the accuracy of the model in predicting disease geometric parameters, and the structural consistency loss is used to constrain the structural consistency of features from different modalities. In order to make the model automatically balance the contributions of each branch in different training stages, an adaptive weight adjustment strategy is combined to dynamically adjust the weights of each part in the loss function according to the training situation of each task, thereby improving the convergence speed and generalization ability of the model.

[0185] After the model is deployed, an online iterative optimization mechanism is established. Through the feedback data collected on site, the model is fine-tuned and updated iteratively. These feedback data contain various complex situations and new disease types in the actual tunnel environment. By optimizing the model using these data, the model can better adapt to the actual environment. By continuously collecting feedback data and updating the model, a closed-loop training mechanism is formed, further improving the stability and recognition accuracy of the model in the actual environment, ensuring that the model can be continuously and effectively applied to tunnel disease detection work.

[0186] In a specific preferred embodiment, the joint loss function is specifically:

[0187] By adopting cross-entropy loss combined with focal loss to construct the classification loss function L cls , which is used for the classification task of tunnel disease types (such as cracks, seepage water, lining deformation, etc.):

[0188]

[0189] where y is the true label, is the predicted probability, α is the class balance coefficient, and γ is the focus parameter;

[0190] Smooth L1 loss combined with IoU loss is used to construct the regression loss function L reg , which is used for the regression prediction of disease geometric parameters (such as crack length, width, deformation displacement, etc.):

[0191]

[0192] where x i is the true value, is the predicted value, IoU is the intersection over union of the predicted region and the true region, and λ is the weight coefficient;

[0193] The structural consistency loss is constructed to constrain the structural consistency of point cloud and image features, ensuring that the features after multi-modal fusion retain the tunnel geometric structure information:

[0194] Based on the consistency of the normal vector of the point cloud, the normal vector cosine value of the predicted point cloud and the real point cloud is calculated to obtain the point cloud structure loss L point :

[0195]

[0196] The perception loss is adopted, the feature map is extracted by pre-training the VGG network, the high-level semantic feature distance of the predicted image and the real image is calculated to construct the image structure loss:

[0197]

[0198] Wherein, f l (·) is the feature extraction function of the lth layer of the VGG network, C l ,H l ,W l is the dimension of the feature map;

[0199] Then the total loss function is:

[0200] L total =ω cls L cls +ω reg L reg +ω point L point +ω img L img

[0201] Wherein, ω cls , ω reg , ω point , ω img are the classification loss L cls , the regression loss L reg , the point cloud structure loss L point , the image structure loss L img The weight in the total loss calculation is used to adjust the contribution degree of the corresponding task loss to the total loss.

[0202] (4) Model verification and optimization

[0203] Please refer to Figure 4 and Figure 5The training process starts from the input of three batches of pre-processed image data. The processed gray image, depth image and camera mosaic image are imported into the network in batches to ensure that the format and size of the input data meet the requirements of the model, laying the foundation for subsequent feature extraction. Then, each branch network extracts features from different types of images. The gray image branch extracts surface texture and local disease characteristics through a convolutional neural network, capturing details such as gray scale changes in crack direction and water leakage areas. The depth image branch uses a three-dimensional geometric feature extraction network to analyze the spatial coordinate differences of structural deformation, obtaining geometric information such as lining displacement and cross-section contour changes. The camera mosaic image branch integrates multi-view images through a global feature extraction network, extracting information about the distribution of diseases in the tunnel in the circumferential and longitudinal directions. The extracted features of each branch are input into the fusion module, where they are complementarily fused through an attention mechanism and a multi-scale fusion module. The channel-level and spatial-level weights are calculated to dynamically adjust the feature proportion, and the features of different convolution layers are reorganized by channel concatenation to form a global feature representation containing texture, geometry, spatial position and other multi-dimensional information.

[0204] After the global features are processed by the fully connected layer, the classification or positioning information of the tunnel disease is output. The classification task can identify different disease types such as cracks, spalling and water leakage, while the positioning information determines the specific location of the disease in the tunnel. Then, a joint loss function containing classification loss, regression loss and structural consistency loss is used to calculate the error between the predicted results and the true labels. The classification loss measures the accuracy of type classification, the regression loss evaluates the prediction accuracy of geometric parameters, and the structural consistency loss constrains the structural matching degree of multi-modal features. Through the backpropagation algorithm, the error is propagated along the network levels, updating the network parameters of each layer and optimizing the model's ability to extract and express disease features.

[0205] In the verification and evaluation stage, the cross-validation method is used to divide the training set and test set. The confusion matrix is used to analyze the recognition of each type of disease by the model, and the precision, recall and other indicators are calculated to quantify the model performance. In addition, field test data is introduced to verify the adaptability of the model to changes in lighting, complex structures and other scenarios in actual tunnel environments, and to evaluate its robustness in real working conditions, ensuring that the model can stably and accurately complete the disease identification task in actual application.

[0206] Embodiment 2

[0207] For reference Figure 6 The embodiment 2 provides a training system of a tunnel disease identification model based on point cloud and image, comprising:

[0208] The data acquisition and preprocessing unit is used for synchronously collecting and calibrating and registering the tunnel point cloud and image data to realize spatial alignment, pre-processing the point cloud to generate a gray image and a depth image, and collecting continuous images by a line array camera to generate a tunnel internal splicing image, and performing multi-image space synchronization on the gray image, the depth image and the camera splicing image; and the large-size image is divided into blocks around the disease area;

[0209] The feature extraction and multi-modal fusion unit is used for constructing a multi-branch network including a gray image branch, a depth image branch and a camera splicing image branch, extracting point cloud structure geometric features by PointNet / 3D CNN, extracting image texture color features by CNN / Transformer architecture, and realizing semantic collaborative fusion by an intermediate layer fusion module.

[0210] The intelligent training strategy implementation unit is used for automatically adjusting parameters according to prediction and detection errors by an adaptive training method, reducing dependence on manual labeling by combining methods including transfer learning, data enhancement and active learning, and accelerating model convergence and improving generalization ability by an online iterative optimization mechanism, a joint loss function and an adaptive weight strategy.

[0211] The model verification and optimization unit is used for evaluating model recognition accuracy, recall rate, real-time performance and robustness by field testing, cross-validation and confusion matrix in various tunnel environments, and optimizing the model structure according to test data to ensure engineering feasibility and efficiency.

[0212] Embodiment 3

[0213] The embodiment 3 also provides a computer readable storage medium, and the computer readable storage medium stores a computer program. The computer program is executed by a processor to implement any step of the training method of the tunnel disease identification model based on point cloud and image.

[0214] The computer readable storage medium can include a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various storage program codes.

[0215] The computer readable storage medium provided in the present application is introduced in the above method embodiments, and the present application will not be repeated here.

[0216] Those skilled in the art can easily understand that the above description is only a preferred embodiment of the present application and is not used to limit the present application. Any modification, equivalent replacement and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for training a tunnel disease identification model based on a point cloud and an image, characterized in that, Comprise: S1. Synchronize the acquisition of tunnel point cloud and image data calibrated registration to realize spatial alignment; Preprocess the point cloud and generate gray scale and depth maps, and generate a continuous image by a line array camera to generate a tunnel interior stitching image; synchronize the gray scale, depth maps and camera stitching image in time and space; block the large size image around the disease area; S2. Construct a multi-branch network including gray scale, depth maps and camera stitching image branches, respectively extract point cloud structure geometric features by PointNet / 3DCNN, image texture color features by CNN / Transformer architecture, and realize semantic collaborative fusion through intermediate layer fusion module; S3. Through the adaptive training method, automatically adjust the parameters according to the prediction and detection error, reduce the dependence on manual annotation by combining methods including transfer learning, data enhancement, active learning, and through online iterative optimization mechanism, joint loss function and adaptive weight strategy to accelerate model convergence and improve generalization ability; S4. In various tunnel environments, evaluate the model recognition accuracy, recall rate, real-time performance and robustness through field test, cross-validation, confusion matrix and other indicators, and optimize the model structure according to the test data to ensure engineering feasibility and efficiency.

2. The method of claim 1, wherein the method comprises: The specific method for preprocessing the point cloud and generating gray scale and depth maps in S1 is: Collect single frame point cloud data frame and encoder data encl by three-dimensional mobile laser measurement system; set single picture size and select step length point cloud according to encoder data encl, extract cross section point cloud by equal interval △y, and obtain total frame point cloud; fit the cross section by least square method to obtain information including elliptic center coordinate, major axis, minor axis, horizontal axis and deflection angle, smooth tunnel center line by tunnel cross section center point and fitting step length, divide step length grid, extract cross section with thickness d y from tunnel center line as center, convert nearest point intensity information into corresponding picture gray value after converting cross section point cloud into grid information, correct spliced picture by feature point matching and generate gray scale diagram by interpolation; obtain point cloud coordinate set P corresponding to each pixel, calculate optimal value of P point before and after T pixel point cloud coordinates, obtain current P point distance difference value and convert RGB information, and further generate depth map.

3. The method of claim 2, wherein the method further comprises: The specific method for preprocessing the point cloud and generating gray scale and depth maps in S1 is: The single frame point cloud data frame is specifically:

4. The method of claim 2, wherein the method further comprises: The data scanned by a laser radar in one circle, including the x1, y1, z coordinates and intensity information i of each point relative to the center of the laser radar, and the real three-dimensional coordinate data x, y, z; the Y direction of the coordinate system is the tunnel center line direction, the Z direction is the height direction of the tunnel section, and the X direction is the horizontal direction of the tunnel section, and the tunnel section refers to the section perpendicular to the tunnel center line. The specific method for preprocessing the point cloud and generating gray scale and depth maps in S1 is: The single frame point cloud data frame contains the geometric features of the tunnel section, and the tunnel wall conforms to the elliptical feature; the tunnel bottom information is filtered through the angle information: θ = arctan2(z - z0, x - x0) θ Let the coordinates of any point in the single frame point cloud data be (x, y, z), take the center point O(x0, y0, z0) of the tunnel section as the reference, convert the point cloud coordinates to relative polar coordinates (r, θ, z), wherein: The bottom rail area angle range is set as θ∈[θ min ,θ max ], and the filtering condition is: if , the point cloud is retained; otherwise, it is rejected. θ is the angle of the point relative to the horizontal axis of the tunnel section; The tunnel wall angle range θ ∈ [θ min ,θ max ] is equally divided into N segments, and the angle interval of each segment is The angle range of the i-th segment is: θ i ∈ [θ min + (i - 1) Δθ, θ min + i Δθ], i = 1, 2,..., N For the i-th segment of point cloud {(x j ,z j )}, an ellipse equation is fitted using least square method: The tunnel wall contains pipelines and support interference, so the pipelines and supports of the tunnel wall are filtered by piecewise fitting: Constraint: B 2 -4AC < 0; Calculate the point (x j ,z j ) to the distance d from the fitted ellipse j , using the point to quadratic curve distance formula: Set distance threshold T = kσ, σ is the standard deviation of the distance of the segment point cloud, k is the coefficient; filtering condition is: if d j ≤ T, keep the point cloud; otherwise, determine as interference and eliminate. 5.The method of claim 2, wherein, Wherein the parameters (A, B, C, D, E, F) are obtained by solving the following optimization problem: Let the section point cloud data {(x, y, z, x1, y1, z1, i}, the tunnel section center point O(x0, y0, z0), the picture height H, the pixel resolution δ, the distance proportion k d Intensity extreme i min ,i max ; The specific process of converting the section point cloud to grid information, taking the nearest point intensity information, and converting it to the corresponding picture gray scale value is: Coordinate conversion and grid mapping: (Δx, Δz) = (x-x0, z-z0) The angle θ of the point relative to the tunnel center is: θ = arctan2(Δz, Δx) where k d is a mapping coefficient of the pixel to the actual distance, and δ is the actual length represented by a unit pixel. The row coordinate Y and column coordinate X of the grid are determined by the angle θ and the picture height H: For all points in grid (X, Y) (j = 1, 2,..., n), filter the point with the smallest distance: The intensity values i min are mapped to grey values G: The coordinates of the point cloud relative to the center of the laser radar are (x1, y1, z1), and the distance from the center of the laser radar is: 6.The method of claim 2, wherein, Then the gray scale value corresponding to the grid (X, Y) is G. The specific method for preprocessing the point cloud and generating gray scale and depth maps in S1 is: Select the edge width of the two pictures before and after the region as the overlapping part, which is used for subsequent feature point matching and offset calculation; The SURF method is used for feature point detection in the overlapping area, and the specific process includes: constructing an image pyramid, identifying extreme points in different scale spaces; calculating the direction vector of the feature point to ensure rotation invariance; generating a feature descriptor to describe the gradient distribution characteristics of the feature point neighborhood; Feature point matching is performed using FlannBasedMatcher, and reliable matching pairs are selected through the following steps: calculating the Euclidean distance or cosine distance of the feature point descriptors of the current image and the previous image; using Lowe's ratio test to retain matching pairs with a nearest neighbor distance to second nearest neighbor distance ratio less than t, and eliminating false matches; For the matching feature point pairs, the least squares method is used to fit the homography matrix H, which describes the projection transformation relationship between the two images; the translation component in the Y direction is extracted from the homography matrix, denoted as Delta y, which reflects the degree of misalignment during image stitching; Translation correction image: according to the offset Delta y, the current image is translated, the formula is: (x', y') = (x, y + Delta y), where (x, y) is the pixel coordinate before correction, (x', y') is the coordinate after correction, and the feature points in the overlapping area are aligned; Remove the overlapping area and splice: splice the corrected current image with the previous image, use the pixel value of the previous image in the overlapping area, and generate a continuous tunnel wall image; Detect missing pixels: detect null points by traversing pixel coordinates; Four-neighbor interpolation: for each missing pixel (i, j), take the gray values of its four adjacent valid pixels (i±1, j) and (i, j±1), and interpolate them as follows: When all four neighbors are valid, take the arithmetic mean: When some neighbors are invalid, only take the average of the valid neighbors; When all neighbors are invalid, expand to eight neighbors; After all missing pixels are interpolated, a gray-scale image is obtained, which restores the intensity distribution of the tunnel wall surface in 1:1, which is used for subsequent disease identification.

7. The method of claim 1, wherein the method further comprises: The semantic collaborative fusion in S2 is realized by an intermediate layer fusion module, specifically: When introducing a multi-modal fusion module into the intermediate layer to realize semantic collaborative fusion of the two branches, a three-layer processing architecture including feature alignment, cross interaction and semantic enhancement can be constructed; Firstly, the spatial transformation network STN is used to align the texture features extracted from the image branch and the geometric features output from the point cloud branch, and an affine transformation matrix is generated using the three-dimensional spatial coordinate information of the point cloud branch to perform operations including rotation and scaling on the image feature map, eliminating feature misalignment caused by differences in data acquisition angles; Then, the cross attention mechanism is used to realize information interaction between modalities, taking the geometric features of the point cloud branch as the query Query, the texture features of the image branch as the key Key and the value Value, calculating the attention weight of the geometric features to each texture feature, and weighting and aggregating to obtain the geometric features of the fused texture information; at the same time, taking the image features as the query and the point cloud features as the key and the value, the texture features of the fused geometric structure are generated, and the cross-modal feature pairs containing "geometric-texture" correlation are constructed through bidirectional interaction; Finally, the semantic enhancement module is used for feature distillation. The double-channel feature pyramid structure is used to fuse the low-level fine-grained texture features and the high-level abstract geometric features across scales. The gating mechanism is used to automatically learn the contribution weight of different modal features at each semantic level. Finally, a fusion feature vector containing multi-dimensional semantic information is output. 8.The method of claim 1, wherein, The joint loss function in S3 is specifically: By adopting cross-entropy loss combined with focal loss to construct classification loss function L cls , for the classification task of tunnel disease types: where y is the true label, is the predicted probability, a is the class balance coefficient, and g is the focus parameter. The regression loss function L is constructed by using a smooth L1 loss combined with an IoU loss reg For regression prediction of disease geometric parameters (such as crack length, width, deformation displacement, etc.): wherein x i is a true value, is a predicted value, IoU is an intersection over union of the predicted region and the true region, and λ is a weight coefficient. The structure consistency loss is constructed to constrain the structure consistency of the point cloud and the image features, and to ensure that the geometric structure information of the tunnel is retained after the multi-modal fusion: Based on the consistency of the normal vector of the point cloud, the normal vector cosine value of the predicted point cloud and the real point cloud is calculated to obtain the point cloud structure loss L point : The perception loss is used to extract feature maps through a pre-trained VGG network, and to calculate the distance between the predicted image and the real image to construct the image structure loss: wherein f l (·) is the feature extraction function of the VGG network at the i-th layer, C l ,H l ,W l is the dimension of the feature map; The total loss function is: L total = ω cls L cls + ω reg L reg + ω point L point + ω img L img Among them, ω cls 、ω reg 、ω point 、ω img They are classification loss L cls , regression loss L reg , point cloud structure loss L point , image structure loss L img The weight in the total loss calculation is used to adjust the contribution of the corresponding task loss to the total loss. 9.A training system of a tunnel disease identification model based on a point cloud and an image, characterized in that, It includes: The data acquisition and preprocessing unit is used for synchronous acquisition of tunnel point cloud and image data, and for spatial alignment through calibration and registration; The point cloud is preprocessed to generate a grayscale image and a depth image. The continuous image is acquired by a line array camera to generate a tunnel interior stitching image. The grayscale image, the depth image, and the camera stitching image are synchronized in time and space. The large-size image is divided into blocks around the disease area; The feature extraction and multi-modal fusion unit is used for constructing a multi-branch network including a grayscale image branch, a depth image branch, and a camera stitching image branch. The point cloud structure geometric features are extracted by PointNet / 3D CNN, the image texture color features are extracted by CNN / Transformer architecture, and the semantic collaborative fusion is realized through an intermediate layer fusion module; The intelligent training strategy implementation unit is used for automatic parameter adjustment according to the prediction and detection error through an adaptive training method. The dependence on manual annotation is reduced through methods including transfer learning, data enhancement, and active learning. The online iterative optimization mechanism, the joint loss function, and the adaptive weight strategy are used to accelerate the model convergence and improve the generalization ability. The model verification and optimization unit is used for evaluating the model recognition accuracy, recall rate, real-time performance, and robustness through field testing, cross-validation, and confusion matrix in various tunnel environments. The model structure is optimized according to the test data to ensure the engineering feasibility and efficiency.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the training method of the tunnel disease identification model based on point cloud and image according to any one of claims 1-8.

Images