Crack feature extraction and quantification method and device based on point cloud large language model

Abstract

The application discloses a crack feature extraction and quantification method and device based on a point cloud large language model, and relates to the technical field of civil engineering structure health monitoring. The method comprises the following steps: according to a two-dimensional image data set and a three-dimensional point cloud data set, performing cross-modal semantic analysis by using a crack feature analysis large model to obtain a crack semantic recognition result; performing skeleton line fitting by using a curve fitting method according to the crack semantic recognition result and the three-dimensional point cloud data set; performing crack feature quantification according to a crack local point cloud subset data set and a three-dimensional skeleton curve of the crack to obtain a crack structured feature; and performing time sequence comparison reasoning by using the crack feature analysis large model according to the crack structured feature and a feature description text based on a crack trend inquiry text to obtain a crack development trend prediction result. The application is a crack feature extraction and quantification method with high precision and strong explainability by combining three-dimensional point cloud data processing and a large language model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of civil engineering structure health monitoring, in particular to a crack feature extraction and quantification method and device based on a point cloud large language model. BACKGROUND

[0002] In recent years, with the breakthrough of Multimodal Large Language Model (MLLM) technology, a series of general models capable of processing unstructured data such as images and point clouds and outputting natural language have emerged, such as 3D Large Language Model (3D-LLM), 3D Large Language and Vision Assistant (3D-LLaVA), LiDAR Large Language Model (LiDAR-LLM: Exploring the Potential of Large Language Models for 3D LiDAR Understanding), and Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Following. These models combine spatial perception, semantic understanding, and conversational reasoning capabilities, providing a new technical path for infrastructure crack "detection, positioning, quantification, and interpretation". For example, the LiDAR-LLM model converts outdoor laser point cloud understanding problems into language modeling problems, gradually aligns three-dimensional point cloud features to the text embedding space of LLM through a three-stage training strategy, and designs an alignment transformer that embeds Bird's Eye View (BEV) location information to enhance the model's spatial understanding capabilities. Models such as Point-LLM explore the combination of point cloud encoders and LLMs to learn from massive automatically generated point cloud-text instruction data, enabling the model to recognize objects in the point cloud and generate textual descriptions.

[0003] However, in the specific engineering application of structural crack detection, there is still a lack of methods to deeply integrate the above point cloud large language models with field requirements. Existing research has not addressed how to enable LLMs to output high-precision crack spatial analysis and geometric quantification results, especially lacking deep modeling of crack center skeletons, width profiles, and scale quantification elements. This limits the usability and precision advantages of large language models in engineering crack automatic detection.

[0004] In the prior art, there is a lack of a crack feature extraction and quantification method with high precision and strong explainability combining three-dimensional point cloud data processing and large language models. SUMMARY

[0005] To solve the technical problem of how to make the LLM output high-precision crack spatial analysis and geometric quantification results, especially the lack of deep modeling of crack center skeleton, width profile, and scale quantification elements, the embodiments of the present application provide a crack feature extraction and quantification method and device based on a point cloud large language model. The technical solution is as follows:

[0006] On the one hand, a crack feature extraction and quantification method based on a point cloud large language model is provided, which is realized by a crack feature extraction and quantification device, and the method comprises:

[0007] A multi-time and multi-view data acquisition of a target structure is performed by a drone carrying a binocular camera and a laser radar to obtain an original two-dimensional image data set, an original three-dimensional point cloud data set, and a drone pose information set;

[0008] Based on a preset improved LiDAR-camera extrinsic calibration method, the original two-dimensional image data set and the original three-dimensional point cloud data set are preprocessed according to the drone pose information set to obtain a two-dimensional image data set and a three-dimensional point cloud data set;

[0009] A to-be-trained crack feature analysis large model is constructed according to the model structure of the point cloud large language model;

[0010] A training data set is obtained; based on a prompt optimization strategy and a data enhancement method, the to-be-trained crack feature analysis large model is subjected to three-stage optimization training according to the training data set to obtain a crack feature analysis large model;

[0011] Cross-modal semantic analysis is performed on the two-dimensional image data set and the three-dimensional point cloud data set using the crack feature analysis large model to obtain a crack semantic recognition result;

[0012] Based on a three-dimensional bounding box method, a Hessian matrix eigenvalue analysis method, and a Laplacian Contraction algorithm, a skeleton line fitting is performed using a curve fitting method according to the crack semantic recognition result and the three-dimensional point cloud data set to obtain a crack local point cloud subset data set and a crack three-dimensional skeleton curve;

[0013] Based on a geometric statistical calculation method and an alpha-Shape algorithm, crack feature quantification is performed according to the crack local point cloud subset data set and the crack three-dimensional skeleton curve to obtain crack structured features; semantic description is performed on the crack structured features by the crack feature analysis large model to obtain a feature description text;

[0014] obtain the crack development trend prediction result.

[0015] In another aspect, a crack feature extraction and quantification device based on a point cloud large language model is provided. The device is applied to a crack feature extraction and quantification method based on a point cloud large language model. The device comprises:

[0016] A crack data acquisition module is configured to acquire multi-time and multi-view data of a target structure by a drone carrying a binocular camera and a laser radar, to obtain an original two-dimensional image data set, an original three-dimensional point cloud data set, and a drone pose information set.

[0017] A data preprocessing module is configured to preprocess the original two-dimensional image data set and the original three-dimensional point cloud data set based on a preset improved LiDAR-camera extrinsic calibration method according to the drone pose information set, to obtain a two-dimensional image data set and a three-dimensional point cloud data set.

[0018] An analysis model construction module is configured to construct a to-be-trained crack feature analysis large model according to a model structure of a point cloud large language model.

[0019] An analysis model training module is configured to acquire a training data set, to perform three-stage optimization training on the to-be-trained crack feature analysis large model based on a prompt optimization strategy and a data enhancement method according to the training data set, and to obtain a crack feature analysis large model.

[0020] A crack semantic recognition module is configured to perform cross-modal semantic analysis using the crack feature analysis large model according to the two-dimensional image data set and the three-dimensional point cloud data set, and to obtain a crack semantic recognition result.

[0021] A crack skeleton extraction module is configured to perform skeleton line fitting using a curve fitting method based on a three-dimensional bounding box method, a Hessian matrix eigenvalue analysis method, and a Laplacian Contraction algorithm according to the crack semantic recognition result and the three-dimensional point cloud data set, to obtain a crack local point cloud subset data set and a crack three-dimensional skeleton curve.

[0022] A crack feature quantification module is configured to perform crack feature quantification based on a geometric statistical calculation method and an alpha-shape algorithm according to the crack local point cloud subset data set and the crack three-dimensional skeleton curve, to obtain a crack structured feature, and to perform semantic description through the crack feature analysis large model according to the crack structured feature, to obtain a feature description text.

[0023] The crack evolution analysis module is configured to obtain a crack trend inquiry text; and based on the crack trend inquiry text, crack development trend prediction results are obtained by using a crack feature analysis large model to perform time sequence comparison reasoning according to crack structured features and feature description texts.

[0024] In another aspect, a crack feature extraction and quantification device is provided, which includes a processor and a memory having computer readable instructions stored thereon, the computer readable instructions being executed by the processor to implement any one of the above crack feature extraction and quantification methods based on a point cloud large language model.

[0025] In another aspect, a computer readable storage medium is provided, which stores at least one instruction, the at least one instruction being loaded and executed by a processor to implement any one of the above crack feature extraction and quantification methods based on a point cloud large language model.

[0026] The technical solutions provided by the embodiments of the present application have at least the following beneficial effects:

[0027] The present application provides a crack feature extraction and quantification method based on a point cloud large language model, which uses point cloud data to completely retain the spatial geometric information of cracks, avoiding the position and shape distortion caused by the lack of depth in two-dimensional image methods; through a multi-modal large language model, image texture and point cloud geometry are deeply fused, and on the basis of obtaining fine 3D reconstruction, rich semantic labels are assigned to the results, realizing the unity of "geometric accuracy" and "semantic understanding" in crack detection;

[0028] A skeleton-section joint crack morphology expression method is proposed, the skeleton line extraction summarizes the overall trend and penetration of the crack, and the width section quantifies the change of the crack opening degree along the length; this multi-level representation can comprehensively depict the crack morphology features, which is helpful for engineers to understand the development state of the crack and can provide a basis for structural stress analysis;

[0029] The present application can output quantitative indicators such as the length, width and spatial position of the crack, as well as inference results (such as whether it is through) based on these indicators; among them, the synchronous quantification of width and position overcomes the information asynchronization problem caused by traditional measurement in two dimensions and three dimensions respectively, realizing the synchronous automatic quantification of crack position and size; the high-precision geometric quantification makes the crack evaluation more objective and reliable, and also facilitates the subsequent use of data for maintenance decision-making;

[0030] Introducing a large language model enables the system to not only provide data but also "interpretations." The model can describe crack characteristics using technical terminology and answer user questions about cracks (such as "Does this crack need repair?" "What might have caused the crack?"). This question-and-answer human-computer interaction greatly improves the system's intelligence, allowing non-technical users to obtain useful information through natural language questions; the explanatory text output by the model aids in understanding the results and writing reports.

[0031] This invention is designed to meet the crack detection needs of various civil engineering concrete structures. With appropriate training and optimization, it can be widely applied to different scenarios such as bridge beams, bridge piers, tunnel linings, subway segments, and building walls. Due to the adoption of the cross-modal transfer capability of a large model, it has good generalization ability to new scenarios, reducing the workload of re-annotation and training for each engineering instance. Through the multi-temporal data monitoring and predictive analysis module, this invention can be seamlessly extended for long-term health monitoring, providing decision support for digital twins and structural maintenance.

[0032] This invention significantly improves upon existing crack detection methods in terms of spatial accuracy, information richness, and intelligence, possessing high engineering application value and effectively ensuring the service safety and durability of major engineering structures. This invention is a high-precision and highly interpretable crack feature extraction and quantification method that combines 3D point cloud data processing with a large language model. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 This is a flowchart of a crack feature extraction and quantization method based on a point cloud large language model provided in an embodiment of the present invention;

[0035] Figure 2 This is a block diagram of a crack feature extraction and quantization device based on a point cloud large language model provided in an embodiment of the present invention;

[0036] Figure 3 This is a schematic diagram of the structure of a crack feature extraction and quantification device provided in an embodiment of the present invention. Detailed Implementation

[0037] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0038] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0039] In the embodiments of this invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning. Similarly, the terms "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning.

[0040] In this embodiment of the invention, sometimes a subscript such as W1 may be written in a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.

[0041] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0042] This invention provides a method for crack feature extraction and quantization based on a point cloud large language model. This method can be implemented using a crack feature extraction and quantization device, which can be a terminal or a server. Figure 1 The flowchart shown is for a crack feature extraction and quantization method based on a point cloud large language model. The processing flow of this method may include the following steps:

[0043] S1. By using a drone equipped with a binocular camera and a lidar, multi-time and multi-view data are collected on the target structure to obtain the original two-dimensional image dataset, the original three-dimensional point cloud dataset, and the drone pose information set.

[0044] In one feasible implementation, this invention uses a reinforced concrete pier of an elevated highway in a city as the inspection object. The pier surface has visible vertical fine cracks. The method proposed in this invention is applied to collect data from the pier cracks using a drone inspection, and the crack features are extracted, quantified, and output.

[0045] Using drones equipped with binocular cameras and LiDAR, a close-range fly-around scan of the bridge piers was conducted. Multiple images of the pier surface were acquired with a resolution of 4000×3000 pixels, covering multiple perspectives of the crack area. The LiDAR point cloud contained approximately 2 million points with a point spacing on the order of millimeters, comprehensively recording the three-dimensional morphology of the pier surface. Simultaneously, this invention collected, recorded, and saved data from multiple time points for subsequent crack evolution analysis.

[0046] S2. Based on the preset improved LiDAR camera extrinsic calibration method, the original two-dimensional image dataset and the original three-dimensional point cloud dataset are preprocessed according to the UAV pose information set to obtain the two-dimensional image dataset and the three-dimensional point cloud dataset.

[0047] Optionally, based on a pre-defined improved LiDAR camera extrinsic calibration method, the original 2D image dataset and the original 3D point cloud dataset are preprocessed according to the UAV pose information set to obtain the 2D image dataset and the 3D point cloud dataset, including:

[0048] Based on the UAV pose information set, preliminary data alignment is performed on the original two-dimensional image dataset and the original three-dimensional point cloud dataset to obtain the first two-dimensional image dataset and the first three-dimensional point cloud dataset.

[0049] Based on the preset improved LiDAR camera extrinsic calibration method, the extrinsic parameters are reprojected and aligned according to the first two-dimensional image dataset and the first three-dimensional point cloud dataset to obtain the second two-dimensional image dataset and the second three-dimensional point cloud dataset.

[0050] The second 3D point cloud dataset is cleaned to obtain a third 3D point cloud dataset containing only the surface of the target structure; the third 3D point cloud dataset is downsampled to obtain a fourth 3D point cloud dataset.

[0051] The second two-dimensional image dataset is defined as two-dimensional image data; the fourth three-dimensional point cloud data is defined as a three-dimensional point cloud dataset.

[0052] In one feasible implementation, the image and point cloud are registered using the UAV pose information set and the UAV's GPS / IMU data and feature matching method: based on the initial alignment of the UAV attitude, an improved LiDAR-camera extrinsic parameter calibration method (e.g., combining SuperGlue feature matching and optimization algorithm) is used to accurately solve the transformation matrix between the point cloud coordinate system and the image pixel coordinate system, and each laser point is projected onto the corresponding image to obtain the spatial correspondence between the point cloud and the image.

[0053] The point cloud data was preprocessed to filter out isolated stray points and remove ground points at the bottom of the bridge piers, cropping the point cloud to include only the surface of the piers. Voxel downsampling (5 mm voxel size) was then performed on the cropped point cloud to reduce the number of points to approximately 500,000, reducing the data size while preserving crack details. Statistical filtering was used to remove outliers and noise, and irrelevant parts such as the ground were cropped to retain only the point cloud of the structural surface area as the data basis for subsequent processing.

[0054] S3. Construct a large model for crack feature analysis to be trained based on the model structure of the point cloud large language model;

[0055] In one feasible implementation, the present invention proposes a large-scale crack feature analysis model based on a point cloud large language model.

[0056] The large-scale model for crack feature analysis is a fusion model architecture that integrates image and point cloud data. It can automatically identify crack regions in point clouds, achieve three-dimensional semantic segmentation, bridge the modal gap between two-dimensional images and three-dimensional point cloud information, and make full use of the complementary advantages of image texture and point cloud depth information to locate crack positions with high precision.

[0057] By using a large language model to generate natural language descriptions of crack features, and supporting users to ask questions about crack conditions, the model can provide inferential answers based on the detection results (such as whether the crack is continuous, possible causes, etc.), thereby improving the human-computer interaction and interpretability of the monitoring system.

[0058] S4. Obtain the training dataset; Based on the prompting optimization strategy and data augmentation method, perform three-stage optimization training on the large model of crack feature analysis to be trained according to the training dataset to obtain the large model of crack feature analysis.

[0059] Optionally, based on the prompting optimization strategy and data augmentation methods, the large-scale crack feature analysis model to be trained is subjected to three-stage optimization training according to the training dataset to obtain the large-scale crack feature analysis model, including:

[0060] Based on the large-scale point cloud-text pair dataset publicly available in the training dataset, cross-modal alignment optimization training is performed on the large model for crack feature analysis to be trained, and the first optimized large model is obtained.

[0061] Based on the prompting optimization strategy, the first optimized large model is semantically fine-tuned according to the publicly available crack detection dataset in the training dataset to obtain the second optimized large model;

[0062] Based on data augmentation methods, the second optimized large model is trained by following the instructions based on the pre-set crack analysis instruction dataset in the training dataset to obtain the crack feature analysis large model.

[0063] In one feasible implementation, the present invention employs a three-stage training strategy to gradually endow the model with cross-modal understanding and instruction reasoning capabilities. The first stage is cross-modal alignment training, which uses general large-scale point cloud-text pairs (partially generated through automatic annotation) to train the alignment between the image / point cloud encoder and the model, enabling the model to generate 3D feature representations corresponding to the language space.

[0064] The second stage is perception training, which involves adding labeled data for crack detection and supervising the model to enable it to output crack segmentation results (this stage is similar to the "perception" task training, ensuring that the model masters the specific semantic concept of crack).

[0065] The third stage is high-level instruction fine-tuning, which involves designing a set of instructions for crack analysis (such as querying crack size, determining crack type, etc.) to train the model to follow instructions, enabling the model to provide conversational answers to crack-related questions.

[0066] Through the above three-stage training, the model's semantic alignment, pattern awareness, and reasoning abilities are gradually enhanced.

[0067] In addition, data augmentation and suggestion optimization strategies were introduced during the training process: 3D modeling software was used to generate synthetic point cloud scenes with cracks, and GPT-4 and other technologies were used to generate corresponding descriptions to expand the training data.

[0068] Design a unified crack description template and a professional terminology dictionary to constrain the model output to use standardized engineering terms (such as using terms like "crack width" and "through crack" according to specifications) to improve the readability and professionalism of the output results.

[0069] The large-scale crack feature analysis model includes a visual Transformer encoder, a 3D sparse convolutional neural network, a position-aware Transformer module, a cross-modal attention transformer, and a large-scale crack semantic reasoning model.

[0070] In one feasible implementation, the present invention uses a LiDAR-LLM model with Bird's Eye View spatial embedding as the model basis. The image is processed by a Vision Transformer (ViT) encoder to extract two-dimensional texture features; the point cloud is first processed by a 3D sparse convolutional neural network (such as SparseConv3D or VoxelNet) to extract three-dimensional voxel features, and then compressed along the vertical axis to form a BEV feature plane with height distribution.

[0071] A Position-Aware Transformer (PAT) module is introduced to explicitly inject pre-constructed BEV mesh position codes into point cloud features. These features are then fused with image features via a cross-attention mechanism using a cross-modal attention transformer. This fusion Transformer outputs a series of visual feature vectors incorporating spatial semantics, which are projected into the word embedding space of a large-scale crack semantic reasoning model. This enables the pre-trained language model to "understand" the semantic information corresponding to the 3D crack features.

[0072] S5. Based on the two-dimensional image dataset and the three-dimensional point cloud dataset, use the crack feature analysis large model to perform cross-modal semantic parsing and obtain crack semantic recognition results.

[0073] Optionally, based on the two-dimensional image dataset and the three-dimensional point cloud dataset, a large-scale crack feature analysis model is used to perform cross-modal semantic parsing to obtain crack semantic recognition results, including:

[0074] Two-dimensional texture features are obtained by extracting features from a two-dimensional image dataset.

[0075] Feature extraction is performed on the 3D point cloud dataset to obtain 3D voxel features; feature height distribution compression along the vertical axis is performed on the 3D voxel features to obtain the bird's-eye view feature plane.

[0076] Based on the spatial location embedding of bird's-eye view from 6 preset perspectives, the feature plane of the bird's-eye view is explicitly injected with position encoding to obtain the feature plane of the bird's-eye view after injection.

[0077] Based on the cross-attention mechanism, feature fusion is performed on two-dimensional texture features and the feature plane of the injected bird's-eye view to obtain spatial-semantic fusion visual features;

[0078] Based on a preset prompt text template, the crack semantic reasoning is performed according to the spatial-semantic fusion visual features to obtain the crack semantic recognition result.

[0079] In one feasible implementation, the processed point cloud and corresponding image stitched together are used as input and fed into a large crack feature analysis model for analysis.

[0080] The model's image encoder extracts deep feature representations for each image, while the point cloud encoder compresses 500,000 points into a bird's-eye view feature map of approximately 180×180 pixels, with each pixel containing geometric features within its corresponding spatial grid. The position-aware Transformer module embeds the BEV positions from six pre-defined viewpoints into the point cloud features, enabling the model to recognize spatial information from different directions. During semantic alignment training, this embedding helps the model understand relative directional descriptions such as "left front" and "right rear," eliminating ambiguity in spatial descriptions. The fused multimodal feature sequence is fed into the large language model core (based on the LLaMA-7B model), and through several layers of self-attention mechanisms, the model ultimately generates an understanding of the input scene.

[0081] A pre-designed prompt text template guides the model to output the identification results of crack regions. For example, the model can be given instructions such as "Please identify the locations of all cracks in the point cloud," and after parsing, it outputs results representing the crack regions. The model generates the required output through adapter fine-tuning. One approach is to introduce a segmentation prediction head in the last layer of the crack semantic reasoning model to classify and label the point cloud voxels; another approach is to have the crack semantic reasoning model output the coordinate range or other encoding of the crack regions, which is then parsed by the system into a segmentation mask. This invention is not limited to a specific implementation method; any method that can obtain the identification results of crack regions from the LLM falls within the scope of this solution.

[0082] The model outputs the probability that each point belongs to either "crack" or "background." After threshold filtering, a set of crack points is obtained, resulting in a 3D point cloud segmentation result representing the area where the crack is located, i.e., marking which points are crack point clouds. From the above steps, a cluster of points belonging to the crack can be obtained in the pier point cloud. Based on its internal understanding of the crack concept, the large-scale crack semantic reasoning model also generates a text description: "A crack extending vertically was detected, with a height of approximately 0.22 m, starting at the bottom edge of the pier. The crack width increases from bottom to top." This description is inferred by the model based on the segmentation results and spatial information and can be used to assist in manual verification of the results.

[0083] S6. Based on the three-dimensional bounding box method, Hessian matrix eigenvalue analysis method and LaplacianContraction algorithm, according to the crack semantic recognition results and the three-dimensional point cloud dataset, the skeleton line is fitted by the curve fitting method to obtain the crack local point cloud subset dataset and the crack three-dimensional skeleton curve.

[0084] Optionally, based on the 3D bounding box method, Hessian matrix eigenvalue analysis method, and Laplacian Contraction algorithm, and according to the crack semantic recognition results and the 3D point cloud dataset, a curve fitting method is used to fit the skeleton line to obtain a local point cloud subset dataset of the crack and the 3D skeleton curve of the crack, including:

[0085] Based on the semantic recognition results of cracks, crack region data is extracted from the 3D point cloud dataset to obtain the crack point cloud dataset.

[0086] Based on the 3D bounding box method, the crack point cloud dataset is finely segmented to obtain a subset dataset of crack local point clouds.

[0087] A three-dimensional Euclidean distance field is constructed based on a subset of local point cloud data of the crack, and the node distance field values ​​of the three-dimensional Euclidean distance field are calculated.

[0088] Based on the node distance field value, and according to the three-dimensional Euclidean distance field, local maximum detection is used to screen crack skeleton points to obtain an initial skeleton point set;

[0089] Based on the Hessian matrix eigenvalue analysis method, ridge points are selected and connected according to the initial skeleton point set to obtain the initial skeleton path;

[0090] Based on the initial skeleton path, the Laplacian Contraction algorithm is used to perform boundary geometric shrinkage according to the crack local point cloud subset dataset to obtain the skeleton center point zone.

[0091] Based on the curve fitting method, the skeleton line is fitted according to the center point zone of the skeleton to obtain the three-dimensional skeleton curve of the crack.

[0092] In one feasible implementation, after obtaining the point cloud region of the crack, a detailed geometric morphology analysis is performed to extract the crack skeleton lines. This invention employs a method based on a three-dimensional distance field and geometric contraction to extract the skeleton.

[0093] A three-dimensional Euclidean distance field is constructed for the point cloud of the crack region, that is, the distance from each voxel in space to the nearest point on the crack surface is calculated. The distance values ​​of points inside the crack (at the crack opening) are larger, while the distance values ​​of points near the edge are smaller. Local maxima in the distance field often correspond to the center of the crack.

[0094] The distance field will form a "distance spine" inside the crack, that is, the region with the largest distance value. By extracting the contour surface of the distance field and analyzing the Hessian matrix, the points where the distance field gradient is zero and the curvature is maximum are found; these points correspond to the central axis of the crack. Connecting these points according to their spatial proximity yields a discontinuous initial path of the skeleton.

[0095] To smooth the skeleton path, the Laplacian Contraction algorithm was applied to iteratively shrink the point cloud boundary: initially, the crack point cloud was treated as a slender "mesh" structure, and the boundary was gradually tightened to approach the centerline. The topology remained unchanged during the shrinkage process to prevent path breakage. After the shrinkage algorithm was completed, the skeleton point positions were further fine-tuned to the extreme points of the distance field. Finally, these skeleton points were fitted with cubic spline curves to obtain a smooth and continuous skeleton line. The skeleton line starts at the crack initiation point at the bottom of the pier and ends at the crack's end. The model calculated the skeleton line length to be approximately 215.7 mm, which agrees well with the results obtained through visual inspection.

[0096] S7. Based on geometric statistical calculation methods and the α-Shape algorithm, crack features are quantified according to the crack local point cloud subset dataset and the crack 3D skeleton curve to obtain crack structured features; based on the crack structured features, semantic description is performed through a large crack feature analysis model to obtain feature description text.

[0097] Optionally, based on geometrical statistical calculation methods and the α-Shape algorithm, crack features are quantized according to a subset dataset of local crack point clouds and the three-dimensional skeleton curve of the crack to obtain the structured features of the crack, including:

[0098] Based on geometric statistical calculation methods and the α-Shape algorithm, crack features are quantified according to the crack local point cloud subset dataset and the crack three-dimensional skeleton curve to obtain crack structure features.

[0099] The spatial location range of the crack is obtained by performing 3D coordinate boundary statistics on a subset of the crack's local point cloud dataset.

[0100] Sampling point statistics are performed based on the three-dimensional skeleton curve of the crack to obtain skeleton point data;

[0101] The crack length is calculated based on the three-dimensional skeleton curve of the crack to obtain the total crack length;

[0102] The crack orientation vector is calculated based on the crack's three-dimensional skeleton curve to obtain the crack's orientation direction.

[0103] Based on the preset target structural parameters, the crack penetration status is obtained by judging the penetration based on the three-dimensional skeleton curve of the crack.

[0104] An orthogonal profile plane is constructed based on the normal direction of each point of the crack 3D skeleton curve to obtain a set of crack point cloud profiles.

[0105] Based on the crack point cloud profile set, the local point cloud subset dataset of the crack is projected into the corresponding profile, and the boundary points are extracted using the α-Shape algorithm to obtain the left and right boundary contours of the crack.

[0106] The crack width is calculated based on the left and right boundary contours of the crack, resulting in a sequence of width values ​​along the crack direction; crack feature statistics are then performed based on the width value sequence to obtain crack width statistics.

[0107] In one feasible implementation, an orthogonal plane is intercepted at regular intervals along the skeleton line (e.g., every 5 mm or adaptively based on the point cloud density), and the width of the crack within this plane is statistically analyzed. For a point P on the skeleton line, its tangent direction t is taken, and a pair of normal lines n1 and n2 are determined to form a local coordinate system together with the tangent. Under this coordinate system, a local plane (n1-n2 plane) is established with P as the origin. Points within a certain range (e.g., radius 20 mm) from P in the crack point cloud are transformed to this local coordinate system and projected onto the plane. The result is a series of projected points on the two-dimensional plane, which roughly form two point bands corresponding to the left and right edges of the crack.

[0108] The left and right boundary contours of the crack within the plane are obtained using the α-Shape algorithm or convex hull-based edge detection. The maximum distance between the left and right boundaries is calculated as the crack width at that cross-section. By repeating the above process at various locations along the skeleton line, a sequence of crack width values ​​along the length direction is obtained. The maximum, minimum, and average values ​​of these cross-sectional width values ​​are calculated to obtain the statistics of the crack opening width.

[0109] In this invention, the maximum width of the vertical crack in the bridge pier was calculated to be approximately 4.3 mm, and the average width was approximately 2.1 mm. Plotting the width as a function of the frame height shows that the width first increases and then decreases along the height direction. The width reaches its peak in the middle of the crack and gradually narrows towards the end. This width distribution analysis suggests that stress concentration may exist near the middle of the crack, causing it to open wider, while the crack does not extend to the full height of the structure (the width tends to decrease again at the top).

[0110] Based on the above analysis results, a structured output of the crack characteristics of the reinforced concrete piers of an elevated bridge in a certain city is generated. The output feature description text can be formatted as a table or data file, including the following:

[0111] Spatial location range: The three-dimensional coordinate bounding box of the area where the crack is located, for example, using ( , ), ( , ), ( , The form represents the extent of the crack's coverage in the structural coordinate system. In the actual detection of this invention, the crack was mainly located in the middle of the bridge pier, with a height of... The range is approximately 2.3–2.5 m. The directional dimension is very small.

[0112] Total length: The length of the crack space calculated along the skeleton line is 215.7 mm.

[0113] Width range: minimum, maximum and average crack opening width, 1.2 mm–4.3 mm (average 2.1 mm).

[0114] Number of skeleton points: The number of sampling points on the skeleton curve, which to some extent reflects the curvature of the skeleton curve or the extraction resolution. In this invention, it is 38 points.

[0115] Orientation: The approximate spatial orientation of the crack skeleton, represented by a direction vector or inclination angle. The actual crack skeleton line is nearly vertical, with a direction vector of approximately (-0.06, 0.95, -0.29), indicating that the crack mainly extends upwards in a vertical direction with a slight inclination.

[0116] Whether the crack is continuous: This is determined based on whether the crack extends from one side of the structure to the other. Since the crack length in a reinforced concrete pier is limited and does not extend through the entire thickness of the component, it is judged as "no".

[0117] The calculated crack parameters are summarized, and the crack characteristics are described using natural language to generate a detection report. The report includes a crack location diagram (the crack point cloud can be highlighted and overlaid on the 3D structural model or 2D image), a crack skeleton and width distribution map, and a structured parameter list.

[0118] The report outputs a text description of the bridge pier crack characteristics based on the aforementioned measurements. The report also includes a text description generated by the model: "A thin, vertically developing crack appeared in the middle of the pier, approximately 0.22m long and 4.3mm wide at its maximum. The crack gradually narrows from bottom to top, and its end does not reach the top edge of the pier. Based on the current width and length, the crack has not yet seriously affected the structural bearing capacity, but its development needs further monitoring." These descriptions are automatically generated by the model based on the detection data, using professional language that conforms to engineering conventions (the model has been fine-tuned with a specialized corpus to master the use of terms such as "vertical crack" and "not continuous").

[0119] S8. Obtain crack trend query text; Based on crack trend query text, according to crack structured features and feature description text, use crack feature analysis big model to perform time series comparison reasoning to obtain crack development trend prediction results.

[0120] In one feasible implementation, the present invention supports multi-time-based historical data comparison and trend prediction: by collecting data from multiple inspections, the current crack characteristics can be output while automatically calculating changes by comparing with previous data. Comparing the current inspection results with the periodic inspection record from 6 months ago reveals that the crack length has increased by approximately 12% compared to the previous instance, while the width has changed by less than 0.5 mm.

[0121] The comprehensive analysis of the large-scale crack characteristic analysis model yielded the following prediction: "If environmental and load conditions remain unchanged, the crack length is expected to continue to increase by about 10% in the next six months. Regular monitoring and grouting repair as needed are recommended." This prediction provides valuable reference for maintenance decisions. The entire process is automated, fully demonstrating the application effect of this invention in practical engineering inspection. In different engineering applications, this invention can achieve similar results simply by fine-tuning the model and setting parameters according to the specific sensor configuration and target structural characteristics, demonstrating broad application prospects.

[0122] This invention proposes a crack feature extraction and quantization method based on a point cloud big language model. It utilizes point cloud data to completely preserve the spatial geometric information of cracks, avoiding the position and shape distortion caused by the lack of depth in two-dimensional image methods. By deeply fusing image texture and point cloud geometry through a multimodal big language model, it endows the results with rich semantic labels on the basis of obtaining fine 3D reconstruction, thus achieving the unity of "geometric accuracy" and "semantic understanding" in crack detection.

[0123] A skeleton-section combined crack morphology representation method is proposed. The skeleton line extracts and summarizes the overall direction and penetration of the crack, while the width section quantifies the change in crack opening along the length. This multi-level representation can comprehensively depict the crack morphology characteristics, help engineers understand the development state of cracks, and provide a basis for structural stress analysis.

[0124] This invention can output quantitative indicators such as the length, width, and spatial location of cracks with one click, as well as inferences based on these indicators (such as whether the crack is continuous or not). Among them, the synchronous quantification of width and location overcomes the problem of information asynchrony caused by traditional measurements in two-dimensional and three-dimensional dimensions, and realizes synchronous automatic quantification of crack location and size. The high-precision geometric quantification makes crack assessment more objective and reliable, and also facilitates the subsequent use of data for maintenance decisions.

[0125] Introducing a large language model enables the system to not only provide data but also "interpretations." The model can describe crack characteristics using technical terminology and answer user questions about cracks (such as "Does this crack need repair?" "What might have caused the crack?"). This question-and-answer human-computer interaction greatly improves the system's intelligence, allowing non-technical users to obtain useful information through natural language questions; the explanatory text output by the model aids in understanding the results and writing reports.

[0126] This invention is designed to meet the crack detection needs of various civil engineering concrete structures. With appropriate training and optimization, it can be widely applied to different scenarios such as bridge beams, bridge piers, tunnel linings, subway segments, and building walls. Due to the adoption of the cross-modal transfer capability of a large model, it has good generalization ability to new scenarios, reducing the workload of re-annotation and training for each engineering instance. Through the multi-temporal data monitoring and predictive analysis module, this invention can be seamlessly extended for long-term health monitoring, providing decision support for digital twins and structural maintenance.

[0127] This invention significantly improves upon existing crack detection methods in terms of spatial accuracy, information richness, and intelligence, possessing high engineering application value and effectively ensuring the service safety and durability of major engineering structures. This invention is a high-precision and highly interpretable crack feature extraction and quantification method that combines 3D point cloud data processing with a large language model.

[0128] Figure 2 This is a block diagram of a crack feature extraction and quantization device based on a point cloud large language model, provided in an embodiment of the present invention. This device is used for crack feature extraction and quantization based on a point cloud large language model. (Refer to...) Figure 2 The device includes a crack data acquisition module 210, a data preprocessing module 220, an analysis model construction module 230, an analysis model training module 240, a crack semantic recognition module 250, a crack skeleton extraction module 260, a crack feature quantification module 270, and a crack evolution analysis module 280. Among them:

[0129] The crack data acquisition module 210 is used to acquire multi-time and multi-view data of the target structure by using a drone equipped with a binocular camera and a lidar to obtain the original two-dimensional image dataset, the original three-dimensional point cloud dataset and the drone pose information set.

[0130] The data preprocessing module 220 is used to preprocess the original two-dimensional image dataset and the original three-dimensional point cloud dataset based on the preset improved LiDAR camera extrinsic calibration method and the UAV pose information set to obtain the two-dimensional image dataset and the three-dimensional point cloud dataset.

[0131] The analysis model building module 230 is used to build a large model for crack feature analysis to be trained based on the model structure of the point cloud large language model;

[0132] The analysis model training module 240 is used to obtain the training dataset; based on the prompting optimization strategy and data augmentation method, the large crack feature analysis model to be trained is optimized and trained in three stages according to the training dataset to obtain the large crack feature analysis model.

[0133] The crack semantic recognition module 250 is used to perform cross-modal semantic parsing based on a two-dimensional image dataset and a three-dimensional point cloud dataset, using a large crack feature analysis model to obtain crack semantic recognition results.

[0134] The crack skeleton extraction module 260 is used to perform skeleton line fitting based on the crack semantic recognition results and the 3D point cloud dataset, using the 3D bounding box method, Hessian matrix eigenvalue analysis method and Laplacian Contraction algorithm, to obtain the crack local point cloud subset dataset and crack 3D skeleton curve.

[0135] The crack feature quantization module 270 is used to quantify crack features based on geometric statistical calculation methods and the α-Shape algorithm, according to the crack local point cloud subset dataset and the crack 3D skeleton curve, to obtain crack structured features; based on the crack structured features, semantic description is performed through the crack feature analysis large model to obtain feature description text.

[0136] The crack evolution analysis module 280 is used to obtain crack trend query text; based on the crack trend query text, according to the crack structured features and feature description text, the crack feature analysis big model is used to perform time-series comparative reasoning to obtain crack development trend prediction results.

[0137] Optionally, the crack data acquisition module 210 is further used for:

[0138] Based on the UAV pose information set, preliminary data alignment is performed on the original two-dimensional image dataset and the original three-dimensional point cloud dataset to obtain the first two-dimensional image dataset and the first three-dimensional point cloud dataset.

[0139] Based on the preset improved LiDAR camera extrinsic calibration method, the extrinsic parameters are reprojected and aligned according to the first two-dimensional image dataset and the first three-dimensional point cloud dataset to obtain the second two-dimensional image dataset and the second three-dimensional point cloud dataset.

[0140] The second 3D point cloud dataset is cleaned to obtain a third 3D point cloud dataset containing only the surface of the target structure; the third 3D point cloud dataset is downsampled to obtain a fourth 3D point cloud dataset.

[0141] The second two-dimensional image dataset is defined as two-dimensional image data; the fourth three-dimensional point cloud data is defined as a three-dimensional point cloud dataset.

[0142] Optionally, the analysis model training module 240 is further used for:

[0143] Based on the large-scale point cloud-text pair dataset publicly available in the training dataset, cross-modal alignment optimization training is performed on the large model for crack feature analysis to be trained, and the first optimized large model is obtained.

[0144] Based on the prompting optimization strategy, the first optimized large model is semantically fine-tuned according to the publicly available crack detection dataset in the training dataset to obtain the second optimized large model;

[0145] Based on data augmentation methods, the second optimized large model is trained by following the instructions based on the pre-set crack analysis instruction dataset in the training dataset to obtain the crack feature analysis large model.

[0146] The large-scale crack feature analysis model includes a visual Transformer encoder, a 3D sparse convolutional neural network, a position-aware Transformer module, a cross-modal attention transformer, and a large-scale crack semantic reasoning model.

[0147] Optionally, the crack semantic recognition module 250 is further used for:

[0148] Two-dimensional texture features are obtained by extracting features from a two-dimensional image dataset.

[0149] Feature extraction is performed on the 3D point cloud dataset to obtain 3D voxel features; feature height distribution compression along the vertical axis is performed on the 3D voxel features to obtain the bird's-eye view feature plane.

[0150] Based on the spatial location embedding of bird's-eye view from 6 preset perspectives, the feature plane of the bird's-eye view is explicitly injected with position encoding to obtain the feature plane of the bird's-eye view after injection.

[0151] Based on the cross-attention mechanism, feature fusion is performed on two-dimensional texture features and the feature plane of the injected bird's-eye view to obtain spatial-semantic fusion visual features;

[0152] Based on a preset prompt text template, the crack semantic reasoning is performed according to the spatial-semantic fusion visual features to obtain the crack semantic recognition result.

[0153] Optionally, the crack skeleton extraction module 260 is further used for:

[0154] Based on the semantic recognition results of cracks, crack region data is extracted from the 3D point cloud dataset to obtain the crack point cloud dataset.

[0155] Based on the 3D bounding box method, the crack point cloud dataset is finely segmented to obtain a subset dataset of crack local point clouds.

[0156] A three-dimensional Euclidean distance field is constructed based on a subset of local point cloud data of the crack, and the node distance field values ​​of the three-dimensional Euclidean distance field are calculated.

[0157] Based on the node distance field value, and according to the three-dimensional Euclidean distance field, local maximum detection is used to screen crack skeleton points to obtain an initial skeleton point set;

[0158] Based on the Hessian matrix eigenvalue analysis method, ridge points are selected and connected according to the initial skeleton point set to obtain the initial skeleton path;

[0159] Based on the initial skeleton path, the Laplacian Contraction algorithm is used to perform boundary geometric shrinkage according to the crack local point cloud subset dataset to obtain the skeleton center point zone.

[0160] Based on the curve fitting method, the skeleton line is fitted according to the center point zone of the skeleton to obtain the three-dimensional skeleton curve of the crack.

[0161] Optionally, the crack feature quantization module 270 is further used for:

[0162] Based on geometric statistical calculation methods and the α-Shape algorithm, crack features are quantified according to the crack local point cloud subset dataset and the crack three-dimensional skeleton curve to obtain crack structure features.

[0163] The spatial location range of the crack is obtained by performing 3D coordinate boundary statistics on a subset of the crack's local point cloud dataset.

[0164] Sampling point statistics are performed based on the three-dimensional skeleton curve of the crack to obtain skeleton point data;

[0165] The crack length is calculated based on the three-dimensional skeleton curve of the crack to obtain the total crack length;

[0166] The crack orientation vector is calculated based on the crack's three-dimensional skeleton curve to obtain the crack's orientation direction.

[0167] Based on the preset target structural parameters, the crack penetration status is obtained by judging the penetration based on the three-dimensional skeleton curve of the crack.

[0168] An orthogonal profile plane is constructed based on the normal direction of each point of the crack 3D skeleton curve to obtain a set of crack point cloud profiles.

[0169] Based on the crack point cloud profile set, the local point cloud subset dataset of the crack is projected into the corresponding profile, and the boundary points are extracted using the α-Shape algorithm to obtain the left and right boundary contours of the crack.

[0170] The crack width is calculated based on the left and right boundary contours of the crack, resulting in a sequence of width values ​​along the crack direction; crack feature statistics are then performed based on the width value sequence to obtain crack width statistics.

[0171] This invention proposes a crack feature extraction and quantization method based on a point cloud big language model. It utilizes point cloud data to completely preserve the spatial geometric information of cracks, avoiding the position and shape distortion caused by the lack of depth in two-dimensional image methods. By deeply fusing image texture and point cloud geometry through a multimodal big language model, it endows the results with rich semantic labels on the basis of obtaining fine 3D reconstruction, thus achieving the unity of "geometric accuracy" and "semantic understanding" in crack detection.

[0172] A skeleton-section combined crack morphology representation method is proposed. The skeleton line extracts and summarizes the overall direction and penetration of the crack, while the width section quantifies the change in crack opening along the length. This multi-level representation can comprehensively depict the crack morphology characteristics, help engineers understand the development state of cracks, and provide a basis for structural stress analysis.

[0173] This invention can output quantitative indicators such as the length, width, and spatial location of cracks with one click, as well as inferences based on these indicators (such as whether the crack is continuous or not). Among them, the synchronous quantification of width and location overcomes the problem of information asynchrony caused by traditional measurements in two-dimensional and three-dimensional dimensions, and realizes synchronous automatic quantification of crack location and size. The high-precision geometric quantification makes crack assessment more objective and reliable, and also facilitates the subsequent use of data for maintenance decisions.

[0174] Introducing a large language model enables the system to not only provide data but also "interpretations." The model can describe crack characteristics using technical terminology and answer user questions about cracks (such as "Does this crack need repair?" "What might have caused the crack?"). This question-and-answer human-computer interaction greatly improves the system's intelligence, allowing non-technical users to obtain useful information through natural language questions; the explanatory text output by the model aids in understanding the results and writing reports.

[0175] This invention is designed to meet the crack detection needs of various civil engineering concrete structures. With appropriate training and optimization, it can be widely applied to different scenarios such as bridge beams, bridge piers, tunnel linings, subway segments, and building walls. Due to the adoption of the cross-modal transfer capability of a large model, it has good generalization ability to new scenarios, reducing the workload of re-annotation and training for each engineering instance. Through the multi-temporal data monitoring and predictive analysis module, this invention can be seamlessly extended for long-term health monitoring, providing decision support for digital twins and structural maintenance.

[0176] This invention significantly improves upon existing crack detection methods in terms of spatial accuracy, information richness, and intelligence, possessing high engineering application value and effectively ensuring the service safety and durability of major engineering structures. This invention is a high-precision and highly interpretable crack feature extraction and quantification method that combines 3D point cloud data processing with a large language model.

[0177] Figure 3 This is a schematic diagram of the structure of a crack feature extraction and quantification device provided in an embodiment of the present invention, as shown below. Figure 3 As shown, the crack feature extraction and quantification device may include the above-mentioned Figure 2 The illustrated crack feature extraction and quantization device is based on a point cloud large language model. Optionally, the crack feature extraction and quantization device 310 may include a first processor 2001.

[0178] Optionally, the crack feature extraction and quantization device 310 may also include a memory 2002 and a transceiver 2003.

[0179] The first processor 2001, memory 2002, and transceiver 2003 can be connected via a communication bus.

[0180] The following is combined with Figure 3 A detailed introduction to each component of the crack feature extraction and quantification device 310 is provided below:

[0181] The first processor 2001 is the control center of the crack feature extraction and quantization device 310. It can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0182] Optionally, the first processor 2001 can perform various functions of the crack feature extraction and quantization device 310 by running or executing software programs stored in the memory 2002 and calling data stored in the memory 2002.

[0183] In a specific implementation, as one example, the first processor 2001 may include one or more CPUs, for example... Figure 3 CPU0 and CPU1 are shown in the diagram.

[0184] In a specific implementation, as one example, the crack feature extraction and quantization device 310 may also include multiple processors, for example... Figure 3The first processor 2001 and the second processor 2004 are shown in the diagram. Each of these processors can be a single-core processor or a multi-core processor. Here, a processor can refer to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).

[0185] The memory 2002 is used to store the software program that executes the present invention, and is controlled by the first processor 2001 to execute it. The specific implementation method can be referred to the above method embodiment, and will not be repeated here.

[0186] Optionally, the memory 2002 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory 2002 may be integrated with the first processor 2001 or may exist independently, and may be connected via the interface circuit of the crack feature extraction and quantization device 310. Figure 3 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0187] The transceiver 2003 is used to communicate with network devices or with terminal devices.

[0188] Alternatively, transceiver 2003 may include a receiver and a transmitter. Figure 3 (Not shown separately). The receiver is used to implement the receiving function, and the transmitter is used to implement the transmitting function.

[0189] Optionally, the transceiver 2003 can be integrated with the first processor 2001 or exist independently, and can be connected to the interface circuit of the crack feature extraction and quantization device 310. Figure 3 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0190] It should be noted that, Figure 3 The structure of the crack feature extraction and quantization device 310 shown in the figure does not constitute a limitation on the router. The actual crack feature extraction and quantization device may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0191] Furthermore, the technical effect of the crack feature extraction and quantization device 310 can be referred to the technical effect of the crack feature extraction and quantization method based on the point cloud large language model described in the above method embodiments, and will not be repeated here.

[0192] It should be understood that the first processor 2001 in the embodiments of the present invention may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor, or it may be any conventional processor, etc.

[0193] It should also be understood that the memory in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0194] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0195] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0196] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0197] It should be understood that, in various embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0198] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0199] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0200] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0201] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0202] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0203] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0204] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for crack feature extraction and quantization based on a point cloud large language model, characterized in that, The method includes: By using a drone equipped with a binocular camera and a lidar, multi-time and multi-view data acquisition is performed on the target structure to obtain the original two-dimensional image dataset, the original three-dimensional point cloud dataset, and the drone pose information set. Based on the preset improved LiDAR camera extrinsic calibration method, the original two-dimensional image dataset and the original three-dimensional point cloud dataset are preprocessed according to the UAV pose information set to obtain the two-dimensional image dataset and the three-dimensional point cloud dataset. Construct a large model for crack feature analysis to be trained based on the model structure of the point cloud large language model; Obtain the training dataset; based on the prompting optimization strategy and data augmentation method, perform three-stage optimization training on the large model of crack feature analysis to be trained according to the training dataset to obtain the large model of crack feature analysis. Based on two-dimensional image datasets and three-dimensional point cloud datasets, a large crack feature analysis model is used to perform cross-modal semantic parsing to obtain crack semantic recognition results. Based on the 3D bounding box method, Hessian matrix eigenvalue analysis method and Laplacian Contraction algorithm, according to the crack semantic recognition results and 3D point cloud dataset, the skeleton line is fitted by the curve fitting method to obtain the crack local point cloud subset dataset and crack 3D skeleton curve. Based on geometric statistical calculation methods and the α-Shape algorithm, crack features are quantified using a subset dataset of crack local point clouds and the crack's 3D skeleton curve to obtain crack structured features. Based on these crack structured features, a large crack feature analysis model is used for semantic description to obtain feature description text. Obtain the crack trend query text; based on the crack trend query text, and according to the crack structure characteristics and feature description text, use the crack feature analysis big model to perform time series comparison and inference to obtain the crack development trend prediction results.

2. The crack feature extraction and quantization method based on a point cloud large language model according to claim 1, characterized in that, The improved LiDAR camera extrinsic calibration method based on a preset method preprocesses the original 2D image dataset and the original 3D point cloud dataset according to the UAV pose information set to obtain the 2D image dataset and the 3D point cloud dataset, including: Based on the UAV pose information set, preliminary data alignment is performed on the original two-dimensional image dataset and the original three-dimensional point cloud dataset to obtain the first two-dimensional image dataset and the first three-dimensional point cloud dataset. Based on the preset improved LiDAR camera extrinsic calibration method, the extrinsic parameters are reprojected and aligned according to the first two-dimensional image dataset and the first three-dimensional point cloud dataset to obtain the second two-dimensional image dataset and the second three-dimensional point cloud dataset. The second 3D point cloud dataset is cleaned to obtain a third 3D point cloud dataset containing only the surface of the target structure; the third 3D point cloud dataset is downsampled to obtain a fourth 3D point cloud dataset. The second two-dimensional image dataset is defined as two-dimensional image data; the fourth three-dimensional point cloud data is defined as a three-dimensional point cloud dataset.

3. The crack feature extraction and quantization method based on a point cloud large language model according to claim 1, characterized in that, The aforementioned suggestion-based optimization strategy and data augmentation method, based on the training dataset, perform a three-stage optimization training on the large-scale crack feature analysis model to obtain the large-scale crack feature analysis model, including: Based on the large-scale point cloud-text pair dataset publicly available in the training dataset, cross-modal alignment optimization training is performed on the large model for crack feature analysis to be trained, and the first optimized large model is obtained. Based on the prompting optimization strategy, the first optimized large model is semantically fine-tuned according to the publicly available crack detection dataset in the training dataset to obtain the second optimized large model; Based on data augmentation methods, the second optimized large model is trained by following the instructions based on the pre-set crack analysis instruction dataset in the training dataset to obtain the crack feature analysis large model.

4. The crack feature extraction and quantization method based on a point cloud large language model according to claim 1, characterized in that, The large-scale crack feature analysis model includes a visual Transformer encoder, a 3D sparse convolutional neural network, a position-aware Transformer module, a cross-modal attention transformer, and a large-scale crack semantic reasoning model.

5. The crack feature extraction and quantization method based on a point cloud large language model according to claim 1, characterized in that, The step involves using a large-scale crack feature analysis model to perform cross-modal semantic parsing based on a two-dimensional image dataset and a three-dimensional point cloud dataset to obtain crack semantic recognition results, including: Two-dimensional texture features are obtained by extracting features from a two-dimensional image dataset. Feature extraction is performed on the 3D point cloud dataset to obtain 3D voxel features; feature height distribution compression along the vertical axis is performed on the 3D voxel features to obtain the bird's-eye view feature plane. Based on the spatial location embedding of bird's-eye view from 6 preset perspectives, the feature plane of the bird's-eye view is explicitly injected with position encoding to obtain the feature plane of the bird's-eye view after injection. Based on the cross-attention mechanism, feature fusion is performed on two-dimensional texture features and the feature plane of the injected bird's-eye view to obtain spatial-semantic fusion visual features; Based on a preset prompt text template, the crack semantic reasoning is performed according to the spatial-semantic fusion visual features to obtain the crack semantic recognition result.

6. The crack feature extraction and quantization method based on a point cloud large language model according to claim 1, characterized in that, The method based on the 3D bounding box method, Hessian matrix eigenvalue analysis method, and Laplacian Contraction algorithm, according to the crack semantic recognition results and the 3D point cloud dataset, uses a curve fitting method to fit the skeleton line, obtaining a local point cloud subset dataset of the crack and the 3D skeleton curve of the crack, including: Based on the semantic recognition results of cracks, crack region data is extracted from the 3D point cloud dataset to obtain the crack point cloud dataset. Based on the 3D bounding box method, the crack point cloud dataset is finely segmented to obtain a subset dataset of crack local point clouds. A three-dimensional Euclidean distance field is constructed based on a subset of local point cloud data of the crack, and the node distance field values ​​of the three-dimensional Euclidean distance field are calculated. Based on the node distance field value, and according to the three-dimensional Euclidean distance field, local maximum detection is used to screen crack skeleton points to obtain an initial skeleton point set; Based on the Hessian matrix eigenvalue analysis method, ridge points are selected and connected according to the initial skeleton point set to obtain the initial skeleton path; Based on the initial skeleton path, the Laplacian Contraction algorithm is used to perform boundary geometric shrinkage according to the crack local point cloud subset dataset to obtain the skeleton center point zone. Based on the curve fitting method, the skeleton line is fitted according to the center point zone of the skeleton to obtain the three-dimensional skeleton curve of the crack.

7. The crack feature extraction and quantization method based on a point cloud large language model according to claim 1, characterized in that, The method based on geometric statistical calculation and the α-Shape algorithm quantifies crack features based on a subset dataset of local point clouds and the 3D skeleton curve of the crack, obtaining structured crack features, including: Based on geometric statistical calculation methods and the α-Shape algorithm, crack features are quantified according to the crack local point cloud subset dataset and the crack three-dimensional skeleton curve to obtain crack structure features. The spatial location range of the crack is obtained by performing 3D coordinate boundary statistics on a subset of the crack's local point cloud dataset. Sampling point statistics are performed based on the three-dimensional skeleton curve of the crack to obtain skeleton point data; The crack length is calculated based on the three-dimensional skeleton curve of the crack to obtain the total crack length; The crack orientation vector is calculated based on the crack's three-dimensional skeleton curve to obtain the crack's orientation direction. Based on the preset target structural parameters, the crack penetration status is obtained by judging the penetration based on the three-dimensional skeleton curve of the crack. An orthogonal profile plane is constructed based on the normal direction of each point of the crack 3D skeleton curve to obtain a set of crack point cloud profiles. Based on the crack point cloud profile set, the local point cloud subset dataset of the crack is projected into the corresponding profile, and the boundary points are extracted using the α-Shape algorithm to obtain the left and right boundary contours of the crack. The crack width is calculated based on the left and right boundary contours of the crack, resulting in a sequence of width values ​​along the crack direction; crack feature statistics are then performed based on the width value sequence to obtain crack width statistics.

8. A crack feature extraction and quantization device based on a point cloud large language model, wherein the crack feature extraction and quantization device based on a point cloud large language model is used to implement the crack feature extraction and quantization method based on a point cloud large language model as described in any one of claims 1-7, characterized in that, The device includes: The crack data acquisition module is used to collect multi-time and multi-view data of the target structure by using a drone equipped with a binocular camera and a lidar to obtain the original two-dimensional image dataset, the original three-dimensional point cloud dataset, and the drone pose information set. The data preprocessing module is used to preprocess the original two-dimensional image dataset and the original three-dimensional point cloud dataset based on the preset improved LiDAR camera extrinsic calibration method and the UAV pose information set to obtain the two-dimensional image dataset and the three-dimensional point cloud dataset. The analysis model building module is used to build a large model for crack feature analysis to be trained based on the model structure of the point cloud large language model; The analysis model training module is used to obtain the training dataset; based on the prompting optimization strategy and data augmentation method, the large crack feature analysis model to be trained is optimized and trained in three stages according to the training dataset to obtain the large crack feature analysis model. The crack semantic recognition module is used to perform cross-modal semantic parsing based on a two-dimensional image dataset and a three-dimensional point cloud dataset, using a large crack feature analysis model to obtain crack semantic recognition results. The crack skeleton extraction module is used to perform skeleton line fitting based on the crack semantic recognition results and the 3D point cloud dataset, using the 3D bounding box method, Hessian matrix eigenvalue analysis method and Laplacian Contraction algorithm, to obtain the crack local point cloud subset dataset and crack 3D skeleton curve. The crack feature quantization module is used to quantify crack features based on geometric statistical calculation methods and the α-Shape algorithm, using a subset dataset of local crack point clouds and the 3D skeleton curve of the crack to obtain the structured features of the crack. Based on the structured features of the crack, a large crack feature analysis model is used to perform semantic description to obtain the feature description text. The crack evolution analysis module is used to obtain crack trend query text; based on the crack trend query text, according to the crack structured features and feature description text, the crack feature analysis big model is used to perform time-series comparative reasoning to obtain crack development trend prediction results.

9. A device for crack feature extraction and quantification, characterized in that, The crack feature extraction and quantification device includes: processor; A memory storing computer-readable instructions that, when executed by the processor, implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains program code that can be invoked by a processor to execute the method as described in any one of claims 1 to 7.

Images