Intelligent geometric reasoning and semantic understanding method based on three-dimensional large language model

Abstract

The invention discloses an intelligent geometric reasoning and semantic understanding method based on a three-dimensional large language model, which comprises the following steps of: acquiring point cloud data of a building component through three-dimensional scanning equipment, associating text information, and constructing a multi-modal three-dimensional large language model comprising a geometric perception coding module, a context semantic understanding module and a parameter efficient fine tuning module; a cross-modal contrast loss and task instruction fine tuning strategy is adopted in model training, and finally semantic recognition, attribute completion and historical background analysis results of the building components are output. The method is suitable for building heritage digital protection, intelligent building process monitoring and three-dimensional digital archive management, component function recognition precision and cultural semantic mining capability in a complex scene can be improved, and real-time semantic updating and interactive response of a dynamic construction environment are supported.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology. More specifically, the present invention relates to an intelligent geometric reasoning and semantic understanding method based on a three-dimensional large language model. Background Art

[0002] With the widespread application of emerging technologies such as Building Information Modeling (BIM), City Information Modeling (CIM), and digital twins in the fields of architecture and urban management, the expression and processing capabilities of spatial data have ushered in unprecedented development opportunities. Three-dimensional point clouds, as a highly accurate, non-contact data collection method, have been widely used in various scenarios, including intelligent construction, architectural heritage preservation, and digital reconstruction, due to their ability to fully record the shape, structure, and spatial relationships of buildings. Three-dimensional point clouds not only enhance the visualization and digitization of physical building structures, but also provide key support for structural monitoring, construction assistance, and virtual restoration. In the field of architectural heritage preservation, the introduction of 3D point cloud technology has fundamentally transformed traditional surveying and archival recording methods, enabling the complete preservation of the geometric information of historical buildings. However, geometric information alone cannot meet the deep-seated needs of heritage preservation for historical value, cultural connotations, and semantic logic. Traditional point cloud analysis methods usually focus on geometric feature extraction and component reconstruction, relying on manual annotation and rule-driven algorithms for semantic segmentation, making it difficult to achieve component function identification, structural logic reasoning, and cultural context understanding. In particular, when faced with large-scale, complex scenes of heritage buildings, existing technologies have significant limitations in processing efficiency and analysis accuracy. In the field of intelligent construction, with the increasing demand for real-time perception and status recognition of construction scenes, traditional rule-based point cloud processing processes have become difficult to meet intelligent needs such as automatic identification of construction components, status updates, and semantic monitoring. Current component recognition methods are mostly based on low-dimensional geometric features or shallow neural networks for classification and positioning, lacking the ability to deeply model the functional semantics of components. This results in low recognition accuracy and insufficient generalization capabilities in dynamic construction or complex environments, which seriously restricts the scalability of intelligent construction systems.

[0003] Furthermore, existing point cloud understanding methods are mostly limited to single-modal information processing, such as using only geometric or image data for component identification. They lack mechanisms for integrating heterogeneous information from multiple sources, such as architectural drawings, historical texts, and regulatory standards. This makes it difficult to provide comprehensive analysis capabilities that balance geometric accuracy and semantic depth for building lifecycle management. Especially in multidimensional knowledge scenarios involving architectural heritage, these models are unable to effectively understand the historical context, artistic style, and cultural connotations underlying architectural components, significantly reducing the value of data-based preservation and reuse. The rapid development of artificial intelligence (AI) technology in recent years, particularly the rise of large language models (LLMs) and multimodal learning techniques, has provided new avenues for semantic understanding and geometric reasoning of architectural point cloud data. Large language models possess powerful language understanding and generation capabilities, capable of processing massive amounts of language information, such as historical documents, drawing descriptions, and regulatory texts, to extract semantic elements related to architectural components. Multimodal learning technology can jointly model and semantically fuse multiple modal data such as point clouds, images and text, realize cross-modal information collaboration and knowledge reasoning, and provide a basis for building intelligent systems with understanding, generalization and reasoning capabilities.

[0004] Based on this, an intelligent geometric reasoning and semantic understanding method is developed for three-dimensional point cloud data, integrating large language models and multimodal learning capabilities. It can not only realize the efficient identification and spatial structure analysis of building components, but also deeply explore their cultural, historical and functional semantics, and improve the intelligent level of architectural heritage protection. At the same time, it also provides general and scalable technical solutions for related fields such as intelligent construction, operation and maintenance management, and digital archives, which has broad technical prospects and application value. Summary of the Invention

[0005] An object of the present invention is to solve at least the above problems and to provide at least the advantages which will be described hereinafter.

[0006] Another object of the present invention is to provide an intelligent geometric reasoning and semantic understanding method based on a three-dimensional large language model, which solves the technical defects of existing three-dimensional point cloud analysis methods that rely on manual annotation and rule-driven algorithms, making it difficult to achieve functional identification, structural logic reasoning and cultural semantic understanding of components in complex architectural scenes, and lacking multimodal information fusion capabilities, resulting in insufficient real-time semantic updates and interactive responses in dynamic construction environments; the present invention is aimed at practical scenarios such as intelligent construction, digital protection of architectural heritage, intelligent analysis of construction processes, semantic recognition of spatial components and natural language interaction systems, and aims to break through key bottlenecks in traditional three-dimensional semantic understanding, such as low geometric understanding accuracy, weak cross-modal alignment capabilities, and insufficient interactive intelligence, and comprehensively improve semantic modeling and intelligent response capabilities in complex three-dimensional scenes.

[0007] In order to achieve these purposes and other advantages according to the present invention, a method for intelligent geometric reasoning and semantic understanding based on a three-dimensional large language model is provided, which includes the following steps: Step 1: Collect original 3D point cloud data of building components through 3D scanning equipment; generate text data based on the original 3D point cloud data; Step 2: Preprocess the original 3D point cloud data; Step 3: Build a multimodal 3D large language model, which includes a geometric perception encoding module, a contextual semantic understanding module, and a parameter efficient fine-tuning module; Among them, the geometric perception encoding module calculates geometric features through local density and global center distance, constructs a shared semantic embedding space and a cross-modal cross-attention module; The contextual semantic understanding module is based on a bidirectional transformer encoder and is pre-trained with a masked language model and the next sentence prediction task; The parameter-efficient fine-tuning module combines low-rank adaptation with structured hint vectors to configure an incremental model update mechanism; Step 4: Use the preprocessed 3D point cloud data obtained in step 2 and the text data obtained in step 1 as training sample sets to train and optimize the multimodal 3D large language model; Step 5: Input the 3D point cloud data and text data to be tested into the trained and optimized multimodal 3D large language model, and output the semantic recognition, attribute completion and historical background analysis results of the building components.

[0008] Preferably, the intelligent geometric reasoning and semantic understanding method based on the three-dimensional large language model, step 2 is specifically: De-noising, standardization and segmentation of original 3D point cloud data, extracting the geometric features of each building component and assigning a unique attribute ID; The uniform downsampling algorithm is used to downsample the original 3D point cloud data, and the redundant information is removed through isolated point detection and abnormal density point screening strategies.

[0009] Preferably, in the intelligent geometric reasoning and semantic understanding method based on a three-dimensional large language model, the training optimization of the multimodal three-dimensional large language model in step 4 specifically includes: Based on the training sample set formed by preprocessed 3D point cloud data and text data, the geometry perception encoding module and the contextual semantic understanding module are jointly pre-trained using the cross-modal contrast loss function and the structure preservation loss function to align the geometric features with the text features in a shared semantic embedding space. During the training process, the cross-modal attention module performs bidirectional interaction between the point cloud features output by the geometric perception encoding module and the text features output by the contextual semantic understanding module to generate a joint feature representation that integrates multimodal semantics. Based on the pre-trained model, the adaptability of the parameter efficient fine-tuning module is optimized through task instruction examples: a. Dynamically encode the features of the newly added component region using a structured hint vector, which is generated based on the component's functional attributes and spatial region ID; b. Update the local parameters corresponding to the newly added component area through the incremental model update mechanism, and freeze the remaining parameters.

[0010] Preferably, the intelligent geometric reasoning and semantic understanding method based on the three-dimensional large language model, the geometric perception encoding module calculates geometric features by local density and global center distance, and constructs a shared semantic embedding space and a cross-modal cross attention module, specifically: 1) Based on the local density ρ of each point in the point cloud data i Distance d from the global center i , construct a dual-scale geometric modulation mechanism; Local density , where K is the number of neighboring points, N(i) is the K nearest neighbor points of point i, and p i 、p j are the spatial coordinates of point i and its neighborhood point j respectively; Global center distance , where c is the coordinate of the global center point of the point cloud; 2) Set ρ i with d i As the geometric perception weight, the geometric adjustment function g(d i , d j , ρ i , ρ j ) Dynamically adjust the self-attention score. The specific formula is: , where q i is the query vector of point i; k j is the key vector of point j; λ is the geometric feature weight adjustment coefficient; g is the geometric adjustment function, which is responsible for calculating the spatial relationship and density relationship between two points; the index τ represents the vector inner product operation; 3) Based on the attention score e ij , and further obtain the normalized attention weight through the Softmax function, and aggregate the value vector v of the neighborhood points j , and after feature transformation, output geometric feature vector , the specific calculation formula is: , Where, e ij is the attention score; v jis the value vector of the jth point, which is learned by learning the geometric features and semantic features of the point; z i is the vector output by the self-attention mechanism, W1 and W2 are network weights; b1 and b2 are bias vectors, is the geometric eigenvector; 4) Geometry-aware feature vector It maps the image to a shared semantic embedding space through linear projection, performs cross-modal attention calculation with text modality features, and generates a joint feature representation that integrates geometry and semantics.

[0011] Preferably, the intelligent geometric reasoning and semantic understanding method based on the three-dimensional large language model, the contextual semantic understanding module is based on a bidirectional transformer encoder, and is pre-trained through a masked language model and the next sentence prediction task, specifically: Pre-training based on text data, where: a. In the masked language model task, 15% of the architectural terminology tokens in the input text are masked. The masking strategies include: i. 80% of the masked tokens are replaced with mask symbols; ii. 10% of the masked tokens are replaced with random architectural terms; iii. 10% of the masked tokens remain unchanged; b. In the next sentence prediction task, the positive samples are actual continuous sentence pairs in the architectural text, and the negative samples are randomly combined non-continuous sentence pairs; During the fine-tuning phase, the semantic vectors output by BERT are supervised and aligned using the building component attribute labels.

[0012] Preferably, the intelligent geometric reasoning and semantic understanding method based on the three-dimensional large language model, the parameter efficient fine-tuning module combines low-rank adaptation and structured hint vectors, and configures an incremental model update mechanism, specifically: A low-rank adaptive mechanism is introduced into the fully connected layer of the multimodal 3D large language model. Specifically: The multimodal three-dimensional large language model weight matrix W∈R m×n , decomposed into W=W0+ΔW, where W0 is the frozen original weight, ΔW is the trainable low-rank matrix, ΔW=A×B T , A∈R m×r , B∈R n×r , and rank r< <min(m,n); Only perform gradient updates on the low-rank matrices A and B, keeping the W0 parameter fixed; Generate structured hint vector T∈R based on component functional attributes and spatial region ID k×d, where k is the number of cue vectors and d is the vector dimension. The cue vector is embedded into the model input by concatenating T with the point cloud feature vector to form an enhanced input Xin = Concat(Xpoint, T); Xin is mapped to the model latent space through a learnable projection matrix; Configure an incremental model update mechanism, specifically: locate the corresponding local low-rank matrix Anew, Bnew and hint vector Tnew based on the spatial area ID of the newly added component; freeze the original model parameters W0 and the low-rank matrix and hint vector of the non-newly added area, and only fine-tune Anew, Bnew and hint vector Tnew; normalize and fuse the model output after the local parameter update with the global model features.

[0013] Preferably, the intelligent geometric reasoning and semantic understanding method based on the three-dimensional large language model, the original three-dimensional point cloud data collected in steps 1 and 2, and the data to be measured in step 5, are stored and managed through a spatial database, specifically including: Build a spatial database linked to a multimodal 3D large language model: Design a component data table and a point cloud data table. The component data table links the component identification codes in the point cloud data table through the component identification codes, forming a one-to-many data mapping relationship. The point cloud data table stores preprocessed point cloud data, including 3D coordinates (X, Y, Z), color information (RGB), and associated component ID fields. A spatial index is constructed on the 3D coordinate fields of the point cloud data table, using an R-tree or octree structure to optimize the following operations: In response to user semantic query commands, the point cloud data of the target component is quickly located through the spatial index and transmitted to the multimodal 3D large language model for semantic parsing; during incremental model updates, the corresponding point cloud data is partitioned and retrieved based on the spatial region ID of the newly added component and loaded into the training process; A dynamic data synchronization mechanism is configured. When the point cloud data of a newly added component is collected and preprocessed by a three-dimensional scanning device, a spatial region identifier is automatically assigned according to the component ID, a corresponding component data table record is generated, and the record is associated with the preprocessed point cloud data and stored. Subsequently, the incremental update process of the parameter efficient fine-tuning module is triggered to synchronously update the spatial index and model parameters.

[0014] The present invention provides an intelligent geometric reasoning and semantic understanding device based on a three-dimensional large language model, which includes: A data acquisition module collects original three-dimensional point cloud data of building components through a three-dimensional scanning device; and obtains text data associated with the original three-dimensional point cloud data; A data preprocessing module, which is used to preprocess the original three-dimensional point cloud data; A model building module, which is used to build a multimodal three-dimensional large language model, including a geometric perception encoding module, a contextual semantic understanding module, and a parameter efficient fine-tuning module; Among them, the geometric perception encoding module calculates geometric features through local density and global center distance, constructs a shared semantic embedding space and a cross-modal cross-attention module; The contextual semantic understanding module is based on a bidirectional transformer encoder and is pre-trained with a masked language model and the next sentence prediction task; The parameter-efficient fine-tuning module combines low-rank adaptation with structured hint vectors to configure an incremental model update mechanism; A model training and optimization module is used to train and optimize the multimodal three-dimensional large language model using the pre-processed three-dimensional point cloud data obtained in step 2 and the text data obtained in step 1 as training sample sets; The parsing module is used to input the 3D point cloud data and text data to be tested into the trained and optimized multimodal 3D large language model, and output the semantic recognition, attribute completion and historical background analysis results of building components.

[0015] The present invention also provides an electronic device, comprising a processor and a memory, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the program.

[0016] The present invention also provides a computer-readable storage medium storing a computer program, which implements the steps of the above method when executed by a processor.

[0017] The present invention has at least the following beneficial effects: 1. This invention significantly improves the semantic recognition accuracy and cultural context analysis capabilities of complex building components by integrating a multimodal large language model with 3D point clouds and text. It supports real-time updates and interactive responses in dynamic construction scenarios, providing an efficient solution for architectural heritage protection and intelligent construction. 2. This invention adopts uniform downsampling and outlier screening strategies to effectively reduce data redundancy and noise interference, improve preprocessing efficiency, enhance the model's robustness to large-scale point cloud data, and lay a high-quality data foundation for subsequent semantic segmentation and feature alignment; 3. This invention uses a cross-modal contrast loss and an incremental fine-tuning mechanism to achieve efficient alignment of geometric and textual features, while supporting the rapid adaptation of new components, significantly improving the model's generalization ability and deployment efficiency in dynamic environments. 4. This invention uses a dual-scale geometric modulation mechanism to combine local density and global distance features to enhance the model's ability to perceive the spatial structure of complex building components, improve the accuracy of capturing the semantic differences of nested structures (such as brackets), and support accurate component function reasoning; 5. This invention uses masking strategies and supervised alignment methods in the architectural field to enhance the BERT model's ability to understand professional terminology and historical context, improve the correlation between semantic vectors and component attributes, and provide highly discriminative text features for cross-modal fusion. 6. This invention significantly reduces the number of fine-tuning parameters through low-rank adaptation and structured hint vector technology. Combined with the incremental update mechanism, it achieves rapid adaptation of new components (average time < 4 minutes), significantly reduces computing resource consumption, and meets the real-time requirements of construction sites. 7. This invention optimizes the storage and retrieval efficiency of point cloud data through spatial indexing and dynamic synchronization mechanisms, supports real-time semantic query and incremental learning processes, and ensures the stability and response speed of the system in high-concurrency scenarios (latency < 500ms); 8. The present invention realizes end-to-end multimodal data processing and reasoning through modular design, reduces the complexity of system integration, and improves the deployment flexibility and scalability in intelligent construction and heritage protection scenarios.

[0018] Other advantages, objectives and features of the present invention will be reflected in part from the following description and will be understood by those skilled in the art through study and practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a flow chart of the intelligent geometric reasoning and semantic understanding method based on a three-dimensional large language model described in the present invention. DETAILED DESCRIPTION

[0020] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments so that those skilled in the art can implement the invention with reference to the description.

[0021] It should be understood that terms such as “having”, “including” and “comprising” used herein do not preclude the existence or addition of one or more other elements or combinations thereof.

[0022] It should be noted that the experimental methods described in the following embodiments are conventional methods unless otherwise specified, and the reagents and materials can be obtained from commercial channels unless otherwise specified.

[0023] In the description of the present invention, the terms "horizontal", "longitudinal", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like to indicate orientations or positional relationships are based on the orientations or positional relationships shown in the accompanying drawings and are only for the convenience of describing the present invention and simplifying the description. They do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation, and therefore should not be understood as limiting the present invention.

[0024] The present invention provides an intelligent geometric reasoning and semantic understanding method based on a three-dimensional large language model, which comprises the following steps: Step 1: Collect the original 3D point cloud data of the building components using a 3D scanning device (Faro scanner, Leica scanner, total station, or binocular depth camera); generate text data based on the original 3D point cloud data (the text data includes the component name, historical background, functional attributes, and dynasty information); Step 2: Preprocess the original 3D point cloud data; Step 3: Build a multimodal 3D large language model, which includes a geometric perception encoding module, a contextual semantic understanding module, and a parameter efficient fine-tuning module; Among them, the geometric perception encoding module calculates geometric features through local density and global center distance, constructs a shared semantic embedding space and a cross-modal cross-attention module; The contextual semantic understanding module (BERT) is based on a bidirectional transformer encoder and is pre-trained using the masked language model (MLM) and next sentence prediction (NSP) tasks. The parameter-efficient fine-tuning module (PEFT) combines low-rank adaptation (LoRA) with structured hint vectors to configure an incremental model update mechanism; Step 4: Use the preprocessed 3D point cloud data obtained in step 2 and the text data obtained in step 1 as training sample sets to train and optimize the multimodal 3D large language model; Step 5: Input the 3D point cloud data and text data to be tested into the trained and optimized multimodal 3D large language model, and output the semantic recognition, attribute completion and historical background analysis results of the building components.

[0025] The 3D scanning device of the present invention can be a Faro Focus S 350 laser scanner, with a scanning accuracy of 0.1 mm and a point cloud density that can be set to 1,000 to 5,000 points per square meter. The device can be mounted on a tripod, and horizontal calibration is used to ensure spatial consistency of the scanned data. The collected raw point cloud data is transferred to an HP Z8 G4 workstation via a USB 3.0 interface and associated with text data. This text data includes component names (e.g., "dougong" and "yanzhu"), historical dates (e.g., "Ming Dynasty" and "Qing Dynasty"), and functional attributes (e.g., "load-bearing" and "decoration"). The text is stored in a JSON structured format. The association logic is based on a one-to-one mapping of point cloud file IDs with component IDs in a text database, with a mapping error tolerance of ±2%. A similar process can be referenced in the process of associating POS (positioning and orientation system) with image data in drone aerial surveys. Technical Effect: Through high-precision scanning and structured text mapping, the model provides geometrically and semantically aligned input data, supporting subsequent cross-modal feature fusion.

[0026] The preprocessing of point metadata in this invention includes denoising, normalization, and segmentation. Denoising uses a statistical outlier removal algorithm, with a noise threshold σ set to 0.05 and an isolated point density threshold set to less than 5 points per cubic meter. Normalization uses Z-score normalization to scale point cloud coordinates to the range [-1, 1]. Segmentation uses a region growing algorithm, with seed points selected from regions with a curvature less than 0.01 and a normal deviation angle less than 15°. Segmented components are assigned unique IDs (e.g., "COMP_001"). Downsampling uses a voxel grid method with a voxel size of 5 cm³, achieving a 60% redundant point elimination rate. Preprocessing was run on a Dell Precision 7865 workstation and took approximately 10 minutes per GB of point cloud data. A similar process can be seen in CT data segmentation and downsampling in medical imaging. Technical Effect: Through noise filtering and uniform sampling, data quality is improved, the computational load during model training is reduced, and key geometric features are preserved. The multimodal three-dimensional large language model of the present invention consists of a geometric perception encoding module, a contextual semantic understanding module, and an efficient parameter fine-tuning module. The local density calculation of the geometric encoding module uses K=32 nearest neighbors, and the global center distance is based on the coordinates of the point cloud centroid. During training, the batch size is 16 in the first stage and 64 in the second stage, the learning rates are 1e-3 and 4e-5 respectively, and the training rounds (epochs) are 50 rounds. The hardware uses the NVIDIA A800 GPU, and the video memory occupies approximately 40 GB. Cross-modal alignment uses a contrast loss function with a temperature coefficient τ=0.07. During incremental fine-tuning, the update of the parameters of the newly added component area takes no more than 4 minutes, and only 5% of the model parameters need to be adjusted. A similar method can be compared to the multimodal pre-training process of the CLIP model.

[0027] This method improves the semantic recognition accuracy of complex architectural components and reduces the error rate in historical context analysis by combining high-precision 3D scanning with structured text, standardizing preprocessing procedures, and training a large multimodal language model. In dynamic construction scenarios, the model supports five concurrent queries per second, with a stable response latency of less than 500 milliseconds, meeting the real-time interaction requirements of architectural heritage preservation and intelligent construction.

[0028] In another technical solution, the intelligent geometric reasoning and semantic understanding method based on the three-dimensional large language model, step 2 is specifically as follows: De-noising, standardization and segmentation of original 3D point cloud data, extracting the geometric features of each building component and assigning a unique attribute ID; The uniform downsampling algorithm is used to downsample the original 3D point cloud data, and the redundant information is removed through isolated point detection and abnormal density point screening strategies.

[0029] The denoising method of this invention uses a statistical outlier removal algorithm. The noise threshold σ can be set to 0.05, and the isolated point density threshold is set to less than 5 points per cubic meter. Normalization is performed by scaling the point cloud coordinates to the range [-1, 1] using Z-score normalization. The normalization parameters are calculated based on the global mean and standard deviation of the point cloud. Segmentation uses a region growing algorithm, with seed points selected based on regions with a curvature less than 0.01 and a normal deviation angle less than 15°. After segmentation, each component is assigned a unique attribute ID (e.g., "COMP_001"). The algorithm can be run on a Dell Precision 7865 workstation equipped with an NVIDIA RTX A6000 graphics card and 128GB of memory. Similar processes can be referenced in the point cloud preprocessing process for 3D scanning of industrial parts. Technical Effect: Through noise filtering and normalization, the interference of data outliers on model training is reduced. The uniqueness of the segmented component ID provides a structural foundation for subsequent semantic alignment. The downsampling method of this invention uses a voxel grid method, with a voxel size that can be set to 5 cm³, and a redundant point elimination rate of up to 60%. Outlier point screening is achieved through density clustering (DBSCAN algorithm), with the neighborhood search radius ε set to 0.1 m and the minimum number of neighborhood points, MinPts, set to 10. Isolated point detection is based on a KD tree spatial index, filtering out isolated points that are more than 0.3 m away from their nearest neighbors. The algorithm is integrated into the CloudCompare open-source software, running on the Ubuntu 20.04 system environment, and data is stored on a Samsung 980 PRO NVMe solid-state drive. Similar methods can be used for LiDAR point cloud compression and noise filtering in autonomous driving scenarios. Technical Effect: Through uniform sampling and density clustering screening, the data size is reduced while retaining key geometric features, improving preprocessing efficiency and model robustness. This method significantly reduces the redundancy and noise in point cloud data through denoising, normalization, segmentation, uniform downsampling, and outlier screening. This reduces the amount of preprocessed data, shortens model training time, and improves the retention of key component geometric features, providing high-quality input for subsequent semantic segmentation and cross-modal alignment.

[0030] In another technical solution, the intelligent geometric reasoning and semantic understanding method based on a three-dimensional large language model, in step 4, the training optimization of the multimodal three-dimensional large language model specifically includes: Based on the training sample set formed by preprocessed 3D point cloud data and text data, the geometry perception encoding module and the contextual semantic understanding module are jointly pre-trained using the cross-modal contrast loss function and the structure preservation loss function to align the geometric features with the text features in a shared semantic embedding space. During the training process, the cross-modal attention module performs bidirectional interaction between the point cloud features output by the geometric perception encoding module and the text features output by the contextual semantic understanding module to generate a joint feature representation that integrates multimodal semantics. Based on the pre-trained model, the adaptability of the parameter efficient fine-tuning module is optimized through task instruction examples: a. Dynamically encode the features of the newly added component region using a structured hint vector, which is generated based on the component's functional attributes and spatial region ID; b. Through the incremental model update mechanism, the local parameters corresponding to the newly added component area are updated and the remaining parameters are frozen to maintain global semantic consistency.

[0031] In the cross-modal contrastive loss function of the present invention, the temperature coefficient τ can be set to 0.07, and the negative sample sampling ratio is 1:3. The structure-preserving loss weight λ is set to 0.5 to constrain the integrity of the local geometric structure of the point cloud. The pre-training batch size is 16 per stage, the learning rate is 1e-3, and the training rounds are 50. The hardware platform can be an NVIDIA A800 GPU with a video memory capacity of 80 GB, and the training data is stored on a Seagate Exos X20 enterprise-class hard drive. During the pre-training process, the point cloud and text features are aligned in a shared semantic embedding space, and the embedding dimension d is set to 768. The contrastive loss is calculated using the CosineSimilarity module of the PyTorch framework, and the similarity threshold is set to 0.8. Similar methods can refer to the image-text alignment pre-training process in multimodal retrieval. Technical effect: Through the joint optimization of geometric and text features, the model's ability to understand the cross-modal semantic associations of building components is enhanced, laying the foundation for subsequent incremental learning. The cross-attention module of the present invention adopts a two-way interaction mechanism, the number of attention heads h is set to 32, and the key value dimension is 64. The number of interaction layers between point cloud features and text features is 12, and each layer contains self-attention and feedforward networks. In the attention calculation, point cloud features are extracted by the geometric perception encoding module, and text features are generated by the BERT module. The dimension of the fused features after interaction is maintained at 768, and the residual connection weight is initialized to 0.02. The gradient clipping threshold is set to 1.0 during training to prevent gradient explosion. The module runs in a Docker container environment under the Linux system, and the container image is compiled based on CUDA 11.8. A similar process can be seen in the multimodal feature fusion design in the visual-language model. Technical effect: The deep fusion of geometric and semantic information is achieved through the two-way attention mechanism, which improves the model's reasoning ability for complex component functions and historical backgrounds. The efficient parameter fine-tuning module of this invention utilizes low-rank adaptation (LoRA) technology. The rank r of the low-rank matrix is ​​set to 8, the number of hint vectors k is set to 10, and the dimension d is 128. During incremental updates, only the low-rank matrices Anew and Bnew corresponding to the newly added component regions are trained, with a learning rate set to 4e-5 and 10 fine-tuning rounds. Structured hint vectors are dynamically generated based on component functional attributes (e.g., "dougong-load-bearing") and spatial zone IDs (e.g., "ZONE_05") and fused with point cloud features via splicing. The fine-tuning process is completed on a single NVIDIA A800 GPU, with a single update taking no more than 4 minutes and using less than 8 GB of memory. This approach is analogous to the local parameter update strategy in federated learning. Technical Effect: Through local parameter freezing and incremental fine-tuning, rapid adaptation of newly added components is achieved while maintaining global model stability, significantly shortening the deployment cycle in dynamic construction scenarios. This invention effectively improves the model's generalization capabilities in dynamic environments through cross-modal comparative pre-training, bidirectional attention interaction, and incremental fine-tuning. Fine-tuning of new components is reduced to minutes, significantly enhancing the model's semantic recognition consistency across multiple scenarios while reducing resource consumption, meeting the real-time response requirements of intelligent construction and heritage preservation scenarios.

[0032] In another technical solution, the intelligent geometric reasoning and semantic understanding method based on the three-dimensional large language model is described. The geometric perception encoding module calculates geometric features by local density and global center distance, constructs a shared semantic embedding space and a cross-modal cross-attention module, specifically: 1) Based on the local density ρ of each point in the point cloud data i Distance d from the global center i , construct a dual-scale geometric modulation mechanism; Local density , where K is the number of neighboring points, N(i) is the K nearest neighbor points of point i, and p i 、p j are the spatial coordinates of point i and its neighborhood point j respectively; Global center distance , where c is the coordinate of the global center point of the point cloud; 2) Set ρ i with d i As the geometric perception weight, the geometric adjustment function g(d i , d j , ρ i , ρ j ) Dynamically adjust the self-attention score. The specific formula is: , where q i is the query vector of point i; k jis the key vector of point j; λ is the geometric feature weight adjustment coefficient; g is the geometric adjustment function, which is responsible for calculating the spatial relationship and density relationship between two points; index Represents a vector inner product operation; 3) Based on the attention score e ij , and further obtain the normalized attention weight through the Softmax function, and aggregate the value vector v of the neighborhood points j , and after feature transformation, output geometric feature vector , the specific calculation formula is: , Where, e ij is the attention score; v j is the value vector of the jth point, which is learned by learning the geometric features and semantic features of the point; z i is the vector output by the self-attention mechanism, W1 and W2 are network weights; b1 and b2 are bias vectors, is the geometric eigenvector; 4) Geometry-aware feature vector It maps the image to a shared semantic embedding space through linear projection, performs cross-modal attention calculation with text modality features, and generates a joint feature representation that integrates geometry and semantics.

[0033] The local density ρ of the present invention i The calculation uses K=32 nearest neighbor points, and the neighborhood is quickly searched through the KD tree algorithm. The neighborhood radius threshold is set to 0.2m. The global center distance d i The centroid coordinates of the point cloud are calculated using the mean of all point coordinates. In the density calculation formula, the denominator is the sum of the Euclidean distances between neighboring points, and the numerator, K, is fixed at 32. The coordinates of the global center point, c, are updated once per batch of training data. The calculation process can be run on an Intel Xeon Platinum 8360Y processor with 256GB of memory, using Samsung DDR4 ECC memory for data caching. Similar methods can be referenced in the rock porosity density calculation process used in geological exploration. Technical Effect: By leveraging the dual-scale features of local density and global distance, the model's ability to perceive the spatial distribution of building components is enhanced, providing fundamental support for the semantic differentiation of nested structures (such as brackets). The geometric adjustment function g(d i , d j , ρ i , ρ j ) is expressed as g = (d i +d j ) / (ρ i +ρ j +1e-5 ), the weight coefficient λ is set to 0.3. In the calculation of self-attention score, the query vector q i With the key vector k j The dimension is 64, and the inner product operation is implemented by matrix multiplication. The attention weight normalization adopts the Softmax function, and the temperature coefficient T is set to 0.1. The output feature dimension of the geometric perception encoding module is 768, and feature fusion is achieved through residual connection. The residual weight is initialized to 0.01. NVIDIA A800 GPU can be used for module training, and the video memory occupies about 12GB. A similar process can be seen in the geometric feature weighted matching method in point cloud registration. Technical effect: By dynamically adjusting the attention weight, the model can more accurately capture the relationship between the local details of the component and the global space, and improve the semantic difference recognition ability of complex structures such as brackets. The feature vector z after self-attention aggregation of the present invention i After two layers of feedforward neural networks (FFNs), the hidden layer dimension is 3072, and the activation function is ReLU. The feature transformation weight matrices W1 and W2 are initialized using the Xavier normal distribution, and the bias terms b1 and b2 are initialized to zero. The geometric feature vectors are mapped to a shared semantic space via a linear projection layer. The projection matrix dimension is 768×768, and the parameters are updated using the Adam optimizer. The projected features interact with the text modality features in a cross-modal cross-attention module. The number of attention heads h is set to 16, and the key-value dimension is 64. Similar methods can be compared to the multi-scale feature fusion network in three-dimensional object detection. Technical effect: Through the joint mapping of geometric features and semantic space, accurate reasoning of the functional properties of building components is achieved, supporting structural stability analysis and historical value assessment. This method significantly enhances the model's understanding of the spatial structure of complex architectural components through local density and global distance calculations, geometric adjustment function optimization, and feature aggregation and projection. A dual-scale geometric modulation mechanism effectively distinguishes semantic differences between nested components (such as brackets and eaves), providing reliable technical support for component function reasoning and cultural heritage preservation.

[0034] In another technical solution, the intelligent geometric reasoning and semantic understanding method based on a three-dimensional large language model, the contextual semantic understanding module (BERT) is based on a bidirectional transformer (Transformer) encoder and is pre-trained through the masked language model (MLM) and next sentence prediction (NSP) tasks. Specifically: Pre-training is performed based on text data (including historical documents, drawing descriptions, and regulatory texts), where: a. In the Masked Language Model (MLM) task, 15% of the architectural terminology tokens in the input text are masked. The masking strategies include: i. 80% of the masked tokens are replaced with mask symbols; ii. 10% of the masked tokens are replaced with random architectural terms; iii. 10% of the masked tokens remain unchanged; b. In the next sentence prediction (NSP) task, the positive samples are actual continuous sentence pairs in architectural texts, and the negative samples are randomly combined non-continuous sentence pairs; During the fine-tuning stage, the semantic vectors output by BERT are supervised and aligned using architectural component attribute labels (including component type, dynasty, and function) to enhance the ability to extract contextual features in the architectural field.

[0035] In the Masked Language Model (MLM) task of this invention, the input text is masked at a rate of 15%. The masking mark replacement strategy is as follows: 80% is replaced with the [MASK] symbol, 10% is replaced with random architectural terms (such as "mortise and tenon" and "eaves"), and the remaining 10% remains unchanged. The random term library is derived from the digitized text of the "Dictionary of Ancient Chinese Architecture" and the "Construction Methods," and contains over 5,000 terms. Professional terms (such as "dougong" and "liangjia") are prioritized for masking, and noun phrases are identified using part-of-speech tagging tools (such as HanLP). Training data is stored on Seagate IronWolf NAS hard drives, and the processing platform can be selected from Huawei's Atlas 800 training server equipped with the Ascend 910B AI chip. Similar methods can be referenced in the professional term masking strategy used in medical text pre-training. Technical Effect: By optimizing masking for architectural terms, the model's contextual reasoning ability for professional terms is enhanced, improving the domain discrimination of semantic vectors. In the next sentence prediction (NSP) task of this invention, positive samples are pairs of consecutive sentences in architectural text (e.g., "Dougong is a typical component of Ming and Qing architecture" and "Its load-bearing function is achieved through layers of overhanging"). Negative samples are generated by randomly combining non-consecutive sentences (e.g., "The column base is carved with lotus patterns" and "The Song Dynasty's 'Yingzaofashi' records tile-making techniques"). The positive-to-negative sample ratio is set to 1:1, and the maximum length of a single sentence is limited to 128 characters. The text data comes from the Palace Museum archive digitization project and the electronic version of "A History of Chinese Architecture." Regular expressions are used to clean the data to filter non-Chinese symbols. During training, the batch size is 64, the learning rate is set to 2e-5, and the gradient clipping threshold is 1.0. A similar process can be seen in the task of determining the coherence of contract clauses in legal texts. Technical Effect: By learning the semantic association between the previous and next sentences, the model's ability to model long-range dependencies on the architectural historical context and construction logic is enhanced. The fine-tuning phase of the present invention uses component attribute labels (such as type, dynasty, and function) for supervised alignment. Label data is generated using an expert annotation platform (such as Label Studio), and the annotation consistency is verified using a Kappa coefficient threshold of 0.75. The alignment loss function uses the cosine similarity metric, with a target similarity value set to 0.9. The AdamW optimizer is used with a weight decay coefficient of 0.01. The fine-tuning hardware can be configured with an NVIDIA A100 GPU with 40 GB of video memory, and training data loading is accelerated using the Apache Arrow format. The model outputs a semantic vector with a dimension of 768. Cross-modal alignment with point cloud features uses a bilinear interaction layer, with interaction weights initialized to a normal distribution (μ = 0, σ = 0.02). Similar methods can be compared to the fine-tuning process for image-text feature alignment in cross-modal retrieval. Technical Effect: Through the strong supervisory signal of attribute labels, the correlation between semantic vectors and attributes such as component function and era is improved, providing highly discriminative text feature representations for cross-modal fusion.

[0036] This paper significantly improves the BERT model's understanding of ancient architectural terminology and historical context through architectural masking strategies, next-sentence prediction tasks, and supervised alignment fine-tuning. The correlation between semantic vectors and component attributes is enhanced, cross-modal feature alignment errors are reduced, and the deep integration of 3D point clouds and text data is supported, providing a reliable semantic analysis foundation for the digital preservation of architectural heritage.

[0037] In another technical solution, the intelligent geometric reasoning and semantic understanding method based on a 3D large language model, the parameter efficient fine-tuning module (PEFT) combines low-rank adaptation (LoRA) with structured hint vectors, and configures an incremental model update mechanism, specifically: The low-rank adaptation (LoRA) mechanism is introduced in the fully connected layer of the multimodal 3D large language model. Specifically: The multimodal three-dimensional large language model weight matrix W∈R m×n , decomposed into W=W0+ΔW, where W0 is the frozen original weight, ΔW is the trainable low-rank matrix, ΔW=A×B T , A∈R m×r , B∈R n×r , and rank r< <min(m,n); Only perform gradient updates on the low-rank matrices A and B, keeping the W0 parameter fixed; Generate structured hint vector T∈R based on component functional attributes and spatial region ID k×d , where k is the number of cue vectors and d is the vector dimension. The cue vector is embedded into the model input by concatenating T with the point cloud feature vector to form an enhanced input Xin = Concat(Xpoint, T); Xin is mapped to the model latent space through a learnable projection matrix; Configure an incremental model update mechanism, specifically: locate the corresponding local low-rank matrix Anew, Bnew and hint vector Tnew based on the spatial area ID of the newly added component; freeze the original model parameters W0 and the low-rank matrix and hint vector of the non-newly added area, and only fine-tune Anew, Bnew and hint vector Tnew; normalize and fuse the model output after the local parameter update with the global model features.

[0038] The fully connected layer weight matrix W of the multimodal three-dimensional large language model of the present invention can be decomposed into W = W0 + ΔW, where ΔW = BA. The rank r of the low-rank matrices A and B is set to 8, and the matrix dimensions are 768×8 and 8×768, respectively. The original weight W0 is frozen during training, and only gradient updates are performed on A and B. The low-rank matrix is ​​initialized using a normal distribution with a mean of 0 and a standard deviation of 0.02, and the learning rate is set to 4e-5. The hardware can be selected using an NVIDIA A800 GPU, reducing the video memory usage to 15% of the original model, and the memory requirement for a single fine-tuning does not exceed 8 GB. Matrix factorization calculations are implemented using linear layer reparameterization in the PyTorch framework, and the code is integrated into Hugging Face's PEFT tool library. Similar methods can be referenced in the adapter fine-tuning technique in natural language processing. Technical effect: Low-rank decomposition reduces the number of trainable parameters, enabling efficient adaptation of new components while maintaining global semantic consistency of the model. The structured hint vector T of this invention is generated based on component functional attributes (such as "column-load-bearing") and spatial zone IDs (such as "ZONE03"). The number of vectors, k, is set to 10, and the dimension, d, is 128. Hint vectors are loaded from a predefined dictionary using a table lookup method. The dictionary contains combinations of 50 functional attributes and 100 spatial zone IDs. After concatenating the hint vector with the point cloud features, it is mapped to the model latent space using a learnable projection matrix (dimension 256×768). The projection matrix is ​​initialized using the Xavier uniform distribution, with the bias term initialized to zero. Samsung 980 PRO NVMe solid-state drives can be used for data storage, with a read speed of up to 7000MB / s. A similar process can be seen in the design of user feature embedding in recommendation systems. Technical Effect: Structured hints enhance the semantic encoding capabilities of newly added components and improve the model's adaptability to dynamic changes in construction scenarios. During incremental updates, the present invention locates the corresponding low-rank matrices Anew and Bnew and the prompt vector Tnew based on the spatial region ID of the newly added component. Parameters in non-newly added regions (W0, Aold, Bold) are all frozen, and only Anew, Bnew, and Tnew are fine-tuned. The fine-tuning rounds are set to 10, with a batch size of 64. The updated local parameters are fused with the global model via weighted averaging, with the weight coefficient α set to 0.7. The fine-tuning process is completed on a single NVIDIA A800 GPU, taking an average of no more than 4 minutes and using less than 4GB of CPU memory. The update process is automated using a Python script deployed in a Docker container (the image is based on Ubuntu 22.04). Similar methods can be compared to local model update strategies in edge computing. Technical Effect: By freezing local parameters and integrating incremental fusion, computing resource consumption is significantly reduced, supporting the real-time semantic update requirements of construction sites. This method achieves rapid adaptation of newly added components through low-rank adaptive decomposition, structured hint embedding, and incremental parameter updates. The number of fine-tuning parameters is reduced to less than 5% of the original model, and the time required for a single update is controlled to minutes. GPU memory usage is significantly reduced, meeting the dual requirements of real-time performance and resource efficiency in intelligent construction scenarios.

[0039] In another technical solution, the intelligent geometric reasoning and semantic understanding method based on the three-dimensional large language model, the original three-dimensional point cloud data collected in steps one and two, and the data to be measured in step five, are stored and managed in a spatial database, specifically including: Build a spatial database linked to a multimodal 3D large language model: Design component data tables and point cloud data tables. The component data table associates the component identification code (COMP_name) in the point cloud data table through the component identification code (COMP_ID), forming a one-to-many data mapping relationship. The point cloud data table stores preprocessed point cloud data, including 3D coordinates (X, Y, Z), color information (RGB), and associated component ID fields. A spatial index is constructed on the 3D coordinate fields of the point cloud data table, using an R-tree or octree structure to optimize the following operations: In response to user semantic query commands, the point cloud data of the target component is quickly located through the spatial index and transmitted to the multimodal 3D large language model for semantic parsing; during incremental model updates, the corresponding point cloud data is partitioned and retrieved based on the spatial region ID of the newly added component and loaded into the training process; A dynamic data synchronization mechanism is configured. When the point cloud data of a newly added component is collected and preprocessed by a three-dimensional scanning device, a spatial region identifier is automatically assigned according to the component ID, a corresponding component data table record is generated, and the record is associated with the preprocessed point cloud data and stored. Subsequently, the incremental update process of the parameter efficient fine-tuning module is triggered to synchronously update the spatial index and model parameters.

[0040] The component data table of the present invention is associated with the point cloud data table through the component identification code (COMP_ID), adopting a one-to-many mapping relationship. The fields of the component data table include component ID, name, dynasty, and functional attributes, and the data type is VARCHAR (64). The point cloud data table stores pre-processed point cloud data, and the fields include three-dimensional coordinates (X, Y, Z, data type FLOAT), color information (RGB, data type INT), and component ID foreign key. The database management system can use MySQL 8.0, the storage engine is InnoDB, and the transaction isolation level is set to READ COMMITTED. The data table partitioning strategy is based on component ID hash sharding, and the maximum capacity of a single table is limited to 1 TB. The data storage hardware can be configured with a Dell PowerEdge R750xa server equipped with a Seagate ExosX20 18 TB hard drive and a RAID 5 redundant array. Similar methods can refer to the design of a database for associating goods and warehousing information in logistics management. Technical effect: Through structured data table design and foreign key constraints, efficient associative query of point cloud data and component attributes is ensured, supporting the semantic parsing requirements of multimodal models. The three-dimensional coordinate fields of the point cloud data table of the present invention use an R-tree index, with a branching factor set to 50 and a node capacity of 100 records. The depth of the octree index is set to 5 layers, and the minimum number of leaf nodes is 100. Index construction is implemented using the spatial extension module of MySQL, and the spatial query range error tolerance is 0.01 m. The index update strategy is asynchronous batch update, and an index rebuild is triggered every time 1,000 new point cloud data are accumulated. The hardware can use Intel Optane persistent memory PMem 200 series to accelerate index loading speed. The index optimization algorithm is based on a greedy strategy, preferentially merging nodes with a spatial overlap rate of more than 70%. A similar process can be seen in the optimization of terrain elevation data indexes in geographic information systems (GIS). Technical effect: Through the hybrid indexing of R-tree and octree, the spatial retrieval efficiency of massive point cloud data is improved, the response time is shortened to milliseconds, and real-time interactive queries are supported. After collecting point cloud data for new components, the present invention automatically assigns a spatial zone identifier (e.g., ZONE_09) based on the component ID and generates a corresponding component data table record. Data synchronization utilizes a dual-write mechanism, writing data to both the MySQL master database and the Redis cache, with a cache expiration time set to 24 hours. The incremental update process is implemented via triggers. When a new record is inserted into the point cloud data table, the incremental training script of the Parameter Efficient Fine-tuning Module (PEFT) is automatically invoked. The script is deployed in a Kubernetes cluster, with a single node configured with a quad-core CPU and 16GB of memory. Synchronization process logs are stored in Elasticsearch 8.0, with a 30-day rolling deletion policy. This approach is comparable to the edge-to-cloud synchronization architecture used in real-time processing of data streams from IoT devices. Technical Effect: Through automated synchronization and triggering mechanisms, the end-to-end latency from new data collection to model update is less than 10 minutes, meeting the real-time requirements of dynamic construction scenarios. This invention significantly improves the storage and retrieval efficiency of massive point cloud data through spatial database construction, hybrid spatial index optimization, and dynamic data synchronization. R-tree indexing speeds up spatial queries, octree optimization reduces storage redundancy, and a dynamic synchronization mechanism ensures the consistency of the incremental learning process, providing efficient and reliable data management support for intelligent construction and heritage preservation scenarios.

[0041] The present invention also provides an intelligent geometric reasoning and semantic understanding device based on a three-dimensional large language model, which includes: A data acquisition module collects raw 3D point cloud data of building components using a 3D scanning device (Faro scanner, Leica scanner, total station, or binocular depth camera); and obtains text data associated with the raw 3D point cloud data (the text data includes component name, historical background, functional attributes, and dynasty information). A data preprocessing module, which is used to preprocess the original three-dimensional point cloud data; A model building module, which is used to build a multimodal three-dimensional large language model, including a geometric perception encoding module, a contextual semantic understanding module, and a parameter efficient fine-tuning module; Among them, the geometric perception encoding module calculates geometric features through local density and global center distance, constructs a shared semantic embedding space and a cross-modal cross-attention module; The contextual semantic understanding module (BERT) is based on a bidirectional transformer encoder and is pre-trained using the masked language model (MLM) and next sentence prediction (NSP) tasks. The parameter-efficient fine-tuning module (PEFT) combines low-rank adaptation (LoRA) with structured hint vectors to configure an incremental model update mechanism; A model training and optimization module is used to train and optimize the multimodal three-dimensional large language model using the pre-processed three-dimensional point cloud data obtained in step 2 and the text data obtained in step 1 as training sample sets; The parsing module is used to input the 3D point cloud data and text data to be tested into the trained and optimized multimodal 3D large language model, and output the semantic recognition, attribute completion and historical background analysis results of building components.

[0042] The data acquisition module of this invention can be equipped with a Faro Focus S 350 laser scanner, offering a scanning accuracy of 0.1 mm and a maximum scanning distance of 350 m. The data preprocessing module runs on a Dell Precision 7865 workstation equipped with an NVIDIA RTX A6000 graphics card and 128 GB of memory. The preprocessing algorithm includes statistical outlier removal (noise threshold σ = 0.05), voxel grid downsampling (voxel size 5 cm³), and region growing segmentation (curvature threshold 0.01). Scanning data is transmitted via a USB 3.0 interface, and preprocessing results are stored on a Samsung 980 PRO NVMe solid-state drive with a read / write speed of 7000 MB / s. Similar processes can be referenced in the 3D scanning and point cloud processing pipeline used in industrial inspection. Technical Effect: Through modular hardware and algorithm integration, end-to-end automation from data acquisition to preprocessing is achieved, reducing the need for manual intervention and improving system deployment efficiency. The model building module of the present invention is based on the PyTorch framework. The local density calculation of the geometric perception coding module adopts K=32 nearest neighbors, and the global center distance is based on the centroid coordinates. The contextual semantic understanding module adopts the BERT-base architecture, the pre-training batch size is 16, and the learning rate is 1e-3. The parameter efficient fine-tuning module is configured with a low-rank matrix rank r=8 and a structured prompt vector dimension d=128. The training hardware can use an NVIDIA A800 GPU cluster with a single-node video memory of 80 GB, and distributed training is implemented through the NCCL communication library. The model optimization uses the AdamW optimizer, the weight decay coefficient is set to 0.01, and the gradient clipping threshold is 1.0. Similar methods can be seen in the model training pipeline design of the multimodal AI platform. Technical effect: Through standardized model architecture and distributed training support, rapid iteration and optimization of complex models are achieved to adapt to the deployment requirements of scenarios of different scales. The parsing module of this invention is deployed in a B / S architecture system. The front-end uses the Three.js library for 3D visualization, and the back-end is built with a RESTful API service based on Node.js, running on the Ubuntu 22.04 operating system. The user interface supports natural language queries (e.g., "Mark all brackets"), and the semantic parsing response latency threshold is set to 500 ms. The system hardware can be configured with the Huawei Atlas 800 inference server, equipped with the Ascend 910BAI chip, supporting ≥5 concurrent users. Data caching uses Redis 6.2 with a cache expiration time of 10 minutes, and database queries are accelerated using MySQL 8.0 spatial indexing. Similar systems can be compared to 3D visualization management platforms in smart cities. Technical Effect: Through a lightweight front-end and back-end separation design, high-concurrency real-time interaction is achieved, meeting the diverse application requirements of construction sites and heritage protection scenarios. This invention achieves end-to-end multimodal data processing and reasoning through modular hardware and algorithm integration, a standardized model training pipeline, and a lightweight interactive system. The system automates the entire process, from data acquisition to semantic parsing, reducing integration complexity. Furthermore, through a distributed architecture and efficient caching mechanisms, it enhances deployment flexibility and scalability in intelligent construction and heritage conservation scenarios.

[0043] The present invention provides an electronic device comprising a processor and a memory, wherein the memory stores a computer program, and when the processor executes the program, the steps of the above-described method are implemented. The electronic device is a device comprising a processor (CPU / MCU / SOC) and a memory (ROM / RAM), such as a desktop computer, a laptop computer, or a smartphone. In particular, the memory stores a computer program, and when the processor loads and executes the computer program, it implements all or part of the steps of the above-described intelligent geometric reasoning and semantic understanding method based on a large three-dimensional language model.

[0044] The present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the aforementioned method. The storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks. The computer program, when loaded and executed by a processor, implements all or part of the steps of the aforementioned method for intelligent geometric reasoning and semantic understanding based on a large three-dimensional language model.

[0045] like Figure 1 As shown, this invention focuses on the semantic understanding needs of 3D point cloud data in complex architectural spaces and builds a multimodal 3D large language model method framework that integrates deep learning and natural language processing technologies. The overall system is divided into three core modules: 1. Multimodal 3D Large Language Model Architecture Design. This paper proposes the 3D-MELL (3D-Multimodal Efficient Language Learning) architecture for semantic understanding of building component point clouds, aiming to overcome the modal limitations between 3D spatial geometry and natural language expression. Unlike existing models such as Point-BERT and CLIP-3D, which rely solely on shallow embedding similarity alignment, 3D-MELL innovatively introduces a spatial-semantic consistency learning mechanism, constructing a shared semantic embedding space and a cross-modal cross-attention module to achieve an explicit mapping between the spatial structure of building components in point clouds and their textual descriptions. The architecture consists of a geometry-aware encoding module (incorporating local density and global center perception mechanisms), a contextual semantic understanding module (BERT), and a parameter-efficient fine-tuning mechanism (PEFT). The model employs a two-stage training strategy: the first stage uses structure-preserving and contrastive loss for cross-modal feature alignment; the second stage incorporates instruction-based language interaction examples for fine-tuning, enhancing the model's language generation and interaction capabilities, enabling complex tasks such as semantic recognition, attribute completion, and language description of building components. To address model failure issues caused by dynamic changes in construction site point clouds, this paper further proposes an incremental model update mechanism within the PEFT architecture. This mechanism, combined with component ID management and regional difference analysis, allows for rapid fine-tuning of local parameters of newly added components in an average of four minutes, eliminating the need for full retraining. This enables real-time semantic fusion and structural updates in dynamic scenarios.

[0046] 2. Method for constructing a 3D point cloud semantic dataset. Considering the professional and high cost issues of semantic annotation of point clouds in complex architectural scenes, this paper constructs a 3D semantic point cloud dataset for architectural components, covering multiple sample categories such as ancient architectural heritage, modern construction, and structural components. The dataset is constructed using a human-machine collaborative annotation mechanism: initial annotation is performed using an automatic annotation tool based on rules (such as component size, normal direction, spatial distribution, etc.), and then high-quality semantic labels are generated through expert interactive correction. This mechanism supports incremental semantic expansion and continuous optimization, and can automatically integrate into the existing annotation process when introducing new scenes or new component sampling data, effectively improving the scalability and diversity of the dataset, and providing a solid semantic foundation for model training and generalization.

[0047] 3. A 3D interactive system integrating semantic question answering and spatial reasoning. Leveraging the semantic capabilities of 3D-MELL, this paper further constructs a 3D semantic question answering and spatial reasoning system platform, enabling integrated interaction for component recognition, semantic querying, geometric analysis, and visual response. The system utilizes a browser / server (B / S) architecture. The front-end uses Three.js and WebGL to load, interactively select, and highlight 3D scene components. The back-end integrates a lightweight natural language understanding module and a spatial reasoning computation engine, allowing users to pose questions in natural language (e.g., "Please mark the brackets," "Which era does this column belong to?", "How are these structures connected?"). The system automatically performs semantic parsing and geometric feedback, enhancing the interpretability and interactivity of point cloud data. At the deployment level, the system supports model pruning, attention sparsification, and cache optimization strategies. In tests on an RTXA800 GPU platform, the system achieved an average response latency of less than 500ms and a concurrent processing capacity of ≥5 users, meeting the engineering requirements for high responsiveness and real-time interaction in architectural scenarios.

[0048] Design of a multimodal encoding framework 3D-MELL The 3D-MELL network is a complex architecture that combines multi-source data and a large language model, specifically designed to understand and describe architectural structural elements. The overall architecture consists of multiple modules, each with its own unique functionality and design, ensuring that point cloud data and text information can be effectively integrated. First, the point cloud data used for building scanning must undergo a series of preprocessing steps such as denoising, normalization, and segmentation to extract individual building elements and assign them unique attribute IDs. After preprocessing, the point cloud data is represented as p∈R N×D , where N is the number of points and D is the feature dimension of each point. The text data is converted into vector form through natural language processing technology and used as input together with the point cloud data for subsequent processing. The point cloud data is processed by the point encoder to generate a point feature sequence X = (x1, x2, x3..., x m )∈R m×c, where m is the number of point features and c is the feature dimension. These point feature sequences are then further converted into embedding vectors through point token embedding and passed to subsequent modules. The large language model can obtain high-dimensional representations of each point, providing more accurate input features for subsequent tasks. After embedding, the point tokens are input to the Geometry-Aware Encoder Block. This module innovates by combining the self-attention mechanism with geometry perception. By introducing a geometric scale adjustment mechanism, the module adjusts self-attention weights based on geometric information such as distance between points and local density, achieving better results in spatial structure modeling. All processed features are ultimately passed to the task-specific head, which generates the final interactive results based on the different requirements of the task. The task-specific head deeply analyzes multimodal information to generate precise task outputs, such as the classification of architectural elements and historical context, thereby achieving a comprehensive description and analysis of architectural data.

[0049] Efficient fine-tuning of PEFT network technology The proposed PEFT strategy combines LoRA with a prompt tuning mechanism to optimize adaptability specifically for 3D building component recognition and semantic generation tasks. Unlike common PEFT methods in the NLP field, the PEFT module in 3D-MELL incorporates structured prompt vectors based on component functional attributes (such as "column-eaves-bracket" logic) and dynasty-specific information to enhance semantic transfer capabilities. It also supports online incremental learning of new components in dynamic construction scenarios. This mechanism allows the model to quickly fine-tune and restore global semantic consistency when new construction point clouds are added, using only local geometry updates and minor prompt adjustments. This improves the system's adaptability and deployment efficiency in real-world construction sites. Furthermore, since the PEFT approach only requires adjusting a small number of newly added learnable parameters (such as token prompts), it avoids the need to fully train large pre-trained models, significantly reducing computational resources and storage overhead. When fine-tuning multiple tasks, PEFT avoids the need to train and store fine-tuned models separately for each task, accelerating model transfer and deployment. By only fine-tuning the newly added parameters, PEFT avoids the large amount of training time required by traditional full-scale fine-tuning methods, thereby significantly improving the efficiency of the large language model training phase.

[0050] To address the potential for noise and redundant information in point cloud data from actual architectural scans, this paper introduces a density-controlled point screening strategy and a uniform downsampling algorithm (Formula 2-7) during the encoding phase. This strategy also improves model stability by removing isolated points and points with abnormal density. To verify the model's robustness in noisy scenarios, simulated point cloud testing experiments with varying noise levels (σ = 0.01 to 0.1) were conducted. The results show that the model maintains semantic recognition accuracy exceeding 85% under moderate to high noise conditions, an improvement of over 9% compared to a baseline model without noise reduction, demonstrating strong anti-interference capabilities, as shown in Formula 2-1.

[0051] F0=TokenEmbed(P),F0∈R M×D (2-1) Where N represents the number of embedded markers; D is the feature dimension of each marker, which converts the original point cloud into point feature markers that are convenient for further processing by the feature extraction module. To enhance the transformer model's ability to perceive the spatial information of point cloud data, a point priority prompt module is introduced. This module uses the original point cloud data P to generate prompt tokens with spatial location information. The calculation formula is as follows: P0=P 2 -Prompt(P), P0∈R K×D (2-2) In the formula, K is the number of indicator tokens. After obtaining the point optimization hint, it is combined with the point marking feature F0 of the previous step so that the transformer module can integrate this information for more accurate feature extraction, as shown in Formula 2-3.

[0052] C0=Concat(P0,F0),C0∈R (M+K)×D (2-3) At this time, the input feature C0 combines the local geometric features and spatial position information of the point cloud. The transformer block processes the spliced ​​feature C0 through the self-attention mechanism and the feed-forward network. There are 12 layers in total. Each layer contains the self-attention mechanism (Self-Attention) and the feed-forward neural network (FFN). For the i-th layer (1≤i≤12) transformer module, its input feature is represented by C i , perform self-attention calculation on the point cloud vector features, as shown in formula 2-4.

[0053] C i ′ =Self-Attn.(C i )+C i (2-4) After further processing by the feedforward neural network, as shown in Formula 2-5, each transformer block extracts features layer by layer, enhancing the model's understanding of the details and global structure in the building point cloud.

[0054] F i =FFN(C i ′ ) + C i ′ , F i ∈R (M+K)×D (2-5) The output feature F of each layer transformer i It will be used as part of the input features of the next layer transformer (i.e. C i+1 ), thereby gradually and deeply extracting the spatial structural features of the building point cloud data. During the processing of the transformer module, the local feature extraction module is also used to capture the local detail information in the building point cloud, especially the area with complex component decoration. Let the local area be represented by the point set P local ⊆P, then the feature extraction formula for local features is shown in 2-6.

[0055] F local = (P local ), F local ∈R Mlocal×D (2-6) Furthermore, in order to reduce the computational burden of large-scale point cloud processing and improve model efficiency, a uniform sampling module is used to downsample the original point cloud data P. The calculation formula is as follows: P sampled =UniformSample(P,M′),P sample ∈R M′×3 , M′<N (2-7) For local features F local Perform pooling operation to extract global structural feature representation, as shown in Formula 2-8.

[0056] F pool =Pooling(F local ), F pool ∈R 1×D (2-8) To further explore the complex spatial dependencies in architectural point cloud data, we introduce Graph Convolutional Networks (GCN) to conduct in-depth analysis of the topological relationships between points. Assuming that the point cloud data is represented by a graph structure G = (V, E), where V is a set of point nodes and E is an edge set representing the relationship between nodes, the graph convolution calculation formula is: H out = σ (AHinW), H out =R ∣V∣×D (2-9) Where A is the adjacency matrix, which describes the spatial topological relationship between points; H in is the input point feature matrix; W is the learnable parameter; σ is a nonlinear activation function. After GCN processing, the high-order graph feature H out This helps the network to better understand the structural stability and spatial form of the building. D is the feature dimension of each tag. Finally, the features output by the transformer module, the local pooling features, and the graph features processed by GCN are integrated to form the final fusion feature representation F. final , as shown in Formula 2-10, is used as the input of the task-specific head.

[0057] Y pred =TaskHead(F final ), Y pred ∈R T (2-10) Where T is a multidimensional feature vector, which provides basic support for semantic reasoning of the three-dimensional large language model. In summary, the PEFT network architecture uses point cloud input as source data and gradually completes feature extraction of building point cloud data through multiple steps such as label embedding, point priority prompting, transformer feature learning, local feature extraction, uniform sampling, pooled feature extraction, graph convolutional network analysis, and task-specific head prediction, thereby achieving accurate cognition of building structures and efficient protection decision support. Compared with the LoRA / Adapter application in general NLP, 3D-MELL introduces a structured prompt mechanism based on component functional attributes and spatial area ID in the PEFT structure (Formula 2-1), and combines the changing characteristics of the construction stage to propose a "local area fine-tuning + imperative token update" strategy, which can achieve incremental adaptation and rapid integration of component semantics without retraining the entire model.

[0058] Contextual Understanding BERT Encoding Structure Design The encoder transforms the input sequence into a series of contextual representation vectors and consists of multiple identical layers. Each layer consists of two sublayers: a self-attention layer and a feed-forward fully connected layer. The Transformer model represents a major innovation in natural language processing. It is based entirely on the attention mechanism, abandoning the previously widely used recurrent neural networks (RNNs) and convolutional neural networks (CNNs). The core feature of the Transformer is the self-attention layer, which processes each element of the input sequence and compares it with all other elements in the sequence to generate a contextualized representation vector. This mechanism not only improves the model's ability to handle long-range dependencies but also significantly improves efficiency through parallel computation. Building on the success of the Transformer, the BERT model further develops this foundation. Its unique feature lies in its fully bidirectional Transformer encoder structure, which allows the model to simultaneously consider contextual information at all layers. The BERT encoder utilizes a multi-head attention mechanism, which allows each attention head to focus on different aspects of the same information, thereby enriching the model's ability to capture linguistic features.

[0059] The BERT encoder uses the Transformer Encoder as its foundational language model and employs BERT to conduct in-depth analysis of architectural text, extracting information about building features, construction details, and more. The encoder employs multi-head attention, which is used to output multiple attentions and focus on different features within the text. The input query (Q), key (K), and value (V) are mapped to multiple subspaces using different linear transformation matrices. Within each subspace, the attention output of the corresponding head is independently calculated. Ultimately, the outputs of all heads are concatenated and a linear transformation is performed to produce the final attention output, as shown in Formula 2-11.

[0060] MultiHead(Q,K,V)=Concat(head1,…head i …head n )W 0 head i =Attention(QW i Q , KW i K , VW i V ) (2-11) Where W i Q , W iK , W i V The linear transformation matrix of the head; W 0 is the output linear transformation matrix; h is the number of heads, i∈(1,2,3,…,h). In this paper, h is set to 32. The pre-training task processing stage mainly includes two pre-training tasks: the Masked Language Model (MLM) and the Next Sentence Prediction (NSP). The MLM masks 15% of the ancient building marker words in the input, replacing 80% with the [MASK] tag, 10% with random words, and 10% unchanged. This allows the use of both left and right context to understand and predict labels. The NSP selects 50% of the sentences in the ancient building markers and their corresponding next sentences to form the upper and lower sentences as positive samples. For the remaining 50%, a random sentence is selected instead of the next sentence to form the upper and lower sentences as negative samples. This method has the ability to abstract long continuous sequence features, enhancing the BERT model's understanding of ancient architecture. In the fine-tuning stage, overfitting is avoided by adjusting parameters such as the learning rate and epochs.

[0061] The BERT model encoder is based on a bidirectional Transformer encoding structure, which can simultaneously focus on the contextual semantics of architectural terms in the text, making the model more comprehensive and accurate in understanding professional architectural terms, structural features, historical background and other information. BERT's built-in multi-head self-attention mechanism allows the model to capture the rich and complex linguistic features in point cloud data descriptions in parallel. Each attention head can focus on different aspects of the ancient architectural text, thereby achieving deep semantic modeling of point cloud data. In addition, BERT's pre-training fine-tuning strategy greatly improves the generalization ability for specialized terminology and expressions in the architectural field, enabling the model to more effectively adapt to the association learning between point cloud scenes and text descriptions, and better understand and extract the structural and layout features of ancient buildings.

[0062] Principle of geometric perception coding module The geometry-aware encoder is designed specifically for architectural 3D point clouds, aiming to accurately extract spatial geometric structure information and convert it into a high-dimensional semantic representation suitable for language model processing. Unlike methods such as PointTransformer, which only build attention weights based on the local relative positions between point pairs, 3D-MELL introduces a "dual-scale geometric modulation mechanism" that comprehensively considers the local density of points (ρ i ) and the Euclidean distance from the point to the global center (d i), dynamically adjusting the attention score (Formula 2-13) to achieve spatial relationship modeling with greater component-level understanding. This mechanism improves the model's ability to capture the semantic differences between multi-layered nested structures (such as brackets, eaves, and column bases), providing a more discerning encoding method for architectural structural modeling. (2-12) Where N(i) represents the K nearest neighbor points of point i; p i Represents the spatial coordinates of point i (x i ,y i , z i );p i Represents the spatial coordinates of point j in the neighborhood of point i; K is the number of neighboring points; ρ i represents the local density of point i; c is the coordinate of the global center point; d i Represents the distance information from a point to a local region or global center point. A geometrically-aware weighting mechanism is introduced to optimize the self-attention weight by combining local density and global center distance. The specific formula is 2-13.

[0063] (2-13) Where q i is the query vector of point i; k j is the key vector of point j; λ is the geometric feature weight adjustment coefficient; g is the geometric adjustment function, which is responsible for calculating the spatial relationship and density relationship between two points; the exponent τ represents the vector inner product operation. Through this process, the model can not only capture the semantic similarity between points, but also accurately reflect the intrinsic relationship between spatial position and geometric structure. Based on the attention score e ij , further obtain the normalized attention weight through the Softmax function, and obtain the aggregated features of the i-th point cloud token, as shown in Formula 2-14.

[0064] (2-14) Where, v j is the value vector of the jth point, which is obtained by learning the geometric features and semantic features of the point. The vector z output by the self-attention mechanism is i Then pass through a two-layer feedforward neural network (with ReLU nonlinear activation function) to perform feature transformation. The calculation formula is as follows: (2-15) Where W1 and W2 are network weights; b1 and b2 are bias vectors. This process is used to enhance the nonlinear ability of the model feature expression and enhance the encoder's perception and expression ability of the geometric structure. The geometric feature vector after the feedforward neural network processing is , further through the adapter structure h i=Adapter( ) is mapped to a more compact and efficient feature space, which is convenient for the subsequent fusion operation of the large language model. Finally, the feature vector h output by the geometric perception encoder is i Mapped to the space of the 3D large language model through a linear projector.

[0065] Example In a Ming and Qing dynasty ancient building complex, many damaged bracket sets needed to be repaired. Traditional methods relied on manual surveying and empirical judgment, which was time-consuming and prone to missing details. This technical solution was adopted, as follows: Step 1: Data collection and preprocessing Equipment: A Faro Focus S 350 laser scanner (0.1 mm accuracy) was used in conjunction with a Leica TS16 total station for spatial positioning.

[0066] Data Acquisition: Scan and acquire bracket arch point clouds (approximately 8 million points per component), simultaneously capturing text data (e.g., "Bracket Arch - Ming Dynasty - South Hall - Number of Overhanging Floors 3"). RTK positioning records the component's global coordinates (error <2 cm).

[0067] Preprocessing: Denoising: A statistical outlier removal algorithm (σ=0.05) was used to remove noise points blocked by scaffolding (reducing the data volume by approximately 15%).

[0068] Downsampling: Voxel grid method (voxel size 3 cm³) compresses the data to 3 million points, preserving the details of the bracket and mortise and tenon joints.

[0069] Segmentation: Based on curvature (threshold 0.008) and normal consistency (angle difference <10°), the main bracket and the damaged part (ID: DG_01_Defect) are separated.

[0070] Step 2. Multimodal model training and inference Model input: Point cloud: preprocessed bracket data (ID: DG_01).

[0071] Text: Associated component properties ("Ming Dynasty", "Load-bearing", "South Hall").

[0072] Cross-modal alignment: The 3D-MELL model calculates the distance between the local density (K=32 nearest neighbors) and the global centroid to generate a geometric feature vector (dimension 768). The BERT module parses the historical description of "Ming Dynasty Dougong" in the text and outputs a semantic vector (dimension 768). A cross-modal contrastive loss (τ=0.07) is used to align the two features, with a similarity threshold of 0.85.

[0073] Semantic reasoning: Input query: "Mark all Ming Dynasty bracket sets with ≥3 overhanging layers".

[0074] Model output: Highlight components that meet the requirements and associate repair plans (such as "mortise and tenon reinforcement" and "wood antiseptic treatment").

[0075] 3. Dynamic Incremental Learning and Repair Verification New component adaptation: Discovered a previously undocumented Qing Dynasty eaves column bracket (ID: DG_New), scanned the point cloud, and associated it with the text ("Qing Dynasty - West Wing - Decorative"). Triggered PEFT incremental fine-tuning: Only the low-rank matrix (rank r=8) and the structured hint vector ("Bracket - Decorative - ZONE_12") were updated, taking 3 minutes and 42 seconds.

[0076] Real-time interactive verification: Construction workers use the B / S system to ask questions such as: "Does this bracket have load-bearing function?" Based on the fine-tuned model, the system responded: “No, the brackets in the west wing during the Qing Dynasty are decorative components. We recommend using lightweight restoration materials.

[0077] The technical solution provided by this invention enables automated semantic recognition of bracket sets, historical context analysis, and restoration plan recommendations. The intelligent geometric reasoning and semantic understanding method based on a three-dimensional large language model has been applied to a national cultural heritage project, successfully restoring 23 bracket sets, saving over 60% in labor costs, and adapting to four newly discovered components through dynamic incremental learning. The semantic labels output by the system achieve 98% consistency with historical archives, providing a reusable technical paradigm for the digital preservation of ancient buildings.

[0078] The number of devices and processing scales described herein are intended to simplify the description of the present invention. Applications, modifications, and variations of the present invention will be readily apparent to those skilled in the art.

[0079] Although the embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the description and implementation methods. They can be fully applied to various fields suitable for the present invention. For those familiar with the art, additional modifications can be easily implemented. Therefore, without departing from the general concept defined by the claims and the scope of equivalents, the present invention is not limited to the specific details and illustrations shown and described herein.

Claims

1. An intelligent geometric reasoning and semantic understanding method based on a three-dimensional large language model, characterized by: The following steps are involved: Step 1: Collect original 3D point cloud data of building components through 3D scanning equipment; Generate text data based on the original 3D point cloud data association; Step 2: Preprocess the original 3D point cloud data; Step 3: Build a multimodal 3D large language model, which includes a geometric perception encoding module, a contextual semantic understanding module, and a parameter efficient fine-tuning module; Among them, the geometric perception encoding module calculates geometric features through local density and global center distance, constructs a shared semantic embedding space and a cross-modal cross-attention module; The contextual semantic understanding module is based on a bidirectional transformer encoder and is pre-trained with a masked language model and the next sentence prediction task; The parameter-efficient fine-tuning module combines low-rank adaptation with structured hint vectors to configure an incremental model update mechanism; Step 4: Use the preprocessed 3D point cloud data obtained in step 2 and the text data obtained in step 1 as training sample sets to train and optimize the multimodal 3D large language model; Step 5: Input the 3D point cloud data and text data to be tested into the trained and optimized multimodal 3D large language model, and output the semantic recognition, attribute completion and historical background analysis results of the building components.

2. The intelligent geometric reasoning and semantic understanding method based on a three-dimensional large language model according to claim 1, characterized in that: Step 2 is as follows: De-noising, standardization and segmentation of original 3D point cloud data, extracting the geometric features of each building component and assigning a unique attribute ID; The uniform downsampling algorithm is used to downsample the original 3D point cloud data, and the redundant information is removed through isolated point detection and abnormal density point screening strategies.

3. The intelligent geometric reasoning and semantic understanding method based on a three-dimensional large language model according to claim 1, characterized in that: The training optimization of the multimodal 3D large language model in step 4 specifically includes: Based on the training sample set formed by preprocessed 3D point cloud data and text data, the geometry perception encoding module and the contextual semantic understanding module are jointly pre-trained using the cross-modal contrast loss function and the structure preservation loss function to align the geometric features with the text features in a shared semantic embedding space. During the training process, the cross-modal attention module performs bidirectional interaction between the point cloud features output by the geometric perception encoding module and the text features output by the contextual semantic understanding module to generate a joint feature representation that integrates multimodal semantics. Based on the pre-trained model, the adaptability of the parameter efficient fine-tuning module is optimized through task instruction examples: a. Dynamically encode the features of the newly added component region using a structured hint vector, which is generated based on the component's functional attributes and spatial region ID; b. Update the local parameters corresponding to the newly added component area through the incremental model update mechanism, and freeze the remaining parameters.

4. The intelligent geometric reasoning and semantic understanding method based on a three-dimensional large language model according to claim 1, characterized in that: The geometry-aware encoding module calculates geometric features by local density and global center distance, constructs a shared semantic embedding space and a cross-modal cross-attention module, specifically: 1) Based on the local density ρ of each point in the point cloud data i Distance d from the global center i , construct a dual-scale geometric modulation mechanism; Local density , where K is the number of neighboring points, N(i) is the K nearest neighbor points of point i, and p i 、p i are the spatial coordinates of point i and its neighborhood point j respectively; Global center distance , where c is the coordinate of the global center point of the point cloud; 2) Set ρ i with d i As the geometric perception weight, the geometric adjustment function g(d i , d j , ρ i , ρ j ) Dynamically adjust the self-attention score. The specific formula is: , where q i is the query vector of point i; k j is the key vector of point j; λ is the geometric feature weight adjustment coefficient; g is the geometric adjustment function, which is responsible for calculating the spatial relationship and density relationship between two points; the index τ represents the vector inner product operation; 3) Based on attention score e ij , and further obtain the normalized attention weight through the Softmax function, and aggregate the value vector v of the neighborhood points j , and after feature transformation, output geometric feature vector , the specific calculation formula is: ， Where, e ij is the attention score; v j is the value vector of the jth point, which is learned by learning the geometric features and semantic features of the point; z i is the vector output by the self-attention mechanism, W1 and W2 are network weights; b1 and b2 are bias vectors, is the geometric eigenvector; 4) Geometry-aware feature vector It maps the image to a shared semantic embedding space through linear projection, performs cross-modal attention calculation with text modality features, and generates a joint feature representation that integrates geometry and semantics.

5. The intelligent geometric reasoning and semantic understanding method based on a three-dimensional large language model according to claim 1, characterized in that: The contextual semantic understanding module is based on a bidirectional transformer encoder and is pre-trained with a masked language model and the next sentence prediction task. Specifically: Pre-training based on text data, where: a. In the masked language model task, 15% of the architectural terminology tokens in the input text are masked. The masking strategies include: i. 80% of the masked tokens are replaced with mask symbols; ii. 10% of the masked tokens are replaced with random architectural terms; iii. 10% of the masked tokens remain unchanged; b. In the next sentence prediction task, the positive samples are actual continuous sentence pairs in the architectural text, and the negative samples are randomly combined non-continuous sentence pairs; During the fine-tuning phase, the semantic vectors output by BERT are supervised and aligned using the building component attribute labels.

6. The intelligent geometric reasoning and semantic understanding method based on a three-dimensional large language model according to claim 1, characterized in that: The parameter efficient fine-tuning module combines low-rank adaptation with structured hint vectors to configure an incremental model update mechanism, specifically: A low-rank adaptive mechanism is introduced into the fully connected layer of the multimodal 3D large language model. Specifically: The multimodal three-dimensional large language model weight matrix W∈R m×n , decomposed into W=W0+ΔW, where W0 is the frozen original weight, ΔW is the trainable low-rank matrix, ΔW=A×B T , A∈R m×r , B∈R n×r , and rank r<< min(m,n); Only perform gradient updates on the low-rank matrices A and B, keeping the W0 parameter fixed; Generate structured hint vector T∈R based on component functional attributes and spatial region ID k×d , where k is the number of cue vectors and d is the vector dimension. The cue vector is embedded into the model input by concatenating T with the point cloud feature vector to form an enhanced input Xin = Concat(Xpoint, T); Xin is mapped to the model latent space through a learnable projection matrix; Configure an incremental model update mechanism, specifically: locate the corresponding local low-rank matrix Anew, Bnew and hint vector Tnew based on the spatial region ID of the newly added component; freeze the original model parameters W0 and the low-rank matrix and hint vector of the non-newly added area, and only fine-tune Anew, Bnew and hint vector Tnew; The model output after local parameter update is normalized and fused with the global model features.

7. The intelligent geometric reasoning and semantic understanding method based on a three-dimensional large language model according to claim 1, characterized in that: The original 3D point cloud data collected in steps 1 and 2, and the data to be measured in step 5, are stored and managed in a spatial database, specifically including: Build a spatial database linked to a multimodal 3D large language model: Design a component data table and a point cloud data table. The component data table links the component identification codes in the point cloud data table through the component identification codes, forming a one-to-many data mapping relationship. The point cloud data table stores preprocessed point cloud data, including 3D coordinates (X, Y, Z), color information (RGB), and associated component ID fields. A spatial index is constructed on the 3D coordinate fields of the point cloud data table, using an R-tree or octree structure to optimize the following operations: In response to user semantic query commands, the point cloud data of the target component is quickly located through the spatial index and transmitted to the multimodal 3D large language model for semantic parsing; during incremental model updates, the corresponding point cloud data is partitioned and retrieved based on the spatial region ID of the newly added component and loaded into the training process; A dynamic data synchronization mechanism is configured. When the point cloud data of a newly added component is collected and preprocessed by a three-dimensional scanning device, a spatial region identifier is automatically assigned according to the component ID, a corresponding component data table record is generated, and the record is associated with the preprocessed point cloud data and stored. Subsequently, the incremental update process of the parameter efficient fine-tuning module is triggered to synchronously update the spatial index and model parameters.

8. An intelligent geometric reasoning and semantic understanding device based on a three-dimensional large language model, characterized by: include: A data acquisition module, which collects original three-dimensional point cloud data of building components through a three-dimensional scanning device; Obtain text data associated with original 3D point cloud data; A data preprocessing module, which is used to preprocess the original three-dimensional point cloud data; A model building module, which is used to build a multimodal three-dimensional large language model, including a geometric perception encoding module, a contextual semantic understanding module, and a parameter efficient fine-tuning module; Among them, the geometric perception encoding module calculates geometric features through local density and global center distance, constructs a shared semantic embedding space and a cross-modal cross-attention module; The contextual semantic understanding module is based on a bidirectional transformer encoder and is pre-trained with a masked language model and the next sentence prediction task; The parameter-efficient fine-tuning module combines low-rank adaptation with structured hint vectors to configure an incremental model update mechanism; A model training and optimization module is used to train and optimize the multimodal three-dimensional large language model using the pre-processed three-dimensional point cloud data obtained in step 2 and the text data obtained in step 1 as training sample sets; The parsing module is used to input the 3D point cloud data and text data to be tested into the trained and optimized multimodal 3D large language model, and output the semantic recognition, attribute completion and historical background analysis results of building components.

9. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a computer program, and when the processor executes the program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that A computer program is stored, and when the program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images