Method and system for automatic generation of high-fidelity 3D models based on multi-modal data

Abstract

This application relates to a method and system for automatically generating high-fidelity 3D models based on multimodal data. The method includes: acquiring multimodal acquisition data including multi-view images, sparse point clouds, dense laser point clouds, panoramic images, and inertial measurement signals; performing multimodal spatiotemporal joint registration on the multimodal acquisition data to obtain fused multimodal data; generating dense 3D geometric representations based on the fused multimodal data, and then performing semantic recognition and occlusion detection to obtain semantic recognition results including building component category labels and fire protection facility category labels, as well as occluded areas; performing dual-constraint AI repair on the occluded areas, using the semantic recognition results as type priors and a domain-specific geometric parameter library as size priors; and outputting a semantically encoded 3D model based on the repaired geometric representations. Through the dual-constraint AI repair mechanism, the repair results simultaneously pass semantic type verification and standard size verification, enabling the output model to be directly used in compliance scenarios such as fire protection inspections.

Description

Technical Field

[0001] This application relates to the field of digital modeling, and in particular to a method and system for automatically generating high-fidelity 3D models based on multimodal data. Background Technology

[0002] Automatic 3D model generation is a key technology in fields such as building digital twins, fire safety inspections, and intelligent operation and maintenance, used to map physical building spaces into computable virtual models. Existing methods in this field utilize multimodal acquisition devices such as UAV oblique photography, multi-view vision, and handheld LiDAR devices to collect multi-view images and point cloud data, and automatically reconstruct dense 3D geometric models of buildings based on this data. AI-driven geometric repair methods are introduced to fill in missing parts, addressing unavoidable occlusions and blind spots during the scanning process.

[0003] However, when facing the high-fidelity reconstruction requirements of architectural scenes, existing methods struggle to unify heterogeneous and time-series data such as UAV oblique photography, multi-view vision, laser point clouds, and inertial measurement into the same spatial coordinate system because the multimodal acquisition devices operate independently. As a result, the geometric accuracy of the reconstruction is limited by the misalignment between modes. Furthermore, since the geometric repair of blind spots and occluded areas relies solely on geometric neighborhood information or general AI models, it lacks the utilization of semantic priors such as building component categories and fire protection facility categories. In addition, the absence of building design codes, fire protection technical codes, and other standards as geometric dimensional constraints makes it easy to generate geometric components that appear semantically reasonable but do not conform to actual building codes. Consequently, the final output 3D model cannot be directly used in compliance scenarios such as fire protection inspections, and manual verification and rework are still required as a fallback. Summary of the Invention

[0004] In order to unify and integrate multimodal acquisition data into the same spatial coordinate system, and to repair occluded areas by joint constraints of semantic priors and domain specifications, thereby outputting semantically meaningful, compliant and usable high-fidelity 3D models, this application provides a method and system for automatically generating high-fidelity 3D models based on multimodal data.

[0005] Firstly, this application provides a method for automatically generating high-fidelity 3D models based on multimodal data, which adopts the following technical solution: A method for automatically generating high-fidelity 3D models based on multimodal data includes the following steps: S1. Acquire multimodal acquisition data, which includes multi-view images, sparse point clouds, dense laser point clouds, panoramic images, and inertial measurement signals.

[0006] S2. Perform multimodal spatiotemporal joint registration on the multimodal acquired data to obtain fused multimodal data located in a unified spatial coordinate system.

[0007] S3. Generate a dense 3D geometric representation of the target scene based on fused multimodal data.

[0008] S4. Perform semantic recognition on the dense 3D geometric representation to obtain semantic recognition results, which include building component category labels and fire protection facility category labels.

[0009] S5. Perform occlusion detection on dense 3D geometric representations to determine occlusion regions.

[0010] S6. Perform dual-constraint AI repair on the occluded area to obtain the repaired geometric representation. Dual-constraint AI repair uses semantic recognition results as type priors and the domain-specific geometric parameter library as size priors, so that the repaired geometric representation passes the verification under both type and size priors.

[0011] S7. Output a semantic 3D model based on the repaired geometric representation. The semantic 3D model includes the repaired geometric representation and semantic labels bound to the repaired geometric representation.

[0012] By adopting the above technical solution, multimodal spatiotemporal joint registration is used as a pre-fusion step, unifying heterogeneous and time-series data such as UAV oblique photography, multi-view vision, laser point cloud, and inertial measurement into the same spatial coordinate system, avoiding geometric reconstruction errors caused by intermodal misalignment. Semantic recognition results and domain-specific geometric parameter libraries are used as dual priors for AI repair of occluded areas, ensuring that the repair results pass both semantic type verification and standard size verification, avoiding the generation of illusory components with reasonable shapes but inconsistent with building codes. As a result, the final output 3D model carries component-level and facility-level semantic tags bound to geometric representations, enabling direct connection to compliance scenarios such as fire safety inspections without the need for manual verification and rework.

[0013] Optionally, the domain specification geometric parameter library stores geometric templates, size constraint intervals, and topological constraint rules corresponding to multiple semantic categories; the type prior includes building component category labels or fire protection facility category labels from the semantic recognition results; the size prior is obtained by indexing from the domain specification geometric parameter library based on the building component category labels or fire protection facility category labels, and the size prior includes the indexed geometric templates, size constraint intervals, and topological constraint rules; the dual-constraint AI repair uses the type prior as the category condition input for the repair network, and uses the geometric templates, size constraint intervals, and topological constraint rules as the joint constraints of the repair network, so that the repaired geometric representation passes the shape verification based on the geometric template, the value verification based on the size constraint interval, and the position verification based on the topological constraint rules; in response to the repaired geometric representation failing any of the shape verification, value verification, and position verification, the occluded area corresponding to the repaired geometric representation is marked as an area to be repaired again.

[0014] By adopting the above technical solution, semantic category priors and canonical geometric priors are jointly verified in the output space of the repair network, so that repair results that pass the single prior are explicitly rejected, while suppressing two types of illusory components: semantically reasonable but size-violated, and size-reasonable but semantically misaligned.

[0015] Optionally, during the dual-constraint AI repair process, a comprehensive confidence score is calculated for each occluded region. The comprehensive confidence score is obtained by weighting the semantic confidence score corresponding to the semantic recognition result, the consistency score between the repaired geometric representation and the neighborhood geometry, and the matching score of the repaired geometric representation to the domain-specific geometric parameter library. Based on the comprehensive confidence score, a graded treatment is performed on the occluded regions. The graded treatment includes assigning a level to each occluded region. The value of the level is determined by the level range in which the comprehensive confidence score corresponding to the occluded region is located.

[0016] By adopting the above technical solution, the multi-source confidence level is unified into a comprehensive confidence level, and the confidence level of each occluded area is recorded by level, which serves as a unified input for subsequent differentiated processing and iterative closed loop according to level.

[0017] Optionally, the gear level can be a first gear level, a second gear level, and a third gear level. The first gear level, the second gear level, and the third gear level correspond to the first gear level interval, the second gear level interval, and the third gear level interval, respectively. The overall confidence level corresponding to the first gear level interval is higher than that corresponding to the second gear level interval, and the overall confidence level corresponding to the second gear level interval is higher than that corresponding to the third gear level interval. In response to the gear level being the first gear level, fine repair is performed on the occluded area. The fine repair generates a fine mesh consistent with the size constraint interval and topology constraint rules based on the geometric template. In response to the gear level being the second gear level, resolution reduction interpolation repair is performed on the occluded area. The resolution reduction interpolation repair generates a low-resolution geometric representation consistent with the size constraint interval and topology constraint rules. In response to the gear level being the third gear level, geometric detail filling is not performed on the occluded area. Only the placeholder volume of the occluded area is generated, and a mark for manual verification is added to the placeholder volume.

[0018] By adopting the above technical solution, geometric inputs of different granularities are applied to the occluded areas according to the level, high-confidence areas are finely repaired, medium-confidence areas are subjected to low-resolution geometric interpolation, and low-confidence areas are left blank with manual verification marks, so as to avoid low-confidence repair from polluting the overall model credibility.

[0019] Optionally, semantic recognition and dual-constraint AI repair are performed on the dense 3D geometric representation for at least two iterations. Each iteration includes: performing semantic recognition based on the geometric representation input in this iteration to obtain the semantic recognition result corresponding to this iteration; performing dual-constraint AI repair based on the semantic recognition result corresponding to this iteration to obtain the repaired geometric representation corresponding to this iteration; calculating the comprehensive confidence level corresponding to this iteration for each occluded region, and updating the gear level corresponding to the occluded region according to the comprehensive confidence level corresponding to this iteration; wherein, the second and subsequent iterations only perform dual-constraint AI repair on occluded regions whose corresponding gear level is not the first gear level; for occluded regions whose corresponding gear level is the first gear level, the repaired geometric representation currently corresponding to the occluded region is frozen, so that the frozen repaired geometric representation no longer participates in this dual-constraint AI repair; the termination criterion for the iteration is: the gear level corresponding to all occluded regions no longer increases between two consecutive iterations, or the number of iterations reaches the preset iteration cycle limit.

[0020] By adopting the above technical solution, the identification and repair form a local iterative closed loop based on the gear level. The occluded area that has been identified as a high gear level is frozen and no longer participates in subsequent iterations, so that the iteration efficiency increases with each round and avoids the high gear area from being contaminated by low-quality re-repair.

[0021] Optionally, the domain specification geometry parameter library includes parameterizable geometric constraints that are respectively bound to multiple semantic categories. These parameterizable geometric constraints are mapped to soft constraint loss terms in the loss function of the repair network, and the soft constraint loss terms participate in the end-to-end optimization of the repair network.

[0022] By adopting the above technical solution, the domain specifications are injected into the end-to-end optimization process of the repair network in the form of parameterizable loss terms, so that the repair network will deposit the specification knowledge into the network weights during the training phase, and the candidates output during the inference phase will likely pass the post-verification in one go.

[0023] Optionally, the domain specification geometric parameter library is divided into multiple sub-libraries corresponding to different building types, and the domain specification geometric parameter library supports hot loading; in response to the switching of building type identification results, the currently effective sub-library is switched to the sub-library corresponding to the switched building type identification results; the building type identification results are obtained by performing classification on the distribution characteristics of building component category labels and fire protection facility category labels in the semantic identification results.

[0024] By adopting the above technical solution, the code sub-library can be automatically switched according to building type, and the same repair network can be reused among design codes and fire protection codes for different building types without retraining.

[0025] Optionally, S2 includes sub-steps S21-S23: S21. Using dense laser point cloud as the reference anchor point, the multi-view image, sparse point cloud and panoramic image are iteratively registered to the unified spatial coordinate system corresponding to the reference anchor point through feature point matching, so as to obtain the finely registered multi-view image, finely registered sparse point cloud and finely registered panoramic image.

[0026] S22. Perform high-frequency attitude compensation on the acquisition motion of the dense laser point cloud using inertial measurement signals, project the dense laser point cloud onto the reference frame at the same time, and obtain the compensated dense laser point cloud.

[0027] S23. Based on the benchmark anchor point, the finely registered multi-view image, the finely registered sparse point cloud, the finely registered panoramic image, and the compensated dense laser point cloud, a fused multimodal data is constructed.

[0028] By adopting the above technical solution, dense laser point cloud is used as the reference anchor point to unify the spatial coordinate system of image and point cloud mode. High-frequency inertial measurement signal is used to compensate for the attitude deviation of the acquisition motion, so that multimodal data from different sources and time sequences can be projected onto the reference frame at the same time and then fused.

[0029] Optionally, at least two acquisitions are performed on the same target scene to obtain multiple multimodal acquisition data. The multiple multimodal acquisition data are cached as time-series cumulative data according to the identification code of the target scene. When performing dual-constraint AI repair, the semantic recognition results corresponding to each acquisition are injected into the repair network as type prior extensions. The repaired geometric representation of the occluded area that has been identified as the first level in each acquisition is used as the initial value of the corresponding position in this acquisition.

[0030] By adopting the above technical solution, the results of multiple scans of the same target scene are accumulated into a temporal prior. The high-level area of ​​each scan is frozen successively, and the occluded area and the area to be manually verified decrease with the number of collections.

[0031] Optionally, the multi-view images and sparse point clouds in the multimodal acquisition data come from the first acquisition device, while the dense laser point clouds, panoramic images, and inertial measurement signals in the multimodal acquisition data come from the second acquisition device; the topological features acquired by the first acquisition device are used as the topological prior for the local fine scanning of the second acquisition device, and the local geometric details acquired by the second acquisition device are used as the detail prior for the missing viewpoints of the first acquisition device.

[0032] By adopting the above technical solution, the two acquisition devices are mutually prior at the topology layer and detail layer. The first acquisition device guides the second acquisition device to enter the low confidence region for fine scanning, and the geometric details of the second acquisition device are backfilled into the missing viewpoint of the first acquisition device.

[0033] Optionally, after S5, the following steps are also performed: based on the topological constraint rules corresponding to each semantic category in the domain specification geometric parameter library, the actual distribution of identified building components and fire protection facilities is statistically analyzed; in response to the deviation between the actual distribution and the topological constraint rules exceeding a preset back-inference threshold, the missing or non-compliant locations in the actual distribution are identified as supplementary occlusion areas, and the supplementary occlusion areas are incorporated into the occlusion areas to perform dual-constraint AI repair.

[0034] By adopting the above technical solutions, the prior knowledge of the standard is transformed from passive verification to active reverse inference, which identifies the locations where there are no geometrically apparent voids but the semantically required facilities are missing, thus making up for the blind spot of geometric void detection in terms of standard-level missing information.

[0035] Optionally, the termination criteria for the iteration also include: all occluded areas are not at the third level, and the coverage of each building component category and fire protection facility category in the semantic recognition results under the topological constraint rules of the domain specification geometric parameter library reaches a preset coverage threshold.

[0036] By adopting the above technical solution, the specification coverage rate is superimposed as an additional hard termination condition on the gear stability criterion, so that the iteration must converge on both the confidence level and the specification coverage rate, thus avoiding the premature termination situation where the confidence level converges but the specification layer is missing.

[0037] Optionally, S4 includes sub-steps S41-S42: S41. Perform component category identification on dense 3D geometric representations to obtain category labels, which include walls, doors, windows, and stairs.

[0038] S42. In response to the major category label being "door", perform subclass recognition on the geometric region identified as "door" to obtain subclass labels including fire door, ordinary door and explosion-proof door; in response to the major category label being "window", perform subclass recognition on the geometric region identified as "window" to obtain subclass labels including fire window and ordinary window; incorporate the major category label and subclass label into the semantic recognition result.

[0039] By adopting the above technical solution, semantic recognition is carried out in two levels: major component categories and facility subcategories. Major category recognition provides component-level geometric boundaries, while subcategory recognition further subdivides fire protection attributes within the major category, thereby improving both recognition accuracy and training efficiency.

[0040] Optionally, S5 includes sub-steps S51-S52: S51. Determine the geometric hole occlusion region based on the connected regions where the point cloud density is lower than the preset density threshold in the dense 3D geometric representation.

[0041] S52. Based on the topological constraint rules of each semantic category in the domain specification geometric parameter library, determine the expected topological position, and determine the connected regions in the semantic recognition results that have no component labels but have the expected topological position as topological missing occlusion regions; determine the geometric hole occlusion regions and the topological missing occlusion regions together as occlusion regions.

[0042] By adopting the above technical solution, the occlusion detection is extended from a single geometric hole dimension to a topological expectation dimension, so that the missing facility areas that are missed by the geometric hole dimension but whose locations are required by the domain specification are included in the repair object.

[0043] Optionally, S5 further includes: back-projecting dense 3D geometric representations onto the respective image planes of the multi-view image and the panoramic image to obtain a back-projection mask; in response to the number of images covered by the back-projection mask on a certain connected region being lower than a preset view coverage threshold, determining the connected region as an occlusion region with insufficient view coverage, and incorporating the occlusion region with insufficient view coverage into the occlusion region.

[0044] By adopting the above technical solution, the back projection coverage of multi-view images and panoramic images is used as a supplementary occlusion criterion, so that areas that are not visible at the same time by multi-view projection are identified as occluded areas with insufficient view coverage.

[0045] Optionally, S3 also includes: when generating dense 3D geometric representation, applying surface closure constraints to both the abrupt changes in the normal gradient of the dense laser point cloud and the boundaries of building component categories determined in the previous iteration of the dense 3D geometric representation, so that the dense 3D geometric representation does not extend across the structural boundaries.

[0046] By adopting the above technical solution, the geometric reconstruction process receives the semantic boundary of the previous iteration as a surface closure constraint, avoiding cross-boundary extension of the reconstruction result at the component boundary and improving the alignment between the reconstructed geometry and the semantic boundary.

[0047] Secondly, this application provides a high-fidelity 3D model automatic generation system based on multimodal data, which adopts the following technical solution: A high-fidelity 3D model automatic generation system based on multimodal data includes: The multimodal acquisition module is configured to acquire multimodal acquisition data, which includes multi-view images, sparse point clouds, dense laser point clouds, panoramic images, and inertial measurement signals. The registration and fusion module is configured to perform multimodal spatiotemporal joint registration on multimodal acquired data to obtain fused multimodal data located in a unified spatial coordinate system; The geometry reconstruction module is configured to generate a dense 3D geometric representation of the target scene based on fused multimodal data; The semantic recognition module is configured to perform semantic recognition on dense 3D geometric representations to obtain semantic recognition results, which include building component category labels and fire protection facility category labels. The occlusion detection module is configured to perform occlusion detection on dense 3D geometric representations to determine occlusion areas; The dual-constraint repair module is configured to perform dual-constraint AI repair on the occluded area to obtain the repaired geometric representation. The dual-constraint AI repair uses the semantic recognition result as the type prior and the domain-specific geometric parameter library as the size prior, so that the repaired geometric representation passes the verification under both the type prior and the size prior. The model output module is configured to output a semantic 3D model based on the repaired geometric representation. The semantic 3D model includes the repaired geometric representation and semantic labels bound to the repaired geometric representation. The output of the multimodal acquisition module is connected to the input of the registration and fusion module, the output of the registration and fusion module is connected to the input of the geometric reconstruction module, the output of the geometric reconstruction module is connected to the input of the semantic recognition module and the input of the occlusion detection module, the output of the semantic recognition module and the output of the occlusion detection module are connected together to the input of the dual-constraint repair module, and the output of the dual-constraint repair module is connected to the input of the model output module.

[0048] By adopting the above technical solution, each step of the above method is encapsulated into interconnected functional modules. The multimodal acquisition module and the model output module are connected in series along the data flow direction, so that the multimodal spatiotemporal joint registration, dual-constraint AI repair and semantic output at the method level have corresponding physical or logical implementations at the system level.

[0049] Optionally, the dual-constraint repair module is further configured to: calculate a comprehensive confidence score for each occluded region, the comprehensive confidence score being a weighted average of the semantic confidence score corresponding to the semantic recognition result, the consistency score between the repaired geometric representation and the neighborhood geometry, and the matching score of the repaired geometric representation to the domain-specific geometric parameter library; and perform graded processing on the occluded regions based on the comprehensive confidence score; the graded processing includes assigning a level to each occluded region, the value of which is determined by the level interval in which the comprehensive confidence score corresponding to the occluded region is located; the system also includes an iteration control module, which is configured to schedule the semantic recognition module and the dual-constraint repair module to perform at least two iterations on the dense 3D geometric representation, the termination criterion for the iteration being that the level corresponding to all occluded regions no longer increases between two consecutive iterations, or the number of iterations reaches a preset upper limit for the number of iterations.

[0050] By adopting the above technical solution, the iterative control module is introduced at the system level in conjunction with the semantic recognition module and the dual-constraint repair module in multiple rounds of calls, enabling the system to perform local iterations according to gears, which corresponds to the iterative closed loop at the method level.

[0051] In summary, this application includes at least one of the following beneficial technical effects: 1. By using multimodal spatiotemporal joint registration as a prerequisite for reconstruction and semantic recognition results and domain-specific geometric parameter libraries as dual priors for AI repair of occluded areas, heterogeneous and time-series data are uniformly fused, and the repair results are simultaneously verified by type and standard size, thus avoiding the generation of hallucinatory components.

[0052] 2. By classifying and processing the repair results according to their overall confidence level and constructing a local iterative closed loop based on the level, high-level areas are frozen and low-level areas are left blank with manual verification marks. This improves the iteration efficiency with each round and avoids low-confidence repairs from contaminating the overall model's credibility.

[0053] 3. By injecting domain specifications into the end-to-end optimization of the repair network as parameterizable loss terms and automatically switching specification sub-libraries according to building type, the repair network learns not to generate geometry that violates the specifications during the training phase and can adapt to different building types without retraining. Attached Figure Description

[0054] Figure 1 A flowchart illustrating an embodiment of the present application shows a method for automatically generating high-fidelity 3D models based on multimodal data.

[0055] Figure 2 The diagram illustrates the module connection of a high-fidelity 3D model automatic generation system based on multimodal data in one embodiment of this application. Detailed Implementation

[0056] The present application will be further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the scope of the application.

[0057] This application provides a method for automatically generating high-fidelity 3D models based on multimodal data, used to map physical building spaces into semantic 3D models that can be used for fire safety inspection. This method takes heterogeneous, multimodal data acquisition as input, unifies the spatial coordinate system through multimodal spatiotemporal joint registration, and uses semantic recognition results and a domain-specific geometric parameter library as dual priors for AI repair of occluded areas. This ensures that the repair results simultaneously pass semantic type verification and standard size verification, and classifies each occluded area according to its comprehensive confidence level, thereby outputting a high-fidelity 3D model carrying component-level and facility-level semantic labels bound to geometric representations.

[0058] For ease of subsequent description, the core technical terms involved in the embodiments of this application will be explained in advance below.

[0059] Multimodal acquisition data refers to the raw data set obtained by multiple heterogeneous sensing devices simultaneously or at different time periods from the same target scene. In the embodiments of this application, it includes five types of data: multi-view images obtained by UAV oblique photography, sparse point clouds obtained by visual stereo matching, dense laser point clouds obtained by laser scanning, panoramic images obtained by panoramic cameras, and inertial measurement signals obtained by inertial measurement units.

[0060] The domain-specific geometric parameter library is a database that pre-models the geometric shape, size range, and topological distribution rules of various building components and fire protection facilities according to national or industry standards such as building design codes and fire protection technical codes. It serves as a priori source of dimensions for subsequent AI repair.

[0061] Dual-constraint AI repair refers to the repair process performed on occluded regions with geometric defects, which is jointly constrained by semantic category priors and domain-specific geometric priors. The semantic category priors come from the semantic recognition results of the geometric representations, while the geometric priors come from the index of the domain-specific geometric parameter library.

[0062] The overall confidence score is a quantifiable measure of the reliability of the repair results for each occluded region. It is weighted by the semantic recognition confidence score, the geometric neighborhood consistency score, and the matching score to the domain-specific geometric parameter library. The overall confidence score falling into three preset numerical ranges corresponds to the first, second, and third levels, respectively, and repair processing of the occluded region is performed at different granularities accordingly.

[0063] The following describes a specific application scenario in detail. This application scenario involves a building with a floor area of ​​approximately 35,000 m². 2 The eight-story commercial complex includes a shopping mall, office building, and underground parking garage, and is equipped with fire hydrants, automatic sprinkler heads, fire-resistant roller shutters, smoke detectors, emergency lighting, evacuation signs, and other fire protection facilities. Subsequent numerical examples will be based on this baseline scenario. First, multimodal data acquisition is performed on the target scenario to be reconstructed.

[0064] S1. Acquire multimodal acquisition data, which includes multi-view images, sparse point clouds, dense laser point clouds, panoramic images, and inertial measurement signals.

[0065] Specifically, the multi-view images are obtained by a drone with controllable attitude equipped with an oblique photography camera, imaging the target scene from multiple angles along a preset flight path. Each image has a resolution of at least 8K, and adjacent images have a field-of-view overlap of at least 60% to ensure subsequent feature point matching. The sparse point cloud is obtained by performing stereo vision matching on the multi-view images, reflecting the 3D positions of edge points and corner points with significant texture in the target scene. The dense laser point cloud is obtained by a handheld LiDAR scanner with scanning capabilities, scanning each area segment by segment along the internal movement path of the target scene, reflecting the 3D positions of various surfaces in the indoor space with millimeter to centimeter precision. A single frame scan covers an area of ​​approximately 60m. 2 The accuracy can reach 2cm. The panoramic image is synchronously acquired by a panoramic camera mounted coaxially or on the same base as the LiDAR scanner using a 360-degree spherical projection method, serving as the visual source for subsequent semantic rendering and texture mapping. The inertial measurement signal is output by the inertial measurement unit built into the second acquisition device at a high-frequency sampling rate, including three-axis acceleration and three-axis angular velocity, with a sampling frequency of approximately 1000Hz, to compensate for the operator's movement posture during handheld scanning.

[0066] For the aforementioned commercial complex scenario, field personnel first operate drones to fly along the perimeter of the target scene in a figure-eight pattern, collecting multi-view images and sparse point clouds covering the building facade, roof, and external site; then, office personnel use handheld LiDAR scanners to enter the target scene and scan in segments along the circulation routes of the shopping mall corridor, office building standard floor corridor, underground parking garage passage, etc., simultaneously collecting dense LiDAR point clouds, panoramic images, and inertial measurement signals.

[0067] The five types of data play different registration roles in the subsequent S2: dense laser point clouds serve as a reference anchor point to provide millimeter-level geometric references; multi-view images, sparse point clouds, and panoramic images are precisely registered to the unified spatial coordinate system corresponding to the reference anchor point through feature point matching; and inertial measurement signals are used to eliminate the disturbance of the dense laser point cloud to the motion posture of the handheld device.

[0068] In some embodiments, multi-view images and sparse point clouds are obtained from a first acquisition device, i.e., an unmanned aerial vehicle (UAV) platform, while dense laser point clouds, panoramic images, and inertial measurement signals are obtained from a second acquisition device, i.e., a handheld LiDAR scanner. The topological features of the first acquisition device serve as topological priors for the local fine scanning of the second acquisition device, and the local geometric details of the second acquisition device serve as detail priors for the views missing from the first acquisition device. This implementation achieves the goal of acquiring complete multimodal data at both the topological and detail layers during the S2 registration and fusion stage through dual-device coverage complementarity and bidirectional prior backflow.

[0069] S2. Perform multimodal spatiotemporal joint registration on the multimodal acquired data to obtain fused multimodal data located in a unified spatial coordinate system.

[0070] The "spatiotemporal joint registration" in S2 includes two dimensions: spatial anchoring and temporal alignment. Spatial anchoring refers to projecting all data that were originally in five independent acquisition coordinate systems to the same spatial coordinate system; temporal alignment refers to performing inter-frame compensation on data sampled by different devices at different timestamps under the same time reference system, so that when performing fusion, each modal data describes the state of the target scene at the same moment.

[0071] The input to S2 is the five types of raw multimodal acquisition data obtained from S1, and the output is fused multimodal data located in a unified spatial coordinate system. Each spatial point in the fused multimodal data carries geometric position, visual texture, and inertial compensation information. The multi-view images and panoramic images acquired by S1 have pixel-level color and semantic information but lack precise 3D position. Sparse point clouds have a small number of high-confidence geometric points but the density is insufficient to reconstruct the surface. Dense laser point clouds have millimeter-level geometric positions but lack texture and semantics. Inertial measurement signals reflect the attitude changes of the acquisition device but do not carry scene information itself. These five types of data need to be fused by S2 to provide a complete geometric + texture + temporal input for S3 geometric reconstruction.

[0072] Specifically, S2 includes sub-steps S21-S23.

[0073] S21. Using dense laser point cloud as the reference anchor point, the multi-view image, sparse point cloud and panoramic image are iteratively registered to the unified spatial coordinate system corresponding to the reference anchor point through feature point matching, so as to obtain the finely registered multi-view image, finely registered sparse point cloud and finely registered panoramic image.

[0074] Specifically, dense laser point clouds, due to their millimeter- to centimeter-level geometric accuracy and independence from lighting conditions, are used as reference anchor points for a unified spatial coordinate system.

[0075] S22. Perform high-frequency attitude compensation on the acquisition motion of the dense laser point cloud using inertial measurement signals, project the dense laser point cloud onto the reference frame at the same time, and obtain the compensated dense laser point cloud.

[0076] Specifically, the inertial measurement signal outputs triaxial acceleration and triaxial angular velocity at a high-frequency sampling rate of 1000Hz, provided by the inertial measurement unit built into the second acquisition device. During the handheld LiDAR scanner's walking scan along the movement line, the operator's hand and body postures cause attitude disturbances to the scanner, resulting in millisecond-level pose shifts between adjacent frames in the dense LiDAR point cloud. In S22, the sampling timestamp of each dense LiDAR point cloud frame is interpolated using the attitude time series synchronously output by the inertial measurement unit to recover the instantaneous pose of the scanner at the sampling time of each frame, and each point cloud frame is projected onto a preset reference frame, which is typically the pose at the start of the scanning movement. For example, in the scanning scenario of an underground parking garage in a commercial complex, the scanner moves at a walking speed of about 1 m / s, and the single-frame scanning cycle is about 100 ms. During this period, the inertial measurement unit samples 100 attitude samples. By using spline interpolation to restore the continuous pose trajectory within the scanning cycle, the point cloud that was originally scattered under different poses at different times is reprojected onto the reference frame. The stitching error between the geometric overlap areas of adjacent frames is reduced from the centimeter level to the millimeter level.

[0077] S23. Based on the benchmark anchor point, the finely registered multi-view image, the finely registered sparse point cloud, the finely registered panoramic image, and the compensated dense laser point cloud, a fused multimodal data is constructed.

[0078] Specifically, the fused multimodal data is organized into a dense point cloud data structure with semantic and texture channels according to a unified spatial coordinate system of the target scene. The fields of each spatial point include 3D position, geometric accuracy from the compensated dense laser point cloud, color and texture from the finely registered multi-view images and panoramic images, visual feature descriptors from the finely registered sparse point cloud, and acquisition time reference from the inertial measurement signal. The fusion of multimodal data allows subsequent S3 geometric reconstruction to directly use the three channels of "geometric accuracy + texture + temporal sequence" as input, without having to go back to the original data to extract them again.

[0079] S3. Generate a dense 3D geometric representation of the target scene based on fused multimodal data.

[0080] Specifically, dense 3D geometric representations can be implemented in various engineering forms. In some embodiments, dense 3D geometric representations adopt a dense point cloud form, where each spatial point is directly composed of the position and density fields from the fused multimodal data; in other embodiments, dense 3D geometric representations adopt a triangular mesh form, generated by performing surface reconstruction algorithms such as Poisson reconstruction or MarchingCubes on the dense point cloud in the fused multimodal data; in still other embodiments, dense 3D geometric representations adopt a neural implicit field form, such as a continuum representation of NeRF or SDF networks. Regardless of the specific form, dense 3D geometric representations cover the geometric positions of all observable areas in the target scene, serving as a unified input for subsequent S4 semantic recognition and S5 occlusion detection.

[0081] In some embodiments, S3 further includes applying surface closure constraints to both the abrupt changes in the normal gradient of the dense laser point cloud and the boundaries of building component categories determined in the previous iteration of the dense 3D geometric representation, preventing the dense 3D geometric representation from extending across the structural boundaries. The surface closure constraint incorporates a penalty term for surface normal discontinuities near the boundary into the reconstruction optimization objective, resulting in a discontinuous surface aligned with the boundary at the structural boundary. This optional embodiment serves as a semantic guidance supplement to the geometric reconstruction of S3, enabling the geometric boundary and semantic boundary to jointly constrain the current reconstruction when connecting from the previous iteration.

[0082] S4. Perform semantic recognition on the dense 3D geometric representation to obtain semantic recognition results, which include building component category labels and fire protection facility category labels.

[0083] Specifically, the set of values ​​for the building component category labels includes common structural component categories in the target scenario of this case, such as walls, floors, beams, columns, doors, windows, stairs, elevator shafts, and roofs. The set of values ​​for the fire protection facility category labels includes equipment categories that need to be included in the fire protection inspection, such as fire hydrants, automatic sprinkler heads, fireproof roller shutters, smoke detectors, heat detectors, emergency lighting, evacuation signs, fire extinguisher boxes, and fire doors. Both types of labels together cover all identifiable objects in the target scenario within the fire protection inspection context. The semantic recognition used in S4 can be a deep learning model based on point clouds, such as PointNet++, or an SDF classification network based on neural implicit fields. This application embodiment does not limit the selection of a specific recognition model.

[0084] For the aforementioned commercial complex scenario, S4 outputs a pair of category labels for each spatial point or surface element in the dense 3D geometric representation, recording the category of the building component to which it belongs and whether the location is covered by fire protection facilities. For example, the floor surface of the atrium area on the first floor of the shopping mall is identified as a "floor slab" building component, and several discrete locations at a certain height above it are identified as "automatic sprinkler heads" fire protection facilities; the ceiling surface of the underground parking garage passage is identified as a "floor slab", and the grid-like discrete locations on it are identified as "emergency lighting" and "evacuation signs".

[0085] The semantic recognition results play a dual role in subsequent steps. First, in S6, they are injected as type priors into the dual-constraint AI repair—for each occluded area, the corresponding geometric template is indexed from the domain specification geometric parameter library according to the corresponding building component category or fire protection facility category, serving as a reference for the generation of the repair network. Second, in S7, they are bound to the repaired geometric representation as semantic labels in the output—ensuring that each spatial point in the final output semantic 3D model carries both geometric location and category attributes, which can be directly used for object recognition and compliance checks in fire protection inspection tools.

[0086] In engineering implementation, when the semantic confidence score of the recognition model for a certain spatial point is lower than the preset semantic confidence score threshold, the category label of the spatial point is set to "to be confirmed," and it is treated with low weight when calculating the comprehensive confidence score in S6. This handling of boundary cases allows the uncertainty carried in the semantic recognition result to be passed on to subsequent hierarchical processing, preventing recognition results with low confidence scores from being misused as high-confidence priors.

[0087] In some embodiments, S4 includes sub-steps S41-S42.

[0088] S41. Perform component category identification on dense 3D geometric representations to obtain category labels, which include walls, doors, windows, and stairs.

[0089] Specifically, in stage S41, a broad category recognition model is used to output a broad category label for each spatial region in the dense 3D geometric representation. Broad category recognition only distinguishes component categories with significantly different main geometric shapes, without delving into subcategories based on fire safety attributes. For example, in a shopping mall's ground-floor corridor scene, continuous surfaces along both sides of the corridor are identified as "walls," openings connecting the corridor to shops and their top and bottom edges are identified as "doors," glass curtain wall openings connecting the corridor to the external site are identified as "windows," and stepped surfaces at the end of the corridor connecting to the second floor are identified as "stairs." The output of S41 only contains these four broad category labels.

[0090] S42. Responding to the major category label "door", perform sub-class recognition on the geometric region identified as a door to obtain sub-class labels including fire door, ordinary door, and explosion-proof door. Responding to the major category label "window", perform sub-class recognition on the geometric region identified as a window to obtain sub-class labels including fire window and ordinary window. Incorporate both the major category label and the sub-class labels into the semantic recognition result.

[0091] Specifically, in stage S42, subclass identification is performed on the geometric areas identified as doors or windows in S41. Subclass identification only applies to the two main categories related to fire safety attributes: doors and windows, without further subdivision of walls and stairs. Subclass identification is based on the geometric features of doors and windows and visual features in the panoramic image. Geometric features of doors and windows include, for example, fire doors typically have fire-resistant edges and door closer tracks, while explosion-proof doors typically use thicker plates and reinforced structures; visual features include the "fire door" marking on the door panel and the fire resistance rating label on the nameplate. The identification results are recorded as subclass labels and incorporated into the semantic identification results along with the parent category labels.

[0092] For the aforementioned commercial complex scenario, a typical identification situation in S42 is as follows: Openings connecting shopping malls to fire escape stairwells are identified as "Doors" by S41, and further subdivided into "Fire Doors" by S42 based on the Class A fire door nameplate and door closer characteristics on the door panel; Entrances to office spaces on standard floors of office buildings are identified as "Doors" by S41, and further subdivided into "Ordinary Doors" by S42 based on the absence of fire-resistant nameplates and door closer tracks. Although both types of doors share the same category label in S41, the subcategory label output by S42 allows the fire inspection tool to further differentiate the compliance requirements of the two types of doors. Specifically, fire doors must meet certain fire resistance ratings, while ordinary doors are not within the scope of fire inspection.

[0093] By dividing component identification into two levels, major categories and subcategories, major category identification can focus on the main geometric shape, while subcategory identification can focus on the detailed features of fire protection attributes. The objectives of the two identification tasks are more focused, and the required accuracy can be achieved by using models trained for their respective tasks.

[0094] S5. Perform occlusion detection on dense 3D geometric representations to determine occlusion regions.

[0095] Specifically, the occluded area refers to a connected spatial region in the target scene where structural surfaces or fire-fighting facilities should exist, but the dense 3D geometric representation fails to fully present this geometric information. In this embodiment, the occluded area includes two levels and four engineering forms: The first level is the lack of acquisition end, including acquisition blind spots (areas that cannot be directly observed by the acquisition device due to physical path limitations, such as the mezzanine above the ceiling, the inside of the cable tray, and the space inside the equipment room cabinet), low-density areas (connected areas where the point cloud density or surface element density in the dense 3D geometric representation is significantly lower than the surrounding areas, reflecting incomplete acquisition and observation of the above areas), and areas with insufficient view coverage (areas where some data is acquired, but insufficient view coverage results in a certain face or side of the geometric representation being unreconstructable); The second level is the lack of specification end, i.e., the topological missing area, which refers to the location of a certain type of facility that should exist according to the domain specification but is not identified in the semantic recognition results, even though it is geometrically complete.

[0096] For the aforementioned commercial complex scenarios, such as automatic sprinkler heads installed above the ceiling on the first floor of a shopping mall, the physical obstruction of the ceiling panel prevents drone oblique photography and handheld LiDAR scanners from directly capturing the geometric position of the sprinkler head itself—only the lower surface of the ceiling panel and the portion of the sprinkler nozzle extending downwards from the ceiling panel can be captured. In S5, the height range between the ceiling panel and the sprinkler head itself is detected as a blind spot and marked as an obstructed area. Similarly, in underground parking garages where main fire hydrant pipes run along the ceiling, there is a gap of approximately 30cm between the pipe and the ceiling. Due to the limited scanning angle, the brackets, clamps, connecting flanges, and other auxiliary components within this gap are not fully captured, forming a low-density area, which is also marked as an obstructed area.

[0097] The occluded regions, as input to the S6 dual-constraint AI repair, are organized as connected regions within a unified spatial coordinate system of the target scene. Each occluded region carries its boundary definition in the unified spatial coordinate system, its corresponding building component category or fire protection facility category (inherited from the labels of adjacent identified regions in the S4 semantic recognition results), and low-density quantization parameters reflecting the completeness of the data collection. Subsequently, S6 performs repair on each occluded region based on these parameters.

[0098] In some embodiments, S5 includes sub-steps S51-S52.

[0099] S51. Determine the geometric hole occlusion region based on the connected regions where the point cloud density is lower than the preset density threshold in the dense 3D geometric representation.

[0100] Specifically, the preset density threshold is set according to the nominal scanning accuracy during target scene acquisition, for example, 60m. 2 The expected average point density of a dense laser point cloud with an accuracy of 2cm is approximately 2500 points / m. 2 The preset density threshold can be set to 500 points / m².2 This is 20% of the desired density. S51 performs spatial block density statistics on dense 3D geometric representations, merging adjacent spatial blocks with point densities below a preset density threshold into connected regions, and recording each connected region as a geometric void occlusion region. For example, in the scenario of a fire pump room in an underground parking garage, due to the obstruction of equipment cabinets, approximately 8m behind the pump room... 2 The point density in the ceiling area is only about 200 points / m². 2 It was identified as a geometrically occluded area.

[0101] S52. Based on the topological constraint rules of each semantic category in the domain-specific geometric parameter library, determine the desired topological location. Identify connected regions in the semantic recognition results that lack component labels but have the desired topological location as topologically missing occluded regions. Combine geometrically empty occluded regions and topologically missing occluded regions as occluded regions.

[0102] Specifically, topological constraint rules record the location patterns that various fire protection facilities should have in the building space. For example, automatic sprinkler heads are required by regulations to be spaced every 12m. 2 Up to 16m 2 Install one smoke detector per 30m floor level, evenly distributed below the floor slab. 2 One fire hydrant is installed on the ceiling surface. A set of fire hydrants is installed every 50m along the corridor, located on the corridor columns or outside the walls, as required by regulations. In S52, based on the building component category identification results of the current target scene, such as geometric statistics like floor area and corridor length, and combined with the topological constraint rules in the corresponding sub-library of the building type, the expected topological location of various fire protection facilities is calculated. Then, locations in the expected location without the corresponding fire protection facility category label are marked as topological missing occlusion areas.

[0103] For the aforementioned commercial complex scenario, such as a shopping mall with a ground floor atrium area of ​​approximately 600 square meters... 2 According to the standard, every 30m 2 Suppose a smoke detector is expected to be deployed with 20 units, but the S4 semantic recognition results only identify 16 smoke detector locations. The remaining 4 expected locations that should exist according to the specifications but were not identified are marked as topological missing occlusion areas by S52.

[0104] The geometric void occlusion area output by S51 reflects incompleteness at the acquisition end, while the topological missing occlusion area output by S52 reflects non-compliance at the specification end. The geometric void dimension only covers the missing parts at the acquisition end; semantic-level missing parts such as "the ceiling was acquired, but the facilities required by the specification below the ceiling were not identified" will not trigger a flag because the geometric acquisition of the ceiling itself is complete. After introducing the topological missing dimension, specification-level missing parts are also included in the occlusion area definition, enabling subsequent S6 remediation to uniformly handle missing parts from two different sources.

[0105] In some embodiments, S5 further includes back-projecting dense 3D geometric representations onto the respective image planes of the multi-view image and the panoramic image to obtain a back-projection mask. In response to the back-projection mask covering fewer images in a connected region than a preset view coverage threshold, the connected region is determined as an occlusion region with insufficient view coverage, and this occlusion region is incorporated into the occlusion region. In this embodiment, the preset view coverage threshold is set to 2, meaning that each spatial point needs to be covered by at least two images in different poses. This optional embodiment, as a supplement to the view dimension of occlusion detection in S5, together with the geometric hole dimension and the topological missing dimension, constitutes the three sources of occlusion regions.

[0106] In some embodiments, after S5, the following is further performed: The actual distribution of identified building components and fire protection facilities is statistically analyzed according to the topological constraint rules corresponding to each semantic category in the domain specification geometric parameter library. In response to a deviation between the actual distribution and the topological constraint rules exceeding a preset back-calculation threshold, locations in the actual distribution that are missing or have spacing that does not conform to the specifications are identified as supplementary occlusion areas, and these supplementary occlusion areas are incorporated into the occlusion area for dual-constraint AI repair. Specifically, the back-calculation threshold in this embodiment includes two criteria: the first criterion is a shortage of more than one facility, and the second criterion is a deviation in facility spacing exceeding 30% of the allowable spacing specified in the specifications. Meeting either criterion constitutes a deviation exceeding the back-calculation threshold. For example, a shopping mall's first-floor corridor is approximately 280m long. According to specifications, a fire hydrant should be installed every 50m, with an expected deployment of 6 fire hydrants. The S4 identification result only identifies 4 fire hydrant locations, with 2 missing. The expected topological locations of the 2 missing fire hydrants are deduced, for example, next to columns approximately 40m from each end of the corridor. These locations are defined as supplementary occlusion areas and incorporated into the occlusion area. This optional embodiment serves as a supplementary proactive backtesting of the S5 occlusion detection specification, enabling locations that should exist according to the specification but have not been marked in either the S5 geometric hole dimension or the topological missing dimension to be included in the repair object.

[0107] S6. Perform dual-constraint AI repair on the occluded area to obtain the repaired geometric representation. Dual-constraint AI repair uses semantic recognition results as type priors and the domain-specific geometric parameter library as size priors, so that the repaired geometric representation passes the verification under both type and size priors.

[0108] Specifically, the technical forms of type prior, size prior, and double pass verification are as follows: Type prior is a description of the component or facility category to which each occluded area belongs, derived from the S4 semantic recognition results. In this embodiment, it is input into the repair network in the form of one-hot encoding or semantic embedding vectors of component or facility category labels, serving as the category condition when the repair network generates the geometric shape. Size prior is a set of geometric templates, size constraint intervals, and topological constraint rules corresponding to the category, derived from the domain specification geometric parameter library. In this embodiment, it is input into the repair network in the form of parameterized geometric constraint terms, serving as the size condition when the repair network generates the geometric shape. Double pass verification refers to three independent rounds of judgment on the repair result generated by the repair network: shape verification based on geometric templates, value verification based on size constraint intervals, and position verification based on topological constraint rules. Shape verification determines whether the generated geometric shape falls within the neighborhood of the geometric template of the category to which it belongs; value verification determines whether the generated geometric dimensions are within the size constraint interval; and position verification determines whether the generated topological position conforms to the topological constraint rules. The repair result is accepted only when all three rounds of verification pass.

[0109] The injection path from the S4 semantic recognition results and the domain-specific geometric parameter library to the repair network forms two parallel channels. The first channel extracts the component or facility category label corresponding to each occluded area from the S4 output, encodes it into a type prior vector, and then sends it to the semantic condition input of the repair network. The second channel indexes the corresponding geometric template and parameter constraint terms from the domain-specific geometric parameter library according to the above categories, assembles them into a size prior vector, and sends it to the geometric condition input of the repair network. During the generation stage, the repair network receives two condition inputs simultaneously, outputs a set of candidate repair geometries, and then filters out the repair geometries that pass the dual verification under type prior and value and position verification under size prior as the repaired geometric representation.

[0110] For the aforementioned commercial complex scenario, such as the repair task of the sprinkler head above the ceiling on the first floor of a shopping mall, the type prior comes from the "automatic sprinkler head" category label output by S4, and the size prior comes from the geometric template, size constraint range, and topology constraint rules corresponding to the "automatic sprinkler head" category in the specification library. The geometric template is the standard structure of the sprinkler cap, connecting pipe, and temperature sensing element; the size constraint range is 15mm to 25mm for the sprinkler cap diameter and 50mm to 80mm for the connecting pipe length; the topology constraint rules require the connecting pipe axis to be perpendicular to the lower surface of the floor slab and the sprinkler cap to face downwards. After the repair network generates candidate sprinkler head geometry, it first confirms that the generated shape belongs to the standard topology combination of the sprinkler head, connecting pipe, and temperature sensing components through morphological verification based on the geometric template. Then, it confirms that the dimensions of each component are within the constraint range through value verification based on the size constraint range, and confirms that the connecting pipe axis direction meets the vertical requirement through position verification based on the topology constraint rules. After all three verifications pass, the candidate geometry becomes the geometric representation of the repaired component.

[0111] When a single prior is used as a constraint for repair, the output space of the repair network is constrained only in a single dimension. When only the type prior is used as a constraint, the geometry generated by the repair network conforms to the category topology but the size is not constrained. This may result in output geometry with a reasonable shape but deviating from the standard size, such as an unconventional sprinkler head with a diameter of 50mm. When only the size prior is used as a constraint, the geometry generated by the repair network is compliant in size but unconstrained in category affiliation. This may result in output geometry with compliant size but misaligned category, such as generating a "fire hydrant and corridor" combination based on corridor location and pipeline size constraints, but the actual location should be a sprinkler head. Dual-constraint AI repair introduces two priors for joint verification, ensuring that the repair result simultaneously satisfies the correct category at the type level and the compliance of shape, value, and location at the geometric level. Candidates with compliant geometric parameters but shapes deviating from the category template are rejected by shape verification; candidates with compliant shapes but non-compliant sizes are rejected by value verification; and candidates with compliant shapes and sizes but non-compliant topological locations are rejected by location verification.

[0112] By using dual prior joint verification, the candidate output space of the repair network is compressed to the intersection of type, shape and size compliance, thus suppressing illusionary components that are compliant in one dimension but violate joint dimensions.

[0113] Specifically, the domain specification geometry parameter library stores geometric templates, size constraint intervals, and topological constraint rules corresponding to multiple semantic categories.

[0114] Type priors include building component category labels or fire protection facility category labels in the semantic recognition results.

[0115] Dimensional priors are obtained by indexing from the domain specification geometry parameter library based on building component category labels or fire protection facility category labels. Dimensional priors include the indexed geometric templates, dimensional constraint intervals, and topological constraint rules.

[0116] The dual-constraint AI repair uses type prior as the category condition input for the repair network, and geometric template, size constraint interval and topology constraint rule as the joint constraints of the repair network, so that the repaired geometric representation can pass the morphological verification based on geometric template, the value verification based on size constraint interval and the position verification based on topology constraint rule.

[0117] If the repaired geometric representation fails any of the shape verification, value verification, or position verification, the occluded area corresponding to the repaired geometric representation is marked as an area to be repaired again.

[0118] Specifically, the engineering format for geometric templates, dimensional constraint intervals, and topological constraint rules is as follows: Geometric templates are stored in the form of parametric geometric descriptions. Each category corresponds to a set of key geometric features and the nominal values ​​of these features. Key geometric features include component skeletons, connection topologies, and the relative positions of sub-components. The nominal values ​​of key geometric features serve as morphological references when generating repair networks. Dimensional constraint intervals are stored as a set of intervals consisting of the minimum and maximum allowable values ​​of each geometric feature in each category. Topological constraint rules are stored in the form of first-order logic assertions or graph theory descriptions, such as "fire doors must include door closers", "fire hydrants must be arranged along the outside of corridor columns", and "automatic sprinkler heads must be perpendicular to the lower surface of the floor slab".

[0119] The indexing paths for type prior and size prior are as follows. For each occluded region, the component category or facility category to which it belongs is first determined. The determination method includes inheriting labels from adjacent S4 semantic recognition results or obtaining the topological expectation category from active back-inference. The determined category label is encoded as the type prior vector; using the determined category as the index key, the corresponding category's geometric template, size constraint interval, and topological constraint rule are retrieved from the domain specification geometric parameter library. The retrieval results are then assembled into the size prior vector.

[0120] The thresholds and judgment processes for morphological verification, value verification, and position verification are as follows. Morphological verification compares the distance in parameter space between the candidate geometry output by the repair network and the geometric template. If the distance does not exceed the morphological threshold, it is considered to pass. For example, the morphological threshold is twice the standard deviation of the nominal geometric parameter set for the "automatic sprinkler head" category. Value verification determines the range of each dimension parameter and dimension constraint interval in the candidate geometry. If all parameters are within the interval, it is considered to pass. Position verification determines the compliance of the position and orientation parameters of the candidate geometry in the unified spatial coordinate system with the topology constraint rules. If all rules are satisfied, it is considered to pass.

[0121] Regarding the fire door repair example in the aforementioned commercial complex scenario: the opening connecting the fire escape stairwell on the first floor of the shopping mall was identified as the "Door" category by S4 and as the "Fire Door" subcategory by S42. It was then actively back-engineered and included in the obstructed area because the door panel was obscured by stacked goods, resulting in incomplete point cloud data. The type prior is the "Fire Door" category label. The size prior is indexed from the specification library to the geometric template, size constraint range, and topological constraint rules for the "Fire Door" category. The geometric template includes the standard structure of the door leaf, door frame, door closer, and fireproof edge sealing. The size constraint range is: door leaf thickness 50 mm to 80 mm, door frame width 1000 mm to 1500 mm, and door closer trajectory length 200 mm to 400 mm. The topological constraint rules require the inclusion of a door closer, fireproof edge sealing, and an outward opening direction towards the corridor side. After the candidate door geometry was generated through network repair, morphological verification confirmed that the candidate door included three components: door leaf, door frame, and door closer. The spatial distance from the standard fire door template was within the threshold. Value verification confirmed that the candidate door leaf thickness (65 mm), door frame width (1100 mm), and door closer trajectory length (280 mm) were all within the specified range. Position verification confirmed that the candidate door included a door closer and fire-resistant edge sealing, and the outward opening direction conformed to the rules. With all three verifications passed, the candidate became the repaired geometric representation.

[0122] The "re-repair" mechanism rejects repair results that only satisfy a single priori requirement: if a candidate door generated by the repair network passes the morphological verification (i.e., contains a reasonable topology including the door leaf, frame, and closer), but the door leaf thickness is estimated to be 35mm during the value verification (below the 50mm lower limit for fire doors), then the value verification fails, the entire candidate is deemed unsuccessful, and the corresponding obstructed area is marked as an area to be re-repaired and sent back to the repair network for regeneration; similarly, if the candidate door passes the value verification (thickness compliant), but the closer's position and orientation are incorrect during the position verification, the candidate is also deemed unsuccessful. Both types of failure modes trigger the "re-repair" marking.

[0123] The domain-specific geometric parameter library, acting as a size prior, injects specification-level topological constraint rules into the repair network, thus constraining the output space of the repair network in the specification dimension. For example, for the fire door category, conditions such as "must include a door closer" and "must include fireproof sealing" in the rules are injected into the repair process as generation constraints, ensuring that the generated door geometry includes these two types of components, avoiding the generation of non-compliant doors with reasonable dimensions but lacking door closers and fireproof sealing.

[0124] In some embodiments, the domain specification geometry parameter library includes parameterizable geometric constraints that are respectively bound to multiple semantic categories. These parameterizable geometric constraints are mapped to soft constraint loss terms in the loss function of the repair network, and the soft constraint loss terms participate in the end-to-end optimization of the repair network.

[0125] The domain specification geometry parameter library is divided into sub-libraries corresponding to multiple building types, and the domain specification geometry parameter library supports hot loading.

[0126] In response to a change in building type identification results, the currently active sub-library is switched to the sub-library corresponding to the changed building type identification results.

[0127] The building type identification results are obtained by classifying the distribution characteristics of building component category labels and fire protection facility category labels in the semantic identification results.

[0128] Specifically, parameterizable geometric constraints are defined in the form of differentiable mathematical expressions, with each constraint corresponding to one or a set of specification requirements. For example, the dimensional constraint "the thickness of the fire door leaf is within the range of 50mm to 80mm" can be parameterized as a piecewise function. ,in For thickness constraint loss term, To correct the door thickness in the network output, and The lower limit is 50mm and the upper limit is 80mm, respectively; the topological constraint term "automatic sprinkler head orientation perpendicular to the lower surface of the floor slab" can be parameterized as direction cosine deviation. ,in To the direction of the constraint loss term, Let be the unit vector along the axis of the sprinkler head connection. The normal vector to the lower surface of the floor slab; the position constraint term "fire hydrants are arranged along the outer side of the corridor columns" can be parameterized as a projection distance penalty. ,in For the location constraint loss term, This is the distance from the center of the fire hydrant to the outer side of the column. To allow for an upper limit on offset, the granularity of the sub-library is coarsened according to building type. In this embodiment, sub-libraries are included for several typical building types such as residential, commercial, industrial, school, hospital, and underground parking garage. The parameterizable constraints in each sub-library are defined according to the design and fire protection technical specifications corresponding to each building type.

[0129] The path by which parameterizable constraint terms are mapped to the soft constraint loss term in the repair network loss function is as follows. The loss function of the repair network consists of a linear weighted sum of a data fidelity term and a soft constraint loss term. The data fidelity term measures the consistency between the repair result and the observed portion of the collected data, while the soft constraint loss term measures the degree of conformity between the repair result and the specification requirements. The two are weighted according to... Weighted combination. Among them, To repair the total network loss, For data fidelity items, This is the sum of all soft-constraint loss terms. These are soft constraint weights. The parameterizable constraint terms corresponding to each category in the currently active sublime are automatically loaded. One component, along with the data fidelity term, participates in the backpropagation of the repair network. During the training phase, the repair network uses the soft constraint loss term as part of the optimization objective, enabling the network to learn to converge the output towards the required direction during weight updates. During the inference phase, the soft constraint loss term no longer participates in weight updates, but serves as a reference metric to verify the compliance of candidate repair results.

[0130] Parameterizable geometric constraints participate in the end-to-end optimization of the repair network as soft constraints. During the training phase, normative knowledge is incorporated into the network weights through backpropagation of the soft constraint loss term; during the inference phase, the soft constraint loss term serves as a reference metric to verify the compliance of candidate repair results. This mechanism, together with the aforementioned ex-post verification mechanism, constitutes two layers of normative constraints: the former ensures the repair network is compliant before generation, while the latter provides final oversight after generation.

[0131] The triggering conditions and granularity for sub-library switching are as follows. Sub-library switching is triggered by changes in the building type identification results. When the semantic recognition results output by S4 statistically exhibit distribution characteristics of building component categories and fire protection facility categories that are inconsistent with the currently effective sub-library, the building type identification module determines that the building type has been switched. The input to the building type identification module is the statistical distribution of each category label in the S4 semantic recognition results, and the output is the building type to which the current target scene belongs. For example, in the identification results of shopping malls and shopping centers, the density of "automatic sprinkler heads," "fireproof roller shutters," and "emergency lighting" is high, which is significantly different from the identification results of residential buildings.

[0132] The sub-library switching runtime sequence for the aforementioned commercial complex scenario is as follows: After the first execution of S4, the semantic recognition results densely contain commercial facilities such as automatic sprinkler heads, fireproof roller shutters, and emergency lighting. Based on this, the building type identification module determines the building type as "commercial" and switches the currently active sub-library to the commercial sub-library through a hot-loading mechanism. When the project is transferred to the next target scenario within the same system, such as a university teaching building, the semantic recognition results of the first execution of S4 show dense smoke detectors but no fireproof roller shutters, and include many classroom and laboratory doors. The building type identification module determines the building type to switch to "school." The hot-loading mechanism unloads the commercial sub-library and loads the school sub-library without restarting the system, ensuring that the size constraint range and topology constraint rules used for subsequent repair of obstructed areas are switched to the corresponding version of the school specifications.

[0133] The parameterized canonical library enables canonical information to participate in the backpropagation of the repair network. The repair network learns the boundaries of the canonical framework during the training phase, and the candidates output during the inference phase have a high probability of passing the verification on the first attempt, with post-inference verification serving as a final safeguard.

[0134] In some embodiments, during the dual-constraint AI repair process, a comprehensive confidence score is calculated for each occluded region. The comprehensive confidence score is obtained by weighting the semantic confidence score corresponding to the semantic recognition result, the consistency score between the repaired geometric representation and the neighborhood geometry, and the matching score of the repaired geometric representation to the domain-specific geometric parameter library.

[0135] Specifically, the three components of the overall confidence score are derived as follows: Semantic confidence score is directly taken from the output probability value of the S4 semantic recognition model for the component or facility category to which the occluded area belongs. Neighborhood geometric consistency score is a comprehensive measure of the boundary continuity, normal continuity, and density continuity between the repaired geometric representation and the dense 3D geometric representations already collected around the occluded area. Specification library matching score is a comprehensive measure of the degree of agreement between the repaired geometric representation and the morphological verification under type prior, the value verification under size prior, and the position verification in S6. A score of 1.0 is awarded if all three verifications pass, and the score approaches 0 the further it deviates from the threshold. The formula for calculating the overall confidence score is:

[0136] in, To assess the overall confidence level, For semantic confidence, The neighborhood geometric consistency score, To standardize library matching scores, , , These are the preset weights for the corresponding components. In this embodiment, the weights are set as follows: , , The sum of the three weights is 1. The value falls within the range of 0 to 1.

[0137] The obscured areas are handled in a tiered manner based on the overall confidence level.

[0138] In response to the overall confidence level being in the first range, fine repair is performed on the occluded area. The fine repair generates a fine mesh that is consistent with the size constraint range and topological constraint rules based on the geometric template.

[0139] In response to the overall confidence level being in the second range, a resolution reduction interpolation repair is performed on the occluded region. The resolution reduction interpolation repair generates a low-resolution geometric representation consistent with the size constraint range and topological constraint rules.

[0140] In response to the overall confidence level being in the third range, geometric detail filling is not performed on the occluded area; only the placeholder volume of the occluded area is generated, and a mark for manual verification is added to the placeholder volume.

[0141] Each occlusion region is assigned a level, with values ​​for a first level, a second level, and a third level. The first, second, and third levels correspond one-to-one with the first, second, and third level intervals, respectively. The level value is determined by the level interval in which the overall confidence score corresponding to the occlusion region falls.

[0142] The interval is a range of comprehensive confidence scores, used for comparison and classification of comprehensive confidence scores; the level is a discrete label corresponding to the interval, used to record the confidence level in the metadata of the occluded area, so as to facilitate operation according to the level label in subsequent iterative loop closure and multi-scan time-series fusion.

[0143] Among them, the overall confidence level corresponding to the first interval is higher than that corresponding to the second interval, and the overall confidence level corresponding to the second interval is higher than that corresponding to the third interval.

[0144] Specifically, the boundary values ​​of the three-tier intervals are determined based on the impact of the reliability of the repair results on fire safety inspection compliance. In this embodiment, the three-tier intervals are set as follows: the first tier interval is... For high-confidence repair results, they can be directly entered into the semantic 3D model as the review object; the second level is... The corresponding repair results from Zhongzhixin provide a geometric approximation but are insufficient for detailed compliance checks; the third tier is... For low-confidence repair results, geometric details are unreliable, and only placeholders are left to prompt manual verification.

[0145] For the aforementioned commercial complex scenario, taking the repair task of an automatic sprinkler head located on the ceiling of the first floor of the shopping mall as a representative case: the semantic recognition of "automatic sprinkler head" is clear. =0.85; the repair result shows good consistency with the normal density of the lower surface of the ceiling. =0.80; The repair result passed the three checks of the specification library. =0.75. The overall confidence level C = 0.4 × 0.85 + 0.3 × 0.80 + 0.3 × 0.75 = 0.805, falling into the first interval. Assign the first position and perform fine repair on the sprinkler head body, outputting a fine mesh including the sprinkler cap, connecting pipes, and temperature sensing element. Other conditions such as underground parking garage pipe support components and unclear wiring on standard office floors correspond to the second and third intervals respectively. These are handled using the reduced resolution interpolation repair method for the second interval and the placeholder marking for manual verification for the third interval.

[0146] The system categorizes repair results by overall confidence level, allowing high-confidence, medium-confidence, and low-confidence repairs to receive geometric inputs of varying granularities. Low-confidence regions explicitly express uncertainty through placeholders and markers indicating pending manual verification. These regions can be automatically skipped by the verification tool or transferred to the manual verification process, preventing low-confidence repair details from entering the final output model as unreliable information and contaminating the overall credibility.

[0147] In some embodiments, semantic recognition and dual-constraint AI repair are performed on the dense 3D geometric representation at least twice. Each iteration includes: performing semantic recognition based on the geometric representation input for this iteration to obtain the semantic recognition result corresponding to this iteration; performing dual-constraint AI repair based on the semantic recognition result corresponding to this iteration to obtain the repaired geometric representation corresponding to this iteration; calculating the comprehensive confidence level corresponding to this iteration for each occluded region, and updating the gear level corresponding to the occluded region according to the comprehensive confidence level corresponding to this iteration.

[0148] In the second and subsequent iterations, double-constraint AI repair is only performed on the occluded areas whose corresponding gear position is not the first gear position.

[0149] For the occluded area corresponding to the first gear position, the repaired geometric representation of the occluded area is frozen, so that the frozen repaired geometric representation will no longer participate in this dual-constraint AI repair.

[0150] The termination criterion for the iteration is: the gear corresponding to all occluded areas no longer increases between two consecutive iterations, or the number of iterations reaches the preset upper limit of iterations.

[0151] Specifically, the input and output of each iteration are connected as follows: The input geometric representation of the first iteration is the dense 3D geometric representation directly generated by S3, and the output is the first round of repaired geometric representation obtained after performing S4 semantic recognition and S6 dual-constraint AI repair based on the input geometric representation of the first iteration; The input geometric representation of the second iteration is the repaired geometric representation output by the first iteration, in which the occlusion area of ​​the first gear has been frozen while the occlusion areas of the second and third gears need to be repaired again, and the output of the second iteration is the second round of repaired geometric representation obtained after performing S4 semantic recognition and S6 dual-constraint AI repair again based on the input geometric representation of the second iteration; and so on for each subsequent iteration.

[0152] For the above-mentioned commercial complex scenario, assuming that 100 occlusion areas are identified in the S5 stage, after the first iteration, the distribution of the tiers is 30 in the first tier, 50 in the second tier, and 20 in the third tier; after multiple iterations, the distribution of the tiers stabilizes at 55 in the first tier, 38 in the second tier, and 7 in the third tier. The tier corresponding to all occlusion areas no longer increases between two consecutive iterations, and the iteration terminates.

[0153] The termination criterion for iteration consists of two conditions. The first condition is gear convergence, meaning that the gear corresponding to all occluded regions no longer increases between two consecutive iterations, indicating that continued iteration no longer brings quality improvement. The second condition is the iteration limit, meaning that the number of iterations reaches a preset iteration limit. In this embodiment, the preset iteration limit is 5 times, used to forcibly terminate the iteration when the gear does not converge for a long period to avoid infinite iteration. The two conditions constitute a "first-come, first-served" relationship; the iteration terminates when either condition is met.

[0154] In some embodiments, the termination criterion for iteration further includes: all occluded areas are not at the third level, and the coverage rate of each building component category and fire protection facility category in the semantic recognition results under the topological constraint rules of the domain specification geometric parameter library reaches a preset coverage rate threshold. Specifically, the method for calculating the specification coverage rate is as follows: for each type of fire protection facility specified in the topological constraint rules of the building type to which the current target scene belongs, calculate the ratio of the number of each type of facility identified in the S4 semantic recognition results to the number of each type of facility required by the specification, and then take the minimum value of the coverage rate of all categories as the overall specification coverage rate. For example, if the above-mentioned commercial complex scene has 2000 automatic sprinkler heads, 930 smoke detectors, and 30 fire hydrants as required by the specification, and the number identified at the end of the iteration is 1980, 911, and 30 respectively, then the coverage rates of each category are 99%, 98%, and 100% respectively, and the overall specification coverage rate is taken as the minimum value of 98%. The preset coverage rate threshold is set to 95% in this embodiment of the application. This optional termination criterion, together with the aforementioned gear convergence criterion, forms an "AND" relationship. The iteration will only terminate when the gear converges, all occluded areas are not in the third gear, and the overall specification coverage reaches the preset coverage threshold; if any condition is not met, the next iteration will be forced to continue.

[0155] In some embodiments, at least two acquisitions are performed on the same target scene to obtain multiple multimodal acquisition data, and the multiple multimodal acquisition data are cached as time-series cumulative data according to the target scene's identifier code. When performing dual-constraint AI repair, the semantic recognition results corresponding to each acquisition are injected into the repair network as type prior extension terms, and the repaired geometric representation corresponding to the occlusion area identified as the first level in each acquisition is used as the initial value of the corresponding position in the current acquisition.

[0156] Specifically, the time-series cumulative data is organized according to the target scene identifier code. Each target scene corresponds to an independent cumulative entry. The entry stores the multimodal acquisition data of each acquisition, the S4 semantic recognition result, the geometric representation after S6 repair, and the evolution history of the level of each occluded region in the order of acquisition time. When performing S6 double-constraint AI repair in the second and subsequent acquisitions, the semantic recognition results of previous acquisitions are read from the cumulative entry as type prior extensions and injected into the repair network. This increases the semantic confidence of occluded regions with the same category in the two acquisitions, while occluded regions with inconsistent categories in the two acquisitions are marked as needing review. At the same time, the repaired geometric representations corresponding to the occluded regions that have reached the first level in previous acquisitions are read from the cumulative entry, so that the current repair network is optimized based on the geometric representations of previous acquisitions. This optional embodiment, as an extension of the iterative closed loop of claim 4 in cross-acquisition scenarios, can gradually converge to the compliance level after multiple scans.

[0157] S7. Output a semantic 3D model based on the repaired geometric representation. The semantic 3D model includes the repaired geometric representation and semantic labels bound to the repaired geometric representation.

[0158] Specifically, the engineering format of the semantic 3D model is as follows: The repaired geometric representation serves as the basic geometric skeleton, and each spatial point or surface element carries a set of semantic label fields in addition to its geometric attributes. These semantic label fields include building component category labels, fire protection facility category labels, sub-category labels, comprehensive confidence scores, grade labels, and markers for manual verification. Specifically, the set of values ​​for the building component category labels includes building component categories such as walls, floors, beams, columns, doors, windows, and stairs; the set of values ​​for the fire protection facility category labels includes fire hydrants, automatic sprinkler heads, fire-resistant roller shutters, and smoke detectors; and the set of values ​​for the sub-category labels includes door subcategories such as fire doors, ordinary doors, and explosion-proof doors. The semantic 3D model can be carried in extended IFC, glTF, or custom point cloud semantic formats; this application embodiment does not limit the specific carrying format.

[0159] For the aforementioned commercial complex scenario, the final output semantic 3D model covers a building area of ​​approximately 35,000 square meters. 2 The model encompasses all structural components and fire protection facilities across three functional areas: a shopping mall, an office building, and an underground parking garage. Each component in the model carries three types of information: geometric location, category, and confidence level. For example, a fire door component on the first floor of the shopping mall is represented in the model as a set of entities with geometric sub-components including wall openings, door leaves, door frames, door closers, and fire-resistant edge sealing. The geometry of these sub-components comes from the S6-repaired geometric representation, and the entire component is bound to the "Fire Door" sub-category label and the first-tier label. Similarly, a fire hydrant component in the underground parking garage is represented as a set of entities with geometric sub-components including hydrant valves, connecting pipes, and housings, bound to the "Fire Hydrant" category label and the first-tier label.

[0160] Because geometric sub-components are tightly bound to semantic tags, the output semantic 3D model can be directly integrated with fire safety inspection tools. The inspection tool directly retrieves all objects of a specific category based on semantic tags; for example, searching for "fire door" returns all fire door entities in the model. It also automatically determines whether each object conforms to regulations based on the dimension fields of the geometric sub-components, such as automatically determining whether the door leaf thickness of each fire door is within the 50mm to 80mm range and whether it includes a door closer and fire-resistant sealing. Objects with the third-tier tag and the "awaiting manual verification" mark are explicitly displayed as pending review in the inspection tool and can only be included in the compliance conclusion after manual verification.

[0161] This application also provides a high-fidelity 3D model automatic generation system based on multimodal data, including: The multimodal acquisition module is configured to acquire multimodal acquisition data, which includes multi-view images, sparse point clouds, dense laser point clouds, panoramic images, and inertial measurement signals. The registration and fusion module is configured to perform multimodal spatiotemporal joint registration on multimodal acquired data to obtain fused multimodal data located in a unified spatial coordinate system; The geometry reconstruction module is configured to generate a dense 3D geometric representation of the target scene based on fused multimodal data; The semantic recognition module is configured to perform semantic recognition on dense 3D geometric representations to obtain semantic recognition results, which include building component category labels and fire protection facility category labels. The occlusion detection module is configured to perform occlusion detection on dense 3D geometric representations to determine occlusion areas; The dual-constraint repair module is configured to perform dual-constraint AI repair on the occluded area to obtain the repaired geometric representation. The dual-constraint AI repair uses the semantic recognition result as the type prior and the domain-specific geometric parameter library as the size prior, so that the repaired geometric representation passes the verification under both the type prior and the size prior. The model output module is configured to output a semantic 3D model based on the repaired geometric representation. The semantic 3D model includes the repaired geometric representation and semantic labels bound to the repaired geometric representation.

[0162] The output of the multimodal acquisition module is connected to the input of the registration and fusion module. The output of the registration and fusion module is connected to the input of the geometric reconstruction module. The output of the geometric reconstruction module is connected to the input of the semantic recognition module and the occlusion detection module, respectively. The outputs of the semantic recognition module and the occlusion detection module are connected together to the input of the dual-constraint repair module. The output of the dual-constraint repair module is connected to the input of the model output module.

[0163] Each module corresponds one-to-one with each step of the above method: the multimodal acquisition module corresponds to S1, the registration and fusion module corresponds to S2, the geometric reconstruction module corresponds to S3, the semantic recognition module corresponds to S4, the occlusion detection module corresponds to S5, the dual-constraint repair module corresponds to S6, and the model output module corresponds to S7. The specific configuration and working mechanism of each module follow the expansion of each step of the above method, and will not be repeated here.

[0164] In some embodiments, the dual-constraint repair module is further configured to: calculate a comprehensive confidence score for each occluded region, the comprehensive confidence score being a weighted average of the semantic confidence score corresponding to the semantic recognition result, the consistency score between the repaired geometric representation and the neighborhood geometry, and the matching score of the repaired geometric representation against the domain-specific geometric parameter library; and perform graded processing on the occluded regions based on the comprehensive confidence score; the graded processing includes assigning a level to each occluded region, the value of which is determined by the level range in which the comprehensive confidence score corresponding to the occluded region is located.

[0165] The system also includes an iteration control module, which is configured to schedule the semantic recognition module and the dual-constraint repair module to perform at least two iterations on the dense 3D geometric representation. The termination criterion for the iteration is that the gear corresponding to all occluded areas no longer increases between two consecutive iterations, or the number of iterations reaches the preset upper limit of iterations.

[0166] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0167] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for automatically generating high-fidelity 3D models based on multimodal data, characterized in that, Includes the following steps: S1. Acquire multimodal acquisition data, which includes multi-view images, sparse point clouds, dense laser point clouds, panoramic images, and inertial measurement signals; S2. Perform multimodal spatiotemporal joint registration on the multimodal acquired data to obtain fused multimodal data located in a unified spatial coordinate system; S3. Generate a dense 3D geometric representation of the target scene based on the fused multimodal data; S4. Perform semantic recognition on the dense 3D geometric representation to obtain semantic recognition results, which include building component category labels and fire protection facility category labels; S5. Perform occlusion detection on the dense 3D geometric representation to determine the occlusion area; S6. Perform dual-constraint AI repair on the occluded area to obtain the repaired geometric representation. The dual-constraint AI repair uses the semantic recognition result as the type prior and the domain-specific geometric parameter library as the size prior, so that the repaired geometric representation passes the verification under both the type prior and the size prior. S7. Output a semantic 3D model based on the repaired geometric representation, wherein the semantic 3D model includes the repaired geometric representation and semantic tags bound to the repaired geometric representation.

2. The method for automatically generating high-fidelity 3D models based on multimodal data according to claim 1, characterized in that, The domain-specific geometric parameter library stores geometric templates, size constraint intervals, and topological constraint rules corresponding to multiple semantic categories. The type prior includes the building component category label or the fire protection facility category label in the semantic recognition result; the size prior is obtained by indexing from the domain specification geometric parameter library according to the building component category label or the fire protection facility category label, and the size prior includes the indexed geometric template, the size constraint interval and the topology constraint rule; The dual-constraint AI repair uses the type prior as the category condition input for the repair network, and the geometric template, the size constraint interval, and the topology constraint rule as the joint constraints of the repair network, so that the repaired geometric representation passes the morphological verification based on the geometric template, the value verification based on the size constraint interval, and the position verification based on the topology constraint rule.

3. The method for automatically generating high-fidelity 3D models based on multimodal data according to claim 2, characterized in that, During the dual-constraint AI repair process, a comprehensive confidence score is calculated for each occluded region. The comprehensive confidence score is obtained by weighting the semantic confidence score corresponding to the semantic recognition result, the consistency score between the repaired geometric representation and the neighborhood geometry, and the matching score of the repaired geometric representation to the domain-specific geometric parameter library. The occlusion area is subjected to graded processing based on the comprehensive confidence level. The graded processing includes assigning a level to each occlusion area. The value of the level is determined by the level range in which the comprehensive confidence level corresponding to the occlusion area is located.

4. The method for automatically generating high-fidelity 3D models based on multimodal data according to claim 3, characterized in that, The gear values ​​include a first gear, a second gear, and a third gear. The first gear, the second gear, and the third gear correspond to a first gear interval, a second gear interval, and a third gear interval, respectively. The overall confidence level corresponding to the first gear interval is higher than the overall confidence level corresponding to the second gear interval, and the overall confidence level corresponding to the second gear interval is higher than the overall confidence level corresponding to the third gear interval. In response to the gear being the first gear, fine-grained repair is performed on the occluded area; In response to the gear being the second gear, a resolution reduction interpolation repair is performed on the occluded area; In response to the gear being the third gear, no geometric detail filling is performed on the occluded area; only a placeholder volume enclosing the occluded area is generated, and a mark for manual verification is added to the placeholder volume.

5. The method for automatically generating high-fidelity 3D models based on multimodal data according to claim 4, characterized in that, The dense 3D geometric representation is subjected to at least two iterations of semantic recognition and dual-constraint AI repair. In each iteration, the semantic recognition is performed based on the geometric representation input in this iteration to obtain the semantic recognition result corresponding to this iteration, and the dual-constraint AI repair is performed based on the semantic recognition result corresponding to this iteration to obtain the repaired geometric representation corresponding to this iteration. Then, the gear corresponding to the occluded area is updated according to the comprehensive confidence level corresponding to this iteration. The second and subsequent iterations only perform the dual-constraint AI repair on the occluded areas whose corresponding gear position is not the first gear position. For the occluded areas whose corresponding gear position is the first gear position, the repaired geometric representation of the occluded area is frozen so that the frozen repaired geometric representation no longer participates in the current dual-constraint AI repair. The termination criterion for the iteration is: the gear corresponding to all the occluded areas no longer increases between two consecutive iterations, or the number of iterations reaches a preset upper limit for iterations.

6. The method for automatically generating high-fidelity 3D models based on multimodal data according to claim 2, characterized in that, The domain-specific geometric parameter library includes parameterizable geometric constraints that are respectively bound to the plurality of semantic categories. The parameterizable geometric constraints are mapped to soft-constraint loss terms in the loss function of the repair network, and the soft-constraint loss terms participate in the end-to-end optimization of the repair network.

7. The method for automatically generating high-fidelity 3D models based on multimodal data according to claim 2, characterized in that, The domain-specific geometric parameter library is divided into multiple sub-libraries corresponding to different building types, and the domain-specific geometric parameter library supports hot loading. In response to a change in the building type identification result, the currently effective sub-library is switched to the sub-library corresponding to the changed building type identification result; The building type identification result is obtained by classifying the distribution characteristics of the building component category labels and the fire protection facility category labels in the semantic identification result.

8. The method for automatically generating high-fidelity 3D models based on multimodal data according to claim 1, characterized in that, S2 includes the following sub-steps: S21. Using the dense laser point cloud as the reference anchor point, the multi-view image, the sparse point cloud, and the panoramic image are precisely registered to the unified spatial coordinate system corresponding to the reference anchor point through feature point matching and iteration, so as to obtain the precisely registered multi-view image, the precisely registered sparse point cloud, and the precisely registered panoramic image. S22. Perform high-frequency attitude compensation on the acquisition motion of the dense laser point cloud using the inertial measurement signal, and project the dense laser point cloud onto the reference frame at the same time to obtain the compensated dense laser point cloud; S23. Based on the reference anchor point, the finely registered multi-view image, the finely registered sparse point cloud, the finely registered panoramic image, and the compensated dense laser point cloud, construct the fused multimodal data.

9. A high-fidelity 3D model automatic generation system based on multimodal data, characterized in that, It includes: A multimodal acquisition module is configured to acquire multimodal acquisition data, which includes multi-view images, sparse point clouds, dense laser point clouds, panoramic images, and inertial measurement signals. The registration and fusion module is configured to perform multimodal spatiotemporal joint registration on the multimodal acquired data to obtain fused multimodal data located in a unified spatial coordinate system; A geometry reconstruction module is configured to generate a dense 3D geometric representation of the target scene based on the fused multimodal data; A semantic recognition module is configured to perform semantic recognition on the dense 3D geometric representation to obtain semantic recognition results, which include building component category labels and fire protection facility category labels. An occlusion detection module is configured to perform occlusion detection on the dense 3D geometric representation to determine the occlusion region; A dual-constraint repair module is configured to perform dual-constraint AI repair on the occluded region to obtain a repaired geometric representation. The dual-constraint AI repair uses the semantic recognition result as a type prior and the domain-specific geometric parameter library as a size prior, so that the repaired geometric representation passes the verification under both the type prior and the size prior. The model output module is configured to output a semantic 3D model based on the repaired geometric representation, the semantic 3D model including the repaired geometric representation and semantic tags bound to the repaired geometric representation; The output of the multimodal acquisition module is connected to the input of the registration and fusion module, the output of the registration and fusion module is connected to the input of the geometric reconstruction module, the output of the geometric reconstruction module is connected to the input of the semantic recognition module and the input of the occlusion detection module, the output of the semantic recognition module and the output of the occlusion detection module are connected together to the input of the dual-constraint repair module, and the output of the dual-constraint repair module is connected to the input of the model output module.

10. The high-fidelity 3D model automatic generation system based on multimodal data according to claim 9, characterized in that, The dual-constraint repair module is further configured to: calculate a comprehensive confidence score for each occluded region, the comprehensive confidence score being a weighted average of the semantic confidence score corresponding to the semantic recognition result, the consistency score between the repaired geometric representation and the neighborhood geometry, and the matching score of the repaired geometric representation against the domain-specific geometric parameter library; and perform tiered processing on the occluded regions based on the comprehensive confidence score; the tiered processing includes assigning a level to each occluded region, the value of which is determined by the level interval in which the comprehensive confidence score corresponding to the occluded region is located; The system further includes an iteration control module, which is configured to schedule the semantic recognition module and the dual-constraint repair module to perform at least two iterations on the dense 3D geometric representation. The termination criterion for the iteration is that the gear corresponding to all the occluded regions no longer increases between two consecutive iterations, or the number of iterations reaches a preset upper limit for iterations.

Images