Multimodal fusion and generative repair: a 3D modeling method and system for harsh environments

Abstract

This invention belongs to the field of 3D modeling technology, specifically proposing a method and system for 3D modeling of harsh environments using multimodal fusion and generative inpainting. The method includes: fusing features from the original image and original point cloud to complete initial semantic segmentation, obtaining an initial semantic segmentation map; using a dual-branch mechanism to obtain a confidence map corresponding to the initial semantic segmentation map; obtaining an object segmentation mask based on the original point cloud; optimizing the object segmentation mask by combining the initial semantic segmentation map and the corresponding confidence map, and generating a guiding map based on the optimized segmentation mask; inpainting the original image based on the generated guiding map, obtaining a repaired image; fusing features from the original image and the repaired image, and using an adaptive inference mechanism to obtain a semantic output map; and combining the semantic output map and the repaired image to construct a 3D environment model of the environment to be modeled. This invention improves the completeness, geometric accuracy, and semantic rationality of 3D models of complex industrial environments.

Description

Technical Field

[0001] This invention belongs to the field of 3D modeling technology, and in particular relates to a method and system for 3D modeling of harsh environments using multimodal fusion and generative repair. Background Technology

[0002] Accurate and reliable 3D modeling of the internal structure and material forms of complex industrial environments is fundamental to enabling robots to autonomously perform tasks such as operations, inspections, and monitoring. This environmental model provides robots with crucial spatial and semantic priors, which are the fundamental guarantee for their accurate perception, reliable planning, and safe execution.

[0003] Currently, 3D modeling techniques for industrial environments primarily rely on visual sensors or LiDAR. In clean, well-lit, structured environments, vision-based simultaneous localization and mapping (V-SLAM) or LiDAR point cloud stitching techniques can effectively complete 3D modeling. However, in many typical harsh industrial environments, such as bulk carrier holds, mine tunnels, underground pipelines, fire and disaster relief sites, and dusty factory workshops, sensing sensors face severe challenges. Aerosol media such as dust, smoke, and water vapor in the environment can severely degrade the quality of sensing data.

[0004] (1) Due to severe scattering and absorption of light, the visual sensor suffers from a sharp decrease in image contrast and obscures texture details, making it difficult to extract stable and effective visual features;

[0005] (2) When the laser beam emitted by the lidar penetrates such a medium, it will be severely scattered and attenuated, which will not only cause the point cloud on the surface of the target object to be distorted and the distance measurement to be inaccurate, but also generate a large amount of false point cloud noise formed by the reflection of the medium itself.

[0006] Furthermore, existing cross-modal fusion and generative repair methods also face bottlenecks in such environments: multi-source sensors often fail under harsh conditions, resulting in unreliable data fusion foundations; while generative models that rely on a large amount of clean data for training have poor generalization ability to complex and disordered degradation patterns, making it difficult to perform effective repair. Summary of the Invention

[0007] To overcome the shortcomings of the existing technologies, this invention provides a method and system for 3D modeling of harsh environments using multimodal fusion and generative repair. This method effectively improves the problems of model incompleteness and geometric distortion that occur in traditional 3D modeling in harsh environments, and significantly enhances the integrity, geometric accuracy, and semantic rationality of 3D models of complex industrial environments, providing a more reliable environmental perception foundation for automation applications in a wide range of fields.

[0008] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0009] The first aspect of this invention provides a method for three-dimensional modeling of harsh environments using multimodal fusion and generative repair.

[0010] A 3D modeling method for harsh environments using multimodal fusion and generative repair includes the following steps:

[0011] Collect the original images and original point clouds of the environment to be modeled;

[0012] A cross-modal attention fusion method is used to fuse features between the original image and the original point cloud and to complete the initial semantic segmentation to obtain the initial semantic segmentation map; at the same time, a dual-branch mechanism is used to obtain the confidence map corresponding to the initial semantic segmentation map.

[0013] The object segmentation mask is obtained based on the original point cloud; the object segmentation mask is optimized by combining the initial semantic segmentation map and the corresponding confidence map, and a guide map is generated based on the optimized segmentation mask;

[0014] Based on the generated guide map, the original image is repaired to obtain the repaired image;

[0015] Feature fusion is performed on the original image and the restored image to obtain feature-level fused features, which in turn yields a semantic prediction map; an adaptive inference mechanism is then used to combine the semantic prediction map to obtain a semantic output map.

[0016] By combining the semantic output image and the repaired image, a 3D environment model of the environment to be modeled is constructed through multi-frame semantic fusion and dense 3D modeling technology.

[0017] The second aspect of this invention provides a three-dimensional modeling system for harsh environments that combines multimodal fusion and generative repair.

[0018] A 3D modeling system for harsh environments that integrates multimodal fusion and generative repair, including:

[0019] The raw data acquisition module is configured to: acquire the raw images and raw point clouds of the environment to be modeled;

[0020] The initial semantic segmentation module is configured to: use a cross-modal attention fusion method to fuse features between the original image and the original point cloud, and complete the initial semantic segmentation to obtain an initial semantic segmentation map; at the same time, a dual-branch mechanism is used to obtain the confidence map corresponding to the initial semantic segmentation map.

[0021] The guidance graph generation module is configured to: obtain an object segmentation mask based on the original point cloud; optimize the object segmentation mask by combining the initial semantic segmentation map and the corresponding confidence map; and generate a guidance graph based on the optimized segmentation mask.

[0022] The image restoration module is configured to restore the original image based on the generated guide image to obtain a restored image;

[0023] The segmentation optimization module is configured to: perform feature fusion on the original image and the repaired image to obtain feature-level fused features, and then obtain a semantic prediction map; and use an adaptive inference mechanism to combine the semantic prediction map to obtain a semantic output map.

[0024] The semantic fusion modeling module is configured to combine the semantic output image and the repaired image to construct a three-dimensional environment model of the environment to be modeled through multi-frame semantic fusion and dense 3D modeling technology.

[0025] The above one or more technical solutions have the following beneficial effects:

[0026] This invention provides a method and system for 3D modeling in harsh environments using multimodal fusion and generative inpainting. By introducing millimeter-wave radar with strong dust penetration capabilities, it fundamentally overcomes the problems of data degradation and failure in harsh environments encountered by traditional vision and lidar methods, providing the system with stable and reliable spatial geometric information. Combined with cross-modal fusion and generative inpainting techniques, it can effectively fill model gaps caused by missing data, generating a complete and continuous 3D environmental model. This solves the fundamental defect of traditional methods in handling large-area model gaps in harsh environments, improving the perception robustness and model integrity under harsh conditions.

[0027] The cross-modal attention fusion mechanism designed in this invention achieves deep complementarity and enhancement of visual texture and radar geometric and physical features. This not only suppresses noise but also enhances feature extraction capabilities in low-contrast environments. Subsequent generative inpainting based on a diffusion model, guided by radar semantic information, can reasonably repair missing regions that conform to both physical structure and semantic logic, avoiding the distortion caused by simple geometric interpolation. This greatly improves the usability of the model and achieves high-quality reconstruction that is both geometrically accurate and semantically reasonable.

[0028] This invention constructs a closed-loop enhancement system from perception to repair, possessing adaptive optimization capabilities. It innovatively integrates the concept of "perception-repair-reperception" into the system design. Reliable areas are assessed through confidence levels, and targeted repairs are performed on damaged parts using a large visual model and a generative model. The repair results are then used to enhance the perception model, forming a positive loop of continuous performance improvement. The system's adaptive inference mechanism dynamically adjusts the output strategy based on the severity of the environment and the quality of the repair, ensuring the reliability of the final result under various complex working conditions. Finally, through multi-frame semantic fusion, a dense 3D environment model with accurate geometry and rich semantics is generated, providing directly usable prior environmental information for robot navigation, task planning, and digital twins.

[0029] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0030] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0031] Figure 1 This is a flowchart of the method in Example 1. Detailed Implementation

[0032] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0033] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations of the present invention.

[0034] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0035] Example 1

[0036] Addressing the common technical challenges described in the background art regarding model loss and distortion due to degraded sensory data in harsh environments, this invention proposes a multimodal fusion and generative inpainting method for 3D modeling in harsh environments:

[0037] First, by introducing millimeter-wave radar with strong penetration capability into non-transparent media, stable point cloud data containing three-dimensional coordinates, radar cross section and Doppler velocity are obtained, providing reliable spatial information for the system to perform three-dimensional modeling in harsh environments;

[0038] Secondly, a cross-modal attention fusion mechanism is designed to deeply complement the geometric and physical information of the radar with visual texture features, providing a complete feature representation foundation for subsequent semantic segmentation tasks.

[0039] Subsequently, based on the confidence level assessment, the missing regions of the identification model are identified, and the generative model is driven to repair the missing parts of the image in a structurally reasonable and semantically consistent manner.

[0040] Finally, by combining multi-frame semantic fusion and dense 3D modeling techniques, the repaired 2D semantic information with radar geometric information is combined to construct a globally consistent, semantically labeled dense 3D environment model.

[0041] The method provided in this embodiment effectively improves the problems of model incompleteness and geometric distortion that occur in traditional 3D modeling in harsh environments. It significantly improves the integrity, geometric accuracy, and semantic rationality of 3D models in complex industrial environments, providing a more reliable environmental perception foundation for automation applications in a wide range of fields.

[0042] like Figure 1 As shown, the 3D modeling method for harsh environments based on multimodal fusion and generative repair includes the following steps:

[0043] Collect the original images and original point clouds of the environment to be modeled;

[0044] A cross-modal attention fusion method is used to fuse features between the original image and the original point cloud and to complete the initial semantic segmentation to obtain the initial semantic segmentation map; at the same time, a dual-branch mechanism is used to obtain the confidence map corresponding to the initial semantic segmentation map.

[0045] The object segmentation mask is obtained based on the original point cloud; the object segmentation mask is optimized by combining the initial semantic segmentation map and the corresponding confidence map, and a guide map is generated based on the optimized segmentation mask;

[0046] Based on the generated guide map, the original image is repaired to obtain the repaired image;

[0047] Feature fusion is performed on the original image and the restored image to obtain feature-level fused features, which in turn yields a semantic prediction map; an adaptive inference mechanism is then used to combine the semantic prediction map to obtain a semantic output map.

[0048] By combining the semantic output image and the repaired image, a 3D environment model of the environment to be modeled is constructed through multi-frame semantic fusion and dense 3D modeling technology.

[0049] The technical solution of this embodiment will be explained in detail below. This embodiment proposes a four-stage fusion repair modeling framework, and the specific steps are as follows:

[0050] Phase 1: Multi-sensor fusion and initial reconstruction.

[0051] The core objective of this phase is to generate an initial scene reconstruction (in the form of a segmentation mask) that is geometrically consistent and possesses basic semantic information by deeply fusing heterogeneous data from cameras and radar. The design of this phase follows the principle of "complementary advantages": leveraging the rich texture of the images and the spatial robustness of the radar to provide a reliable prior foundation for subsequent phases.

[0052] 1.1 Multi-source data acquisition and preprocessing.

[0053] The system employs a rigorous hardware synchronization mechanism to acquire multimodal data streams from both the camera and millimeter-wave radar (hereinafter referred to as radar) in parallel, ensuring spatiotemporal consistency. Precise extrinsic parameter matrices have been obtained between the camera and radar through calibration, ensuring accurate correlation between image pixels and radar point clouds in the spatial coordinate system.

[0054] For visual data streams, the system acquires raw RGB image data. ,in and These represent the height and width of the original image, respectively. To meet the input size requirements of deep learning models, a bilinear interpolation algorithm is used to uniformly scale the image to a fixed resolution of 640×640. At the same time, the pixel values ​​are normalized, linearly transforming them from the original range [0,255] to the interval [0,1].

[0055] For radar data streams, the system acquires raw point cloud data. ,in This represents the original number of radar points in each frame of data. The five dimensions correspond to 3D spatial coordinates (x, y, z), radar cross section (RCS), and Doppler velocity, respectively. Here, x, y, and z represent the coordinate values ​​of the x-axis, y-axis, and z-axis, respectively.

[0056] Considering the significant differences in radar point cloud density under different environments and the model's requirement for consistency in input dimensions, the system employs a dynamic sampling strategy to normalize the point cloud data. Specifically, when At that time, 1000 points are randomly selected from the original point cloud; when If necessary, zero vectors are added to the existing point cloud to bring the total number of points to 1000, ensuring that the output point cloud data has a fixed dimension. .

[0057] 1.2 Cross-modal attention fusion.

[0058] Based on the completed data preprocessing, the system constructs a dual-branch encoder architecture specifically designed to extract and fuse deep feature representations from heterogeneous sensor data.

[0059] This dual-branch encoder architecture consists of a visual coding branch and a radar coding branch, achieving deep interaction and complementarity of information from the two modalities through a carefully designed cross-modal attention mechanism. ResNet is used as the image encoder to extract multi-level visual features, resulting in... PointCNN was used as the radar encoder to extract geometric features from the point cloud, resulting in... .in, Indicates the first The, here meaning the first layer.

[0060] ResNet stands for Residual Neural Network, while PointCNN is a deep learning model specifically designed for point cloud data.

[0061] The core innovation of this phase lies in the introduction of a cross-modal fusion mechanism based on cross-attention. This mechanism uses image features as query vectors and radar features as key-value pairs. By calculating the correlation weights between image spatial locations and radar points, it achieves adaptive fusion of visual and spatial geometric information. The specific fusion process is implemented through the following formula:

[0062]

[0063] in, , , , For the first Layer-learnable projection matrix, , , These represent the query vector, key vector, and value vector, respectively, and are the core components of the attention mechanism; Reshape(·) represents the reshaping operation, which reshapes the image features from... The dimensions were reshaped into The aim is to convert spatial pixel locations into a sequence to meet the computational requirements of attention mechanisms; , , They represent the first The height, width, and number of channels of the layer feature map; For projection dimensions; This represents the attention weight matrix between image pixels and radar points. This indicates the number of radar points, i.e., the number of points in each frame of the point cloud; express A function is used to normalize attention scores into a probability distribution; Indicates transpose; This represents multi-scale fusion features, which are the output of a cross-modal attention mechanism and integrate visual texture and radar geometric information.

[0064] This fusion mechanism enables each image pixel to autonomously retrieve relevant radar spatial information based on its semantic content, while residual connections ensure the complete preservation of the original visual features.

[0065] This finely designed cross-modal feature interaction mechanism generates enhanced multi-scale fusion features. This provides a complete feature representation foundation for subsequent semantic segmentation tasks.

[0066] 1.3 Initial semantic segmentation.

[0067] Building upon the deeply fused multi-scale features, this stage focuses on achieving accurate initial semantic segmentation and quantifying its reliability. This process is accomplished through a dual-path decoding architecture, responsible for semantic segmentation reconstruction and confidence estimation respectively, resulting in a comprehensive understanding of the scene.

[0068] The initial segmentation and reconstruction path employs an improved decoder based on the U-Net architecture, specifically designed to handle multi-scale fused features. This decoder integrates the fused features through progressive upsampling and feature aggregation operations. This is converted into a dense semantic segmentation graph. The reconstruction process can be formally described as follows:

[0069]

[0070] in This represents the initial semantic segmentation map, where each pixel location contains a... A dimensional vector, representing a dimensional vector belonging to The probability distribution of each category; and These represent the height and width of the output segmentation image, respectively. This represents the total number of semantic categories. , , , These represent the fusion features of layers 1 through 4, respectively. This indicates the decoder.

[0071] The decoder combines shallow detail features with deep semantic features through a skip connection mechanism, ensuring accurate semantic segmentation results while maintaining boundary precision.

[0072] 1.4 Confidence estimation.

[0073] The confidence estimation path constructs a parallel evaluation network that provides a reliability metric through a two-branch mechanism. This path first processes radar point cloud features independently, then performs point-level semantic classification using a multilayer perceptron.

[0074]

[0075] in This represents the category confidence distribution for each radar point, i.e., the radar point classification confidence, which indicates the probability distribution of each radar point belonging to each semantic category. The dimension is... . This represents a multilayer perceptron, a fully connected neural network used for nonlinear transformation and classification of radar features.

[0076] Meanwhile, the system calculates the spatial consistency score based on the cross-modal attention weight matrix:

[0077]

[0078] in, For the first Layer attention weight matrix The elements in the image represent the pixels. With radar points The correlation weight between them; For the first Image pixels in layer The spatial consistency score is a scalar value that reflects the average correlation strength between the pixel and all radar points. The higher the score, the more consistent the pixel position is with the radar geometry. Represents the index of an image pixel, with values ​​ranging from 1 to... (i.e., the first) (Total number of pixels in the layer feature map) This represents the index of the radar point, with values ​​ranging from 1 to... ( (Number of radar points).

[0079] This score reflects the average correlation strength between each image location and the radar point.

[0080] The two confidence levels are integrated through a learnable fusion module to generate the final confidence map:

[0081]

[0082] in, This represents the final confidence plot, with dimensions [missing information]. Each pixel value represents the reliability of its segmentation result; As a learnable fusion module, radar point classification confidence is achieved through convolution operations. Projection and interpolation into a dense image space, and spatial consistency score. and the initial semantic segmentation graph Perform weighted fusion.

[0083] Through this reconstruction and evaluation mechanism, the system outputs an accurate initial semantic segmentation map. And it provides the corresponding confidence graph. This provides an important and reliable basis for decision-making and refined processing in subsequent stages.

[0084] Phase 2: Reliable region segmentation and repair guide map generation.

[0085] The initial model and confidence map generated in the first stage identify potential problems, but the repair work requires more precise guidance. The purpose of this stage is to generate a precise repair guidance map based on the preceding results. This map clearly indicates the pixel regions that need repair and their corresponding semantic categories, thereby driving the third-stage generative model to perform targeted and semantically reasonable repairs.

[0086] 2.1 Pseudomask generation based on large visual models.

[0087] This system employs the lightweight FastSAM model (an image segmentation model) as its basic segmentation tool. Its core advantage lies in balancing powerful zero-shot segmentation capabilities with extremely high computational efficiency, making it highly suitable for the real-time requirements of industrial scenarios. Specifically, the radar points projected onto the image coordinate system in the first stage are used as location cues input to the FastSAM model to generate corresponding object segmentation masks. This step fully utilizes the characteristic that radar points can reliably detect objects even in harsh environments, providing a rich pool of candidate regions for subsequent processing.

[0088] 2.2 Noise filtering and mask optimization based on confidence level.

[0089] Although FastSAM can generate detailed segmentation masks, the directly generated masks often contain a large number of false targets and background noise due to the inherent sparsity and noise characteristics of radar point clouds. To solve this problem, this system designs a confidence-based noise filtering unit, the processing flow of which is as follows:

[0090] First, the system starts from the initial semantic segmentation map in the first stage. Extracting high-confidence background regions and specific interference areas Constructing a noise template :

[0091]

[0092] The operation represents AND.

[0093] Next, the mask is optimized through the following sequence of operations:

[0094]

[0095] in, This indicates element-wise multiplication. This represents morphological opening and closing operations, used to eliminate small noise areas and fill voids. This indicates the high-confidence foreground regions selected from the initial segmentation results, ensuring that important targets are not mistakenly filtered out. This represents the mask after noise filtering, which is used for initial denoising driven by confidence level. The mask representing the result of morphological operations; This represents the optimized segmentation mask.

[0096] 2.3 Fixed the generation of the guide map.

[0097] Based on the optimized segmentation mask, the system generates a guidance map to guide subsequent image restoration. This guidance map not only contains binary mask information but also incorporates semantic label information based on radar point classification results.

[0098]

[0099] in, It is a semantic encoding function that combines a binary mask with radar point classification confidence. Combined, a repair guidance diagram with rich semantic information is generated. .

[0100] Each channel of the guiding map corresponds to a semantic category, with an additional channel representing the background area that does not require restoration. By projecting the radar point cloud onto the image plane, the classification confidence vector of each radar point is assigned to its projected image pixel location, and distance-based interpolation diffusion is performed on the surrounding area to generate a dense semantic probability map.

[0101] Through this stage of processing, the system successfully transforms sparse radar detection results into dense and reliable repair guidance information, providing precise spatial guidance and semantic constraints for the generative repair in the third stage, thereby achieving high-quality scene understanding and image repair in complex environments.

[0102] Phase 3: Generative repair and model enhancement based on diffusion model.

[0103] The core objective of this stage is to recover visual information damaged by harsh environments through generative models and to improve the performance of semantic segmentation models using the restored and enhanced data. This stage creatively introduces image inpainting technology into the perception system, achieving closed-loop optimization of "perception-inpainting-reperception".

[0104] 3.1 Zero-sample image inpainting based on diffusion model.

[0105] Custom Diffusion is a technique that enables multi-concept customization by fine-tuning a text-to-image diffusion model. This stage uses a pre-trained Custom Diffusion engine as the inpainting engine, leveraging its powerful zero-shot generation capabilities to recover occluded or degraded visual content. The inpainting process is guided by the inpainting guide map generated in the second stage. and the original image As input, supplemented by semantic prompt text generated based on radar point classification results. Provide high-level semantic guidance: Statistical confidence level of radar point classification The dominant semantic category in the image is used to generate a natural language description, which is then input into the text encoder of the diffusion model to be converted into a semantic embedding vector. This vector is then cross-attentioned with the latent image representation to obtain a repaired image that conforms to both physical structure and semantic logic. .

[0106] This diffusion model gradually recovers the visual content of the damaged region by iteratively denoising in the latent space. Its core iterative optimization process can be formally represented as:

[0107]

[0108] here, Indicates the diffusion time step. Indicates the time step in the diffusion process Potential variables at time; Indicates the previous diffusion time step; Indicates at time step The latent variable at that time is the result after one round of denoising; , , For noise scheduling parameters, For a well-trained denoising network, The noise is standard Gaussian noise. The process iteratively optimizes the latent space to progressively recover the visual content of the damaged area while maintaining consistency with the surrounding environment.

[0109] 3.2 Multimodal feature fusion and enhanced reasoning.

[0110] To fully utilize the complementary information in the original and restored images, a dual-branch feature fusion architecture is designed. This architecture processes the original image separately using two independent Segformer encoders. and image restoration Multi-scale depth features of the original image are obtained. Multi-scale depth features of repaired images .

[0111] The Segformer encoder is a semantic segmentation framework based on Transformer, featuring a hierarchical Transformer encoder and a lightweight MLP decoder.

[0112] Subsequently, the system integrates the two information sources through a feature-level fusion mechanism. This mechanism can dynamically adjust the dependence on each mode according to the signal-to-noise ratio at a specific location. Its core calculation process is as follows:

[0113]

[0114] in, For the sigmoid function, and For learnable gating parameters, This indicates a feature splicing operation. This represents the gate value, which is adaptively weighted according to the feature quality, and its value range is [0,1]. The feature-level fusion feature is a collection of multi-scale feature tensors.

[0115] Gating value When the value is close to 0, the output relies more on the clear features after repair; gating value When the value is close to 1, the original image features are trusted more. This adaptive fusion strategy significantly improves the robustness and accuracy of the final semantic segmentation results in complex and degraded regions.

[0116] This fusion feature It contains multimodal complementary information, and then a lightweight segmentation decoder is used to generate the final semantic segmentation map:

[0117]

[0118] in, This is a segmentation head consisting of a small number of upsampling and convolutional layers, used to transform fused features into a dense semantic prediction map. .

[0119] 3.3 Progressive model enhancement training.

[0120] To continuously improve segmentation performance, the system employs a progressive training strategy. First, a basic segmentation model is trained based on the original image data; then, augmented data after restoration is introduced to fine-tune the model.

[0121] The fine-tuning process employs a specially designed multi-task learning framework to improve performance by jointly optimizing segmentation accuracy and feature consistency. The overall objective function is composed of a weighted average of segmentation loss, consistency loss, and perceptual loss.

[0122]

[0123] in, It represents the total loss during training and is a weighted loss function for multi-task learning; and Indicates weight; It is the standard segmentation loss, used to ensure that the model outputs accurate segmentation results, and is the basic supervision signal for the segmentation task. Perceptual loss is used to constrain the semantic consistency between the restored content and the original image. The core is the consistency loss. The definition is as follows:

[0124]

[0125] in, Indicates the first The, here meaning the first One location; This indicates that the segmentation result based on the original image is in the th... The values ​​at each position (processed by the decoder to achieve fused features) get); The segmentation result based on the repaired image is represented in the first... The value of each position; The confidence plot is shown in the first... The values ​​at each position are used as weights.

[0126] During training, a perceptual loss constraint is introduced to ensure semantic consistency between the repaired content and the original image.

[0127] 3.4 Adaptive reasoning mechanism.

[0128] During the inference phase, the system dynamically adjusts its operating mode based on environmental conditions and confidence level assessment. When environmental conditions are favorable and the confidence level of the repair guidance map is high, the system prioritizes using the output of the repair enhancement path; when the environment is unfavorable or the repair quality is uncertain, the system reverts to the segmentation result based on the original image. This adaptive mechanism ensures the system's reliability under various operating conditions.

[0129]

[0130] in, The semantic output graph is the final semantic segmentation result; This represents the segmentation result based on the original image; The quality score for the guided image in the second stage. and Where is the quality threshold, The high-quality threshold score is a constant. This is the low-quality threshold score, which is also a constant. These are the interpolation weights based on the quality score.

[0131] Through generative inpainting and model enhancement in this phase, the system not only recovered visual information lost due to environmental factors, but also significantly improved the accuracy and robustness of semantic segmentation through multimodal fusion. This innovative architecture, which combines generative and discriminative models, provides an effective solution to visual perception problems in harsh environments.

[0132] Phase 4: Multi-frame semantic fusion and dense 3D modeling.

[0133] The core objective of this stage is to combine the enhanced and repaired frame-by-frame 2D semantic information output from the previous stages with the precise 3D geometric information provided by the radar. Through multi-frame data fusion, a globally consistent and semantically rich dense 3D environment model is ultimately constructed. This stage completes the mapping from 2D perception to the 3D world and is a crucial step in generating the final usable environment model.

[0134] 4.1 Sensor pose estimation and point cloud correlation.

[0135] The system continuously estimates the pose of the camera and radar in the global coordinate system using a radar-assisted SLAM algorithm, i.e., the pose of the camera and radar in the global coordinate system. pose transformation matrix at time step ,in For timestamps, It is a three-dimensional special Euclidean group. Each frame of the radar point cloud... All were transformed to the global coordinate system:

[0136]

[0137] in, Represents a point cloud in the global coordinate system; Indicates that the camera and radar are in The pose transformation matrix at time t.

[0138] This process establishes the spatial correspondence of point clouds between consecutive frames, forming the geometric framework for 3D modeling.

[0139] 4.2 Semantic information projection and voxel fusion.

[0140] The system employs a voxel-based fusion strategy to incrementally construct a global 3D semantic map.

[0141] First, create a global voxel map: initialize a 3D voxel mesh in the global coordinate system, with each voxel... Stores two core pieces of information:

[0142] Geometric Occupancy Probability (Indicates the probability that the voxel is occupied by an object);

[0143] Semantic probability distribution (A D-dimensional vector representing the probability that the voxel belongs to each semantic category). Wherein, Indicates the first Individual factors.

[0144] Then, semantic information back projection is performed: for the first... frame( The frames obtained at each time point will be used to output the high-precision semantic segmentation result in the third stage. With sensor pose The semantic information of each pixel is back-projected onto its corresponding voxel on a 3D ray using a camera model. Specifically, for each voxel, its semantic probability distribution is updated by fusing semantic information from all frames that observed it:

[0145]

[0146] in It is the first frame( The semantic probability provided when the voxel is observed in the frame obtained at time (time). It is the weight of this observation, which can be based on the confidence level of this frame. The sensor distance or other factors can be dynamically adjusted.

[0147] Finally, geometric information fusion is performed: simultaneously, radar point clouds transformed to global coordinates are utilized. We employ a fusion method based on truncated symbolic distance function (TSDF) to incrementally update the geometric occupancy probability of voxels in order to generate smoother and more accurate object surfaces.

[0148] 4.3 Global model extraction and optimization.

[0149] After a sufficient number of data frames have been fused into the voxel map, the system extracts the final, usable 3D model with semantic information. This process includes the following steps:

[0150] First, based on the geometric occupancy probability field The moving cube algorithm is used to extract a continuous triangular mesh surface model. ;

[0151] Then, semantic mapping is performed: each triangular facet is mapped to its corresponding voxel. This association allows the semantic probability distribution stored in voxels to be linked. Assign a mesh model to generate a 3D mesh with semantic labels. .

[0152] This model not only has precise geometric surfaces, but each of its patches also contains a semantic probability vector, providing directly usable high-order environmental prior information for the robot's semantic perception and decision-making.

[0153] 4.4 Real-time incremental output.

[0154] The entire fusion and reconstruction process is designed to be incremental, which can update the global 3D model in real time as the robot moves, providing continuous environmental information support for the robot's online positioning, navigation and decision-making.

[0155] Example 2

[0156] This embodiment discloses a 3D modeling system for harsh environments that combines multimodal fusion and generative repair.

[0157] A 3D modeling system for harsh environments that integrates multimodal fusion and generative repair, including:

[0158] The raw data acquisition module is configured to: acquire the raw images and raw point clouds of the environment to be modeled;

[0159] The initial semantic segmentation module is configured to: use a cross-modal attention fusion method to fuse features between the original image and the original point cloud, and complete the initial semantic segmentation to obtain an initial semantic segmentation map; at the same time, a dual-branch mechanism is used to obtain the confidence map corresponding to the initial semantic segmentation map.

[0160] The guidance graph generation module is configured to: obtain an object segmentation mask based on the original point cloud; optimize the object segmentation mask by combining the initial semantic segmentation map and the corresponding confidence map; and generate a guidance graph based on the optimized segmentation mask.

[0161] The image restoration module is configured to restore the original image based on the generated guide image to obtain a restored image;

[0162] The segmentation optimization module is configured to: perform feature fusion on the original image and the repaired image to obtain feature-level fused features, and then obtain a semantic prediction map; and use an adaptive inference mechanism to combine the semantic prediction map to obtain a semantic output map.

[0163] The semantic fusion modeling module is configured to combine the semantic output image and the repaired image to construct a three-dimensional environment model of the environment to be modeled through multi-frame semantic fusion and dense 3D modeling technology.

[0164] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.

[0165] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A method for harsh environment 3D modeling with multi-modal fusion and generative inpainting, characterized in that, The method comprises the following steps: Collecting original images and original point clouds of an environment to be modeled; Fusing features of the original images and the original point clouds by using a cross-modal attention fusion method, and completing initial semantic segmentation to obtain an initial semantic segmentation map; meanwhile, a confidence map corresponding to the initial semantic segmentation map is obtained by using a double-branch mechanism; Obtaining an object segmentation mask based on the original point clouds; Optimizing the object segmentation mask in combination with the initial semantic segmentation map and the corresponding confidence map, and generating a guide map based on the optimized segmentation mask; Repairing the original images based on the generated guide map to obtain a repaired image; Fusing features of the original images and the repaired image to obtain feature-level fused features, and then obtaining a semantic prediction map; a semantic output map is obtained by using an adaptive inference mechanism in combination with the semantic prediction map; Constructing a three-dimensional environment model of the environment to be modeled by using multi-frame semantic fusion and dense three-dimensional modeling technology in combination with the semantic output map and the repaired image; The generation process of the guide map specifically comprises: Radar points projected to the image coordinate system are inputted into the FastSAM model as position cues to generate the corresponding object segmentation mask ; extracting high-confidence background regions and specific interferent regions from the initial semantic segmentation map constructing a noise template ; Noise template based Object segmentation mask optimization is achieved by the following sequence of operations ​ ； wherein, represents element-wise multiplication, represents a morphological opening-closing operation, represents a high-confidence foreground region filtered from an initial semantic segmentation map ; represents a noise-filtered mask; represents a mask after a morphological opening-closing operation; represents an optimized segmentation mask; represents an OR operation; Using a semantic encoding function, the optimized segmentation mask Confidence of radar point classification Combined, a guide map is generated. .

2. The multi-modal fusion and generative repaired harsh environment 3D modeling method of claim 1, wherein, The original images and the original point clouds of the environment to be modeled are collected, specifically comprising: The original images are collected by a camera, and the original point clouds are collected by a millimeter wave radar; the original point clouds comprise five dimensions, namely 3D space coordinates, radar scattering cross-sectional area and Doppler velocity; The original images are scaled and normalized; The original point cloud data is normalized by using a dynamic sampling strategy.

3. The multi-modal fusion and generative remediation based harsh environment 3D modeling method of claim 1, wherein, The features of the original images and the original point clouds are fused by using a cross-modal attention fusion method, specifically comprising: A double-branch encoder architecture is constructed, including a visual encoding branch and a radar encoding branch; Extracting multi-level visual features from an original image using a visual coding branch , indicates the first layer; Extracting point cloud geometric features from raw point clouds using radar code branches ; Cross-modal fusion mechanism based on cross attention, to multi-level visual features Point cloud geometry features as query vectors As key-value pairs, the adaptive fusion of visual information and spatial geometric information is achieved by calculating the correlation weight between the image spatial position and the radar points, and the multi-scale fusion features are obtained.

4. The multi-modal fusion and generative repaired harsh environment 3D modeling method of claim 3, wherein, The specific acquisition process of the initial semantic segmentation map is as follows: An improved decoder based on the U-Net architecture is adopted to convert the multi-scale fused features into an initial semantic segmentation map through progressive upsampling and feature aggregation operations ; Or, The confidence map corresponding to the initial semantic segmentation map is obtained by using a double-branch mechanism, specifically comprising: One branch in the double-branch mechanism is for point cloud geometric features Independent processing, point-level semantic classification is realized by multi-layer perception, and radar point classification confidence is obtained ; Another branch in the dual-branch mechanism is based on a cross-modal attention weight matrix to compute a spatial consistency score ; The spatial consistency score and the radar point classification confidence are integrated to obtain a confidence map : ； wherein, denotes a learnable fusion module, denotes the first layer a spatial consistency score of the image pixels.

5. The multi-modal fusion and generative remediation based harsh environment 3D modeling method of claim 4, wherein, The original images are repaired based on the generated guide map to obtain a repaired image, specifically comprising: A pre-trained diffusion model is used as a repair engine; The generated guide map and the original image are used as inputs of the diffusion model, and a semantic prompt text generated based on a radar point classification result is used for high-level semantic guidance; by iteratively denoising in a latent space, the visual content of the damaged area is gradually restored to obtain a repaired image.

6. The multi-modal fusion and generative remediation of harsh environment 3D modeling method of claim 5, wherein, The generation process of the semantic prediction map specifically comprises: The original image and the repaired image are respectively processed by two independent Segformer encoders to obtain multi-scale deep features of the original image and multi-scale deep features of the repaired image ; The feature-level fusion mechanism is used to obtain the feature-level fusion feature : ； wherein, is a sigmoid function, and is a learnable gating parameter, denotes a feature concatenation operation; denotes an element-wise multiplication; denotes a gating value; fusing features at a feature level generate a semantic prediction map through a lightweight segmentation decoder Or, The generation process of the semantic output map specifically comprises: A quality score of the guide map is generated; A high-quality threshold score and a low-quality threshold score are set; Adaptively inferring semantic output graphs using the following equation : ； wherein, is a quality score of the guided map, and are a high quality threshold score and a low quality threshold score, respectively, is an interpolation weight; denotes a segmentation result based on the original image.

7. The multi-modal fusion and generative remediation based harsh environment 3D modeling method of claim 1, wherein, The semantic segmentation model is trained by using a progressive training strategy: First, the semantic segmentation model is trained based on original image data; Subsequently, the semantic segmentation model is fine-tuned by introducing repaired image data; The fine-tuning process adopts a multi-task learning framework to jointly optimize segmentation accuracy and feature consistency, and the objective function is as follows: wherein, represents the total loss in the training process; and represents the weight; is the standard segmentation loss, is the perception loss, is the consistency loss.

8. The multi-modal fusion and generative remediation based harsh environment 3D modeling method of claim 1, wherein, The three-dimensional environment model of the environment to be modeled is constructed by using multi-frame semantic fusion and dense three-dimensional modeling technology in combination with the semantic output map and the repaired image, specifically comprising: The radar point clouds in each frame of the original image are converted to a global coordinate system; A global voxel map is created, and a voxel-based fusion strategy is used to incrementally construct a global three-dimensional semantic map; After the global voxel map has fused a sufficient number of data frames, a three-dimensional model with semantic information is extracted from the global voxel map; wherein the semantic probability distribution of each voxel is updated by fusing semantic information of all frames in which it is observed: ； in, It is the first The semantic probability provided when the frame observes the current voxel. It is the weight of the observation; For timestamps; Representing each voxel The semantic probability distribution Indicates the first Individual factors.

9. A multi-modal fusion and generative inpainting based harsh environment 3D modeling system, characterized in that, comprise: a raw data acquisition module configured to collect raw images and raw point clouds of an environment to be modeled; an initial semantic segmentation module configured to perform feature fusion on the raw images and the raw point clouds using a cross-modal attention fusion method, and complete initial semantic segmentation to obtain an initial semantic segmentation map; meanwhile, an initial semantic segmentation map corresponding confidence map is obtained using a double-branch mechanism; a guide map generation module configured to obtain an object segmentation mask based on the raw point clouds; optimizing the object segmentation mask in combination with the initial semantic segmentation map and the corresponding confidence map, and generating a guide map based on the optimized segmentation mask; an image inpainting module configured to inpaint the raw images based on the generated guide map to obtain inpainted images; a segmentation optimization module configured to perform feature fusion on the raw images and the inpainted images to obtain feature-level fused features, and further obtain a semantic prediction map; an adaptive inference mechanism is used to obtain a semantic output map in combination with the semantic prediction map; a semantic fusion modeling module configured to construct a three-dimensional environment model of the environment to be modeled by combining the semantic output map and the inpainted images through multi-frame semantic fusion and dense three-dimensional modeling technology; the generation process of the guide map specifically comprises: Radar points projected into the image coordinate system are inputted into the FastSAM model as position cues to generate the corresponding object segmentation mask ; extracting high-confidence background regions and specific interferent regions from the initial semantic segmentation map constructing a noise template ; Noise template based Object segmentation mask optimization is achieved by the following sequence of operations ​ ； wherein, represents an element-wise multiplication, represents a morphological opening-closing operation, represents a high-confidence foreground region filtered from an initial semantic segmentation map represents a noise-filtered mask; represents a mask after a morphological opening-closing operation; represents an optimized segmentation mask; represents an OR operation;​ The optimized segmentation mask is combined with radar point classification confidence to generate a guidance map .​

Images