Non-static scene reconstruction method and system based on multi-modal occlusion perception scoring

Abstract

The invention discloses a non-static scene reconstruction method and system based on multi-modal occlusion perception scoring. The method comprises the following steps: generating a geometric consistency distribution diagram, static feature points and geometric prior masks through three-dimensional reconstruction of a multi-view image; fusing the geometric prior mask and a semantic segmentation model to extract a semantic mask and a fused semantic feature map; guiding the image segmentation model to generate candidate masks based on static feature point positive point prompt and occlusion area negative frame prompt, and optimizing the consistency by using a grid complementary fusion method; constructing a multi-modal shielding scoring module, and fusing the multi-source features to output a binary static mask; and utilizing a static mask to constrain neural radiation field training, inhibiting dynamic interference and optimizing static scene reconstruction. According to the method, the static region is sensed cooperatively through multi-modal information, the robustness and accuracy of mask generation are improved, the interference of dynamic elements on neural radiation field modeling is effectively inhibited, and high-quality three-dimensional image reconstruction and new view synthesis of a non-static scene are realized.

Description

Technical Field

[0001] This invention relates to the field of computer vision and 3D image reconstruction technology, and in particular to a non-static scene reconstruction method and system based on multimodal occlusion perception scoring. Background Technology

[0002] Neural Radiance Fields (NeRF), with its ability to accurately model the radiation field of a scene using implicit neural networks, has demonstrated outstanding performance in 3D reconstruction and new view synthesis of static scenes. Using multi-view images and corresponding camera poses as input, it can achieve high-precision 3D reconstruction and high-fidelity new view rendering of static scenes, making it a research hotspot in the field of 3D vision. However, the modeling logic used in NeRF heavily relies on the assumption of global scene consistency, meaning that the input multi-view images must satisfy content stability and unambiguous cross-view matching. When "transient interference" exists in the image, such as moving pedestrians, vehicles, or dynamically changing objects in the scene, their presence can cause conflicts in the radiation field descriptions of corresponding regions in different views. This prevents NeRF from learning consistent spatial features during training, ultimately resulting in problems such as blurring, ghosting, and geometric distortion in the reconstruction results. Furthermore, cross-view matching ambiguity in dynamic regions can further interfere with the model's learning of the radiation field of static regions, leading to the loss of static structural details.

[0003] Existing technologies primarily address these issues in two ways. First, they use pre-trained segmentation models to detect transient interference, segmenting the transient interference region from the image and generating a mask. However, the performance of such methods is highly dependent on the prerequisite that the transient object belongs to a predefined semantic category. They cannot effectively detect dynamic elements not included in the pre-training data, leading to the failure of dynamic interference suppression. Second, they use manually preset thresholds to filter static regions based on metrics such as cross-view feature matching consistency, pixel grayscale value change rate, or optical flow vector size. However, these methods have poor robustness and are difficult to adapt to different scenarios. Furthermore, existing technologies mainly process single-modal information or simply fuse multi-modal features, failing to fully explore the complementary characteristics between different modalities, thus making it impossible to accurately locate the static core region.

[0004] Therefore, there is an urgent need for a reconstruction method that can effectively handle dynamic interference in non-static scenes and accurately capture the features of static areas, so as to overcome the application limitations of NeRF in non-static scenes and achieve high-quality 3D reconstruction and view synthesis. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a non-static scene reconstruction method and system based on multimodal occlusion perception scoring. By collaboratively perceiving static regions through multimodal information, the robustness and accuracy of mask generation are improved, effectively suppressing the interference of dynamic elements on neural radiation field modeling, and realizing high-quality 3D image reconstruction and new view synthesis of non-static scenes. The aim is to effectively remove dynamic occlusions and improve the 3D reconstruction effect of neural radiation field (NeRF) in non-static scenes.

[0006] To achieve the above objectives, the present invention provides the following technical solution: In a first aspect, the present invention provides a non-static scene reconstruction method based on multimodal occlusion perception scoring, comprising the following steps: Step 1: Perform 3D reconstruction on the multi-view RGB image to obtain the scene's 3D point cloud and camera pose; generate geometric prior information for the scene's 3D point cloud based on the multi-view matching consistency measure of 2D local feature points in the multi-view RGB image; wherein, the geometric prior information includes a geometric consistency distribution map, static feature points, and a geometric prior mask; Step 2: Use the geometric prior mask as a cue to assist the semantic segmentation model in extracting the semantic mask of the fused semantic feature map and the multi-view RGB image; Step 3: Generate positive point prompts from the static feature points obtained based on 3D reconstruction, and generate negative box prompts from the bounding boxes of the occluded areas extracted by semantic masking. Guide the image segmentation model to output candidate static masks and candidate occlusion masks, and optimize the mask consistency through mesh complementary fusion method to obtain candidate static masks and secondary candidate occlusion masks. Step 4: Using the geometric consistency distribution map, candidate static mask, secondary candidate occlusion mask and fused semantic feature map as multi-source features, perform differential processing and feature enhancement on the multi-source features using the multi-modal occlusion scoring module, and generate a binary static mask through dual-branch weight prediction and region-aware fusion. Step 5: Integrate the binary static mask into the pixel-level loss function of the neural radiation field model. By using the binary static mask, the weight of light rendering error optimization in static areas is enhanced, and the loss contribution of dynamic and occluded areas is suppressed. This drives the training of the neural radiation field model and completes the 3D reconstruction of non-static scenes and the synthesis of new perspectives.

[0007] Preferably, step 1 includes: Perform 3D reconstruction on multi-view RGB images to extract 2D local feature points from each image, and calculate the multi-view matching consistency metric for each 2D local feature point.

[0008] The multi-view matching consistency metric of each 2D local feature point is mapped to the pixel coordinates of the corresponding image, and a single-view geometric consistency distribution map is generated by bilinear interpolation. For each view of the RGB image, select 2D local feature points with a multi-view matching consistency metric ≥ threshold T, and use these as the static feature points of the current image.

[0009] All static feature points in the same image are mapped to the original RGB image, and a geometric prior mask is generated by optimizing the morphological closing operation.

[0010] Preferably, step 2 includes: Each view's RGB image is processed by an encoder of a semantic segmentation model to extract multi-scale semantic feature maps; Align the geometric prior mask with the multi-scale semantic feature map, perform pixel-level multiplication fusion, and generate a multi-scale semantic feature map after fusing the geometric prior. The multi-scale semantic feature maps are upsampled to align the resolution and then concatenated to obtain a fused semantic feature map. The semantic segmentation model's decoder performs semantic classification on the fused semantic feature map, maps it to a high-dimensional semantic category space, and then upsamples it back to the image resolution to obtain the semantic mask of the current view's RGB image.

[0011] Preferably, step 3 includes: Positive cue points are generated by uniformly sampling all static feature points of the RGB image in each view. Boundary boxes of occluded regions are extracted from the semantic mask of the current RGB image and used as negative cue points. Positive cue points and negative cue points are input into the image segmentation model to obtain candidate static masks and candidate occlusion masks. The candidate static mask and the inverse code of the candidate occlusion mask are divided into meshes using the mesh complementary fusion method, and the candidate static mask within the mesh is calculated. Inverse code of candidate occlusion mask Consistency percentage ; Based on consistency percentage Update the inverse codes of all candidate occlusion masks within the grid; merge the updated inverse codes of all candidate occlusion masks to obtain the inverse code of the new candidate occlusion mask; finally, invert the inverse code of the new candidate occlusion mask to obtain the secondary candidate occlusion mask.

[0012] More preferably, the step of basing the proportion of consistency The process of updating the inverse of the candidate occlusion mask within all grids is as follows: Consistency ratio If the grid is consistent, it is considered a consistent grid and no further processing is performed; the original value of the candidate occlusion mask inverse code is retained. If the consistency ratio is... If the result is not consistent, the pixel values ​​are fused according to weights, and then binarized to obtain the inverse code of the new candidate occlusion mask.

[0013] The weighted fusion of pixel values ​​is achieved by normalizing the local confidence weights of the inverse codes of the candidate static mask and the candidate occlusion mask.

[0014] Preferably, step 4 includes: Geometric consistency distribution map, candidate static mask, secondary candidate occlusion mask, and fused semantic feature map are used as multi-source features. The multi-source features are differentiated and then spatial features are extracted using depthwise separable convolution to obtain multi-source enhanced features. The multi-source enhanced features are concatenated and then global average pooling is performed to extract cross-modal global features. A dual-branch weight predictor is used to learn the modal weights for each region in static and occlusion regions respectively. Using the inverted result of the secondary candidate occlusion mask as a region indicator, static weight branch or occlusion weight branch is selected pixel by pixel, and weighted fusion is performed to obtain the fusion feature of each pixel. After dimensionality reduction of the fused features, a pixel-level static confidence map is generated using the Sigmoid function, and then binarized into the final static mask.

[0015] Preferably, in step 5, the static mask is incorporated into the pixel-level loss function of the neural radiation field. In Chinese, the formula is as follows:

[0016] in For pixels In the rendering of colors within a neural radiation field, For pixels The true color, Represents pixels The static mask.

[0017] Secondly, the present invention provides a non-static scene reconstruction system, the system comprising: The image geometric prior information extraction module is responsible for performing 3D reconstruction on multi-view RGB images to obtain the scene's 3D point cloud and camera pose; based on the multi-view matching consistency measure of 2D local feature points of the multi-view RGB images, it generates geometric prior information for the scene's 3D point cloud; wherein, the geometric prior information includes a geometric consistency distribution map, static feature points, and a geometric prior mask; The image semantic mask extraction module is responsible for using the geometric prior mask extracted by the image geometric prior information extraction module as a cue to assist the semantic segmentation model in extracting the semantic mask of the fused semantic feature map and multi-view RGB image; The candidate static mask and secondary candidate occlusion mask extraction module is responsible for generating positive point prompts from the static feature points obtained based on 3D reconstruction, generating negative box prompts by combining the bounding boxes of the occlusion areas extracted by semantic mask, guiding the image segmentation model to output candidate static masks and candidate occlusion masks, and optimizing mask consistency through mesh complementary fusion method to obtain candidate static masks and secondary candidate occlusion masks. The multimodal occlusion scoring module is responsible for using geometric consistency distribution map, candidate static mask, secondary candidate occlusion mask and fused semantic feature map as multi-source features. After performing differential processing and feature enhancement on the multi-source features, it generates a binary static mask through dual-branch weight prediction and region-aware fusion. The scene reconstruction module is responsible for integrating binary static masks into the pixel-level loss function of the neural radiation field model. By using binary static masks, it enhances the weights of light rendering errors in static areas, suppresses the loss contribution of dynamic and occluded areas, drives the training of the neural radiation field model, and completes the 3D reconstruction of non-static scenes and the synthesis of new perspectives.

[0018] Thirdly, the present invention provides an electronic device including a processor and a memory, the memory storing machine-executable instructions executable by the processor, the processor executing the machine-executable instructions to implement the above-described method.

[0019] Fourthly, the present invention provides a machine-readable storage medium storing machine-executable instructions that, when invoked and executed by a processor, cause the processor to implement the above-described method.

[0020] The beneficial effects of the present invention include at least the following: This invention designs a method for generating an initial mask using a geometric-semantic dual-cue guide SAM and a mesh complementary fusion method to correct mask errors. Through the collaborative constraint of positive cue points on static feature points and negative bounding box cue points for occluded regions, the SAM is guided to initially segment static and occluded regions. By combining geometric features and semantic information to perform weighted fusion of inconsistent meshes, the accuracy and consistency of the mask are effectively improved.

[0021] This invention constructs a multimodal occlusion scoring module, which integrates multi-source features such as geometric consistency distribution map and secondary candidate static mask, and dynamically generates static confidence score map and binary static mask through fusion. It automatically learns the correlation between multimodal features, is suitable for complex feature processing in non-static scenes, and overcomes the defect of single-modal feature processing in existing technologies that makes it difficult to accurately identify static regions.

[0022] This invention adds a static mask-guided loss function mechanism, which adjusts the loss weights of neural radiation field training based on the generated binary static mask. By strengthening the static region loss and suppressing dynamic region interference, it enhances the learning of static scene features and solves the problem of decreased reconstruction accuracy caused by dynamic elements in non-static scenes. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 The present invention provides a block diagram of a non-static scene reconstruction method based on multimodal occlusion perception scoring.

[0025] Figure 2 This is an architecture diagram of a non-static scene reconstruction method based on multimodal occlusion perception scoring, according to an embodiment of the present invention.

[0026] Figure 3 This is a flowchart of a method for correcting mask errors based on a geometric-semantic dual-hint guidance SAM method to generate an initial mask and a mesh complementary fusion method, according to an embodiment of the present invention.

[0027] Figure 4 This is a flowchart of a multimodal occlusion scoring method for an embodiment of the present invention. Detailed Implementation

[0028] To more clearly illustrate the technical solution of the present invention, the embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the embodiments described herein are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0029] This application provides a method for reconstructing the neural radiation field of a non-static scene based on multimodal occlusion perception scoring, aiming to effectively remove dynamic occlusions and improve the accuracy and stability of 3D reconstruction of the NeRF model in non-static scenes.

[0030] The above technical solution will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0031] like Figure 1As shown in the figure, this embodiment of the invention provides a block diagram of a non-static scene neural radiation field reconstruction method based on multimodal occlusion perception scoring, including: S1. Acquire multi-view RGB images to reconstruct 3D scene point clouds and camera poses, and construct geometric prior information based on the feature matching relationship between multi-view point clouds.

[0032] S2. Using geometric prior masks as cues, the semantic segmentation model is assisted in extracting semantic masks from the fused semantic feature map and multi-view RGB image.

[0033] S3. A method that guides SAM to generate an initial mask through geometric-semantic dual prompts and corrects mask errors by mesh complementarity fusion, outputting candidate static masks and candidate occlusion masks that tend to be complementary.

[0034] S4. Construct a multimodal occlusion scoring module. Take the geometric consistency distribution map, candidate static mask, secondary candidate occlusion mask and fused semantic feature map as input. After feature preprocessing to unify the dimension, and after depthwise separable convolution to enhance spatial details, a dual-branch weight predictor learns the feature weight preference of static or occlusion regions. After dynamically fusing multi-source features, the module outputs a binary static mask for NeRF training through confidence scoring and binarization.

[0035] S5. Based on binary static mask-guided Mip-NeRF training, loss interference in dynamic regions of the scene is dynamically suppressed and radiation field learning in static regions is enhanced. After training, the target scene is synthesized and rendered from a new perspective to obtain a complete and clear 3D reconstruction result.

[0036] This invention introduces a multimodal fusion mechanism in the mask construction stage, combining geometric consistency, semantic segmentation and scoring mechanisms to improve the accuracy and training stability of static region recognition in NeRF reconstruction in non-static scenes, effectively suppressing the interference of dynamic occlusion on reconstruction quality, and has broad scene adaptability and promotion application value.

[0037] like Figure 2 As shown, Figure 2 An architecture diagram of a nonstatic scene reconstruction method based on multimodal occlusion perception scoring is disclosed. The following will combine... Figure 2 The content details each step: In step S1, multi-view RGB images are acquired to reconstruct the 3D scene point cloud and camera pose. Geometric prior information is constructed based on the feature matching relationship between the multi-view point clouds.

[0038] SfM (Structure from Motion) is a 3D reconstruction technique that recovers the 3D structure of a scene by utilizing feature associations from multiple view images.

[0039] 1-1 Obtain a set of multi-view input RGB images with transient objects Where H is the image height, W is the image width, and N is the number of images.

[0040] 1-2 Use SfM for each RGB image Extract a set of 2D local feature points :

[0041] in It is an appearance descriptor used to encode the visual appearance information of a local image region surrounding a feature point. Indicates its position in the image pixel coordinates in In the image The number of 2D local feature points in the data.

[0042] 1-3 SfM utilizes the feature set of all RGB images Reconstruct a sparse point cloud C representing the 3D structure of the target scene, where the correspondence between 2D feature points in different images depends on whether they are projected from the same 3D point in the sparse point cloud C.

[0043] 1-4 For each 2D local feature point Record its image The number of matching points it possesses is Based on the number of matching points, calculate the multi-view matching consistency metric for this feature point:

[0044] The method for finding the matching point is as follows: 1. Calculate the distance between appearance descriptors for feature points extracted from different images; 2. Set a threshold and filter out feature point pairs with the smallest distance that meet the threshold condition as matching points, i.e., if the image... Middle feature points With another image Middle feature points The descriptor distance satisfies the matching condition, and the two are each other's matching points.

[0045] 1-5 Measure the consistency of multi-view matching for each 2D local feature point Mapped to its pixel coordinates For areas without feature point coverage, bilinear interpolation is used to fill the gaps, generating a single-view geometrically consistent distribution map. ,in Represents pixels Multi-view matching stability at the location.

[0046] Because transient objects in a scene typically have low matching consistency across multiple views and a low matching frequency, they are easier to distinguish from stable, static objects.

[0047] 1-6 Set a threshold for matching consistency Filter out images China satisfies feature point set This means matching feature points with a frequency higher than a threshold and using them as static feature points of the image.

[0048] 1-7 Mapping the set of feature points in the image back to the image Generate a single-view static tooltip mask ,right Performing morphological closing operations fills in the sparse gaps between feature points, resulting in a more coherent static region mask. The results from all images are then aggregated to generate a multi-view geometric prior mask. .

[0049] In step S2, a geometric prior mask is used as a cue to assist the semantic segmentation model in extracting the semantic mask and multi-scale semantic feature map of the image.

[0050] This embodiment uses the SegFormer semantic segmentation model. SegFormer is used to generate semantic masks and semantic feature maps. Compared to the traditional Transformer, SegFormer's encoder MiT has a hierarchical structure with four stages. Each stage progressively reduces feature resolution (1 / 4→1 / 8→1 / 16→1 / 32) while increasing the number of channels (64→128→320→512) through block embedding and Transformer blocks. The number of Transformer blocks and channels varies in different stages, emphasizing a balance between low-resolution high-semantic features and high-resolution low-semantic features. Furthermore, because its hierarchical encoder implicitly includes spatial relationships through block embedding and progressive downsampling, SegFormer completely abandons positional encoding. The decoder uses a lightweight full MLP decoder, achieving multi-scale feature fusion through simple upsampling, concatenation, and MLP dimensionality reduction.

[0051] In this invention, a set of multi-view input images containing transient objects are input. No pixel-level semantic annotations, where H is the image height, W is the image width, and N is the number of images.

[0052] Using the SegFormer encoder MiT, information from local details to global semantics is captured progressively as the resolution decreases. The multi-scale semantic feature maps output from the four stages are as follows:

[0053] Unify the resolution of geometric priors and semantic features. Employ bilinear interpolation to mask the geometric priors. By downsampling to a resolution that matches the multi-scale semantic feature maps output at each stage, we obtain:

[0054] in The downsampling of the feature map indicates that bilinear interpolation will... Scale to target size.

[0055] Geometric prior weighted semantic features. Geometric prior masks at each scale. Corresponding semantic feature map Perform pixel-level multiplication to enhance the semantic feature weights of geometrically inconsistent regions:

[0056] in This represents pixel-level multiplication. This is to integrate the semantic feature map after geometric priors.

[0057] To aggregate multi-scale information, bilinear interpolation is used to fuse the results. , , Upsampling to resolution The formula is:

[0058] Combine the upsampled features with By concatenating the features, a fused semantic feature map is obtained:

[0059] This fused feature map contains multi-scale semantic information, providing a basis for distinguishing between static and occluded regions. The fused semantic feature map is then processed by an MLP decoder. Semantic classification is performed by mapping high-dimensional features to a semantic category space:

[0060] in Represents pixels It was predicted to be a semantic class c.

[0061] Re-apply bilinear interpolation Upsampled to image resolution, the final semantic mask is obtained. .

[0062] In step S3, this invention guides the SAM to generate an initial mask through a geometric-semantic dual-hint method and corrects mask errors using a mesh complementarity fusion method, outputting candidate static masks and candidate occlusion masks that tend to be complementary. Through the collaborative constraint of positive hints for static feature points and negative bounding box hints for occlusion regions, the SAM is guided to initially segment the static and occlusion regions. Furthermore, the mesh complementarity fusion method further corrects the inconsistencies in the initial mask, ensuring that the static and occlusion regions tend to be complementary.

[0063] SAM (Segment Anything Model) is an image segmentation model that unifies the 2D segmentation task by introducing a cue-based segmentation paradigm. Given an image I and a set of cues P as input, SAM can output a corresponding 2D segmentation binary mask. : P can be a point, a box, or a mask. Point hints and box hints are the most common forms, while area hints are used to assist them.

[0064] like Figure 3 As shown in the figure, the present invention provides a flowchart of a method for correcting mask errors based on a geometric-semantic dual-hint guided SAM generation and a mesh complementary fusion method, including: 3-1 Generation of the punctuality prompt. From the image. static feature point set K feature points are uniformly sampled to ensure coverage of different regions of the image, resulting in a set of coordinates for the positive prompts.

[0065] in .

[0066] 3-2 The region is marked as static and input into the image segmentation model SAM.

[0067] 3-3 Generation of semantic box hints. From semantic masks Extract the occluded region from the data, and denote it as... Calculate its minimum bounding rectangle:

[0068] in The coordinates of the top left corner of the region. The coordinates are the lower right corner of the region.

[0069] Use this bounding box as a negative box cue and input it into the image segmentation model SAM.

[0070] 3-4. Collaborative guidance through dual prompts. Including on-time prompts. With negative box prompt The model optimizes the segmentation by combining input SAMs and using the following logic: positive cue constraints constrain SAM to preserve the integrity of static regions, and negative bounding box cue guides SAM to focus. Refine the edges of dynamic or occluded objects.

[0071] SAM ultimately outputs two candidate masks after double hint optimization: Candidate static mask (1 = static candidate region, 0 = non-static) Candidate occlusion mask (1 = non-static candidate region, 0 = static).

[0072] To ensure that the two candidate masks tend to be complementary, this invention proposes a mesh complementarity fusion method to solve the given problem. Its main steps are as follows: right Inverting the result gives It can be determined .

[0073] Will and Perform grid generation, select The pixel grid size, where K is 16, is denoted as [pixel size]. and Each grid contains a set of pixels. .

[0074] Perform pixel-by-pixel comparisons according to the grid, and record the corresponding grid values. The number of pixels is ,calculate and Consistency percentage:

[0075] Set consistency threshold Here it is set to 0.9, if If the result is a consistent grid, no processing is performed, and the original value of the inverse code of the candidate occlusion mask is retained. like If the result is not consistent, it is considered an inconsistent mesh, and subsequent operations will proceed.

[0076] For inconsistent grids, weights are calculated based on the local confidence of the two masks. ,in for SfM point density within the grid for Alignment of semantic boxes within the grid with image edges.

[0077] Normalize the two weights For inconsistent pixels within the grid The formula is as follows:

[0078] Binarize the result if If it is 1, then it is 0; otherwise, it is 0.

[0079] 3-6 Repeat steps 3-5, and after traversing and processing all grid regions, merge them into a new mask. Invert the new mask: .

[0080] Final output candidate static mask and secondary candidate occlusion mask .

[0081] In step S4, a multimodal occlusion scoring module is constructed. The geometric consistency distribution map, candidate static mask, secondary candidate occlusion mask, and fused semantic feature map are used as multi-source feature inputs. After differential processing, the multi-source features are unified in dimension. After enhancing spatial details through depthwise separable convolution, a dual-branch weight predictor is used to learn the feature weight preferences of static and occluded regions. After dynamically fusing multi-source features, a binary static mask is output for NeRF training through confidence scoring and binarization. This retains high-confidence static regions to drive the model to learn scene geometry and appearance, while masking occluded regions to avoid interference, providing accurate regional constraints for transient scene reconstruction.

[0082] like Figure 4 As shown, the system of this embodiment provides a flowchart of a multimodal occlusion scoring method, including: Differentiated processing of input feature information ensures channel uniformity and dimension alignment.

[0083] Geometric Consistency Distribution Map To enhance spatial perception and strengthen geometric evidence in static areas, the formula is as follows:

[0084] in for Gaussian filtering is used to smooth noise. The convolution is increased to 32 channels.

[0085] For candidate static masks and secondary candidate occlusion mask This is achieved by extracting static and occluded edge information. After XORing with the original mask, the result is used... The formula for increasing the dimensionality of convolution to 32 channels is as follows:

[0086]

[0087] in The operator is used to extract mask edges.

[0088] For fused semantic feature maps First, it is upsampled to the same resolution as other features, and then a 1×1 convolution is used to directly compress the channels, quickly adapting to the dimensionality requirements of subsequent multimodal fusion. The formula is as follows:

[0089] Depthwise separable convolutions enhance spatial details. For the four preprocessed 32-channel features, spatial features are extracted using depthwise separable convolutions to enhance local detail representation. Specifically, depthwise separable convolutional layers are employed, including depthwise convolutions. A 3×3 convolution is used independently for each input channel to extract spatial features; pointwise convolutions are then performed using 1×1 convolutions to fuse the depthwise convolution results, maintaining a channel count of 32. The output is the enhanced feature. .

[0090] Region-aware dynamic attention fusion. The four enhancement features are concatenated along the channel dimension to obtain:

[0091] Global average pooling (GAP) is used to extract cross-modal global features, providing a global context for weight prediction.

[0092] A dual-branch weight predictor is designed to learn modal weight preferences for static and occluded regions, respectively. The dual-branch weight predictor includes a static weight branch and an occlusion weight branch. The static weight branch includes... Classifier ( ), The activation function, wherein the occlusion weight branch includes Classifier ( ), Activation function.

[0093] In the static region, the weights of each modality tend to enhance geometric and static mask features, expressed as:

[0094] in These represent the weights of the geometric consistency distribution map, candidate static mask, secondary candidate occlusion mask, and fused semantic feature map under the static region, respectively. The weights of each modality within the occluded region tend to reinforce the occlusion mask and semantic features, as expressed as:

[0095] in These represent the weights of the geometric consistency distribution map, candidate static mask, secondary candidate occlusion mask, and fused semantic feature map under the occlusion region, respectively. Randomly select secondary candidate occlusion masks Inverting it gives Using it as a region indicator, static weight branches or occlusion weight branches are selected pixel by pixel to achieve pixel-level modal adaptation. From the formula:

[0096]

[0097] Final output fused features .

[0098] Static confidence scoring and mask generation. Based on fusion features. To perform feature dimensionality reduction, a 3×3 depthwise separable convolutional layer is used to reduce the 32 channels to 16 channels, and then a 1×1 convolutional layer is used to compress the 16 channels to 1 channel, resulting in the output. .

[0099] Single-channel scoring feature map By applying the Sigmoid activation function, the feature values ​​are mapped to the [0,1] interval to obtain a pixel-level static confidence map:

[0100] in This represents the Sigmoid activation function. Represents pixels Confidence level for areas belonging to the static region.

[0101] Set confidence threshold Generate a static mask using the following formula:

[0102] In step S5, a static mask is used. The training process of the Mip-NeRF model is constrained to prioritize learning the geometric structure and appearance features of static regions, suppressing interference from occluded areas. After training, the model is used to synthesize and render the target scene from a new perspective, outputting a complete and clear 3D static scene reconstruction result.

[0103] Compared to the traditional Neural Radiation Field (NeRF), Mip-nerf's core improvement lies in modeling pixels as tiny regions, with corresponding rays as cones. By integrating multi-scale features, it captures scene details at different resolutions, ensuring that the rendering results remain clear from both near and far perspectives. Its core function is to learn the volume density and radiance of the scene through neural networks and to achieve image rendering from any perspective based on the integral calculation of cone rays. It has become a mature tool for the static structure reconstruction of transient scenes.

[0104] Training guidance and parameters. The final static mask output by the S4 module. A pixel-level loss function is incorporated into Mip-NeRF. Specifically, in the calculation of ray rendering loss, the loss is calculated for each image pixel. The loss item multiplied by That is, the area that is blocked The loss contribution is suppressed in the static region. The loss function is optimized as a key feature, and its form is:

[0105] in Render colors for Mip-NeRF. The input image contains the true colors. The parameters of the original Mip-NeRF network remain unchanged, and the number of training iterations is set to 20k.

[0106] Reconstruction and Rendering. After training, Mip-NeRF's rendering function performs ray casting and color synthesis on any new perspective of the target scene, ultimately outputting the 3D reconstruction result and the new perspective rendered image.

[0107] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A non-static scene reconstruction method based on multimodal occlusion perception scoring, characterized in that, Includes the following steps: Step 1: Perform 3D reconstruction on the multi-view RGB image to obtain the scene's 3D point cloud and camera pose; generate geometric prior information for the scene's 3D point cloud based on the multi-view matching consistency measure of 2D local feature points in the multi-view RGB image; wherein, the geometric prior information includes a geometric consistency distribution map, static feature points, and a geometric prior mask; Step 2: Use the geometric prior mask as a cue to assist the semantic segmentation model in extracting the semantic mask of the fused semantic feature map and the multi-view RGB image; Step 3: Generate positive point prompts from the static feature points obtained based on 3D reconstruction, and generate negative box prompts from the bounding boxes of the occluded areas extracted by semantic masking. Guide the image segmentation model to output candidate static masks and candidate occlusion masks, and optimize the mask consistency through mesh complementary fusion method to obtain candidate static masks and secondary candidate occlusion masks. Step 4: Using the geometric consistency distribution map, candidate static mask, secondary candidate occlusion mask and fused semantic feature map as multi-source features, perform differential processing and feature enhancement on the multi-source features using the multi-modal occlusion scoring module, and generate a binary static mask through dual-branch weight prediction and region-aware fusion. Step 5: Integrate the binary static mask into the pixel-level loss function of the neural radiation field model. By using the binary static mask, the weight of light rendering error optimization in static areas is enhanced, and the loss contribution of dynamic and occluded areas is suppressed. This drives the training of the neural radiation field model and completes the 3D reconstruction of non-static scenes and the synthesis of new perspectives.

2. The non-static scene reconstruction method based on multimodal occlusion perception scoring according to claim 1, characterized in that, Step 1 includes: Perform 3D reconstruction on multi-view RGB images to extract 2D local feature points of each image, and calculate the multi-view matching consistency metric of each 2D local feature point; The multi-view matching consistency metric of each 2D local feature point is mapped to the pixel coordinates of the corresponding image, and a single-view geometric consistency distribution map is generated by bilinear interpolation. 2D local feature points with a multi-view matching consistency metric ≥ threshold T are selected from the RGB images of each view and used as static feature points of the current image. All static feature points in the same image are mapped to the original RGB image, and a geometric prior mask is generated by optimizing the morphological closing operation.

3. The non-static scene reconstruction method based on multimodal occlusion perception scoring according to claim 1, characterized in that, Step 2 includes: Each view's RGB image is processed by an encoder of a semantic segmentation model to extract multi-scale semantic feature maps; Align the geometric prior mask with the multi-scale semantic feature map, perform pixel-level multiplication fusion, and generate a multi-scale semantic feature map after fusing the geometric prior. The multi-scale semantic feature maps are upsampled to align the resolution and then concatenated to obtain a fused semantic feature map. The semantic segmentation model's decoder performs semantic classification on the fused semantic feature map, maps it to a high-dimensional semantic category space, and then upsamples it back to the image resolution to obtain the semantic mask of the current view's RGB image.

4. The non-static scene reconstruction method based on multimodal occlusion perception scoring according to claim 1, characterized in that, Step 3 includes: Positive cue points are generated by uniformly sampling all static feature points of the RGB image in each view. Boundary boxes of occluded regions are extracted from the semantic mask of the current RGB image and used as negative cue points. Positive cue points and negative cue points are input into the image segmentation model to obtain candidate static masks and candidate occlusion masks. The candidate static mask and the inverse code of the candidate occlusion mask are divided into meshes using the mesh complementary fusion method, and the candidate static mask within the mesh is calculated. Inverse code of candidate occlusion mask Consistency percentage ; Based on consistency percentage Update the inverse codes of all candidate occlusion masks within the grid; merge the updated inverse codes of all candidate occlusion masks to obtain the inverse code of the new candidate occlusion mask; finally, invert the inverse code of the new candidate occlusion mask to obtain the secondary candidate occlusion mask.

5. The non-static scene reconstruction method based on multimodal occlusion perception scoring according to claim 4, characterized in that, The proportion based on consistency The process of updating the inverse of the candidate occlusion mask within all grids is as follows: If the consistency percentage If the grid is consistent, it is considered a uniform grid and no further processing is performed; the original value of the candidate occlusion mask inverse code is retained. If the consistency ratio is... If the result is inconsistent, the pixel values ​​are fused according to weights, and then binarized to obtain the inverse code of the new candidate occlusion mask. The weighted fusion of pixel values ​​is achieved by normalizing the local confidence weights of the inverse codes of the candidate static mask and the candidate occlusion mask.

6. The non-static scene reconstruction method based on multimodal occlusion perception scoring according to claim 1, characterized in that, Step 4 includes: Geometric consistency distribution map, candidate static mask, secondary candidate occlusion mask, and fused semantic feature map are used as multi-source features. The multi-source features are differentiated and then spatial features are extracted using depthwise separable convolution to obtain multi-source enhanced features. The multi-source enhanced features are concatenated and then global average pooling is performed to extract cross-modal global features. A dual-branch weight predictor is used to learn the modal weights for each region in static and occlusion regions respectively. Using the inverted result of the secondary candidate occlusion mask as a region indicator, static weight branch or occlusion weight branch is selected pixel by pixel, and weighted fusion is performed to obtain the fusion feature of each pixel. After dimensionality reduction of the fused features, a pixel-level static confidence map is generated by the Sigmoid function, and then binarized into a final binary static mask.

7. The non-static scene reconstruction method based on multimodal occlusion perception scoring according to claim 1, characterized in that, In step 5, the static mask is incorporated into the pixel-level loss function of the neural radiation field. In Chinese, the formula is as follows: ； in For pixels In the rendering of colors within a neural radiation field, For pixels The true color, Represents pixels The static mask.

8. A non-static scene reconstruction system implementing the method of any one of claims 1-7, characterized in that, The system includes: The image geometric prior information extraction module is responsible for performing 3D reconstruction on multi-view RGB images to obtain the scene's 3D point cloud and camera pose; based on the multi-view matching consistency measure of 2D local feature points of the multi-view RGB images, it generates geometric prior information for the scene's 3D point cloud; wherein, the geometric prior information includes a geometric consistency distribution map, static feature points, and a geometric prior mask; The image semantic mask extraction module is responsible for using the geometric prior mask extracted by the image geometric prior information extraction module as a cue to assist the semantic segmentation model in extracting the semantic mask of the fused semantic feature map and multi-view RGB image; The candidate static mask and secondary candidate occlusion mask extraction module is responsible for generating positive point prompts from the static feature points obtained based on 3D reconstruction, generating negative box prompts by combining the bounding boxes of the occlusion areas extracted by semantic mask, guiding the image segmentation model to output candidate static masks and candidate occlusion masks, and optimizing mask consistency through mesh complementary fusion method to obtain candidate static masks and secondary candidate occlusion masks. The multimodal occlusion scoring module is responsible for using geometric consistency distribution map, candidate static mask, secondary candidate occlusion mask and fused semantic feature map as multi-source features. After performing differential processing and feature enhancement on the multi-source features, it generates a binary static mask through dual-branch weight prediction and region-aware fusion. The scene reconstruction module is responsible for integrating binary static masks into the pixel-level loss function of the neural radiation field model. By using binary static masks, it enhances the weights of light rendering errors in static areas, suppresses the loss contribution of dynamic and occluded areas, drives the training of the neural radiation field model, and completes the 3D reconstruction of non-static scenes and the synthesis of new perspectives.

9. An electronic device, characterized in that, The method includes a processor and a memory, the memory storing machine-executable instructions that can be executed by the processor, the processor executing the machine-executable instructions to implement the method of any one of claims 1-7.

10. A machine-readable storage medium, characterized in that, The machine-readable storage medium stores machine-executable instructions that, when invoked and executed by a processor, cause the processor to implement the method of any one of claims 1-7.

Images