Rainy day multi-view feedforward gaussian reconstruction method and system based on weather factor gating
By using a weather factor-gated multi-view feedforward Gaussian reconstruction method, the instability problem in multi-view reconstruction under rainy weather was solved, and efficient and stable 3D reconstruction and new perspective rendering were achieved under severe weather conditions.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HANGZHOU DIANZI UNIV
- Filing Date
- 2026-04-29
- Publication Date
- 2026-05-29
AI Technical Summary
In existing technologies for multi-view reconstruction in rainy weather, cross-view matching is easily affected by transient degradation interference such as rain streak occlusion, fog scattering and global illumination changes, resulting in unstable 3D reconstruction. Furthermore, scene-by-scene optimization methods have high computational costs and are difficult to converge stably under severe weather conditions, making it difficult to balance reconstruction quality and engineering realism.
A multi-view feedforward Gaussian reconstruction method based on weather factor gating is adopted for rainy days. The method predicts transmittance and rain occlusion probability through a weather factor decomposition network, constructs a pixel-level matching reliability map, and performs cross-view matching and aggregation. It combines a 3D reconstruction network and a differentiable renderer for feedforward prediction and introduces rain domain cyclic consistency constraints for training.
It outputs a clean and stable 3D Gaussian scene representation in a single forward inference, improving the robustness and consistency of 3D reconstruction and new perspective rendering under rain conditions, simplifying the calculation process and improving inference efficiency.
Smart Images
Figure CN122115689A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer vision and 3D reconstruction, specifically relating to a multi-view feedforward Gaussian reconstruction method and system for rainy days based on weather factor gating. Background Technology
[0002] 3D scene reconstruction and novel perspective synthesis are fundamental problems in computer vision and computer graphics, widely serving applications such as robot perception, autonomous driving, mapping and modeling, and immersive content generation. With the development of neural scene representation, especially representation and rendering methods represented by 3D Gaussian splashing, the outstanding performance in reconstruction quality and rendering efficiency has made efficient 3D reconstruction possible under limited viewpoint conditions. However, in real-world outdoor environments, adverse weather conditions such as rain, fog, and overcast skies are common, and image degradation significantly disrupts cross-viewpoint consistency and geometric inference, thus affecting the reliability and practicality of 3D reconstruction.
[0003] Existing technologies mainly fall into two categories: The first is based on scene-by-scene optimization using neural radiation fields or 3D Gaussian optimization methods, which iteratively optimize a single scene to obtain high-quality reconstruction results; some works also attempt to introduce weather modeling during the optimization process to improve robustness under severe weather conditions. The second is a feedforward sparse-view 3D reconstruction method oriented towards efficient inference, which directly predicts the 3D Gaussian representation through cross-view feature matching and aggregation (e.g., Transformer-based cross-view inference, planar scan / cost volume matching, etc.), thus completing reconstruction and rendering without scene-by-scene optimization. Furthermore, for degradation such as rain streaks and raindrops, 2D image deraining techniques are well-developed, and in engineering, a two-stage process of "first 2D deraining, then multi-view reconstruction" is often adopted to improve input quality.
[0004] However, the aforementioned existing technologies still have shortcomings in multi-view reconstruction scenarios under rainy conditions: On the one hand, many feedforward multi-view reconstruction methods typically assume that the input observations are approximately clean, and their cross-view matching is easily affected by transient degradation interference such as rain streaks, fog scattering, and global illumination changes, leading to instability in cost volume construction and geometric inference, which in turn propagates weather artifacts into the 3D representation and new view rendering results. On the other hand, the 2D deraining in the two-stage scheme is usually processed independently on a single viewpoint, with the optimization target biased towards single-frame appearance restoration, making it difficult to guarantee the consistency of light and geometry between multiple views, which may introduce new cross-view inconsistencies, thereby affecting the reliability of subsequent 3D matching and fusion. At the same time, scene-by-scene optimization methods are relatively disadvantageous in terms of inference efficiency and deployment cost, and may also have difficulty distinguishing between stable structures and transient weather degradation under rainy degraded input, making it difficult to balance reconstruction quality and engineering realism requirements. Summary of the Invention
[0005] To address the shortcomings of existing technologies, this invention proposes a weather factor-gated multi-view feedforward Gaussian reconstruction method and system for rainy days. Without performing scene-by-scene iterative optimization, it uses interpretable weather factors to explicitly model rainy day degradation, constructs pixel-level matching reliability, and injects cross-view matching and aggregation, enabling the 3D reconstruction network to output a clean and stable 3D Gaussian scene representation in a single forward inference and support new viewpoint rendering.
[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:
[0007] The first aspect is a multi-view feedforward Gaussian reconstruction method for rainy days based on weather factor gating, which includes the following steps:
[0008] S1. Acquire multiple rainy weather context images from different perspectives and their camera parameters, and determine the target perspective camera parameters;
[0009] S2. Input the rainy weather context image into the weather factor decomposition network to generate predicted weather factors; construct a matching reliability map based on the predicted weather factors to represent the confidence level of cross-view correspondence.
[0010] S3. Extract multi-scale features from the rainy weather context image; generate gated features based on the multi-scale features, matching reliability map, and reliability gating; and generate a fusion representation by aggregating multiple views based on the gated features.
[0011] S4. Input the fused representation into the 3D reconstruction network for feedforward prediction to obtain a 3D Gaussian set; input the 3D Gaussian set into a differentiable renderer and combine it with the target view camera parameters to render a clean predicted image of the target view.
[0012] S5. Obtain clean context images from multiple perspectives and construct rainy day training samples; construct a loss function that includes clean reconstruction supervision, weather factor supervision and rain domain cyclic consistency constraints, and train the weather factor decomposition network and the 3D reconstruction network in combination with the rainy day training samples.
[0013] Preferably, in S2, the predicted weather factors include atmospheric light, rain streaks, transmittance for indicating scattering degradation, and rain shading probability for indicating shading degradation.
[0014] Preferably, in S2, the encoding end of the weather factor decomposition network consists of several concatenated convolutional blocks, with downsampling achieved through convolutions with a stride of 2; the decoding end employs bilinear upsampling and concatenates it with the corresponding scale features from the encoding end to form skip connections; the pixel-level output head uses... Convolution generates pixel-level transmittance, rain occlusion probability, and rain pattern layers; the global output head performs global average pooling on the deepest features and outputs the predicted atmospheric and channel affine parameters after passing through two fully connected layers.
[0015] As a preferred embodiment, in S2, the formula for generating the matching reliability map is as follows:
[0016]
[0017] in, Transmittance, The probability of being blocked by rain. For element-wise multiplication, Indicates the first A rainy day context image, and These represent the image height and width, respectively.
[0018] Preferably, S3 includes:
[0019] S31. For each scale, the matching reliability map is upsampled and aligned to the spatial resolution of the multi-scale features to obtain the gated features;
[0020] S32. Select a rainy weather context view as a reference view; for each candidate depth, project the source view gating features onto the reference view according to the camera parameters and calculate the dot product similarity with the reference gating features to form a similarity volume; aggregate the similarity volumes of multiple source views to obtain a matching representation; encode the matching representation using two-dimensional U-Net to form a fused representation.
[0021] Preferably, S4 includes:
[0022] In reference view pixels The system predicts scale parameters, rotation quaternions, opacity parameters, and color parameters; it generates camera coordinate system rays based on pixel coordinates and camera intrinsic parameters, and obtains the Gaussian center position by combining depth, and then transforms it to the world coordinate system through camera extrinsic parameters; it constructs the covariance from the scale parameters and rotation quaternions and maps it to the world coordinate system.
[0023] Preferably, S5 includes generating rainy day training samples from clean context images through a controlled degradation process, and simultaneously obtaining weather factor labels; the formula for the controlled degradation process is as follows:
[0024]
[0025] in, For element-wise multiplication, For the global overcast tone mapping operator, the first row of the above formula is used to form the global overcast appearance, the second row is used to describe the effect of fogging scattering on contrast and brightness, and the third row is used to describe the effect of rain pattern overlay on transient texture; transmittance label Rain occlusion probability label Together, they determine the reliability of subsequent cross-perspective matching. For a clean context image, For luminescent labels, For rain pattern layer labels.
[0026] As a preferred option, the loss function in S5 is:
[0027]
[0028] in, and These are the weighting coefficients; For clean reconstruction supervision, clean predicted images are used to supervise the target viewpoint. With clean truth image Consistency between them; For weather factor monitoring, it is used to monitor the consistency between predicted weather factors and weather factor labels; Rain domain cyclic consistency constraints are used to ensure the clean context image is rendered. Differentiable regenerative weathering operator Map back to the rain region and with the rainy day context image Calculate consistency constraints.
[0029] Preferably, the rain domain cycle consistency constraint includes performing channel affine correction on the rendered clean context view for each rainy day context view, then combining the transmittance and atmospheric light synthesis fogging scattering results, and finally combining the rain texture layer to perform pixel-by-pixel screen synthesis to obtain a mapped rainy day image; and then combining the mapped rainy day image with the rainy day context view. Calculate the consistency loss.
[0030] Secondly, a multi-view feedforward Gaussian reconstruction system for rainy days based on weather factor gating includes:
[0031] The data acquisition module is used to acquire multiple rainy weather context images from different perspectives and their camera parameters, and to determine the target perspective camera parameters.
[0032] The weather factor decomposition module is used to input the rainy weather context image into the weather factor decomposition network to generate predicted weather factors; and to construct a matching reliability map based on the predicted weather factors to represent the confidence level of cross-view correspondence.
[0033] The image feature encoding module extracts multi-scale features from rainy weather context images;
[0034] The reliable gating module is used to generate gating features based on multi-scale features, matching reliability maps, and reliability gating.
[0035] The cross-view matching and aggregation module is used to generate fused representations by aggregating multiple views based on gated features.
[0036] The 3D Gaussian representation prediction module is used to input the fused representation into the 3D reconstruction network for feedforward prediction to obtain a 3D Gaussian set;
[0037] The differentiable rendering module is used to input a 3D Gaussian set into the differentiable renderer and render a clean predicted image of the target viewpoint by combining the target viewpoint camera parameters.
[0038] The training module is used to acquire clean context images from multiple perspectives and construct rainy day training samples; it constructs a loss function that includes clean reconstruction supervision, weather factor supervision and rain domain cycle consistency constraints, and trains the weather factor decomposition network and the 3D reconstruction network in combination with the rainy day training samples;
[0039] The weather factor-gated multi-view feedforward Gaussian reconstruction system for rainy days is used to implement the weather factor-gated multi-view feedforward Gaussian reconstruction method and its steps for rainy days as described in the first aspect.
[0040] Compared with the prior art, the beneficial effects of the present invention are reflected in:
[0041] 1. Unlike traditional multi-view reconstruction techniques that rely solely on cross-view matching based on rainy weather observations, depend on photometric consistency, and struggle to handle rain streaks and occlusion degradation, this invention employs a "decomposition of interpretable weather factors + matching reliability gating" technique, predicting transmittance from weather factors. Probability of rain obstruction And so on, and based on this, construct a pixel-level reliability map. Furthermore, gating of multi-scale features and participation in cross-view aggregation explicitly suppresses the interference of low-reliability regions such as strong rain patterns and heavy occlusion on matching and aggregation, making the cross-view correspondence more stable. This reduces reconstruction drift and rendering ghosting caused by mismatches, and improves the robustness and consistency of 3D reconstruction and new view rendering under rainy conditions.
[0042] 2. Unlike traditional 3D reconstruction techniques that rely on scene-by-scene iterative optimization, incur high computational costs, and struggle to achieve stable convergence under adverse weather conditions, this invention employs a "feedforward 3D Gaussian ensemble prediction + differentiable rendering" approach. This approach directly predicts the 3D Gaussian ensemble from cross-view fusion representations. and through a differentiable renderer It outputs target view images, enabling the representation of 3D scenes and the generation of new perspectives without scene-by-scene iteration. The calculation process is simpler, the reasoning efficiency is higher, and continuous and stable 3D representation and rendering results can still be obtained even when degradation occurs in rainy weather.
[0043] 3. Unlike traditional technical solutions that only supervise the observation domain and lack decoupling constraints on "stable scene components" and "weather degradation components," resulting in insufficient generalization, this invention introduces rain domain cyclic consistency constraints and weather factor supervision during the training phase. This allows the weather factor decomposition network and the 3D reconstruction network to mutually correct each other within the same optimization framework, promoting the consistent separation of "stable scene components explained by 3D Gaussian and transient weather components explained by weather factors." This enhances the generalization ability and rendering credibility on real rainy day data.
[0044] 4. By constructing pixel-level matching reliability using transmittance and occlusion probability obtained from weather factor decomposition, and gating cross-view features with reliability, the interference of rain ripple occlusion and scattering degradation on matching and aggregation is suppressed from the source, making cross-view correspondence estimation more stable, thereby significantly reducing the risk of rain and fog artifacts propagating to 3D representation and rendering results. At the same time, by mapping clean rendering results back to the rain domain through differentiable re-weathering and aligning them with the closed loop of observed rain map, the interpretable separation of "stable scene components - transient weather components" is strengthened in end-to-end training, alleviating degradation caused by relying solely on clean reconstruction supervision and improving geometric and appearance consistency. Therefore, the method of this invention can output a renderable clean 3D Gaussian scene representation in a single forward inference, and is suitable for real-time 3D perception and new perspective rendering tasks under severe weather conditions. Attached Figure Description
[0045] Figure 1 This is a flowchart of the method in Embodiment 1 of the present invention;
[0046] Figure 2 This is a network architecture diagram of Embodiment 2 of the present invention;
[0047] Figure 3 This describes the process of synthesizing a rainy day dataset in Embodiment 1 of the present invention;
[0048] Figure 4 This is a diagram illustrating the rain removal effect on a synthetic rainy day dataset constructed based on RealEstate10K (RE10K) and ACID datasets, according to Embodiment 1 of the present invention.
[0049] Figure 5 This is a diagram illustrating the rain removal effect of Embodiment 1 of the present invention on a real rainy day dataset. Detailed Implementation
[0050] To make the technical means, inventive features, objectives, and effects of the invention readily understandable, the invention is further described below with reference to specific illustrations. However, the invention is not limited to the embodiments described below.
[0051] It should be noted that the structures, proportions, sizes, etc., illustrated in the accompanying drawings of this specification are only used to complement the content disclosed in the specification for those skilled in the art to understand and read, and are not intended to limit the conditions under which the present invention can be implemented. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in the proportions, or adjustments to the size, without affecting the effects and objectives that the present invention can produce, should still fall within the scope of the technical content disclosed in the present invention.
[0052] This method employs a multi-step process. First, it acquires the rainy weather context view and camera parameters. During the training phase, it constructs controllable rain and fog degradation for the context view and generates weather factor supervision signals. Then, it predicts weather factors such as transmittance, atmospheric light, rain streaks, and occlusion probability using a weather factor decomposition network. Next, it calculates pixel-level matching reliability using the predicted factors, gates the cross-view feature / cost construction, and completes matching aggregation. Finally, it predicts a 3D Gaussian ensemble using feedforward and performs differentiable rendering. Joint optimization is achieved using clean reconstruction supervision, weather factor supervision, and rain domain cyclic consistency constraints. This ensures that stable scene components are explained by the 3D Gaussian ensemble, while transient weather components are explained by weather factors, thereby improving the robustness and consistency of multi-view reconstruction in rainy weather. The invention will be further described in detail below with reference to the accompanying drawings.
[0053] Example 1:
[0054] like Figures 1 to 2 The method for multi-view feedforward Gaussian reconstruction in rainy weather based on weather factor gating, as shown, includes the following steps:
[0055] S1. Acquire multiple rainy weather context images from different perspectives and their camera parameters, and determine the target perspective camera parameters;
[0056] Acquire at least two rainy context images of the same scene and their camera parameters, and determine the target viewpoint camera parameters. Let the number of rainy context images be... ,and ;use Indicates the first A context view. The rainy weather context image is denoted as... ,in , and These represent the image height and width, respectively; the corresponding camera intrinsic and extrinsic parameters are respectively... and ,in For camera intrinsic parameter matrix, For camera extrinsic parameters / pose. Target view camera parameters are denoted as... In qualitative evaluations of real-world rain data, when precise camera parameters are lacking, approximate values can be obtained using camera parameter estimation methods. To complete the rendering and display.
[0057] S2. Input the rainy weather context image into the weather factor decomposition network to generate predicted weather factors; construct a matching reliability map based on the predicted weather factors to represent the confidence level of cross-view correspondence.
[0058] like Figure 2 As shown, each rainy day context image Input the Weather Factor Decomposition Network WeatherNet, and denote the network as... The output includes at least the transmittance. , gas light Rain veils and the probability of rain obstruction One or more of the following weather forecasting factors:
[0059]
[0060] in and The pixel-level spatial distribution indicates scattering degradation and occlusion degradation, respectively. and Used to depict the overall appearance of atmospheric light and rain patterns; and These are the channel affine parameters used for rain domain cycle consistency constraints during the training phase (they may not be explicitly output during the inference phase).
[0061] Weather Factor Decomposition Network It employs a lightweight encoder / decoder architecture. The encoder consists of several concatenated convolutional blocks, with downsampling achieved through convolutions with a stride of 2; each convolutional block preferably consists of two layers. The algorithm consists of convolution, group normalization (GroupNorm), and SiLU activation. The decoder employs bilinear upsampling and concatenates it with corresponding scale features from the encoder to form skip connections, thus balancing global degradation estimation with local detail preservation. The pixel-level output header uses... Convolution produces pixel-level factors , and and through Sigmoid , Constrained to The global output head performs global average pooling on the deepest features and outputs global parameters after passing through two fully connected layers. and channel affine parameters , ,in Constrained by Sigmoid The channel affine parameters can be obtained through hyperbolic tangent. Limit its magnitude to ensure numerical stability of subsequent re-weathering processes.
[0062] Constructing a pixel-level matching reliability map based on predicted weather factors This is used to quantify the confidence level corresponding to different viewpoints and serves as the control signal for subsequent gating. The matching reliability map is jointly determined by scattering degradation and occlusion degradation, and is generated according to the following formula:
[0063]
[0064] The lower the reliability, the less suitable the pixel is for cross-view matching; conversely, the higher the reliability, the more likely the pixel is to provide a stable response.
[0065] S3. Extract multi-scale features from the rainy weather context image; generate gated features based on the multi-scale features, matching reliability map, and reliability gating; and generate a fusion representation by aggregating multiple views based on the gated features.
[0066] S31. Normalize the rainy weather context image and input it into the image encoder. Multi-scale features are extracted and reliability gating is introduced before cross-view matching to obtain gated features. For each rainy-day context image... extract Individual scale features ,in ;set up To upsample and align the matching reliability graph to the first... The scale-feature resolution operator aligns the matching reliability map with multi-scale features and performs matching reliability gating to suppress feature contributions from low-reliability regions: for each scale ,Will Upsample and align to Spatial resolution, to obtain gating features It can be represented as:
[0067]
[0068] S32. Select a rainy weather context image as a reference view (e.g., ), and in Deep Candidates Based on camera parameters Gating features of the source view (No. The gated features (from each viewpoint) are reprojected and aligned to the reference view, and then aligned with the reference gated features. Similarity is calculated for gated features from the reference viewpoint, and these similarities are aggregated across multiple views to form a matching representation. This results in a fused representation that simultaneously encodes consistent appearance and geometric constraint cues from multiple viewpoints, providing input for 3D representation prediction. Since the matching input comes from gated features, the correspondence is less affected by transient degradation, which can improve the stability and detail fidelity of subsequent 3D representation prediction.
[0069] Cross-view matching representation is driven by depth candidates, preferably selecting one of the context views as the reference view. For each candidate depth... Gating features of the source view based on camera geometry Projected onto the reference view and compared with the reference feature The dot product similarity is calculated to form a similarity volume. The similarity volumes from multiple sources are aggregated (e.g., averaged) to obtain a multi-view consistent matching representation. This representation is then encoded using a 2D U-Net to form a fused feature representation that simultaneously includes multi-view consistent appearance cues and geometric constraint cues. Subsequently, a softmax operation is performed on the depth candidate dimension to obtain the probability distribution. And to obtain the desired coarse depth (Or coarse parallax), and then refined by combining the context image and density cues to obtain the final depth and density. The final depth and density are the prediction values decoded from the fused representation through the output head.
[0070] S4. Input the fused representation into the feedforward 3D reconstruction network for feedforward prediction to obtain a 3D Gaussian set representation; input the 3D Gaussian set into a differentiable renderer and combine it with the target view camera parameters to render a clean predicted image of the target view.
[0071] 3D Gaussian set It consists of several three-dimensional Gaussian units, each of which includes at least the spatial position, shape parameters (such as scale and rotation), color parameters, and opacity parameters used for rendering. It is used to interpret the stable scene structure and appearance and support subsequent rendering of new perspectives.
[0072] S41, Three-dimensional Gaussian set It is obtained through two stages: original Gaussian parameter prediction and geometric adaptation. (In the reference view pixels) Predicted scale parameters Rotation Quaternions Opacity and color parameters The scale component is limited to a preset range through monotonic mapping and can adaptively scale with depth. The rotation quaternion is normalized to obtain the effective rotation. The geometric adaptation module uses pixel coordinates and camera intrinsics. Generate camera coordinate system rays and combine them with depth. The location of the Gaussian center is obtained, and then the extrinsic parameters are used. Transform to the world coordinate system; construct the covariance from scale and rotation and map it to the world coordinate system; opacity is determined by... This is obtained through monotonic mapping, resulting in a renderable 3D Gaussian set. .
[0073] S42, 3D Gaussian set Input Differentiable Renderer Combined with target view camera parameters The resulting clean predicted image from the target viewpoint is rendered as follows:
[0074]
[0075] S5. Obtain clean multi-view images to construct rainy day training samples; construct a loss function that includes clean reconstruction supervision, weather factor supervision and rain domain cycle consistency constraints, and train the weather factor decomposition network and the 3D reconstruction network in combination with the rainy day training samples.
[0076] S51. In the training process, in order to obtain rainy day supervision data and form interpretable factor constraints, synthetic rainy day training samples are constructed from clean multi-view images.
[0077] Let the clean context image be Rainy day context images are generated through a controlled degradation process. And simultaneously obtain weather factor tags. .in For transmittance labeling, For luminescent labels, For rain-textured labels, A rain occlusion probability label. The degradation process is constructed according to the following formula (symbols...). For element-wise multiplication, (Global overcast tone mapping operator)
[0078]
[0079] The first row of the above formula is used to form the global overcast appearance, the second row is used to describe the effect of haze on contrast and brightness, and the third row is used to describe the effect of rain overlay on transient textures. and Together, they determine the reliability of subsequent cross-view matching and are used to supervise the weather factor decomposition module, thereby allowing for simultaneous constraints on the interpretation of "scene intrinsic appearance" and "transient weather degradation" during training.
[0080] During the training phase, at least one clean ground truth image of the target viewpoint is also acquired. and its camera parameters Used to monitor the rendering results.
[0081] S52. In the training implementation, to ensure that weather factor decomposition, reliability gating, and 3D Gaussian reconstruction form a consistent end-to-end optimization closed loop, the weather factor decomposition network is optimized. The network is jointly trained with a 3D reconstruction network. The training loss consists of clean reconstruction supervision, weather factor supervision, and rainwater cycle consistency constraints, and its overall loss function is preferably written as follows:
[0082]
[0083] in and These are the weighting coefficients; Used to supervise the rendering results of the target viewpoint. With clean truth image Consistency between them; Used to monitor weather factor decomposition output and Figure 3 The consistency between the synthetic tags shown ensures that the source of reliability is explainable and calibrable. Used to render a clean context view Differentiable regenerative weathering operator Mapping back to the rain domain yields and with observed rain maps The consistency constraints are computed to suppress degradation and enhance consistency by simply smoothing out reconstruction errors.
[0084] Rainwater cyclic consistency constraints are achieved through differentiable re-weathering operators. Build. For each context view First, it will be composed of three-dimensional Gaussian sets. The rendered clean context view Perform affine channel correction (by , control):
[0085]
[0086] in, This means cropping the value to between 0 and 1.
[0087] Combined with transmittance With light Synthetic fog scattering results:
[0088]
[0089] Finally, combine the rain streak layer. A pixel-by-pixel screen composite is performed to obtain a mapped rainy day image:
[0090]
[0091] And Images of rainy days Calculate the consistency loss to reinforce the idea that "stable scene components are composed of a three-dimensional Gaussian set". The separation relationship between "explanation of transient weather degradation and explanation of weather factors" is used to improve the stability of cross-view matching and rendering under rainy conditions.
[0092] This invention can employ a two-view training setup, where each forward prediction and backward update is performed using two contextual rainy day views as input. Training can utilize an adaptive moment estimation optimizer with a fixed batch size for iterative optimization. The rainy day observations used in the training samples are all... Figure 3 The synthesis process shown is employed to ensure that the supervisory signal is available and interpretable. The model is trained and optimized using a loss function, and its performance is tested on a test set.
[0093] The reasoning phase only requires completing 3D Gaussian reconstruction and rendering from a new perspective.
[0094] To test the performance of the method of the present invention, qualitative and quantitative experiments were conducted on synthetic rainy day datasets and real rainy day datasets.
[0095] Qualitative experimental results as follows Figure 4 and Figure 5 As shown. Figure 4 This invention demonstrates its ability to effectively suppress rain streaks and fog artifacts on a synthetic rainy day dataset constructed based on the RE10K and ACID datasets. Compared to other classic feedforward 3D Gaussian reconstruction methods, this invention better preserves structural edges and texture details in new perspective rendering. Furthermore, as... Figure 5 As shown, in real rainy day datasets, despite the lack of clean ground truth and accurate camera parameters, this invention can still output clearer visibility and more stable detail recovery results.
[0096] In terms of quantitative analysis, Table 1 presents the overall quantitative comparison results under rainy weather degradation conditions. For the synthetic rainy weather test set of RE10K and ACID, the present invention outperforms the compared feedforward baseline method in all three indices: PSNR, SSIM, and LPIPS.
[0097] Table 1: Performance comparison of RE10K and ACID synthetic rainy day datasets (the best result is shown in this paper).
[0098]
[0099] Meanwhile, under the Mip-NeRF360 synthetic rain setting shown in Table 2, the present invention achieves higher overall performance compared to scene-by-scene optimization methods. Specifically, CSUD is a novel and efficient 2D image deraining model, which can serve as a typical example of a two-stage deraining framework.
[0100] Table 2: Performance comparison of the Mip-NeRF360 synthetic rainy day dataset (the best result is shown in this paper).
[0101]
[0102] In addition, Table 3 presents the ablation results under the RealEstate10K synthetic rainy weather setting, which is used to analyze the interaction between various technical features.
[0103] Table 3: Ablation experiments of matching reliability gating and rain domain cyclic consistency regularization constraint modules (including whether the method of this invention was used; using the method of this invention is the optimal result).
[0104]
[0105] As can be seen from the ablation experiments in the table, the performance of all indicators is improved after introducing a matched reliability gating on the MVSplat baseline; the performance is further improved after adding rain domain cyclic consistency regularization constraints, indicating that the reliability gating and regularization constraints are complementary and help to obtain more stable consistency reconstruction under rain degradation conditions.
[0106] Example 2:
[0107] like Figure 3 As shown, this invention provides a multi-view feedforward Gaussian reconstruction system for rainy weather based on weather factor gating. In this embodiment, the method is collaboratively composed of a weather factor decomposition module, an image feature encoding module, a reliable gating module, a cross-view matching and aggregation module, a 3D Gaussian representation prediction module, and a differentiable rendering module. Specifically, the weather factor decomposition module estimates degradation-related interpretability information from rainy images and generates matching reliability; the reliable gating module suppresses feature contributions from low-reliability regions before cross-view matching; cross-view matching performs robust multi-view information fusion under camera geometric constraints; the 3D Gaussian scene representation prediction module generates a clean, directly renderable 3D Gaussian set; and the differentiable rendering module outputs a clean new perspective image result for the target perspective. These technical features form a closed loop through data flow and constraint relationships, thereby improving reconstruction stability and rendering quality under rainy conditions.
Claims
1. A multi-view feedforward Gaussian reconstruction method for rainy days based on weather factor gating, characterized in that, Includes the following steps: S1. Acquire multiple rainy weather context images from different perspectives and their camera parameters, and determine the target perspective camera parameters; S2. Input the rainy weather context image into the weather factor decomposition network to generate predicted weather factors; construct a matching reliability map based on the predicted weather factors to represent the confidence level of cross-view correspondence. S3. Extract multi-scale features from the rainy weather context image; generate gated features based on the multi-scale features, matching reliability map, and reliability gating; and generate a fusion representation by aggregating multiple views based on the gated features. S4. Input the fused representation into the 3D reconstruction network for feedforward prediction to obtain a 3D Gaussian set; input the 3D Gaussian set into a differentiable renderer and combine it with the target view camera parameters to render a clean predicted image of the target view. S5. Obtain clean context images from multiple perspectives and construct rainy day training samples; construct a loss function that includes clean reconstruction supervision, weather factor supervision and rain domain cyclic consistency constraints, and train the weather factor decomposition network and the 3D reconstruction network in combination with the rainy day training samples.
2. The rainy day multi-view feedforward Gaussian reconstruction method based on weather factor gating according to claim 1, characterized in that, In S2, the predicted weather factors include atmospheric light, rain streaks, transmittance used to indicate scattering degradation, and rain shading probability used to indicate shading degradation.
3. The rainy day multi-view feedforward Gaussian reconstruction method based on weather factor gating according to claim 2, characterized in that, In S2, the encoding end of the weather factor decomposition network consists of several concatenated convolutional blocks, which are downsampled using convolutions with a stride of 2. The decoding end uses bilinear upsampling and is concatenated with the corresponding scale features from the encoding end to form skip connections. The pixel-level output head uses... Convolution generates pixel-level transmittance, rain occlusion probability, and rain pattern layers; the global output head performs global average pooling on the deepest features and outputs the predicted atmospheric and channel affine parameters after passing through two fully connected layers.
4. The rainy day multi-view feedforward Gaussian reconstruction method based on weather factor gating according to claim 2, characterized in that, In S2, the formula for generating the matching reliability map is as follows: in, Transmittance, The probability of being blocked by rain. For element-wise multiplication, Indicates the first A rainy day context image, and These represent the image height and width, respectively.
5. The rainy day multi-view feedforward Gaussian reconstruction method based on weather factor gating according to claim 1, characterized in that, S3 include: S31. For each scale, the matching reliability map is upsampled and aligned to the spatial resolution of the multi-scale features to obtain the gated features; S32. Select a rainy weather context view as a reference view; for each candidate depth, project the source view gating features onto the reference view according to the camera parameters and calculate the dot product similarity with the reference gating features to form a similarity volume; aggregate the similarity volumes of multiple source views to obtain a matching representation; encode the matching representation using two-dimensional U-Net to form a fused representation.
6. The rainy day multi-view feedforward Gaussian reconstruction method based on weather factor gating according to claim 5, characterized in that, S4 include: In reference view pixels The system predicts scale parameters, rotation quaternions, opacity parameters, and color parameters; it generates camera coordinate system rays based on pixel coordinates and camera intrinsic parameters, and obtains the Gaussian center position by combining depth, and then transforms it to the world coordinate system through camera extrinsic parameters; it constructs the covariance from the scale parameters and rotation quaternions and maps it to the world coordinate system.
7. The multi-view feedforward Gaussian reconstruction method for rainy days based on weather factor gating according to claim 1, characterized in that, S5 involves generating rainy day training samples from clean context images through a controlled degradation process, and simultaneously obtaining weather factor labels; the formula for the controlled degradation process is as follows: in, For element-wise multiplication, For the global overcast tone mapping operator, the first row of the above formula is used to form the global overcast appearance, the second row is used to describe the effect of fogging scattering on contrast and brightness, and the third row is used to describe the effect of rain pattern overlay on transient texture; transmittance label Rain occlusion probability label Together, they determine the reliability of subsequent cross-perspective matching. For a clean context image, For luminescent labels, For rain pattern layer labels.
8. The multi-view feedforward Gaussian reconstruction method for rainy days based on weather factor gating according to claim 1, characterized in that, The loss function in S5 is: in, and These are the weighting coefficients; For clean reconstruction supervision, clean predicted images are used to supervise the target viewpoint. With clean truth image Consistency between them; For weather factor monitoring, it is used to monitor the consistency between predicted weather factors and weather factor labels; Rain domain cyclic consistency constraints are used to ensure the clean context image is rendered. Differentiable regenerative weathering operator Map back to the rain region and with the rainy day context image Calculate consistency constraints.
9. The rainy day multi-view feedforward Gaussian reconstruction method based on weather factor gating according to claim 8, characterized in that, Rainfield cycle consistency constraints include performing channel affine correction on each rainy day context view, combining the rendered clean context view with the transmittance and atmospheric light synthesis fog scattering results, and finally combining the rain texture layer to perform pixel-by-pixel screen synthesis to obtain the mapped rainy day image. Map the rainy image to the rainy context view Calculate the consistency loss.
10. A multi-view feedforward Gaussian reconstruction system for rainy days based on weather factor gating, characterized in that, include: The data acquisition module is used to acquire multiple rainy weather context images from different perspectives and their camera parameters, and to determine the target perspective camera parameters. The weather factor decomposition module is used to input the rainy weather context image into the weather factor decomposition network to generate predicted weather factors; and to construct a matching reliability map based on the predicted weather factors to represent the confidence level of cross-view correspondence. The image feature encoding module extracts multi-scale features from rainy weather context images; The reliable gating module is used to generate gating features based on multi-scale features, matching reliability maps, and reliability gating. The cross-view matching and aggregation module is used to generate fused representations by aggregating multiple views based on gated features. The 3D Gaussian representation prediction module is used to input the fused representation into the 3D reconstruction network for feedforward prediction to obtain a 3D Gaussian set; The differentiable rendering module is used to input a 3D Gaussian set into the differentiable renderer and render a clean predicted image of the target viewpoint by combining the target viewpoint camera parameters. The training module is used to acquire clean context images from multiple perspectives and construct rainy day training samples; it constructs a loss function that includes clean reconstruction supervision, weather factor supervision and rain domain cycle consistency constraints, and trains the weather factor decomposition network and the 3D reconstruction network in combination with the rainy day training samples; The weather factor-gated multi-view feedforward Gaussian reconstruction system for rainy days is used to implement the weather factor-gated multi-view feedforward Gaussian reconstruction method and its steps as described in claim 1.