A dual-view de-reflection method based on frame-aware spatial attention and camera pose
By employing a dual-view dereflection method based on frame-aware spatial attention and camera pose, the problem of under-constrained image reflection is solved, achieving efficient removal of reflections while maintaining a realistic background on mobile devices. This method is applicable to scenarios such as mobile photography and automotive vision.
Patent Information
- Application Number
- CN202611254482.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-18
- Publication Date
- 2026-11-17
AI Technical Summary
Existing technologies are under-constrained in removing image reflections, causing reflected content to be mistaken for the real background, resulting in problems such as reflection residue, color drift, and texture loss. Furthermore, existing methods require additional hardware or increase the user's workload.
We employ a dual-view dereflection method based on frame-aware spatial attention and camera pose. By implicitly converging multiple correspondences between viewpoints through frame-aware spatial attention, we utilize relative camera pose as a soft condition to inject into the diffusion network, thereby avoiding the vulnerability of explicit optical flow estimation and making full use of dual-view geometric cues.
It achieves stable removal of complex reflections while maintaining realistic background content without increasing user workload or computational overhead, improving the accuracy of transmissive layer restoration. It is suitable for ordinary mobile terminals and compatible with simultaneous dual-camera, sequential dual-shot, and single-image rollback modes.
Smart Images

Figure CN123118397A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing technology, and in particular to a dual-view dereflection method based on frame-aware spatial attention and camera pose. Background Technology
[0002] In scenarios such as museum display cases, vehicle windshields, building glass curtain walls, and shop windows, the images received by the camera are typically composed of a superposition of the transmissive layer of the target scene and the reflective layer formed by the glass surface. The goal of image dereflection is to recover a clear and clean transmissive layer from a mixed image containing reflections. For a single image, any mixed image may correspond to multiple combinations of transmissive and reflective layers; therefore, single-image dereflection is essentially an underconstrained problem. When the reflection intensity is high, the structure is complex, or there is local overexposure, existing single-image dereflection methods are prone to mistaking reflected content for the real background or incorrectly deleting textures and structures from the real background, leading to problems such as reflection residue, color shift, and texture loss.
[0003] To mitigate the underconstraints of single-image dereflection, existing techniques typically introduce additional information to aid in distinguishing between the transmissive and reflective layers. For example, some methods employ polarizers, near-infrared imaging devices, and flash / non-flash image pairs to obtain additional physical cues; however, these methods require specialized hardware or additional shooting operations, limiting their application on ordinary mobile devices. Other methods involve capturing multiple images continuously with a moving camera, utilizing motion differences between different viewpoints to separate the transmissive and reflective layers. However, multi-frame shooting increases user workload and computational overhead, and usually requires the scene to remain stationary during the shooting process. With the proliferation of dual cameras in smartphones, automotive cameras, and other mobile devices, users can simultaneously capture two images of the same scene from different viewpoints with a single shutter release, or obtain two images with a certain parallax by briefly and slightly moving the camera. Since the real transmissive scene and the virtual image reflected in the glass are typically at different depths, they produce different parallaxes between the two viewpoints. Therefore, dual-view information can provide effective cues for separating the transmissive and reflective layers, and is more convenient than long-sequence shooting.
[0004] Existing dual-view dereflection methods (such as Simon Niklaus, Xuaner Cecilia Zhang, Jonathan T. Barron, et al. Learned Dual-View Reflection Removal. WACV, 2021: 3713–3722) typically first estimate the explicit optical flow between the two images, then deform and align one of the images based on the optical flow, and finally use the aligned image to synthesize the transmission layer (e.g., ...). Figure 1(As shown). However, in reflective regions, a single pixel often contains both the motion of the transmissive layer and the motion of the reflective layer, meaning the same pixel may correspond to two different motion modes, while conventional optical flow can usually only represent one motion. Therefore, in regions with strong reflections, occlusion, repetitive textures, or large parallax, explicit optical flow is prone to estimation errors and propagates alignment errors to subsequent transmissive layer reconstruction processes.
[0005] On the other hand, existing single-image diffusion dereflection methods (Jichen Hu, Chen Yang, Zanwei Zhou, et al. Dereflection Any Image with Diffusion Priors and Diversified Data.arXiv:2503.17347, 2025, implementation methods are as follows) Figure 2 As shown in the figure, although it is possible to utilize the image prior of the pre-trained generative model, it does not fully utilize the complementary observations, inter-layer parallax, and camera motion information provided by the second viewpoint.
[0006] Furthermore, conventional multi-view diffusion attention allows bidirectional interaction between two noisy target features, which can easily cause diffusion noise from different viewpoints to propagate to each other; if a three-dimensional network layer is directly introduced, it will change the feature structure of the pre-trained two-dimensional generative model, making it difficult to fully reuse the pre-trained weights. Summary of the Invention
[0007] This invention addresses the problems of existing technologies by proposing a dual-view dereflection method based on frame-aware spatial attention and camera pose. By implicitly converging multiple possible correspondences between two viewpoints through frame-aware spatial attention, and injecting relative camera pose as a soft condition into the diffusion network, this method avoids the vulnerability of explicit optical flow estimation while making full use of dual-view geometric cues to achieve stable removal of complex reflections and effective preservation of the real background content.
[0008] To achieve the above objectives, the present invention provides the following technical solution:
[0009] This invention provides a dual-view reflection removal method based on frame-aware spatial attention and camera pose, comprising the following steps:
[0010] Step S1, Dual Views and Pose Acquisition:
[0011] Obtain the first reflection-containing image of the same reflection-containing scene. Second image containing reflection And the camera poses corresponding to the two images. , ;
[0012] Step S2, Construction of joint latent variables:
[0013] The clean conditional latent variables of two images containing reflections and the two noisy transmission latent variables are combined into a joint input by channel, and the transmission layers of the two viewpoints are output synchronously by the same denoising network.
[0014] Step S3, Frame-aware spatial attention denoising:
[0015] Noisy transmission features generate only queries, while clean reflection-containing conditional features from two viewpoints generate only keys and values, thereby enabling cross-view information exchange and blocking the mutual propagation of noisy features.
[0016] Step S4, Soft Condition Injection of Pose:
[0017] A 7-dimensional pose is formed by normalizing the relative translation direction and the relative quaternion rotation. After being encoded by MLP, the pose is superimposed according to the scaling factor and embedded at the diffusion time to obtain the global soft condition.
[0018] Step S5, Decoding the dual-view transmission layer:
[0019] The noise at two viewpoints is iteratively predicted under global soft conditions. Two clean transmission latent variables are gradually obtained through a preset diffusion sampler. The transmission layer to be recovered is then output by a frozen or finely tuned VAE decoder. , .
[0020] Furthermore, in step S1, the dual views are acquired synchronously by the two cameras of the mobile terminal, or sequentially by the same camera after it moves over a period of time; the camera pose is given by the device IMU, gyroscope, visual inertial odometry or camera calibration results.
[0021] Furthermore, in step S1, for the first... The hybrid imaging model with reflected images from multiple viewpoints is as follows:
[0022] ;
[0023] in, The transmission layer to be restored. The reflective layer to be suppressed; the addition operation is performed in an approximately linear light intensity space.
[0024] Furthermore, in step S1, for the image that has already undergone tone mapping, its nonlinear mixing relationship is learned end-to-end through the network.
[0025] Furthermore, in step S2, a frozen VAE encoder is used. The two images containing reflections and the corresponding ground truth values from the training phase are mapped to the latent space, and the clean conditional latent variables are obtained from the images containing reflections. , Transmission is truly worthwhile. , When the input image resolution is H×W, the dimensions of each latent variable are 4×h×w, where h=H / 8 and w=W / 8. Adding diffusion noise to the transmission latent variable yields... , And obtained by connecting according to the channel:
[0026] ,
[0027] ,
[0028] ;
[0029] The U-Net in SD v2.1 is expanded from 4 input channels in the first layer to 16 and from 4 output channels in the last layer to 8, enabling a network to predict noise from two viewpoints simultaneously in one forward pass.
[0030] Furthermore, in step S3, the noisy transmission query part With clean, reflective conditions Represented as:
[0031] ;
[0032] ;
[0033] , Rearranged into spatial sequences , Represented as:
[0034] ;
[0035] .
[0036] Furthermore, in step S3, the intermediate features are sampled in the 16×16 and 32×32 upsampling layers of the U-Net. Split into noisy transmission query part With clean, reflective conditions And rearranged into spatial sequences respectively. , ,Will and Connect along the sequence dimension and use query, key, and value projection matrices pre-trained in SD v2.1. , , get , , Subsequently, query Q is extracted only from the noisy transmission position, and key K and value V are extracted only from the clean condition position.
[0037] The query Q, key K, and value V are represented as follows:
[0038] ,
[0039] ,
[0040] ;
[0041] Will and The sequential dimension connection is represented as:
[0042] .
[0043] Further, in step S4, each camera pose is represented as... ,in For translation, The relative pose is calculated based on two viewpoints using unit quaternion rotations. The 7-dimensional relative pose is then input into a two-layer MLP with a hidden dimension of 256 to obtain the pose embedding. and with scaling factor Embedded with diffusion time Adding them together yields the global soft condition. ;
[0044] The relative pose calculated from two viewpoints is represented as follows:
[0045]
[0046] Pose embedding Represented as:
[0047] ;
[0048] Global soft conditions Represented as:
[0049] ;
[0050] .
[0051] Furthermore, the model training employs a two-stage course, progressing from easy to difficult. The first stage uses 768×768 semi-synthetic dual-view data, with reflection intensity... With a 15% probability, use visual Figure 1 Replacement Figure 2 Additionally, there is a 15% probability of using visual... Figure 2 Replacement Figure 1The relative pose is set to zero; in the remaining 70% of the real dual-view samples, the pose is discarded with a conditional probability of 25%, so that the model can degenerate into a single-view prior when there is a lack of effective disparity or pose; in the second stage, the reflection intensity is set to N(0.8,0.2) truncated to [0.4,1.2], and motion blur, chromatic aberration and ghosting are added to enhance robustness to strong reflections.
[0052] In a second aspect, the present invention also provides an image processing system for implementing the method described in any of the above claims, comprising the following modules:
[0053] The dual-view and pose acquisition module is used to acquire the first reflection-containing image of the same reflection-containing scene. Second image containing reflection And the camera poses corresponding to the two images. , ;
[0054] The joint latent variable construction module is used to combine the clean conditional latent variables of two images containing reflection with two noisy transmission latent variables into a joint input by channel, and the transmission layer of the two viewpoints is output synchronously by the same denoising network.
[0055] The frame-aware spatial attention denoising module is used to generate only queries from noisy transmission features and only keys and values from clean reflection-containing conditional features of two viewpoints, thereby enabling cross-view information exchange and blocking the mutual propagation of noisy features.
[0056] The pose soft condition injection module uses normalized relative translation direction and relative quaternion rotation to form 7-dimensional pose, which is then encoded by MLP and superimposed according to scaling factor to be embedded at the diffusion time to obtain global soft conditions.
[0057] The dual-view transmission layer decoding module is used to iteratively predict noise from two viewpoints under global soft conditions. It gradually obtains two clean transmission latent variables through a preset diffusion sampler, and then outputs the transmission layer to be recovered by a frozen or fine-tuned VAE decoder. , .
[0058] Thirdly, the present invention also provides a computer-readable storage medium storing program instructions that, when executed by a processor, implement the method described in any of the preceding claims.
[0059] Fourthly, the present invention also provides an electronic device, including a processor and a memory, wherein the memory stores program instructions that, when executed by the processor, implement the method described in any of the preceding claims.
[0060] Compared with the prior art, the beneficial effects of the present invention are as follows:
[0061] 1. The method of this invention does not require determining a unique explicit optical flow for each pixel, nor does it perform hard deformation based on optical flow. This avoids optical flow ambiguity and error propagation caused by dual-layer motion. It also solves the problem that explicit optical flow can only provide a single correspondence and alignment errors easily propagate to the de-reflection result when there are two motion modes, transmission motion and reflection motion, in reflective regions. Compared to multi-frame methods that rely on scene-specific optimization, dual-view input operations are simpler and inference speed can be improved by approximately 60 times.
[0062] 2. The method of the present invention alleviates the underconstraint of single-image transmission-reflection decomposition by complementary observation from two viewpoints, and solves the problem that existing single-image diffusion dereflection methods cannot utilize the complementary information of two views and depth parallax, and the decomposition is unstable when facing strong reflection and complex structural reflection, thereby improving the accuracy of transmission layer recovery; real data shows that the solution of the present invention improves the baseline of strong single-image diffusion by more than 10%.
[0063] 3. The method of the present invention uses directed attention to make noisy targets take keys and values only from clean conditions, thereby blocking cross-view noise leakage. At the same time, it reuses the projection weights of the pre-trained two-dimensional diffusion model, reducing the training difficulty and maintaining the generation prior, solving the problem of mutual contamination of noisy features in ordinary cross-view joint attention, and realizing cross-view information exchange without destroying the parameter structure of the pre-trained two-dimensional diffusion model.
[0064] 4. The method of the present invention uses translation direction and quaternion rotation for relative pose and injects soft conditions. It is not sensitive to translation scale drift of consumer-grade IMU, does not require high-precision three-dimensional hard constraints, solves the problems of cumulative drift of translation amount in mobile phone IMU and sensitivity of hard geometric constraints to pose accuracy, and enables low-precision pose of ordinary mobile terminals to still provide effective geometric guidance.
[0065] 5. The method of the present invention is compatible with simultaneous dual-camera, sequential dual-shot, poseless, and single-image rollback modes, and can be deployed in mobile phone photography, vehicle vision, exhibition digitization, window monitoring, and computational photography software. Attached Figure Description
[0066] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.
[0067] Figure 1 This presents an implementation scheme for the existing dual-view reflection removal method.
[0068] Figure 2 This presents an implementation scheme for existing single-image diffusion dereflection methods.
[0069] Figure 3 The overall flowchart of the dual-view de-reflection method based on frame-aware spatial attention and camera pose provided in the embodiments of the present invention is shown.
[0070] Figure 4 The image shows the dual-camera mobile terminal shooting settings and anti-reflection effect provided in Embodiment 1 of the present invention. Detailed Implementation
[0071] To better understand this technical solution, the method of the present invention will be described in detail below with reference to the accompanying drawings.
[0072] This invention provides a dual-view de-reflection method based on frame-aware spatial attention and camera pose, such as... Figure 3 As shown, the process mainly includes six steps: dual-view and pose acquisition, joint latent variable construction, frame-aware spatial attention denoising, pose soft condition injection, dual-view transmission layer decoding, and model training.
[0073] 1. Dual Views and Pose Acquisition:
[0074] Obtain the first reflection-containing image of the same reflection-containing scene. Second image containing reflection and the corresponding camera pose , Dual views can be acquired simultaneously by two cameras on a mobile terminal, or sequentially by the same camera after slight movements within a short period. Camera pose is provided by the device's IMU, gyroscope, visual inertial odometry, or camera calibration results.
[0075] For the The hybrid imaging model is as follows: (Multiple viewpoints, hybrid imaging model)
[0076] ;
[0077] in, The transmission layer to be restored. The layer to be suppressed is the reflection layer; the addition operation is performed in an approximately linear intensity space. For images that have already undergone tone mapping, their nonlinear mixing relationships can also be learned end-to-end by the network.
[0078] 2. Construction of joint latent variables:
[0079] Using a frozen VAE encoder The two images containing reflections and the corresponding ground truth values of transmission from the training phase are mapped to the latent space. The images containing reflections yield clean conditional latent variables. , Transmission is truly worthwhile. , When the input resolution is H×W, the dimensions of each latent variable are 4×h×w, where h=H / 8 and w=W / 8. Adding diffusion noise to the transmission latent variable yields... , And connect by channel:
[0080] ,
[0081] ,
[0082] .
[0083] The U-Net in SD v2.1 has its first-layer input channels expanded from 4 to 16 and its last-layer output channels expanded from 4 to 8, enabling a single network to predict noise from two viewpoints simultaneously during a single forward pass. The original 4-channel pre-trained weights are copied to the two noisy transmission channels, and the conditional channels are initialized with a small amount of randomness, thereby maximizing the reuse of pre-trained priors.
[0084] 3. Frame-aware spatial attention:
[0085] In the 16×16 and 32×32 upsampling layers of U-Net, intermediate features are... Split into noisy transmission query part With clean, reflective conditions And rearranged into spatial sequences respectively. , :
[0086] ,
[0087] ,
[0088] ,
[0089] .
[0090] Will and Connect along the sequence dimension and use query, key, and value projection matrices pre-trained in SD v2.1. , , get , , Subsequently, query Q is extracted only from noisy transmission locations, and key K and value V are extracted only from clean condition locations:
[0091] ,
[0092] ,
[0093] ,
[0094] .
[0095] Thus, noisy transmission queries at each viewpoint can implicitly aggregate complementary evidence from clean, reflective conditions at both viewpoints, but cannot extract key-value pairs from noisy targets at the other viewpoint, structurally blocking cross-view noise leakage. Attention outputs are added back to the query features via residual connections, then restored to a spatial arrangement and connected to unchanged conditional features. This scheme does not perform optical flow estimation or hard deformation, enabling soft aggregation of multiple possible corresponding locations.
[0096] 4. Camera pose soft conditions:
[0097] Each camera pose is represented as ,in For translation, Rotation to a unit quaternion. Calculate the relative pose from two viewpoints:
[0098] .
[0099] To suppress translational scale drift caused by IMU double integration, only the translation direction and relative rotation are retained; when the translation norm is less than a preset threshold ε, the translation direction is set to zero vector. The 7-dimensional relative pose is input into a two-layer MLP with a hidden dimension of 256 to obtain the pose embedding. and with scaling factor Embedded with diffusion time Add:
[0100] ,
[0101] , .
[0102] The pose is only used as a global soft condition, without applying forced projection or hard alignment to the pixel position. Therefore, it can tolerate the drift and noise of consumer-grade IMUs and guide the network to distinguish between transmission parallax and reflection parallax based on the actual camera movement.
[0103] 5. Dual-view transmission layer decoding:
[0104] U-Net under conditions The next iteration predicts noise from both viewpoints, gradually obtaining two clean transmission latent variables through a preset diffusion sampler, and then outputting them by a frozen or finely tuned VAE decoder. , The default sampling step count is 50 steps; on edge devices, latent variable consistency distillation can be used to reduce the sampling step count to 4 steps, combined with INT8 / INT4 quantization deployment.
[0105] 6. Model Training:
[0106] The training employs a two-stage curriculum, progressing from easy to difficult. The first stage uses 768×768 semi-synthesized dual-view data, focusing on reflectance intensity. With a 15% probability, use visual Figure 1 Replacement Figure 2 Additionally, there is a 15% probability of using visual... Figure 2 Replacement Figure 1 The relative pose is set to zero; in the remaining 70% of the real dual-view samples, the pose is discarded with a conditional probability of 25%, allowing the model to degenerate to a single-view prior when effective disparity or pose is lacking. In the second stage, the reflection intensity is set to N(0.8, 0.2) truncated to [0.4, 1.2], and motion blur, chromatic aberration, and ghosting are added to enhance robustness to strong reflections. The loss function is expressed as:
[0107] ;
[0108] in, To add Gaussian noise, To predict noise for the network, For view mask, For pose mask, These are clean conditional latent variables obtained from the reflection image. Pose embedding is used. Training data includes semi-synthetic dual views sampled from video trajectories, physically rendered dual views based on optical models, and real-world data acquired for trajectory alignment.
[0109] Training data construction and collection:
[0110] 1) Video compositing: Select frame pairs from clean video with camera trajectory at controlled intervals to ensure viewpoint baseline and scene overlap; construct viewpoint-related reflections using gamma correction, Gaussian blur, ghost mask and random homography transformation, and then mix with the transmission layer.
[0111] 2) Physical rendering: Set the glass refractive index, reflection intensity, indoor and outdoor environment and 2-4 cm baseline and add random perturbation to render a two-view sample with accurate transmission ground truth, reflection ground truth and pose ground truth.
[0112] 3) Real data acquisition: A programmable track slide, removable flat glass, mobile terminal and IMU / gyroscope are used; first, a mixed sequence is acquired along the preset track through the glass, then the glass is removed and the same track is repeated to acquire the transmission true value, and finally the frame and pose are aligned.
[0113] The above overall technical solution is illustrated with examples in specific application environments as follows.
[0114] Example 1: Shooting with a dual-camera mobile terminal
[0115] 1. Using a mobile terminal equipped with dual wide-angle and ultra-wide-angle cameras, an IMU, and a gyroscope, RGB images including reflections from two viewpoints are acquired simultaneously in a single shutter release, and the pose near the exposure time is read. Intrinsic, distortion, and extrinsic parameters are calibrated for the dual cameras; the images are cropped or scaled to 768×768 and color-matched. Typical lens baselines are 15–25 mm, and the target distance for shooting through the lens is 0.3–1.0 m.
[0116] 2. Input the two images containing reflections and their relative poses into the VAE encoder and the pose MLP. Relative translations are normalized to orientations, and relative rotations are represented using quaternions; when the pose is missing, a zero vector is input. Construct a 16-channel joint input and feed it into the modified SD v2.1 U-Net.
[0117] 3. Frame-aware spatial attention is performed on the 16×16 and 32×32 upsampling layers of U-Net: only two clean, reflection-containing conditional features are queried from the two noisy transmission features. Transmission latent variables from both viewpoints are obtained synchronously using 50 sampling steps, and the original image resolution is restored after VAE decoding.
[0118] 4. Output two dereflection images. For real-time preview, a 4-step distillation model is used with INT8 quantization of the network; for high-quality photography, a full 50-step model is used. The output can also be fed into object detection, 3D reconstruction, or image enhancement modules.
[0119] The dual-camera mobile terminal shooting settings and anti-reflection effect in this embodiment are as follows: Figure 4 As shown.
[0120] Example 2: Dereflection of a Single-Shot Slightly Moving or Live Photograph
[0121] 1. Using the same camera, take continuous pictures within about 0.5 seconds and obtain image pairs with an equivalent baseline of 1-3 cm by slightly translating the hand naturally; calculate the pose of each frame using visual inertial odometry, and select two frames from the sequence with sufficient scene overlap and parallax greater than the threshold.
[0122] 2. Hard optical flow alignment is not performed on dynamic object regions; instead, soft matching is performed directly by frame-aware spatial attention. For regions with contradictory correspondences or occlusions, the model regresses to the single-image diffusion prior through random view replacement and pose discarding mechanisms during the training phase.
[0123] 3. Output the transmission layers of the two selected viewpoints respectively; when the viewpoint only undergoes pure rotation or the parallax is insufficient, the system prompts the user to move slightly laterally, or outputs the results in single-image mode.
[0124] Example 3: Model Training and Validation
[0125] 1. The model was trained using 13,181 pairs of dual-view images from the DualRef-R training partition and tested using 2,819 semi-synthetic test pairs, 500 pairs of dual-view images sampled from 195 real sequences, and 20 publicly available real pairs of dual-view images.
[0126] 2. Training is performed on an NVIDIA GPU with at least 24 GB of VRAM, using the AdamW optimizer and a two-stage course strategy; during inference, processing one scene on an RTX 6000-level GPU takes approximately 3 seconds.
[0127] 3. The PSNR, SSIM, LPIPS, CLIPIQA, and MUSIQ were used for evaluation. On the real DualRef-V, the PSNR of this method was 24.17 dB, which is better than the 21.57 dB of the comparative single-map diffusion method DAI; on the semi-synthetic DualRef-R, the PSNR was 28.19 dB.
[0128] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. However, these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
[0129] The image processing system, mobile terminal, computing device, and computer-readable storage medium storing the corresponding program obtained based on the above method of the present invention are all within the protection scope of the present invention.
Claims
1. A dual-view reflection removal method based on frame-aware spatial attention and camera pose, characterized in that, Includes the following steps: Step S1, Dual Views and Pose Acquisition: Obtain the first reflection-containing image of the same reflection-containing scene. Second image containing reflection And the camera poses corresponding to the two images. , ; Step S2, Construction of joint latent variables: The clean conditional latent variables of two images containing reflections and the two noisy transmission latent variables are combined into a joint input by channel, and the transmission layers of the two viewpoints are output synchronously by the same denoising network. Step S3, Frame-aware spatial attention denoising: Noisy transmission features generate only queries, while clean reflection-containing conditional features from two viewpoints generate only keys and values, thereby enabling cross-view information exchange and blocking the mutual propagation of noisy features. Step S4, Soft Condition Injection of Pose: A 7-dimensional pose is formed by normalizing the relative translation direction and the relative quaternion rotation. After being encoded by MLP, the pose is superimposed according to the scaling factor and embedded at the diffusion time to obtain the global soft condition. Step S5, Decoding the dual-view transmission layer: The noise at two viewpoints is iteratively predicted under global soft conditions. Two clean transmission latent variables are gradually obtained through a preset diffusion sampler. The transmission layer to be recovered is then output by a frozen or finely tuned VAE decoder. , .
2. The dual-view dereflection method based on frame-aware spatial attention and camera pose according to claim 1, characterized in that, In step S1, the dual views are acquired synchronously by the two cameras of the mobile terminal, or sequentially by the same camera after it moves over a period of time; the camera pose is given by the device IMU, gyroscope, visual inertial odometry or camera calibration results.
3. The dual-view de-reflection method based on frame-aware spatial attention and camera pose according to claim 1, characterized in that, In step S1, for the first The hybrid imaging model with reflected images from multiple viewpoints is as follows: ; in, The transmission layer to be restored. The reflective layer to be suppressed; the addition operation is performed in an approximately linear light intensity space.
4. The dual-view dereflection method based on frame-aware spatial attention and camera pose according to claim 1, characterized in that, In step S1, for an image that has already undergone tone mapping, its nonlinear mixing relationship is learned end-to-end through the network.
5. The dual-view de-reflection method based on frame-aware spatial attention and camera pose according to claim 1, characterized in that, In step S2, a frozen VAE encoder is used. The two images containing reflections and the corresponding ground truth values from the training phase are mapped to the latent space, and the clean conditional latent variables are obtained from the images containing reflections. , Transmission is truly worthwhile. , When the input image resolution is H×W, the dimensions of each latent variable are 4×h×w, where h=H / 8 and w=W / 8. Adding diffusion noise to the transmission latent variable yields... , And obtained by connecting according to the channel: , , ; The U-Net in SD v2.1 is expanded from 4 input channels in the first layer to 16 and from 4 output channels in the last layer to 8, enabling a network to predict noise from two viewpoints simultaneously in one forward pass.
6. The dual-view dereflection method based on frame-aware spatial attention and camera pose according to claim 5, characterized in that, In step S3, the intermediate features are upsampled in the 16×16 and 32×32 upsampling layers of U-Net. Split into noisy transmission query part With clean, reflective conditions And rearranged into spatial sequences respectively. , ,Will and Connect along the sequence dimension and use query, key, and value projection matrices pre-trained in SD v2.
1. , , get , , ; Then, query Q is extracted only from the noisy transmission position, and key K and value V are extracted only from the clean condition position; Noisy transmission query section With clean, reflective conditions Represented as: ; ; , Rearranged into spatial sequences , Represented as: ; 。 7. The dual-view dereflection method based on frame-aware spatial attention and camera pose according to claim 1, characterized in that, In step S3, the query Q, key K, and value V are represented as follows: , , ; Will and The sequential dimension connection is represented as: 。 8. The dual-view de-reflection method based on frame-aware spatial attention and camera pose according to claim 1, characterized in that, In step S4, the pose of each camera is represented as ,in For translation, The relative pose is calculated based on two viewpoints using unit quaternion rotations. The 7-dimensional relative pose is then input into a two-layer MLP with a hidden dimension of 256 to obtain the pose embedding. and with scaling factor Embedded with diffusion time Adding them together yields the global soft condition. ; The relative pose calculated from two viewpoints is represented as follows: Pose embedding Represented as: ; Global soft conditions Represented as: ; 。 9. The dual-view dereflection method based on frame-aware spatial attention and camera pose according to claim 1, characterized in that, The model training employs a two-stage course, progressing from easy to difficult. The first stage uses 768×768 semi-synthesized dual-view data, focusing on reflection intensity. With a 15% probability, replace view 2 with view 1, and with another 15% probability, replace view 1 with view 2, and set the relative pose to zero. In the remaining 70% of real dual-view samples, the pose is discarded with a conditional probability of 25%, so that the model can degenerate into a single-view prior when there is a lack of effective disparity or pose. In the second stage, the reflection intensity is set to N(0.8,0.2) truncated to [0.4,1.2], and motion blur, chromatic aberration and ghosting are added to enhance robustness to strong reflections.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program instructions that, when executed by a processor, implement the method according to any one of claims 1-9.
