An unsupervised optical flow estimation method based on instance mask guided smoothness loss

Abstract

The application discloses an unsupervised optical flow estimation method based on instance mask guided smoothing loss, combines a M2Flow neural network architecture core part with a brand-new mask guided smoothing loss function, and carries out neural network unsupervised training under a PWC-Lite 4-Frame pyramid framework. The training process is completed by relying on the image photometric consistency constraint and the dynamic warm-up strategy of the loss function, and does not need artificial optical flow data labeling. The trained neural network can realize significant optimization effect: on the boundary sharpening and overflow suppression level, a sharp and accurate flow field fault is formed at the complex instance boundary of traffic signs, fast moving vehicles and the like, and the overflow problem of the flow field to the static background is effectively avoided; on the textureless area robustness level, for large-area sky, vehicle body, reflective glass curtain wall and the like low-texture scene, the texture-independent instance constraint is relied on to suppress the noise and the false image caused by reflection and the error motion diffusion, and the flow field inside the instance is kept uniform and pure.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to an unsupervised optical flow estimation method based on instance mask guided smoothing loss. Background Technology

[0002] Existing unsupervised optical flow estimation methods generally adopt traditional smoothing loss (such as first-order or second-order edge-aware smoothing) as shown in equation (1), the core assumption of which is that adjacent pixels have similar motion. However, this assumption fails severely at the object boundary: when the moving object has a significant motion difference with the background, the traditional smoothing term blindly punishes the optical flow gradient, resulting in blurred boundaries, "overflow" or "shrinkage" of the object outline, which seriously affects the accuracy of downstream tasks (such as video segmentation and motion tracking).

[0003]

[0004] In the field of optical flow estimation, unsupervised optical flow estimation methods have broader engineering application prospects because they do not rely on a large amount of labeled data. However, existing unsupervised optical flow estimation methods generally suffer from core technical defects: when dealing with the boundary region between moving targets and the background, it is difficult to balance the smoothness of the optical flow field and the sharpness of the boundary, often resulting in blurred boundaries and poor consistency of optical flow within moving targets. Specifically, the smoothing loss function used in traditional unsupervised optical flow estimation methods is mostly designed with globally uniform weights, which cannot distinguish the different smoothing requirements of the interior of moving instances and the boundary of instances. If global smoothing is overemphasized, it will lead to the loss of motion boundary information and "blurring" of the optical flow field; if the smoothing constraint is weakened, the optical flow distribution within instances will be messy and inconsistent, ultimately affecting the accuracy of optical flow estimation and failing to meet the high accuracy requirements of optical flow estimation in engineering scenarios such as autonomous driving and video frame matching. Summary of the Invention

[0005] To overcome the shortcomings of existing technologies, this invention provides an unsupervised optical flow estimation method based on instance mask-guided smoothing loss. It combines the core of the M2Flow neural network architecture with a novel mask-guided smoothing loss function, and conducts unsupervised training of the neural network within the PWC-Lite 4-Frame pyramid framework. The training process relies on the photometric consistency constraints of the image and a dynamic warm-up strategy for the loss function, eliminating the need for manual annotation of optical flow data. The trained neural network achieves significant optimization effects: in terms of boundary sharpening and spillover suppression, it forms sharp and accurate flow field faults at the boundaries of complex instances such as traffic signs and fast-moving vehicles, effectively preventing the flow field from spilling over into the static background; in terms of robustness in textureless regions, for low-texture scenes such as large areas of sky, vehicle bodies, and reflective glass curtain walls, it relies on texture-independent instance constraints to suppress noise and reflection-induced artifacts and erroneous motion diffusion, maintaining a uniform and pure flow field within instances.

[0006] The technical solution adopted by this invention to solve its technical problem is as follows: Step 1: Introduce the Mask-GuidedSmoothnessLoss function into the PwC-Lite 4-Frame pyramid framework. Its multi-scale total loss within the pyramid framework is applied only to the first layer, with weights of 0 in the remaining layers, expressed as:

[0007] in: The dynamic total weights for the current training steps employ a linear warm-up strategy: the weights are 0 when the global number of steps is less than 10,000, linearly increasing from 0 to 0.2 between 10,000 and 30,000 steps, and then remaining constant at 0.2. For the first layer, the corresponding multi-scale weight configuration is w_sm_scales_pan = [1.0, 0.0, 0.0, 0.0, 0.0]. This represents the smoothing loss guided by the first-layer mask; The specific calculation form of the first layer is a weighted L1 penalty guided by an instance mask on the second-order gradient of the optical flow field:

[0008] The "smart weight" W is defined as follows:

[0009] In the formula: This is a globally unique Panoptic ID; 1[·] is an indicator function that takes the value 1 if the condition is true, and 0 otherwise; To fix the hyperparameters, Pixels p The right neighbor and the down neighbor; , These represent the components of the optical flow field in the x-direction and the y-direction, respectively, which are the displacements of the pixel in the horizontal and vertical directions. When adjacent pixels belong to different instances, i.e., are located at the boundary, 1[·]=1, and the weight is reduced to exp( 10) ≈ 4.5 × 10 -5 That is, when the machine is close to zero, almost no smoothing penalty is applied; when adjacent pixels belong to the same instance, i.e., are inside, 1[·]=0, weight=1, and a full second-order smoothing penalty is applied. Simultaneously, the forward optical flow F t → t+1 and backward optical flow F t+1 → t Apply a mask to guide the smoothing loss and take the average to form a bidirectional instance smoothing constraint; Step 2: Decode the globally unique Panoptic ID; Input a uint8 type RGB mask image of [B, H, W, 3], and obtain a long type "ID card image" of [B, H, W] according to formula (5), ensuring that the numbers of any two different instances are different;

[0010] Step 3: Calculate the instance boundary indicator diagram; Using semantic-level true boundary responses, we determine whether the right and bottom edges of a pixel are the same instance, and thus determine whether it is an edge. The implementation formula is as follows:

[0011] in, These indicate whether pixel p is an instance boundary in the horizontal direction (x direction) and the vertical direction (y direction), respectively. Step 4: Calculate the exponentially decaying weighted graph; The boundary weights are reduced directly from 1.0 to approximately 0, which is achieved through the following mathematical formula:

[0012] Inside the vehicle body, A value of 0 indicates a weight. At the vehicle body boundary, A value of 1 indicates a weight. ; Step 5: Bidirectional instance smoothing; This is achieved through the following loss function:

[0013] The first item on the right is "forward optical flow using the mask tube of frame t", and the second item is "backward optical flow using the mask tube of frame t+1". This represents the mask for frame t. This represents the mask for the (t+1)th frame; Step 6: Multi-scale mask downsampling; Each layer is processed separately. The mask is downsampled in the nearest mode using F.interpolate, and then re-decoded layer by layer to generate a clear "Panoptic ID" for each layer. At the same time, a multi-scale weight design is adopted to enable instance smoothing only in the first layer, which requires the most structural prior.

[0014] Preferably, the fixed hyperparameter .

[0015] An electronic device includes: a processor and a memory; the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory to enable the electronic device to perform the above-described unsupervised optical flow estimation method.

[0016] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described unsupervised optical flow estimation method.

[0017] A chip includes a processor for calling and running a computer program from a memory, causing a device equipped with the chip to perform the above-described unsupervised optical flow estimation method.

[0018] A computer program product includes a computer storage medium storing a computer program, the computer program including instructions executable by at least one processor, which, when executed by the at least one processor, implement the above-described unsupervised optical flow estimation method.

[0019] The beneficial effects of this invention are as follows: This invention proposes a novel MaskGuidedSmoothnessLoss (MSLoss) for the PWC-Lite 4-Frame pyramid framework. This loss function overcomes the limitations of the globally uniform first-order smoothing loss function in traditional unsupervised optical flow estimation methods. Through a collaborative design involving multi-scale targeted application, a dynamic warm-up weight strategy, second-order smoothing constraints, and instance mask-guided differentiated weight allocation based on Panoptic ID, it achieves the optimization goal of applying strong smoothing within instances and exponentially decaying weights at boundaries. Specifically, it applies the loss only to the coarsest layer of the pyramid to improve training efficiency, ensures stable convergence through dynamic warm-up weights, adapts to the planar perspective projection rules of real-world scenes using second-order smoothing constraints to address the unnatural motion estimation problem in large planar regions, and combines bidirectional smoothing constraints to achieve complete second-order smoothing within instances and near-zero penalty at boundaries, resulting in a differentiated effect. This innovative design precisely addresses the core shortcomings of existing technologies, maintaining the advantages of unsupervised learning without relying on additional labeled data. It effectively preserves the sharpness of the optical flow field at the moving boundary and enhances its internal consistency, significantly improving the consistency, boundary sharpness, and adaptability to real-world scenarios of optical flow estimation. It demonstrates clear innovation and practicality. Attached Figure Description

[0020] Figure 1 This is an overall framework diagram of the present invention; Figure 2 The process of calculating the smoothing term guided by the mask; Figure 3This is a Panoptic ID decoding and weight map visualization (KITTIStep example: white inside, black at the boundary); Figure 4 This is a boundary weight heatmap (KITTIStep example); Figure 5 The ablation experiment curves for the WarmUp strategy are shown in the following figures: (a) the decay curve of the endpoint error, (b) the decay curve of the F1-all index, and (c) the weights of the Pan-Instance Smoothness Loss. Changes throughout the training process.

[0021] Figure 6 Optical flow comparison for complex scenes in KITTI 2015 (sharpness vs. traditional blur). Detailed Implementation

[0022] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0023] To address the engineering challenge of balancing motion boundary sharpness and instance-internal optical flow consistency in existing unsupervised optical flow estimation methods, this invention provides an unsupervised optical flow estimation method based on instance mask-guided smoothing loss. The core technical solution involves employing a customized neural network architecture and specifically designing an instance mask-guided smoothing loss function, as detailed below: 1. Neural Network Architecture Design: This invention adopts the core of the M2Flow neural network architecture, with ARFlow as the backbone network, and focuses on integrating the Motion Information Propagation (MIP) module. It is adapted to the unsupervised optical flow estimation task with four consecutive RGB image inputs and builds the PWC-Lite 4-Frame pyramid framework. Among them, the shared pyramid encoder is used to extract multi-scale feature maps from four input images, providing feature support for subsequent optical flow prediction and motion information modeling; the iterative residual refinement decoder realizes the initial prediction and iterative optimization of bidirectional optical flow (forward flow F1→2, F2→3 and backward flow F0←1, F1←2) between multiple frames through warp operation, correlation calculation and residual prediction; the MIP module, as the core integrated module, realizes cross-frame fusion of motion information between four frames through motion residual calculation and warp propagation, abandoning the constant velocity assumption of traditional methods and accurately capturing the trend of scene motion change.

[0024] 2. Core Innovation: MaskGuidedSmoothnessLoss Design: To solve the problems of existing technologies, this invention introduces a brand-new mask-guided smoothness loss function into the PWC-Lite 4-Frame pyramid framework. The core design logic is to combine the multi-scale characteristics of the pyramid, second-order smoothness constraints and instance mask-guided weighting to achieve differentiated smoothness penalty. For specific technical details, please refer to the technical solution.

[0025] This approach combines the core of the M2Flow neural network architecture with a novel mask-guided smoothing loss function, enabling unsupervised training of the neural network within the PWC-Lite 4-Frame pyramid framework. The training process relies on image photometric consistency constraints and a dynamic warm-up strategy for the loss function, eliminating the need for manual annotation of optical flow data. The trained neural network achieves significant optimizations: in terms of boundary sharpening and spillover suppression, it creates sharp and accurate flow field discontinuities at the boundaries of complex instances such as traffic signs and fast-moving vehicles, effectively preventing the flow field from spilling over into the static background; regarding robustness in textureless regions, for low-texture scenes such as large areas of sky, vehicle bodies, and reflective glass curtain walls, it uses texture-independent instance constraints to suppress noise and reflection-induced artifacts and erroneous motion propagation, maintaining a uniform and pure flow field within instances. Overall, this solution successfully addresses the engineering problems of existing methods, such as the inability to simultaneously balance smoothing constraints and boundary preservation, unnatural motion estimation in large planar areas, and inaccurate estimation of occluded regions. It significantly improves the robustness and accuracy of optical flow estimation, making the results more suitable for practical engineering applications such as autonomous driving and video frame fusion.

[0026] This invention introduces a novel Mask-Guided SmoothnessLoss function into the PWC-Lite 4-Frame pyramid framework. In this function, the total multi-scale loss within the pyramid framework is applied only to the coarsest first layer (l=1), with weights of 0 for the remaining layers. This loss can be expressed as:

[0027] in: The dynamic total weights under the current training steps adopt a linear warm-up strategy: the weights are 0 when the global number of steps is less than 10,000, and linearly increase from 0 to 0.2 between 10,000 and 30,000 steps, and then remain unchanged at 0.2; for the first layer, the corresponding multi-scale weight configuration is w_sm_scales_pan = [1.0, 0.0, 0.0, 0.0, 0.0].

[0028] The specific calculation form of the first layer is a weighted L1 penalty guided by an instance mask on the second-order gradient of the optical flow field:

[0029] The core advantage of using second-order smoothing lies in its ability to allow the optical flow within the same instance to vary linearly (consistent with the planar perspective projection rules in real-world scenes), rather than forcing the optical flow to tend towards a constant using traditional first-order smoothing. This results in more natural motion estimation in large planar areas such as road surfaces and vehicle bodies. The "smart weight" W is defined as follows:

[0030] In the formula: It is a globally unique Panoptic ID, and 1[·] is an indicator function (1 if the condition is true, 0 otherwise). To fix the hyperparameters, These are the right neighbor and the bottom neighbor of pixel p, respectively.

[0031] When adjacent pixels belong to different instances (located at the boundary), 1[·]=1, and the weight decreases to exp( 10) ≈ 4.5 × 10 -5 (Close to machine zero), almost no smoothing penalty is applied; when adjacent pixels belong to the same instance (located inside), 1[·]=0, weight=1, and a full second-order smoothing penalty is applied. To further improve the motion consistency within instances, this invention also applies to the forward optical flow F t → t+1 and backward optical flow F t+1 → t The above mask is applied to guide the second-order smoothing loss, and the average is taken to form a bidirectional instance smoothing constraint.

[0032] (1) Decode the globally unique Panoptic ID; Input a uint8 type RGB mask image of [B, H, W, 3] and obtain a long type "ID card image" of [B, H, W] according to formula (5), ensuring that the numbers of any two different instances are different.

[0033]

[0034] (2) Calculate the instance boundary map; The original image's edge detection method is not suitable because it is affected by factors such as shadows, reflections, and raindrops. Therefore, a semantic-level true boundary response is used. This method determines whether a pixel is an edge by checking if its right and bottom edges are the same instance. The implementation formula is as follows:

[0035] (3) Calculate the exponentially decaying weight map; Its core invention lies in causing the boundary weight to plummet from 1.0 to almost 0. This is achieved through the following mathematical formula:

[0036] in, To fix the hyperparameters, this invention uniformly takes... Inside the vehicle, A value of 0 indicates a weight. This achieves full punishment for unevenness; at the vehicle body boundaries, A value of 1 indicates a weight. It imposes almost no penalty and allows for dramatic shifts. The weight graph is visualized as a "perfect switch graph" where instances are pure white (weight 1.0) inside and black lines at instance boundaries (weight 0.000045). Compared to linear decay, exponential decay allows boundary weights to approach machine zero directly, completely freeing them from boundary issues and avoiding the boundary blurring caused by the slight penalty of linear decay.

[0037] (4) Bidirectional Instance Smoothing; Its core principle is to jointly constrain the preceding and following frames to address the shortcomings of unidirectional smoothing. In the unidirectional mechanism, if only the mask of the first frame is used, when an instance in the first frame (such as a vehicle) is occluded (e.g., half-blocked by a tree), problems arise such as incomplete masking and missing smoothing constraints. The bidirectional mechanism of this invention achieves this through the following loss function:

[0038] The first term is "forward optical flow using the mask tube of frame t", and the second term is "backward optical flow using the mask tube of frame t+1". In simple terms, the forward direction uses the shape of the car in frame 1 to constrain the optical flow "how to move the car to frame 2", and the backward direction uses the shape of the car in frame 2 to constrain the optical flow "how to move the car back to frame 1". This bidirectional coordination can effectively avoid target deformation.

[0039] (5) Multi-scale mask downsampling strategy; This invention proposes a "nearest neighbor downsampling + layer-by-layer decoding" strategy. Specifically, each layer is processed separately. The mask is downsampled using F.interpolate in the nearest mode (ensuring no ID confusion for each pixel, avoiding bilinear interpolation that mixes cars and roads into an average color, leading to ID errors after decoding). Then, each layer is re-decoded to generate a clear "Panoptic ID" specific to each layer. Simultaneously, a multi-scale weight design is employed, enabling instance smoothing only in the first layer, where structural prior is most crucial. This strategy achieves results such as: in the first layer, even though a vehicle has only 10 pixels, its ID is clear, ensuring correct optical flow structure; while in the fifth layer, fine-tuning through photometric loss achieves a combined coarse and fine optimization effect.

[0040] Example: This invention utilizes a PWC-Lite 4-Frame pyramidal convolutional neural network to estimate pixel motion in video sequences, thereby obtaining a high-precision, sharp-boundary optical flow field. The overall framework of the model is as follows: Figure 1 This method innovatively introduces the MaskGuidedSmoothnessLoss module based on the existing unsupervised optical flow framework. This module constructs a globally unique Panoptic ID map using the video instance segmentation mask (VIS Mask) provided by the KITTI-STEP dataset, and then generates an instance-aware boundary indicator map and an exponentially decaying weight map. This achieves structure-aware regularization with "strong smoothing within instances and zero penalty at instance boundaries," solving the problem of motion boundary blurring in traditional smoothing loss. The calculation process of the smoothing term in this method is as follows: Figure 2 Furthermore, by combining bidirectional consistency constraints, multi-scale mask downsampling strategies, and the Adam optimization algorithm, the model achieved a Fluenti score of 11.15% on the KITTI 2015 Flow official test set, representing an improvement of over 16% compared to the baseline method. This study demonstrates that instance-level structural information is crucial for robust regularization of unsupervised optical flow. This method can be widely applied to visual tasks such as autonomous driving, video segmentation, and action recognition. It can also be fused with other sensors (such as LiDAR and millimeter-wave radar) to further enhance perception robustness in complex environments.

[0041] The following is a detailed implementation process of the present invention. This example provides an unsupervised optical flow estimation method based on the PWC-Lite 4-Frame framework and combined with KITTI-STEP instance ground truth. It innovatively introduces a MaskGuidedSmoothnessLoss module after each level of the decoder pyramid, further enhancing the model's ability to express the consistency between motion boundaries and instance interiors. The specific implementation steps are illustrated below with reference to the accompanying figures: Step 1: Parse training parameters. Use the Python standard library `argparse` to parse command-line arguments and obtain hyperparameter configuration information. This provides a complete configuration entry point for subsequent training. Step 2: Load and preprocess the KITTI-STEP dataset. Use the custom datasets.KITTISTEP module to load the 20 training sequences (8008 frames in total) provided by the official documentation. Each frame contains: an RGB image and an RGB-encoded video instance segmentation mask (R=sem, G=Panoptic ID / / 256, B=Panoptic ID%256). Step 3: Create a data loader and perform online augmentation. Use `torch.utils.data.DataLoader` to create a training data loader with `BatchSize=4`. Real-time augmentation strategies include: Geometric consistency transformations: RandomVerticalFlip (vflip=True), RandomHorizontalFlip (hflip=True), and RandomSwap (swap=True). These transformations are applied synchronously to image pairs and instance masks to ensure geometric consistency. Appearance enhancement transformations: Color Jitter (brightness=0.5, contrast=0.0, saturation=0.0, hue=0.0, applied only to image pairs) and Random Gaussian Blur (p=0.5, radius range 0~3, applied only to image pairs). Input transformation: the array is converted into a tensor and uniformly scaled to the target size.

[0042] Step 4: Create a PWC-Lite 4-Frame model. Instantiate a pyramid network (5 layers) using the official networks.pwclite_4f module, enable learned_upsampler=True, and disable traditional smoothing loss (w_smooth=0.0). Simultaneously register a custom PanInstanceLoss module to prepare for integrating instance mask-guided smoothing loss (e.g., ...). Figure 1 Network structure and Figure 2 (Smoothing term calculation process).

[0043] Step 5: Set up the optimizer and learning rate scheduler, using the Adam optimizer (β1=0.9, β2=0.999, weight_decay=1e-6), with an initial learning rate of 2×10⁻⁶. -4The ExponentialLR scheduler is used, decaying every 1000 steps. Gradient clipping (clip=1.0) is enabled to prevent explosion. All trainable parameters participate in optimization, with no frozen layers (e.g., ...). Figure 1 Optimizer settings).

[0044] Step 6: Train the model, executing a complete training loop of 100 epochs. In each iteration, obtain four consecutive frames of images (I0, I1, I2, I3) and their corresponding four RGB instance masks (panoptic_M1_rgb, panoptic_M2_rgb, etc.) from the DataLoader. After forward propagation of the model, calculate the pyramid multi-scale optical flow predictions (pyramid_flows), and then calculate the total loss, including photometric loss and the newly added mask-guided smoothness loss. The total loss formula is: Total loss = .

[0045] Specifically, in the process of calculating the smoothing term of MaskGuidedSmoothnessLoss, the input RGB instance mask is first decoded to obtain a globally unique Panoptic ID map (see formula (5) for details), such as Figure 3 The Panoptic ID decoding and weight map visualization shown is based on a KITTI-STEP dataset example, where the inside of the instance is white and the boundary is represented by black lines. Next, an instance boundary indicator map is calculated, generating a boundary map by comparing whether adjacent pixels have the same Panoptic ID (see formula (6)). Values ​​are 1 at the boundary and 0 inside. Then, an exponentially decaying weight map is calculated based on the boundary indicator map (see formula (7)). α=10 is used as a fixed hyperparameter, causing the boundary weights to rapidly decay to approximately 4.54×10⁻⁶. -5 ,like Figure 4The boundary weight heatmap shown is based on the KITTI-STEP dataset example. In the heatmap, the boundary region is in a cool color to represent low weight, and the interior is in a warm color to represent high weight. Then, the first gradient (dx, dy) of the optical flow field is calculated, and the second gradient (dx2, dy2) is further obtained to implement the second-order smoothing penalty. This design allows the optical flow inside the instance to change linearly rather than being forced to be constant. At the same time, the weight map is clipped to match the shape of the second gradient (e.g., weight_x_clipped= weight_x[:, :, :, 1:]), and finally the weighted L1 loss (loss_x + loss_y) is calculated. To achieve bidirectional constraint, the smoothing loss is applied to both the forward optical flow (using the mask of frame t) and the backward optical flow (using the mask of frame t+1), and the average value is taken (see formula (8) for details). The entire smoothing term calculation process is as follows. Figure 2 The mask-guided smoothing term calculation process is shown (from RGB mask input to second-order weighted loss output).

[0046] Under the multi-scale strategy, the smoothing loss is calculated only in the first coarsest layer of the pyramid (w_sm_scales_pan = [1.0, 0.0, 0.0, 0.0, 0.0]), and a dynamic warm-up weight w_smooth_curr is applied (0 for the first 10,000 steps, then linearly increasing to 0.2). After calculating the total loss, the Adam optimizer is used (β1 = 0.9, β2 = 0.999, weight_decay = 1e-6, initial learning rate 2×10). -4 Backpropagation is performed to update parameters. TensorBoard logs are recorded every 500 steps, including metrics such as smoothing loss (loss_sm) and endpoint error (EPE_val); model checkpoints (ckpt) are saved every 1000 steps. The entire training loop is as follows: Figure 1 The diagram shown is an overall framework diagram of the present invention (including data loading, model forward propagation, loss calculation and optimization update).

[0047] Step 7, Validation and Testing: During the training process, every 1000 steps, EPE, Fl-all and other metrics are calculated on the local validation set KITTI 2015 (train), and the current optimal pre-trained model weights are saved, as shown in Table 1.

[0048] Table 1

[0049] Step 8: Design component ablation experiments and fix them. λ sm =0.2, α=10, retrained 3 times on KITTI-STEP and the average was taken, see Table 2;

[0050] Table 2 validates the proposed... And the effectiveness of its Warmup strategy. Compared to the baseline ( Only), only introduced This significantly reduced the F1-All score from 13.43% to 11.45%, demonstrating the inherent effectiveness of the instance ID-guided smoothing regularization term in addressing the boundary overflow problem. Further introducing an adaptive Warmup strategy resulted in optimal convergence of the model performance, with the F1-All score reaching 11.15% and the EPE dropping to a minimum of 2.828. This confirms the crucial role of the Warmup mechanism in stabilizing training and continuously refining the flow field details.

[0051] Figure 5 (a) shows the decay curve of End-Point Error (EPE). The model using the warmup strategy eventually converges to a significantly lower EPE, further validating the key role of the warmup mechanism in stabilizing training and continuously refining optical flow details. Figure 5 (b) shows the decay curve of F1-all, a core metric for measuring the accuracy of moving boundaries. The model using the warmup strategy (dark red curve) converges to a significantly lower F1-all value, indicating that this strategy promotes optimal convergence and continuously improves the quality of boundary details. Figure 5 (c) Shows the weights of Pan-Instance Smoothness Loss. The weights change linearly from 0 to their maximum value between 10k and 30k steps, ensuring stability in the early stages of training while also... In the later stages of training, it can serve as an effective "booster" to continuously optimize model performance.

[0052] Step 9 Weighted ablation experiment: Table 3

[0053] Table 3 discusses weight Impact on model performance. When the weights are too low ( When the weights are too high (=0.05), the model's boundary constraints are insufficient, and the F1-All score is only 12.2%. When the coefficient of performance (F1-All) is 0.50, although F1-All is still superior, the excessive constraint on the flow field causes a slight rebound in EPE. The final choice... =0.20, which achieves the optimal trade-off between EPE and F1-All, making the model optimal in both overall accuracy and boundary accuracy.

[0054] Step 10, Boundary Sharpening Parameters ablation, fixation =0.2, Use a warmup strategy: Table 4

[0055] Table 4 analyzes Internal boundary sharpening parameters The sensitivity of the flow field. This parameter controls the degree to which discontinuities are allowed at instance boundaries. When Smaller ( When the weight decay is less than 5, the boundary smoothness is too high, and the F1-All performance is slightly worse. Setting it to 10 achieves optimal edge sharpening, reducing F1-All to 11.82%. Too large ( When the regularization penalty is 15, it is too sparse and may introduce small noise at the boundaries, resulting in a slight performance degradation.

[0056] Step 11: Use the best-performing model to predict and view the optical flow prediction map. After training, take the model with the lowest Fl-all score (trained based on KITTI-STEP) and generate an optical flow prediction map on KITTI 2015, such as... Figure 6 .

[0057] Figure 6This paper presents a qualitative comparison of the PGF method of this invention with baseline methods and the classic ARFlow on the KITTI 2015 test set. The results clearly highlight the advantages of PGF in solving two major systemic problems. Boundary sharpening and bleed suppression: In the first two rows, at the boundaries of complex objects such as traffic signs and fast-moving vehicles, both the baseline method and ARFlow exhibit severe optical flow leakage from the foreground to the static background. In contrast, guided by instance IDs, PGF can generate extremely sharp and accurate optical flow discontinuities at object contours, completely eliminating cross-boundary contamination and resulting in a clean, well-defined motion field. Robustness in low-texture areas: The last two rows focus on large low-texture areas such as the sky, vehicle bodies, and reflective glass curtain walls. In these areas, the baseline method and ARFlow produce significant artifacts and erroneous motion propagation due to noise and reflections. PGF, benefiting from texture-independent instance-level constraints, effectively suppresses these artifacts and interferences, maintaining highly uniform and clean optical flow within each instance. This fully demonstrates the significant value of panoramic instance IDs as a robust structural regularization signal.

Claims

1. An unsupervised optical flow estimation method based on instance mask guided smoothing loss, characterized in that, Includes the following steps: Step 1: Introduce the Mask-GuidedSmoothnessLoss function into the PwC-Lite 4-Frame pyramid framework. Its multi-scale total loss within the pyramid framework is applied only to the first layer, with weights of 0 in the remaining layers, expressed as: in: The dynamic total weights for the current training steps employ a linear warm-up strategy: the weights are 0 when the global number of steps is less than 10,000, linearly increasing from 0 to 0.2 between 10,000 and 30,000 steps, and then remaining constant at 0.

2. For the first layer, the corresponding multi-scale weight configuration is w_sm_scales_pan = [1.0, 0.0, 0.0, 0.0, 0.0]. This represents the smoothing loss guided by the first-layer mask; The specific calculation form of the first layer is a weighted L1 penalty guided by an instance mask on the second-order gradient of the optical flow field: The "smart weight" W is defined as follows: In the formula: This is a globally unique Panoptic ID; 1[·] is an indicator function that takes the value 1 if the condition is true, and 0 otherwise; To fix the hyperparameters, Pixels p The right neighbor and the down neighbor; , These represent the components of the optical flow field in the x-direction and the y-direction, respectively, which are the displacements of the pixel in the horizontal and vertical directions. When adjacent pixels belong to different instances, i.e., are located at the boundary, 1[·]=1, and the weight is reduced to exp( 10) ≈ 4.5 × 10 -5 That is, when the machine is close to zero, almost no smoothing penalty is applied; when adjacent pixels belong to the same instance, i.e., are inside, 1[·]=0, weight=1, and a full second-order smoothing penalty is applied. Simultaneously, the forward optical flow F t → t+1 and backward optical flow F t+1 → t Apply a mask to guide the smoothing loss and take the average to form a bidirectional instance smoothing constraint; Step 2: Decode the globally unique Panoptic ID; Input a uint8 type RGB mask image of [B, H, W, 3], and obtain a long type "ID card image" of [B, H, W] according to formula (5), ensuring that the numbers of any two different instances are different; Step 3: Calculate the instance boundary indicator diagram; Using semantic-level true boundary responses, we determine whether the right and bottom edges of a pixel are the same instance, and thus determine whether it is an edge. The implementation formula is as follows: in, These indicate whether pixel p is an instance boundary in the horizontal direction (x direction) and the vertical direction (y direction), respectively. Step 4: Calculate the exponentially decaying weighted graph; The boundary weights are reduced directly from 1.0 to approximately 0, which is achieved through the following mathematical formula: Inside the vehicle body, A value of 0 indicates a weight. At the vehicle body boundary, A value of 1 indicates a weight. ; Step 5: Bidirectional instance smoothing; This is achieved through the following loss function: The first item on the right is "forward optical flow using the mask tube of frame t", and the second item is "backward optical flow using the mask tube of frame t+1". This represents the mask for frame t. This represents the mask for the (t+1)th frame; Step 6: Multi-scale mask downsampling; Each layer is processed separately. The mask is downsampled in the nearest mode using F.interpolate, and then re-decoded layer by layer to generate a clear "Panoptic ID" for each layer. At the same time, a multi-scale weight design is adopted to enable instance smoothing only in the first layer, which requires the most structural prior.

2. The unsupervised optical flow estimation method based on instance mask guided smoothing loss according to claim 1, characterized in that, The fixed hyperparameters .

3. An electronic device, characterized in that, include: Processor and memory; The memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory to cause the electronic device to perform the method as described in any one of claims 1 to 2.

4. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 2.

5. A chip, characterized in that, include: A processor for retrieving and running a computer program from memory, causing a device on which the chip is mounted to perform the method as described in any one of claims 1 to 2.

6. A computer program product, characterized in that, The computer program product includes a computer storage medium storing a computer program, the computer program including instructions executable by at least one processor, which, when executed by the at least one processor, implement the method as described in any one of claims 1 to 2.

Images