Optical flow estimation method based on dynamic mask cross-modal distillation

Abstract

The invention discloses an optical flow estimation method based on dynamic mask cross-modal distillation. The method comprises the steps of data preprocessing, dynamic mask optimization, adaptive temperature cross-modal distillation and optical flow prediction. The method comprises the following steps: firstly, synchronously preprocessing an image sequence and an event stream, encoding the event stream into a 3D voxel grid and enhancing the 3D voxel grid; a dynamic mask is generated based on event high-dynamic features, and loss end-to-end optimization is carried out through triple comparison: student features and dynamically masked teacher features are used as positive samples, and original unprocessed teacher features are used as negative samples; and then the static mask subjected to value inversion processing is fused into the adaptive temperature cross-modal distillation process. According to the method, a robust cross-modal distillation thought is used for reference, an event-driven dynamic mask is used as a knowledge selection mechanism for the first time, and a sample-perceived temperature adaptive strategy is combined, so that the problem of knowledge migration distortion caused by modal difference in a high-dynamic scene in a traditional method is solved, and the estimation robustness and generalization ability in a complex environment are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically to an optical flow estimation method based on dynamic mask cross-modal distillation. Background Technology

[0002] Optical flow estimation is one of the core tasks in computer vision, estimating pixel-level motion vectors from continuous visual data. It is a key underlying technology for real-time systems such as autonomous driving and robot navigation. Traditional methods are mainly based on image sequences, relying on the photometric consistency assumption and spatial smoothness constraints to solve the motion field. However, in complex scenes with high-speed motion, sudden changes in illumination, or motion blur, image information is prone to degradation, leading to a significant decrease in the accuracy of optical flow estimation and severely limiting its applicability in real-world environments.

[0003] The emergence of event cameras offers a new possibility for solving the above problems. These sensors possess characteristics such as high temporal resolution, high dynamic range, and low latency, effectively capturing details of fast-moving events and compensating for the shortcomings of traditional image methods. However, event cameras output asynchronous, sparse streams of events, rather than regular intensity image frames; their data format differs fundamentally from that of images. This modal gap makes it difficult to directly apply traditional image-based optical flow algorithms—whether single-modal modeling or multi-modal fusion strategies—to event data, limiting the full realization of the advantages of event cameras.

[0004] To synergistically leverage the high dynamic characteristics of event data and the semantic richness of images, researchers have primarily explored two technical paths: In the multimodal fusion path, the model still relies on the synchronous input of both image and event data during the inference phase. This presents challenges in practical applications, including high hardware synchronization costs and deployment complexity, particularly limiting its practical potential in single-event source scenarios. In the knowledge distillation path, existing technologies are mostly rooted in knowledge transfer mechanisms between homogeneous modalities (such as image-to-image). When simply transplanted to heterogeneous modalities like images and events, they encounter bottlenecks in knowledge transfer efficiency due to the fundamental mismatch in their feature spaces. More importantly, in complex dynamic scenarios, the image teacher model itself may suffer from degenerate prediction errors due to motion blur, overexposure, or underexposure. Existing distillation frameworks generally lack effective error identification and isolation mechanisms, causing such harmful noise to directly contaminate the event student model through the distillation process, thus weakening the robustness inherent in the event data itself.

[0005] Due to the two core obstacles of strong inference dependence and insufficient reliability of cross-modal distillation, existing technologies lack a cross-modal distillation scheme that can effectively coordinate modal differences and achieve high-quality knowledge transfer. There is an urgent need for an innovative technical approach that can robustly and efficiently transfer useful knowledge from image modalities while preserving the high dynamic perception capability of event data, thereby comprehensively improving the accuracy and stability of optical flow estimation for event modalities in complex scenes. Summary of the Invention

[0006] To address the aforementioned shortcomings in existing technologies, this invention aims to provide an optical flow estimation method based on dynamic masking and adaptive cross-modal distillation. This method aims to offer a flexible and universal knowledge distillation framework that can easily adapt to different teacher-student model combinations. Its core objective is to effectively solve the problem of feature distribution differences between image and event modalities through a collaboratively optimized distillation framework. This allows for the efficient transfer of semantic and structural knowledge rich in image modalities while preserving the high dynamic characteristics of event data, ultimately significantly improving the accuracy and robustness of optical flow estimation for event modalities in complex scenes.

[0007] An optical flow estimation method based on dynamic mask cross-modal distillation includes the following steps:

[0008] Step 1: Data processing. For event stream input data, the event information is divided into different time scales based on the exposure timestamp of the RGB camera and processed into the network input format in sequence.

[0009] Step 2: Construct a distillation framework that includes an image teacher model with parameter freezing, a pre-trained event student model, and a dynamic masking network;

[0010] Step 3: Design a lightweight encoder-decoder structure mask network to generate pixel-level dynamic masks based on the spatiotemporal characteristics of event data; optimize the mask network through triplet contrastive loss to enable it to effectively identify dynamic regions that are semantically similar to event information.

[0011] Step 4: Invert the dynamic mask value obtained in Step 3 to obtain the static mask, which aims to preserve the high-quality semantic information of the static region of the image; construct a cross-modal feature similarity matrix in the static reliable region, introduce adaptive temperature contrastive learning loss to achieve refined knowledge transfer; at the same time, combine optical flow estimation task loss to train the student model end-to-end.

[0012] Step 5: During the inference phase, only the trained event student model is used to process the input event stream, and the pixel-level dense optical flow field is output to achieve efficient and robust optical flow estimation.

[0013] The specific details of each step are as follows:

[0014] In step 1, the input event stream is voxelized. Assume the event stream contains several events within a time window; divide the time dimension into several equally spaced time bins, and linearly accumulate the events into their corresponding time bins according to their timestamps, thus forming a three-dimensional voxel grid representation, which is used as the input to the subsequent neural network; the RGB image remains a two-dimensional raster input, strictly aligned with the event stream in time.

[0015] The multimodal input data in step 1 are as follows: ;

[0016] in, and In order to be in Time and Continuous image frames at time points, for Frame to The event stream within the exposure time. for Frame to The specific format for representing the event stream within the exposure time is as follows: ,in The timestamp of the moment the event occurred. for The location at which the event is generated at any given time, where and These represent the maximum length of the row and column, respectively. express The polarity of events generated at any given time. This represents increasing the brightness by a threshold. This indicates that the brightness is reduced by one threshold.

[0017] Then, the event streams divided into time periods need to be processed as follows: and Discretized A series of consecutive time bins were encoded into a 3D voxel mesh. In the form of ), where This indicates the batch size during the training process. and These represent the height and width of the original input data, respectively, and the specific formulas are as follows:

[0018]

[0019] in and These represent the start and end times of the event data, respectively. Indicates the number of event data. The range is The event data after voxelization is represented as and , and serve as subsequent network input;

[0020] The image teacher model described in step 2 is responsible for extracting high-quality image features; the event student model is used to process event stream data; the dynamic masking network adopts an encoder-decoder structure and fuses multi-scale features through skip connections; the pre-trained student model refers to the model weights obtained by preliminary training on the target training set. This pre-training process enables the student model to have basic feature extraction capabilities, providing a stable optimization starting point for subsequent cross-modal distillation; the pre-training stage uses a standard optical flow estimation loss function to ensure that the student model has a good initial state in the target domain data distribution;

[0021] In step 3, two consecutive event streams are... Event features extracted via student model encoder (where feature dimension) Input a dynamic masking network; this network performs the following operations: first, it inputs event features... The network is spliced ​​along the channel direction to fuse spatiotemporal information from different time points. The spliced ​​features are then fed into a stacked module consisting of convolutional layers, batch normalization layers, and activation functions for processing. The encoder part of the network uses strided convolutions to downsample features and gradually extract high-level semantic features. The decoder part restores spatial resolution through transposed convolutions and skip connections to retain more detailed information. The intermediate layers of the network all contain batch normalization layers and ReLU activation functions to ensure the stability of the training process, and finally generate a dynamic mask that is strictly aligned with the dimension of the feature map. Value greater than or equal to The region is considered a dynamic region. End-to-end training of the dynamic masking network is achieved by minimizing the triple contrastive loss, which is specifically defined as follows:

[0022]

[0023] in The cosine similarity function is used. These represent the extracted features from the student model, the original features from the teacher model, and the teacher model features optimized by masking filtering. The boundary parameter controls the minimum distance between positive and negative samples. This loss function guides the masking network to accurately identify dynamic regions in the scene by pushing negative sample pairs further apart and bringing positive sample pairs closer together, providing a high-quality feature alignment foundation for subsequent cross-modal distillation.

[0024] In step 4, after obtaining the dynamic mask, considering that static regions usually contain richer image semantic information, including texture and edges, this method performs value inversion processing on the dynamic mask: pixel values ​​below a set threshold are processed... The region is determined to be a static reliable feature region (preserving semantic information), which is higher than or equal to The region is then considered a dynamic region (dominated by motion information); thus enabling the discrimination and separation of dynamic and static regions, laying the foundation for subsequent feature alignment and knowledge distillation focusing on static regions;

[0025] Next, in the obtained static reliable region ( Within this scope, the adaptive temperature cross-modal distillation method employed specifically includes the following details:

[0026] First, the feature matching difficulty measurement module calculates the degree of matching between student features and teacher features in the current batch. The matching difficulty... Defined by the following formula:

[0027]

[0028] in This indicates the total number of samples in the batch. and Representing the first Given the high degree of information overlap and spatiotemporal continuity between consecutive video frames in the optical flow estimation task, and the high semantic correlation between their student and teacher feature vectors, this method, to simplify calculations and avoid redundancy, defaults to selecting the feature vector of the first frame in each sample sequence as the representative calculation unit when calculating batch matching difficulty; The value of is in the range of [0,2], and the larger the value, the higher the difficulty of feature matching;

[0029] Secondly, an adaptive temperature regulation mechanism is adopted to dynamically adjust the temperature parameters based on the difficulty of feature matching. Comparative learning and optimization were conducted, focusing on temperature parameters. The adaptive calculation formula is as follows

[0030]

[0031] in The reference temperature is the basic level used to control the distillation intensity. This is an adjustment coefficient used to control the degree to which the matching difficulty affects the temperature; when the feature matching difficulty is high, the temperature parameter... Automatic reduction to increase attention to difficult samples when matching is relatively easy. The intensity is increased accordingly to maintain a relatively moderate distillation intensity.

[0032] Furthermore, the improved cross-modal distillation loss function, executed within this statically reliable region, consists of two parts: intra-modal contrastive loss and cross-modal contrastive loss. Cross-modal feature alignment is achieved through a multi-level contrastive learning mechanism, and its expression is:

[0033]

[0034] Intramodal contrastive loss ensures the coherence and stability of feature representations within the same modality by constructing semantic consistency comparisons within both the teacher and student models. Its expression is:

[0035]

[0036] The cross-modal contrastive loss achieves alignment between the teacher and student models in the semantic feature space by constructing contrastive learning at cross-modal semantic correspondence positions. Its expression is as follows:

[0037]

[0038] in, For the first Normalized vectors of masking features of a sample teacher model. For the first Normalized vectors of student features from a sample, For adaptive temperature parameters.

[0039] Furthermore, the specific process of simultaneously optimizing the student model and dynamic masking network parameters based on the optical flow estimation task in step 4 includes first inputting the data-augmented event voxel grid into the student model for forward inference, obtaining multi-scale optical flow prediction results through the feature extraction network, correlation calculation module, and optical flow regression head, and then comparing the optical flow prediction results with the true optical flow labels to calculate the optical flow estimation supervision loss, using the sequence-weighted L1 loss function:

[0040]

[0041] in This is the sequence weight decay factor. To predict the total length of the sequence, This is the index for the current prediction step.

[0042] Finally, combining optical flow estimation supervision loss, dynamic masking loss, and cross-modal distillation loss, the AdamW optimizer is used to backpropagate the total loss, and gradient pruning technique is used to simultaneously update the parameters of the student model and the dynamic masking network. During the training phase, the total loss function is expressed as:

[0043]

[0044] In step 5, during the inference phase, only the trained event student model is retained and used. The input event stream data is fed into this model, and the optical flow estimation results are directly output.

[0045] Compared with the prior art, the beneficial effects of the present invention are:

[0046] 1. This invention addresses the problems of semantic sparsity, motion ambiguity sensitivity, and difficulties in cross-modal knowledge transfer in optical flow estimation tasks using event data. It proposes a dynamic mask-guided cross-modal distillation method. By separating dynamic and static regions through dynamic masking, feature alignment is performed in the static, reliable region to improve optical flow estimation accuracy. An adaptive temperature contrastive learning mechanism is employed to overcome modal differences and enhance the model's robustness under high-speed motion. A dual-path distillation strategy is combined to simultaneously optimize the optical flow estimation and feature alignment tasks.

[0047] 2. This method effectively improves the optical flow estimation accuracy of event data in complex scenes, suppresses the error propagation of the image teacher model, and realizes high-quality optical flow prediction that relies solely on event data, providing reliable technical support for the practical application of event cameras in the field of motion perception. Attached Figure Description

[0048] Figure 1 This is a flowchart illustrating the entire process of the algorithm proposed in this invention.

[0049] Figure 2 This is a diagram of the dynamic masking network structure and a schematic diagram of the dynamic masking loss proposed in this invention.

[0050] Figure 3 The figure shows the comparison results with advanced image and event optical flow estimation algorithms on the MultiFlow dataset. Detailed Implementation

[0051] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0052] refer to Figures 1-3This paper presents an optical flow estimation method based on dynamic mask cross-modal distillation, with a workflow divided into a training phase and a prediction phase. In the training phase, a student model is first supervised pre-trained on the training dataset to enable it to understand basic features of the target dataset. Then, the teacher model parameters are fixed, and cross-modal knowledge distillation is performed based on dynamic masking and an adaptive temperature mechanism; finally, the optimized event student model is retained. In the prediction phase, based on the trained model parameters, the optical flow estimation task for events in different scenarios is tested. To enhance the generalization ability of the trained model,

[0053] A series of data augmentation techniques were used:

[0054] (1) Random flipping: Horizontal and vertical flipping are performed randomly according to probability.

[0055] (2) Random rotation: Rotate clockwise or counterclockwise by 90, 180 or 270 degrees according to probability.

[0056] (3) Random cropping: randomly crop the region of interest of the image. In the experiment, the size was set to 512 pixels in length and width.

[0057] The above data augmentation techniques are applied to both image frames and event streams. Furthermore, since event data lacks a frame structure, it is first voxelized before the corresponding operations are performed.

[0058] The multimodal input data in the method are as follows: ;

[0059] in, and In order to be in Time and Continuous image frames at time points, for Frame to The event stream within the exposure time. for Frame to The specific format for representing the event stream within the exposure time is as follows: ,in The timestamp of the moment the event occurred. for The location at which the event is generated at any given time, where and These represent the maximum length of the row and column, respectively. express The polarity of events generated at any given time. This represents increasing the brightness by a threshold. This indicates that the brightness is reduced by one threshold.

[0060] Then, the event streams divided into time periods need to be processed as follows: and Discretized A series of consecutive time bins were encoded into a 3D voxel mesh. In the form of ), where This indicates the batch size during the training process. and These represent the height and width of the original input data, respectively, and the specific formulas are as follows:

[0061]

[0062] in and These represent the start and end times of the event data, respectively. Indicates the number of event data. The range is The event data after voxelization is represented as and , and serve as subsequent network input.

[0063] After the event fluidization process is completed, according to Figure 1 The flowchart illustrates the construction of the proposed algorithm network. In this method, the image teacher model uses the UniMatch general optical flow estimation model based on the Transformer architecture, while the event student model employs the TMA architecture with temporal memory capabilities. It should be noted that although the teacher and student models differ significantly in architecture (the former is based on Transformer, the latter on convolutional neural networks), forming a heterogeneous cross-modal distillation framework, this method is still applicable to knowledge transfer between homogeneous models. In practical applications, the combination of teacher-student models with homogeneous architectures may achieve better feature alignment results.

[0064] After the framework is built, the specific data processing flow in the model is as follows: and The inputs are fed into the teacher model and the student model respectively to obtain the teacher features of the output image. and student characteristics (where feature dimension) );

[0065] Then the characteristics of the event and Input a dynamic masking network. This network performs the following operations: First, it inputs event features... The network is spliced ​​along the channel direction to fuse spatiotemporal information from different time points. The spliced ​​features are then fed into a stacked module consisting of convolutional layers, batch normalization layers, and activation functions for processing. The encoder part of the network uses strided convolutions to downsample features and progressively extract high-level semantic features; the decoder part restores spatial resolution through transposed convolutions and skip connections to retain more detailed information. All intermediate layers of the network include batch normalization layers and ReLU activation functions to ensure the stability of the training process. Finally, a dynamic mask that is strictly aligned with the dimension of the feature map is generated. In this dynamic mask In this paper, end-to-end training of the dynamic masking network is achieved by minimizing the triple contrastive loss. The specific definition of this loss function is as follows:

[0066]

[0067] in The cosine similarity function is used. These represent the extracted features from the student model, the original features from the teacher model, and the teacher model features optimized by masking filtering. The boundary parameter controls the minimum distance between positive and negative samples. This loss function works by pushing negative sample pairs further apart and bringing positive sample pairs closer together.

[0068] After obtaining the dynamic mask, considering that static regions typically contain richer image semantic information, including texture and edges, this method performs value inversion on the dynamic mask: pixel values ​​below a set threshold are processed... Regions that are considered static reliable feature regions (preserving semantic information) are classified as such. The region is then considered a dynamic region (dominated by motion information), thus enabling the differentiation and separation of dynamic and static regions, laying the foundation for subsequent feature alignment and knowledge distillation focusing on static regions.

[0069] Next, in the obtained static reliable region ( Within this scope, the adaptive temperature cross-modal distillation method employed specifically includes the following details:

[0070] First, the feature matching difficulty measurement module calculates the degree of matching between student features and teacher features in the current batch. The matching difficulty... Defined by the following formula:

[0071]

[0072] in This indicates the total number of samples in the batch. and Representing the first Given the high degree of information overlap and spatiotemporal continuity between consecutive video frames in the optical flow estimation task, and the high semantic correlation between their student and teacher feature vectors, this method, in order to simplify calculations and avoid redundancy, defaults to selecting the feature vector of the first frame in each sample sequence as the representative calculation unit when calculating batch matching difficulty. The value range is [0,2], and the larger the value, the higher the difficulty of feature matching.

[0073] Secondly, an adaptive temperature regulation mechanism is adopted to dynamically adjust the temperature parameters based on the difficulty of feature matching. Comparative learning and optimization were conducted, focusing on temperature parameters. The adaptive calculation formula is:

[0074]

[0075] in The reference temperature is the basic level used to control the distillation intensity. This is an adjustment coefficient used to control the degree to which the matching difficulty affects the temperature; when the feature matching difficulty is high, the temperature parameter... Automatic reduction to increase attention to difficult samples when matching is relatively easy. The intensity is increased accordingly to maintain a relatively moderate distillation intensity.

[0076] Furthermore, the improved cross-modal distillation loss function, executed within this statically reliable region, consists of two parts: intra-modal contrastive loss and cross-modal contrastive loss. Cross-modal feature alignment is achieved through a multi-level contrastive learning mechanism, and its expression is:

[0077]

[0078] Intramodal contrastive loss ensures the coherence and stability of feature representations within the same modality by constructing semantic consistency comparisons within both the teacher and student models. Its expression is:

[0079]

[0080] The cross-modal contrastive loss achieves alignment between the teacher and student models in the semantic feature space by constructing contrastive learning at cross-modal semantic correspondence positions. Its expression is as follows:

[0081]

[0082] in, For the first Normalized vectors of masking features of a sample teacher model For the first Normalized vectors of student features from a sample, For adaptive temperature parameters.

[0083] To enhance the synergy between the distillation process and the target task, this method introduces optical flow estimation supervision during training. Specifically, a data-enhanced event voxel grid is input into the student model for forward inference, sequentially passing through a feature extraction network, a correlation calculation module, and an optical flow regressor to obtain multi-scale optical flow prediction results. Subsequently, the optical flow prediction values ​​at each scale are compared with the corresponding true optical flow labels, and a sequence-weighted L1 loss function is used to calculate the optical flow estimation supervision loss, the expression of which is as follows:

[0084]

[0085] in This is the sequence weight decay factor. To predict the total length of the sequence, This is the index for the current prediction step.

[0086] Finally, combining optical flow estimation supervision loss, dynamic masking loss, and cross-modal distillation loss, the AdamW optimizer is used to backpropagate the total loss, and gradient pruning technique is used to simultaneously update the student model and dynamic masking network parameters. During the training phase, the total loss function is expressed as:

[0087]

[0088] In the experimental section, to verify the effectiveness of the proposed method, it is compared with several representative state-of-the-art optical flow estimation models in the prior art. These comparative models each have their own characteristics in terms of architecture design, input data utilization, and motion modeling, which are summarized below:

[0089] UniMatch is a unified geometric matching model designed to achieve general and powerful correspondence finding across various visual tasks, such as optical flow and stereo matching. Its key feature is its use of the cross-attention mechanism from the Transformer architecture, performing feature matching at a global or local scale. Specifically, in the optical flow estimation task, this model uses image modalities to predict optical flow values.

[0090] BFlow achieves the estimation of continuous-time dense pixel trajectories from event and image data through collaborative iterative optimization of Bézier curves and multi-view related volumes.

[0091] TMA is an optical flow estimation method focused on using event data for temporal motion aggregation. Its key feature is that the model emphasizes extracting and fusing motion information across different time scales from the event stream.

[0092] As shown in Figure 3, this method not only draws on the advantages of the teacher model (first row), but also avoids the areas where the teacher model's predictions are poor (second row), ultimately achieving better prediction performance than the original student model and teacher model (third row).

[0093] This method uses the publicly available MultiFlow dataset to evaluate the performance of the proposed continuous-time optical flow method. Specifically, the evaluation interval is selected from 0.4 seconds to 0.9 seconds, which contains dense ground truth pixel trajectories. To compare with traditional two-frame optical flow methods, the method uses endpoint error (EPE) as the evaluation metric, which calculates only the average Euclidean distance between the predicted and actual displacements at time 0.9 seconds. A smaller endpoint error indicates more accurate optical flow predictions and better performance. Table 1 shows the test results on the MultiFlow dataset.

[0094] Table 1: Optical flow estimation results on MultiFlow

[0095]

[0096] In the input data, "I" represents image data, "E" represents event data, and "E+I" indicates multimodal data composed of both images and events. Since UniMatch and TMA do not provide pre-trained weights on the MultiFlow dataset, their results on this dataset are obtained through fine-tuning.

[0097] As can be seen from the results in Table 1, this method achieves optimal performance when using only event data.

[0098] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0099] The above description is merely a preferred embodiment of the present invention and does not constitute any limitation on the present invention. Although embodiments of the present invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. An optical flow estimation method based on dynamic mask cross-modal distillation, characterized in that, The method includes the following steps: Step 1: Data preprocessing. The input image sequence and event stream are synchronized and encoded. For the input event stream data, the event information is divided into different time scales based on the camera's exposure timestamp. Step 2: Construct a framework model based on the data characteristics obtained in Step 1. The framework model includes: a parameter-frozen image teacher model, a pre-trained event student model, and a dynamic masking network based on event flow features. Step 3: Based on the high dynamic range characteristics of event information, a dynamic mask is generated using a dynamic masking network, and the dynamic mask is optimized using triple contrastive loss. The positive sample pair consists of student features extracted from the student model and teacher model features filtered by the mask, while the negative sample pair consists of the student features and the original unfiltered teacher model features, in order to identify dynamic regions that are closer to the semantic information of student features. Step four: The dynamic mask determined in step three is inverted to obtain a static mask. Adaptive temperature cross-modal distillation is performed using the high-quality teacher features and student features filtered by the static region. During this process, the parameters of the student model and the dynamic mask network are simultaneously optimized for the optical flow estimation task. Step 5: Use the student model trained in Step 4 to process the voxelized event data and output a high-precision dense optical flow field.

2. The optical flow estimation method based on dynamic mask cross-modal distillation according to claim 1, characterized in that, Step one is as follows: S1. First, read the input image data and event stream; for the event stream, discretize it into several time bins according to the time dimension, and process the data for them; S2. Process the input data obtained in step S1. For the event stream, discretize it into several time bins according to the time dimension and encode them into the form of 3D voxel mesh as the input of the subsequent network. S3. Perform a series of random rotations and cropping operations on the event voxel grid and image frames obtained in step S2 to achieve data augmentation.

3. The optical flow estimation method based on dynamic mask cross-modal distillation according to claim 2, characterized in that, The method for 3D voxel mesh encoding in step S2 is as follows: S2.1, Divide the time dimension of the event information into equal parts. Each timebox contains a certain number of events (the number may vary), and the timestamps of the event information are normalized to 0. between; S2.2 For a coordinate in a timebox obtained in step S2.1, perform a time-weighted sum of all events in that timebox at that coordinate. The closer the timestamp of an event is to the timestamp of the timebox, the greater its weight. S2.

3. Process the entire event information using the method in step S2.2 to obtain the most primitive event representation.

4. The optical flow estimation method based on dynamic mask cross-modal distillation according to claim 1, characterized in that, In step three, the construction method and processing flow of the dynamic masking network are as follows: S3.1 The dynamic mask network adopts an encoder-decoder architecture with skip connections. It takes voxelized event data as input, extracts features through the stacking of convolutional layers, batch normalization layers and ReLU activation functions, and finally outputs a pixel-level dynamic mask image through the sigmoid function. S3.

2. End-to-end training of the masked network is achieved by minimizing the triple contrast loss: positive sample pairs consist of student features and masked teacher features, while negative sample pairs consist of student features and original teacher features. The optimization objective is to reduce the similarity of positive sample pairs and increase the distance between negative sample pairs. The specific form of the loss function is: in The cosine similarity function is used. These represent the extracted features from the student model, the original features from the teacher model, and the teacher model features optimized through masking filtering. This is a boundary parameter that controls the minimum distance between positive and negative samples.

5. The optical flow estimation method based on dynamic mask cross-modal distillation according to claim 1, characterized in that, The specific process of adaptive temperature cross-modal distillation in step four is as follows: S4.

1. After inverting the optimized dynamic mask value, a static mask is obtained and applied to the output features of the teacher model to filter out dynamic noise regions, thus obtaining the masked teacher features. S4.2 Input the statically masked teacher features and student features into the global comparison learning module, calculate the cosine similarity matrix between the two within the static reliable region, and dynamically adjust the temperature parameter based on the feature matching difficulty. Comparative learning and optimization were conducted, focusing on temperature parameters. The adaptive calculation formula is as follows in As the reference temperature, For adjustment coefficients, This is a measure of the difficulty of feature matching. The cross-modal contrastive distillation loss consists of two parts: intra-modal contrastive loss and cross-modal contrastive loss. It achieves cross-modal feature alignment through a multi-level contrastive learning mechanism. The intra-modal contrastive loss aims to enhance semantic consistency within the same modality, while the cross-modal contrastive loss aims to achieve semantic feature alignment between different modalities. The specific calculation formula is as follows: in, For the first Normalized vectors of masking features of a sample teacher model. For the first Normalized vectors of student features from a sample, For adaptive temperature parameters.

6. The optical flow estimation method based on dynamic mask cross-modal distillation according to claim 1, characterized in that, Step four involves the specific process of simultaneously optimizing the student model and dynamic mask network parameters based on the optical flow estimation task, including: S4.3 Input the data-enhanced event voxel grid into the student model for forward inference, and obtain multi-scale optical flow prediction results through the feature extraction network, correlation calculation module and optical flow regression head; S4.

4. Compare the optical flow prediction results with the actual optical flow labels, calculate the optical flow estimation supervision loss, and use the sequence-weighted L1 loss function: in This is the sequence weight decay factor. To predict the total length of the sequence, Index for the current prediction step; S4.

5. Combining optical flow estimation supervision loss, dynamic masking loss and cross-modal distillation loss, the AdamW optimizer is used to backpropagate the total loss, and the parameters of the student model and the dynamic masking network are updated synchronously through gradient pruning technique. S4.6 Repeat the above training process, periodically evaluate the optical flow estimation accuracy of the model on the validation set, and save the optimal parameters for final deployment.

7. The optical flow estimation method based on dynamic mask cross-modal distillation according to claim 1, characterized in that, The optical flow estimation process mentioned in step five is as follows: S5.1 Load the optimal training parameters obtained in step S4.6 into the network, including the student model weights and the dynamic mask network weights; S5.2 Input the preprocessed event voxel mesh data into the network obtained in step S5.1, and obtain the final optical flow estimation result through forward inference; S5.3 Post-process the output optical flow field, including boundary filling, outlier filtering and spatial smoothing operations, to improve the visual quality of optical flow estimation; S5.4 Convert the processed optical flow field into a standard stream format file for easy subsequent visual applications and analysis.

8. The optical flow estimation method based on dynamic mask cross-modal distillation according to claim 1, characterized in that, The method employs a multi-stage training strategy: Phase 1: Pre-train the student model using only optical flow supervised loss to establish basic optical flow estimation capabilities; The second stage involves jointly optimizing the student model and the dynamic masking network. The dynamic masking network employs a random initialization strategy and introduces cross-modal distillation loss to achieve knowledge transfer and performance improvement.

Images