A hybrid expert network-based optical flow estimation method and system

Abstract

The application discloses a mixed expert network-based optical flow estimation method and system, belonging to the field of computer vision and artificial intelligence. First, the input two frames of images are preprocessed, low-resolution features are extracted through a mixed expert feature extractor (MoEE) containing a sparse activation mechanism to reduce redundant calculation; then, dot product operation is performed on the low-resolution feature maps to construct a 4D correlation volume to capture pixel motion matching relationship; subsequently, a mixed expert updater (MoEU) is used to iteratively update the hidden state and regress the residual flow increment through a dynamic expert selection mechanism to optimize the optical flow accuracy; finally, a multi-scale upsampling module is used to reconstruct the high-resolution optical flow field to output the high-precision optical flow result. The algorithm realizes dynamic resource allocation through the MoE architecture, significantly reduces the calculation cost while ensuring the accuracy of optical flow estimation, and can adapt to resource-constrained scenarios such as automatic driving and unmanned aerial vehicles, and can balance efficient inference and flexible deployment.

Description

Technical Field

[0001] This application relates to the fields of computer vision and artificial intelligence, and in particular to a method and system for optical flow estimation based on a hybrid expert network. Background Technology

[0002] This invention relates to the fields of computer vision and artificial intelligence, specifically addressing the need for efficient and high-precision modeling of optical flow estimation tasks, and is applicable to resource-constrained scenarios such as autonomous driving, robot vision, and drone perception.

[0003] Optical flow estimation, a fundamental task in computer vision, aims to compute pixel-level motion vectors (optical flow fields) between adjacent frames of a video, providing core support for downstream motion analysis, target tracking, and scene understanding. Traditional optical flow estimation methods (such as the Horn-Schunck and Lucas-Kanade algorithms) are based on variational optimization or energy minimization principles, resulting in low computational efficiency and poor adaptability to complex scenes. In recent years, deep learning-based optical flow estimation methods (such as RAFT and its improved algorithms) have improved estimation accuracy by constructing 4D related volumes and cyclic update mechanisms, but they require a large amount of computational resources and memory, have a large number of parameters, and are intensive in floating-point operations, making them difficult to deploy on edge devices with limited computing power (such as vehicle terminals and drone controllers).

[0004] To balance accuracy and efficiency, existing technologies attempt two optimization paths: one is to design lightweight networks (such as Fast-FlowNet and LiteFlowNet) to accelerate inference by reducing parameters and computational load, but this is often accompanied by a significant decrease in accuracy; the other is to use Transformer variants or diffusion models (such as MambaFlow and FlowDiffuser) to improve modeling capabilities, but this still does not solve the problem of unreasonable global computing resource allocation - most methods adopt a "static uniform processing" mode, allocating the same computing resources to all image regions, resulting in redundant computation, poor scalability, and inability to adapt to the differences in computing power of different hardware.

[0005] Hybrid expert networks (MoEs) achieve global dynamic computation through the "sparse expert activation" mechanism, which can reduce computational costs while ensuring accuracy (such as the efficient resource allocation capability demonstrated by DeepSeekMoE). However, existing technologies have not yet effectively integrated them into the entire process of optical flow estimation: there is a lack of MoE module design for optical flow tasks (such as expert collaboration mechanisms for feature extraction and iterative optimization), and an end-to-end MoE architecture of "feature extraction-correlation matching-optical flow optimization-upsampling" has not been formed. As a result, optical flow estimation still faces the dual bottlenecks of "insufficient accuracy" and "low efficiency" in resource-constrained scenarios.

[0006] Therefore, there is an urgent need to develop an optical flow estimation algorithm and system based on a hybrid expert network. By combining dynamic resource allocation with lightweight modules, high accuracy can be guaranteed while reducing computational costs, enabling efficient deployment in resource-constrained scenarios. Summary of the Invention

[0007] This application provides a method and system for optical flow estimation based on a hybrid expert network. The technical solution is as follows:

[0008] According to one aspect of this application, a hybrid expert network-based optical flow estimation method is provided, comprising:

[0009] The two input images are preprocessed, and low-resolution features are extracted using a Hybrid Expert Feature Extractor (MoEE) to obtain a low-resolution feature map.

[0010] Dot product operation is performed on the low-resolution feature maps of the two frames to construct 4D related volume information for capturing motion matching relationships between pixels;

[0011] During the training and inference phases, the optical flow is iteratively optimized using a hybrid expert updater (MoEU), the hidden state is updated through a dynamic expert selection mechanism, and the residual flow increment is regressed.

[0012] The optimized low-resolution optical flow field is upsampled by a multi-scale upsampling module to reconstruct the high-resolution optical flow field and output the final high-precision optical flow estimation result.

[0013] Optionally, the preprocessing of the two input frames of images, extracting low-resolution features using a Hybrid Expert Feature Extractor (MoEE), includes:

[0014] The two input images are normalized to the range [0,1] and the image size is adjusted to a multiple of 8.

[0015] The Hybrid Expert Feature Extractor (MoEE) includes a depthwise separable convolutional layer, a layer normalization layer, and a Hybrid Expert (MoE) layer. The Hybrid Expert (MoE) layer has a shared expert and a routing expert. It dynamically allocates computing resources through a Top-K sparse activation strategy to extract a low-resolution feature map with a preset resolution dimension.

[0016] Optionally, the step of performing a dot product operation on the low-resolution feature maps of the two frames to construct 4D related volume information for capturing motion matching relationships between pixels includes:

[0017] Let the low-resolution feature maps of the two images be respectively f 1. f 2. Through the formula Calculate the 4D related volume, where (i,j) is the feature map. fThe pixel coordinates of 1, (k,l) are the feature map. f 2 pixel coordinates, d is the feature channel dimension, corr( f 1, f 2)(i,j,k,l) ​​represents the pixel matching value at the corresponding position in the 4D correlated volume.

[0018] Optionally, during the training and inference phases, the optical flow is iteratively optimized using a hybrid expert updater (MoEU), the hidden state is updated through a dynamic expert selection mechanism, and the residual flow increment is regressed, including:

[0019] Initialize hidden state h 0 and initial optical flow field μ 0;

[0020] The optimization steps are executed iteratively a preset number of times, including:

[0021] Based on the current optical flow field, matching features are found from the 4D related volume, and motion features are calculated by the motion encoder;

[0022] Motion features, the current hidden state, and the context features of the first frame image are input into the hybrid expert updater (MoEU), and a dynamic routing strategy is used to adaptively select experts to update the hidden state.

[0023] The updated hidden state is processed by an optical flow head network to regress the residual flow increment;

[0024] The optical flow field is updated based on the residual flow increment.

[0025] The optional dynamic routing strategy employs a gating mechanism, using the formula... Implementation, in which Let x be the routing weight matrix, and x be the features input to the Hybrid Expert Updater (MoEU). To activate the number of experts, the TopK function is used to select the k experts with the highest weights, and the Softmax function is used to normalize the weights of the selected experts.

[0026] Optional, also includes:

[0027] The hybrid expert updater (MoEU) adopts the ConvNeXt backbone network and combines a dynamic routing strategy to adaptively select the most relevant expert for optical flow optimization calculation based on the motion characteristics and context information of the input features in the temporal dimension.

[0028] The multi-scale upsampling module uses a combination of bilinear interpolation and convolutional layers. First, it performs preliminary upsampling of the low-resolution optical flow field through bilinear interpolation, and then refines the features of the optical flow field after preliminary upsampling through convolutional layers to gradually restore the high-resolution optical flow field.

[0029] On the other hand, a hybrid expert network-based optical flow estimation system is also provided, including:

[0030] The preprocessing and feature extraction module is used to preprocess the two input frames of images, extract low-resolution features through the Hybrid Expert Feature Extractor (MoEE), and output a low-resolution feature map.

[0031] The relevant volume construction module is used to perform dot product operations on the low-resolution feature maps of two frames of images to construct 4D relevant volume information;

[0032] The optical flow optimization module is used to iteratively optimize the optical flow during the training and inference phases using a hybrid expert updater (MoEU), update the hidden state through a dynamic expert selection mechanism, and regress the residual flow increment.

[0033] The high-resolution reconstruction module is used to upsample the optimized low-resolution optical flow field through the multi-scale upsampling module, reconstruct the high-resolution optical flow field, and output the final high-precision optical flow estimation result.

[0034] The optional system also includes a hardware adaptation module, which is used to adjust the number of experts in the hybrid expert network and the number of activated experts in the Top-K sparse activation strategy according to the computing power configuration of different hardware devices, so as to realize the deployment of the method on hardware with different performance.

[0035] On the other hand, a computer-readable storage medium is also provided, wherein a computer program is stored in the computer program, which is loaded and executed by a processor to implement the hybrid expert network-based optical flow estimation method as described above.

[0036] First, the two input image frames are preprocessed. Low-resolution features are extracted using a Hybrid Expert Feature Extractor (MoEE) with a sparse activation mechanism to reduce redundant computation. Then, dot product operations are performed on the low-resolution feature maps to construct a 4D correlation volume to capture pixel motion matching relationships. Subsequently, a Hybrid Expert Updater (MoEU) is used to iteratively update the hidden state and regress the residual flow increment through a dynamic expert selection mechanism to optimize optical flow accuracy. Finally, a high-resolution optical flow field is reconstructed through a multi-scale upsampling module to output high-precision optical flow results. This algorithm achieves dynamic resource allocation through the MoE architecture, significantly reducing computational costs while ensuring optical flow estimation accuracy. It is adaptable to resource-constrained scenarios such as autonomous driving and drones, balancing efficient inference and flexible deployment. Attached Figure Description

[0037] Figure 1 This is a diagram of the overall system architecture.

[0038] Figure 2 Here is a diagram of the MoEE module structure;

[0039] Figure 3 This is a diagram of the MoEE feature extraction process. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0041] In this article, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0042] Example 1

[0043] A hybrid expert network-based optical flow estimation method is provided, the method comprising:

[0044] Step 101: Preprocess the two input frames of images and extract low-resolution features using a Hybrid Expert Feature Extractor (MoEE) to obtain a low-resolution feature map.

[0045] Among them, the two input frames (Initial dimension is) (H is the image height, W is the image width, and 3 is the number of RGB channels) is preprocessed, and low-resolution features are extracted using a Hybrid Expert Feature Extractor (MoEE) to obtain a low-resolution feature map.

[0046] S11, normalize the two input frames to the range [0,1] and adjust the image size to a multiple of 8.

[0047] The two input images Standardization is performed by mapping pixel values ​​from the range [0,255] to the range [0,1] using the formula normalized_image=image / 255.0, eliminating the interference of pixel value magnitude differences on feature extraction. At the same time, the image size is adjusted to an integer multiple of 8 (e.g., if the original resolution is 1920×1080, it is adjusted to 1920×1080; if the original resolution is 1925×1083, it is cropped or padded to 1920×1080) to ensure that the resolution scaling has no remainder during subsequent feature extraction and to avoid feature misalignment.

[0048] S12, the hybrid expert feature extractor (MoEE) includes a depthwise separable convolutional layer, a layer normalization layer, and a hybrid expert (MoE) layer. The hybrid expert (MoE) layer has a shared expert and a routing expert. It dynamically allocates computing resources through a Top-K sparse activation strategy to extract a low-resolution feature map with a preset resolution dimension.

[0049] Specifically, the Hybrid Expert Feature Extractor (MoEE) is built on the ConvNeXt architecture, which includes depthwise separable convolutional layers, layer normalization, and hybrid expert (MoE) layers, and all expert networks adopt the ConvNeXt sub-network structure.

[0050] Depthwise separable convolutional layers. Using 3×3 convolutional kernels, standard convolution is split into "depthwise convolution (channel-wise convolution) + pointwise convolution (1×1 convolution)," reducing the number of parameters and computation by more than 75% compared to standard convolution, achieving lightweight feature extraction; Layer normalization. A layer normalization operation is added after the depthwise separable convolutional layers to standardize the mean and variance of each feature channel, improving training stability and feature consistency; Hybrid expert (MoE) layers. These include a shared expert (1, responsible for handling general features) and routing experts (N, N≥4, responsible for handling specific motion pattern features). Computational resources are dynamically allocated using a Top-K sparse activation strategy (K=2~4, adjustable according to computing power). The gating network first scores the input features, selecting the K experts with the highest weights. Only these K experts are activated for feature computation, while the remaining experts are not involved, reducing redundant computation; Feature output. Finally, a low-resolution feature map is extracted, which retains sufficient motion details while reducing the complexity of subsequent volumetric calculations.

[0051] Step 102: Perform dot product operation on the low-resolution feature maps of the two frames of images to construct 4D related volume information for capturing motion matching relationships between pixels.

[0052] Let the low-resolution feature maps of the two images be respectively f 1. f 2. Through the formula Calculate the 4D related volume, where (i,j) is the feature map. f The pixel coordinates of 1, (k,l) are the feature map. f 2 pixel coordinates, d is the feature channel dimension, corr( f 1, f 2)(i,j,k,l) ​​represents the pixel matching value at the corresponding position in the 4D correlated volume.

[0053] Specifically, a dot product operation is performed on the low-resolution feature maps F(I1) and F(I2) (i.e., f1=F(I1), f2=F(I2)) of the two frames to construct 4D related volume information for capturing motion matching relationships between pixels.

[0054] Let the low-resolution feature maps of the two images be respectively f 1. f 2. Through the formula Calculate the 4D related volume, where (i,j) is the feature map. f The pixel coordinates of 1, (k,l) are the feature map. f 2 pixel coordinates, d is the feature channel dimension, corr( f 1, f 2)(i,j,k,l) ​​represents the pixel matching value at the corresponding position in the 4D correlated volume. A larger value indicates... f1 The (i,j) pixel and f The stronger the motion correlation of (k,l) pixels in 2, the greater the overall dimension of the 4D related volume. (H / 8)×(W / 8)×(H / 8)×(W / 8) This directly provides pixel-level correlation basis for subsequent optical flow matching.

[0055] Step 103: During the training and inference phase, the optical flow is iteratively optimized using a hybrid expert updater (MoEU), the hidden state is updated through a dynamic expert selection mechanism, and the residual flow increment is regressed.

[0056] S31, Initialize hidden state h 0 and initial optical flow field μ 0.

[0057] Initialize hidden state (dimension) B×(H / 8)×(W / 8)×C B is the batch size, C is the number of hidden state channels (values ​​range from 128 to 512), and the matrix is ​​all zeros; initialize the initial optical flow field. (dimension) B×2×(H / 8)×(W / 8) , 2 (corresponding to the horizontal and vertical motion components respectively) is an all-zero matrix to ensure the consistency of parameters at the iteration starting point.

[0058] S32 searches for matching features from the 4D related volume based on the current optical flow field and calculates motion features through a motion encoder.

[0059] Based on the current optical flow field (t is the current iteration number, t∈[0,11]) Perform a "find match" operation from the 4D relevant volume. Based on the indicated motion direction, the potential matching pixel region f2 corresponding to each pixel f1 in the 4D relevant volume is located, and the relevant features of this region are extracted. The extracted relevant features are input into the motion encoder (composed of a 2-layer ConvNeXt sub-network), and the motion features M are calculated through convolution and activation operations. t (dimension) B×(H / 8)×(W / 8)×C This feature integrates pixel matching information with motion trend information.

[0060] S33, input the motion features, the current hidden state and the context features of the first frame image into the hybrid expert updater (MoEU), and use a dynamic routing strategy to adaptively select experts to update the hidden state.

[0061] The dynamic routing strategy employs a gating mechanism, using the formula... Implementation, in which Let x be the routing weight matrix, and x be the features input to the Hybrid Expert Updater (MoEU). To activate the number of experts, the TopK function is used to select the k experts with the highest weights, and the Softmax function is used to normalize the weights of the selected experts.

[0062] Specifically, the motion feature M t Current hidden state h t The contextual features C(I1) of the first frame image (the global contextual features additionally extracted from I1 by the MoEE module, dimension...) B×(H / 8)×(W / 8)×C Concatenate them to form the input feature x (dimension) B ×(H / 8)×(W / 8)×3C The input is a hybrid expert updater (MoEU); the MoEU uses a dynamic routing strategy to adaptively select experts to update hidden states. The dynamic routing strategy employs a gating mechanism, implemented through the formula G(x) = Softmax(TopK(Wg·x,k)), where Wg is the routing weight matrix (dimension 1). N×3C N is the total number of experts in the MoEU, N≥6, used to perform a linear transformation on the input feature x to obtain the score of each expert; x is the concatenated feature input to the Hybrid Expert Updater (MoEU), and k is the number of activated experts (k=2~3), which is adjusted according to the hardware computing power; the higher the computing power, the larger k can be set); the TopK function is used to select the k experts with the largest weights from the scores of N experts, and the Softmax function is used to normalize the weights of the selected k experts (ensuring that the sum of the weights is 1) to obtain the expert selection weights; only the selected k experts are activated to process the input feature x, and the output results of each expert are summed according to their corresponding weights to obtain the updated hidden state h. t+1(dimension) B×(H / 8)×(W / 8)×C ).

[0063] S34 processes the updated hidden state through the optical flow head network and regresses the residual flow increment.

[0064] The updated hidden state h is processed through an optical flow head network (consisting of one 3×3 convolutional layer and one 1×1 convolutional layer). t+1 The process involves processing. A 3×3 convolution is used to refine feature details, while a 1×1 convolution compresses the number of feature channels to 2 (corresponding to the horizontal and vertical directions), ultimately outputting the residual flow increment Δμt (dimension). B×2×(H / 8)×(W / 8) This increment represents the amount of correction made to the optical flow field in the current iteration.

[0065] S35 updates the optical flow field based on residual flow increment.

[0066] The optical flow field is updated based on the residual flow increment Δμt, and the update formula is μ t+1 =μt+Δμt, through 12 iterations, the optical flow deviation is gradually corrected, so that the optical flow field μ 12 The optical flow field after 12 iterations approximates the real optical flow distribution.

[0067] Step 104: Upsample the optimized low-resolution optical flow field using a multi-scale upsampling module to reconstruct the high-resolution optical flow field and output the final high-precision optical flow estimation result.

[0068] The optimized low-resolution optical flow field μ is obtained through a multi-scale upsampling module. 12 (dimension) B×2×(H / 8)×(W / 8) Upsampling is performed to reconstruct the high-resolution optical flow field, outputting the final high-precision optical flow estimation result. First, μ... 12 The optical flow field is upsampled from (H / 8×W / 8) resolution to (H / 4×W / 4) resolution, and the boundary pixel motion information is corrected through the upsampling module. Then, the optical flow field at (H / 4×W / 4) resolution is upsampled to (H / 2×W / 2) resolution to further refine the local motion details. Finally, the optical flow field at (H / 2×W / 2) resolution is upsampled to the original image resolution (H×W) to obtain the final high-resolution optical flow field μ. final (dimension) B ×2×H×W ); for μ final The format is converted to output a visual optical flow diagram (using the HSL color space, where hue represents the direction of motion and saturation represents the amplitude of motion), which can be directly used for downstream tasks such as dynamic obstacle tracking in autonomous driving and scene perception in drones.

[0069] Example 2

[0070] The above embodiments also include the following further details.

[0071] Content 1: The Hybrid Expert Updater (MoEU) adopts the ConvNeXt backbone network and combines a dynamic routing strategy to adaptively select the most relevant expert for optical flow optimization calculation based on the motion characteristics and context information of the input features in the temporal dimension.

[0072] The Hybrid Expert Updater (MoEU) uses the ConvNeXt backbone network as its infrastructure and combines dynamic routing strategies to achieve adaptive selection of experts in the time-series dimension. The specific design is as follows.

[0073] ConvNeXt backbone network configuration. The MoEU ConvNeXt backbone network consists of three ConvNeXt sub-blocks, each composed of "layer normalization → 7×7 depthwise convolution → 1×1 pointwise convolution → Gaussian error linear unit → 1×1 pointwise convolution → scaling layer (LayerScale) → DropPath". The 7×7 depthwise convolution expands the receptive field, capturing a wider range of motion-related information; the 1×1 pointwise convolution achieves dimensionality scaling (first scaling to 4 times the number of channels, then scaling back to the original number of channels), enhancing feature representation; the scaling layer (LayerScale) scales features channel-wise using a learnable parameter γ (initial value 1e-6), dynamically adjusting feature importance; DropPath (with a dropout rate of 0.1) prevents overfitting and improves model generalization ability. Expert adaptive selection logic for the temporal dimension.

[0074] In each iteration (t from 0 to 11), MoEU dynamically adjusts the expert selection strategy based on the motion characteristics of the input features (such as motion amplitude and direction changes) and contextual information (such as image edges and texture density). When the input features correspond to "large displacement motion regions" (such as fast-moving vehicles), the gating network prioritizes activating experts that are good at handling large displacement motions (optimizing expert division of labor through loss feedback during the training phase); when the input features correspond to "complex rotational motion regions" (such as rotating robotic arms), it prioritizes activating experts that are good at handling rotational motions; temporally, the first 3 iterations prioritize activating "global motion experts" to quickly correct overall optical flow deviations; the last 9 iterations prioritize activating "local detail experts" to refine small-scale motion errors, realizing the optimization logic of "global coarse adjustment → local fine adjustment".

[0075] Content 2: The multi-scale upsampling module adopts a combination of bilinear interpolation and convolutional layers. First, it performs preliminary upsampling of the low-resolution optical flow field through bilinear interpolation, and then refines the features of the optical flow field after preliminary upsampling through convolutional layers to gradually restore the high-resolution optical flow field.

[0076] The multi-scale upsampling module adopts a cascaded approach combining bilinear interpolation and convolutional layers to gradually recover the high-resolution optical flow field in three steps, as follows.

[0077] Step 1. Low resolution → Medium-low resolution (H / 8×W / 8 → H / 4×W / 4) First, use bilinear interpolation to adjust μ. 12 Initial upsampling is performed to double the resolution. Bilinear interpolation is calculated using the formula output(i,j)=sum{k,l}input(k,l)×w(k,l,i,j) (where w(k,l,i,j) is the interpolation weight, positively correlated with pixel distance), ensuring the continuity of the optical flow field after upsampling. Then, a 3×3 convolutional layer (padding=1, output channels=2) is used to refine the features of the initially upsampled optical flow field, correcting the "blurring effect" caused by interpolation, and outputting an optical flow field μ with a resolution of (H / 4×W / 4). up1 .

[0078] Step 2. Perform bilinear interpolation upsampling on \(μ_{up1}\) from low to medium resolution (H / 4×W / 4 → H / 2×W / 2), doubling the resolution. Refine the features using a 3×3 convolutional layer (padding=1, output channels=2), while simultaneously introducing the (H / 2×W / 2) resolution contextual features from the first frame (obtained through downsampling by the MoEE module). Concatenate the contextual features with the upsampled optical flow field and then convolve them to enhance the correlation between the optical flow field and the image content, outputting an (H / 2×W / 2) resolution optical flow field μ. up2 .

[0079] Step 3. Medium to high resolution → original resolution (H / 2×W / 2 → H×W) for μ up2 Bilinear interpolation upsampling is performed to double the resolution. A final thinning layer (padding=1, output channels=2) is then applied, along with an upsampling mask (the boundary correction mask output from the MoEU iteration process) to specifically correct the optical flow values ​​at image edges and in occluded areas, avoiding estimation bias in boundary pixel motion. The final output is a high-resolution optical flow field μ at the original resolution. final .

[0080] Furthermore, this application embodiment also includes a third component: optimization strategies for the training phase.

[0081] The aim is to supplement the key design of the training-related loss function to ensure that the algorithm converges to the optimal state during the training phase.

[0082] A combination of "Endpoint Error (EPE) loss + multi-scale loss" is used as the training loss. The mean Euclidean distance between the predicted and actual optical flows is calculated. The EPE losses for four resolutions (H / ×W / 8), (H / 4×W / 4), (H / 2×W / 2), and (H×W) are calculated respectively, and the total loss L is obtained by summing them according to weights (0.1, 0.2, 0.3, 0.4). total To ensure the model learns effective features at all resolutions, the AdamW optimizer and training parameters were configured with an initial learning rate of 1e-4, a weight decay coefficient of 1e-5, and a cosine annealing scheduling method (decreasing by a cosine function after each training round, with a minimum learning rate of 1e-6). The training dataset consisted of FlyingChairs3D (100,000 image pairs) + Sintel training set (12,000 image pairs), with a batch size of 8 and 100 training rounds. Training was stopped and the model parameters were saved when the EPE loss of the validation set (10% of the data in the Sintel training set) did not decrease for 10 consecutive rounds.

[0083] Example 3

[0084] An optical flow estimation system based on a hybrid expert network includes...

[0085] The preprocessing and feature extraction module is used to preprocess the two input frames of images, extract low-resolution features through the Hybrid Expert Feature Extractor (MoEE), and output a low-resolution feature map.

[0086] The relevant volume construction module is used to perform dot product operations on the low-resolution feature maps of two frames of images to construct 4D relevant volume information;

[0087] The optical flow optimization module is used to iteratively optimize the optical flow during the training and inference phases using a hybrid expert updater (MoEU), update the hidden state through a dynamic expert selection mechanism, and regress the residual flow increment.

[0088] The high-resolution reconstruction module is used to upsample the optimized low-resolution optical flow field through the multi-scale upsampling module, reconstruct the high-resolution optical flow field, and output the final high-precision optical flow estimation result.

[0089] Furthermore, the system also includes a hardware adaptation module, which is used to adjust the number of experts in the hybrid expert network and the number of activated experts in the Top-K sparse activation strategy according to the computing power configuration of different hardware devices, so as to realize the deployment of the method on hardware with different performance.

[0090] Example 4

[0091] The following description, in conjunction with the accompanying drawings, describes the embodiments.

[0092] Figure 1 This is a diagram of the overall system architecture.

[0093] This diagram illustrates the entire architecture of an optical flow estimation system based on a hybrid expert network. The core of the system consists of five modules: an input layer, a feature extraction layer, a relevant volume construction layer, an iterative optimization layer, and an output layer. The input layer receives two frames of images to be processed (labeled "inp"); the feature extraction layer processes the input images using a hybrid expert feature extractor (MoEE) and outputs low-resolution motion features (labeled "motionfeature") and context features (labeled "context"); the correlation volume construction layer generates a 4D correlation volume (labeled "4DCostVolume") based on the motion features of the two frames, providing a basis for pixel motion matching; corr is the feature correlation calculation operation; the iterative optimization layer uses "MoEBlock" as the core component and iteratively updates the hidden state (labeled "h'(t)") and initial optical flow (labeled "InitialFlow") using a hybrid expert updater (MoEU) to gradually correct optical flow deviations; the output layer outputs the final optical flow (labeled "FinalFlow") after multi-scale processing, clearly presenting the interaction logic of the MoEE and MoEU modules, and fully covering the end-to-end process of "image input → feature extraction → optical flow optimization → result output". Here, flow refers to iterative optical flow. InitialFlow is the initial optical flow. All N-1 iterations of the initial optical flow are called flow, which is the optical flow obtained during the intermediate iteration process. The final optical flow is obtained after N iterations.

[0094] Figure 2 This is a structural diagram of the MoEE module.

[0095] This diagram breaks down the internal structure of the Hybrid Expert Feature Extractor (MoEE) in detail. Based on the ConvNeXt architecture, it consists of a depthwise separable convolutional layer, a layer normalization (LayerNorm), a hybrid expert (MoE) layer, and residual connection components, arranged sequentially from input to output. The depthwise separable convolutional layer uses a 3×3 convolutional kernel (labeled "DWConv3x3") to achieve lightweight feature extraction. Layer normalization is used to standardize the feature distribution and improve training stability. The MoE layer is the core component, containing shared experts and routed experts. Both shared and routed experts adopt a ConvNeXt sub-block structure (containing a 1×1 pointwise convolution "PWConv1x1", a Gaussian error linear unit, a scaling layer "LayerScale", and a path dropping "DropPath"), and dynamically select activation experts through a Top-K sparse activation strategy. Furthermore, the module prevents overfitting through DropPath, ultimately outputting enhanced low-resolution features, fully demonstrating the feature processing logic of the MoEE module: "convolutional extraction → normalization → dynamic expert calculation".

[0096] Figure 3 This is a diagram of the MoEE feature extraction process.

[0097] This diagram illustrates the process by which the MoEE module extracts image and contextual features in a flowchart format. The input is the original image (labeled "Input"), which undergoes progressive feature refinement through cascading processing of multiple MoEEBlocks. Each MoEEBlock contains a convolutional layer (labeled "Conv", with both 3×3 and 7×7 kernels), stride control (labeled "stride"), and layer normalization (labeled "+LN"). The preceding MoEEBlocks (e.g., 1-8) achieve feature dimensionality reduction and global information capture through large convolutional kernels and reasonable strides (e.g., stride=2), while subsequent MoEEBlocks (e.g., 10-11) focus on optimizing local detail features. The entire process does not rely on a pre-trained model. Through the synergistic effect of multi-stage convolution and MoE layers, the final output is a low-resolution feature map that meets the preset resolution, intuitively demonstrating the transformation path of the MoEE module from the original image to effective features.

[0098] To make the content of this invention clearer and easier to understand, the content of this invention will be described in detail below with reference to specific calculation embodiments.

[0099] Example 1: Algorithm Implementation.

[0100] Step 1: Feature extraction stage.

[0101] Input two frames of images After processing by the MoEE module:

[0102] a) Image preprocessing: normalize to the range [0,1], and adjust the size to a multiple of 8;

[0103] b) Feature extraction: Low-resolution features are extracted using the MoEE module.

[0104]

[0105] c) Correlation volume construction: Calculate 4D correlation volume:

[0106]

[0107] Step 2: Optical flow update phase, initialize hidden state Harmony flow field The following steps are executed iteratively (N times in total).

[0108] Motion characteristic calculation:

[0109] ;

[0110] Hidden state update:

[0111] ;

[0112] MoEU employs a dynamic routing strategy, adaptively selecting experts based on input features.

[0113] Residual Regression:

[0114] ;

[0115] Optical flow update:

[0116] ;

[0117] Step 3: Output phase.

[0118] Reconstructing high-resolution optical flow fields using multi-scale upsampling modules:

[0119] ;

[0120] Calculation Example 2: MoE Module Implementation.

[0121] Expert Network Design. There are three main choices for expert networks: FFN, traditional CNN, and ConvNeXt. Extensive experiments have shown that ConvNeXt performs best.

[0122] Gating network design. Using linear transformation plus Softmax:

[0123] ;

[0124] in Let N be the routing weight matrix, where N is the total number of experts and k is the number of activated experts.

[0125] Sparse activation strategy. Top-K sparse activation is employed, activating only the K experts with the highest weights, significantly reducing computational cost.

[0126] .

[0127] This invention successfully achieves efficient computation and high-precision modeling of optical flow estimation by introducing a hybrid expert network (MoE) architecture and combining a dynamic routing mechanism with a lightweight feature extraction module.

[0128] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0129] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for optical flow estimation based on a hybrid expert network, characterized in that, The method includes: The two input images are preprocessed, and low-resolution features are extracted using a Hybrid Expert Feature Extractor (MoEE) to obtain a low-resolution feature map. This preprocessing includes standardizing the two input images to the range [0,1] and adjusting the image size to a multiple of 8. The MoEE comprises a depthwise separable convolutional layer, a layer normalization layer, and a MoE layer. The MoE layer includes a shared expert and a routing expert, and dynamically allocates computational resources using a Top-K sparse activation strategy to extract a low-resolution feature map with a preset resolution dimension. Perform a dot product operation on the low-resolution feature maps of the two image frames to construct 4D related volume information for capturing motion matching relationships between pixels; the dot product operation on the low-resolution feature maps of the two image frames to construct 4D related volume information for capturing motion matching relationships between pixels includes: assuming the low-resolution feature maps of the two image frames are f1 and f2, and using the formula... Calculate the 4D related volume, where (i,j) are the pixel coordinates of feature map f1, (k,l) are the pixel coordinates of feature map f2, d is the feature channel dimension, and corr(f1,f2)(i,j,k,l) ​​are the pixel matching values ​​at the corresponding positions in the 4D related volume. During the training and inference phases, a Hybrid Expert Updater (MoEU) is used to iteratively optimize the optical flow, updating the hidden state through a dynamic expert selection mechanism and regressing the residual flow increment. This process includes: initializing the hidden state h0 and the initial optical flow field μ0; iteratively executing optimization steps a preset number of times, including: finding matching features from the 4D relevant volume based on the current optical flow field and calculating motion features using a motion encoder; inputting the motion features, the current hidden state, and the context features of the first frame image into the Hybrid Expert Updater (MoEU), and adaptively selecting experts to update the hidden state using a dynamic routing strategy; processing the updated hidden state through an optical flow head network to regress the residual flow increment; and updating the optical flow field based on the residual flow increment. The dynamic routing strategy employs a gating mechanism, using the formula G(x)=Softmax(TopK(W)). g x,k)) is implemented, where Let x be the routing weight matrix, x be the feature input to the hybrid expert updater (MoEU), k be the number of activated experts, the TopK function is used to select the k experts with the largest weights, and the Softmax function is used to normalize the weights of the selected experts. The optimized low-resolution optical flow field is upsampled by a multi-scale upsampling module to reconstruct the high-resolution optical flow field and output the final high-precision optical flow estimation result.

2. The method according to claim 1, characterized in that, Also includes: The hybrid expert updater (MoEU) adopts the ConvNeXt backbone network and combines a dynamic routing strategy to adaptively select the most relevant expert for optical flow optimization calculation based on the motion characteristics and context information of the input features in the temporal dimension. The multi-scale upsampling module uses a combination of bilinear interpolation and convolutional layers. First, it performs preliminary upsampling of the low-resolution optical flow field through bilinear interpolation, and then refines the features of the optical flow field after preliminary upsampling through convolutional layers to gradually restore the high-resolution optical flow field.

3. A hybrid expert network-based optical flow estimation system, used to implement the hybrid expert network-based optical flow estimation method as described in claim 1 or 2, characterized in that, include: The preprocessing and feature extraction module is used to preprocess the two input frames of images, extract low-resolution features through a hybrid expert feature extractor (MoEE), and output a low-resolution feature map; it is also used to normalize the two input frames of images to the range [0,1] and adjust the image size to a multiple of 8; the hybrid expert feature extractor (MoEE) includes a depthwise separable convolutional layer, a layer normalization layer, and a hybrid expert (MoE) layer. The hybrid expert (MoE) layer has a shared expert and a routing expert, and dynamically allocates computing resources through a Top-K sparse activation strategy to extract a low-resolution feature map with a preset resolution dimension; The relevant volume construction module is used to perform dot product operations on the low-resolution feature maps of two frames of images to construct 4D relevant volume information; it is also used to define the low-resolution feature maps of the two frames as f1 and f2, and then use the formula... Calculate the 4D related volume, where (i,j) are the pixel coordinates of feature map f1, (k,l) are the pixel coordinates of feature map f2, d is the feature channel dimension, and corr(f1,f2)(i,j,k,l) ​​are the pixel matching values ​​at the corresponding positions in the 4D related volume. The optical flow optimization module is used to iteratively optimize optical flow during the training and inference phases using a hybrid expert updater (MoEU), updating the hidden state through a dynamic expert selection mechanism, and regressing the residual flow increment. It is also used to initialize the hidden state h0 and the initial optical flow field μ0. The module iteratively executes optimization steps a preset number of times, including: finding matching features from the 4D relevant volume based on the current optical flow field, calculating motion features using a motion encoder; inputting the motion features, the current hidden state, and the context features of the first frame image into the hybrid expert updater (MoEU), and adaptively selecting experts to update the hidden state using a dynamic routing strategy; processing the updated hidden state through an optical flow head network to regress the residual flow increment; and updating the optical flow field based on the residual flow increment. The dynamic routing strategy employs a gating mechanism, using the formula G(x)=Softmax(TopK(W g x,k)) is implemented, where Let x be the routing weight matrix, x be the feature input to the hybrid expert updater (MoEU), k be the number of activated experts, the TopK function is used to select the k experts with the largest weights, and the Softmax function is used to normalize the weights of the selected experts. The high-resolution reconstruction module is used to upsample the optimized low-resolution optical flow field through the multi-scale upsampling module, reconstruct the high-resolution optical flow field, and output the final high-precision optical flow estimation result.

4. The system according to claim 3, characterized in that, The system also includes a hardware adaptation module, which is used to adjust the number of experts in the hybrid expert network and the number of activated experts in the Top-K sparse activation strategy according to the computing power configuration of different hardware devices, so as to realize the deployment of the method on hardware with different performance.

5. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which is loaded and executed by a processor to implement the hybrid expert network-based optical flow estimation method as described in claim 1 or 2 above.

Images