Self-supervised optimization and adaptive deployment method for optical flow network

Abstract

The invention discloses a self-supervised optimization and self-adaptive deployment method for an optical flow network, and the method enables the effective region endpoint of a KITTI data set and the error of a self-built complex scene data set to be effectively reduced through the collaborative design of a self-supervised optimization method and a self-adaptive computing power frame based on the accumulation and circulation consistency constraint of a reverse optical flow, and improves the precision. A feature multiplexing mechanism enables the inference speed of embedded equipment to be improved and the memory occupation to be reduced, a dynamic pyramid iteration module supports flexible balance precision and time delay improvement efficiency according to a computing power threshold, a progressive training strategy effectively inhibits error accumulation, the cross-domain migration precision is improved, and the optical flow direction estimation accuracy in a low-resolution scene is improved. The problem of performance degradation caused by a fixed structure in the prior art is solved, and an optical flow estimation solution considering real-time performance and high precision is provided for mobile platforms such as an unmanned aerial vehicle.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of network supervision, and particularly relates to a method for self-supervised optimization and adaptive deployment of an optical flow network. BACKGROUND

[0002] As a core characterization method for describing the projection of three-dimensional motion in a two-dimensional plane, the optical flow technology can effectively infer the object motion trajectory by analyzing the pixel-level motion trend, and has important application value in target tracking, visual navigation and video frame interpolation. However, when applied to rapidly developing mobile platforms such as unmanned aerial vehicles and unmanned vehicles, the balance between real-time performance and computing resources in embedded scenarios requires higher requirements. The existing high-precision optical flow model has a high computing cost due to the complex network structure, which makes it difficult to meet the lightweight deployment requirements. Although the traditional lightweight model has the advantage of low computing amount, it still faces the bottleneck of insufficient cross-domain migration ability, especially in real scenes.

[0003] In the prior art, the acquisition of optical flow data is different from the conventional image annotation. The optical flow ground truth is difficult to obtain directly through manual annotation. Although laser radar point cloud registration technology can be used to assist optical flow calculation, the generated data density is insufficient to support the modeling requirements of dense optical flow, and the multi-sensor collaborative operation is easy to introduce noise interference and registration errors. Therefore, current research generally relies on virtual scene generated training data. However, although deep network models show strong "simulation to reality" migration ability, lightweight models are difficult to effectively migrate the optical flow inference ability learned in virtual scenes to real scenes due to the limitation of network depth, resulting in a significant decrease in accuracy in actual applications.

[0004] To solve the above problems, there is an urgent need for a solution that can improve the cross-domain adaptability of lightweight optical flow models without ground truth annotation, solve the performance degradation problem of lightweight models in real scenes due to data distribution differences, and especially solve the technical gap in terms of real-time performance and accuracy. SUMMARY

[0005] The technical solution of the present application solves the following technical problems caused by the fixed structure of the existing deep learning optical flow network model:

[0006] 1. Insufficient model adaptability: traditional deep learning optical flow network models need to design network structures separately for different computing power devices (such as cloud servers and embedded platforms), resulting in high development and maintenance costs, and difficulty in balancing high accuracy and low latency requirements;

[0007] 2. Low utilization of computing resources: in the inference process of the existing deep learning optical flow network, repeated calculation and redundant storage of feature maps cause waste of computing resources, especially in resource-constrained edge devices (such as Jetson Orin NX), which significantly reduces real-time performance;

[0008] 3. Weak dynamic scene adaptation capability: The fixed parameter amount model in the existing deep learning optical flow network cannot dynamically adjust the calculation path according to the complexity of the input scene (such as low dynamic range and high texture density area), resulting in low energy efficiency ratio.

[0009] To solve the above technical problems, the technical scheme of the present application provides an optical flow network self-supervised optimization and adaptive deployment method, comprising the following steps:

[0010] Feature extraction is performed on the continuous first continuous image and the second continuous image to obtain 1 / 8 resolution feature maps and 1 / 16 resolution feature maps;

[0011] The 1 / 16 resolution feature map of the first continuous image is taken as a query, and the 1 / 16 resolution feature map of the second continuous image is taken as a key and a value to calculate an attention weight matrix, and normalization is performed to obtain a 1 / 16 resolution initial optical flow field;

[0012] Single-layer detail compensation is adopted, an additional global matching branch is introduced to enhance the 1 / 8 resolution feature map of the single layer, and a 1 / 8 resolution compensation optical flow field is generated through cross-frame feature similarity calculation;

[0013] Pyramid iterative detail compensation is adopted, the 1 / 16 resolution initial optical flow field is sampled to 1 / 8 resolution, fused with the 1 / 8 resolution compensation optical flow field to generate a coarse optical flow field, the 1 / 8 resolution feature map corresponding to the second continuous image is aligned to the feature space of the first continuous image, and a residual optical flow is calculated, and the optical flow field is updated according to the residual optical flow and the coarse optical flow field;

[0014] A cache queue is defined to store the 1 / 8 resolution feature map corresponding to the previous frame continuous image, and the queue is empty at initialization, at which time the 1 / 8 resolution feature map of the first continuous image is completely extracted, when the queue state is not empty, the 1 / 8 resolution feature map of the first continuous image is directly reused when processing the current frame continuous image, and if it is empty, the 1 / 8 resolution feature map corresponding to the current frame continuous image is extracted again;

[0015] When the device memory occupancy exceeds a threshold value, the feature maps in the queue are serialized into a disk file, and high-frequency access data is preferentially loaded when the features are reused;

[0016] The single-layer detail compensation is trained until convergence, the pre-posed parameters are frozen, the Refine module is introduced and the weights are initialized, the Refine module is jointly trained to update only the parameters until the convergence condition is met, and the multi-level resolution switching of the pyramid iterative detail compensation is performed, and the optimization path is activated from bottom to top along the pyramid structure;

[0017] According to the device computing power threshold dynamic configuration optimization path, the input scene complexity is analyzed in real time, and the texture density is calculated to realize the self-supervised optimization and adaptive deployment of the optical flow network.

[0018] Preferably, the feature extraction is performed by a shallow CNN backbone network.

[0019] Preferably, the attention weight matrix is calculated by using a cross-attention mechanism.

[0020] Preferably, the normalization is performed by using a Softmax function.

[0021] Preferably, the 1 / 8 resolution feature map corresponding to the second contact image is aligned to the feature space of the first continuous image by feature distortion.

[0022] Preferably, the convergence condition includes: the EPE drop rate is less than 0.5% and the SNR improvement amplitude is less than 1dB for 3 consecutive epochs.

[0023] The technical scheme of the present application provides a method for self-supervised optimization and adaptive deployment of an optical flow network. Through the collaborative design of the self-supervised optimization method and the adaptive computing power framework, based on the reverse optical flow accumulation and the cyclic consistency constraint, the precision is improved with low error in the effective area endpoints of the KITTI dataset and the self-built complex scene dataset. The feature reuse mechanism improves the inference speed of the embedded device, reduces the memory occupation, and supports the dynamic pyramid iteration module to flexibly balance the precision and time delay according to the computing power threshold to improve the efficiency. The progressive training strategy effectively suppresses error accumulation, improves cross-domain migration accuracy, and improves the accuracy of optical flow direction estimation in low-resolution scenes. The present application solves the performance degradation problem caused by the fixed structure of the prior art, and provides an optical flow estimation solution for mobile platforms such as unmanned aerial vehicles, which takes into account real-time performance and high precision. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 A multi-scale feature extraction and double-path matching flowchart is provided for the embodiments of the present application.

[0025] Figure 2 A feature caching and reuse mechanism flowchart is provided for the embodiments of the present application.

[0026] Figure 3 A progressive training strategy flowchart is provided for the embodiments of the present application. DETAILED DESCRIPTION

[0027] The application will be further described below with reference to specific embodiments. It should be understood that these embodiments are only used to illustrate the application and not to limit the scope of the application. In addition, it should be understood that those skilled in the art can make various modifications or changes to the application after reading the content of the application, and these equivalent forms also fall within the scope defined by the appended claims.

[0028] The terms mentioned in the embodiments of the application and the explanations are as follows:

[0029] 1.CNN(Convolutional Neural Network)

[0030] A deep learning model widely used in image processing tasks (such as classification, detection, segmentation), which extracts local features through convolutional layers and reduces computational complexity using pooling layers. In this application, CNN is used as the backbone network to extract multi-scale features required for optical flow estimation.

[0031] 2.EPE(End-Point Error)

[0032] A core indicator to measure the accuracy of optical flow estimation, defined as the Euclidean distance between the predicted optical flow and the true optical flow at the pixel position. The formula is:

[0033] EPE=(u pred -u gt ) 2 +(v pred -v gt ) 2

[0034] Where u and v represent the horizontal and vertical components of the optical flow, respectively. The smaller the EPE, the more accurate the optical flow estimation.

[0035] 3.Jetson Orin NX

[0036] A high-performance embedded computing platform launched by NVIDIA, designed for edge AI applications, with powerful GPU acceleration capabilities suitable for real-time inference tasks (such as optical flow estimation). In this application, it is used to verify the real-time performance and efficiency of the model.

[0037] 4.Refine module

[0038] A lightweight sub-network structure used to iteratively optimize the initial optical flow field, which gradually improves the accuracy of optical flow through residual correction. It is commonly used in optical flow estimation models (such as RAFT, GMFlow). In this application, multiple Refine modules are stacked using a progressive training strategy.

[0039] 5.FPS(Frames Per Second)

[0040] The speed of the system processing video or image sequence is measured in frames per second (FPS). The higher the FPS, the better the real-time performance of the system. The present invention requires the embedded device to maintain ≥25FPS to meet the real-time scene requirements of drones and the like.

[0041] 6.KITTI

[0042] A commonly used public dataset in the field of autonomous driving, containing label information such as optical flow, depth, and stereo matching in real-world scenarios. The present invention uses the KITTI dataset to verify the performance of the model in real-world scenarios (such as EPE).

[0043] 7.SNR(Signal-to-Noise Ratio)

[0044] The ratio of signal strength to noise strength, used to evaluate data quality or the stability of model output. In the present invention, SNR is one of the convergence criteria for progressive training (training is terminated when the improvement amplitude is <1dB).

[0045] 8.epoch

[0046] In the process of deep learning training, an epoch represents the complete traversal of the training dataset by the model. In the present patent, "three consecutive epochs" refers to the model stopping training when it fails to achieve performance improvement for three consecutive times during the training process.

[0047] 9.ScaleFlow++

[0048] An optical flow estimation algorithm based on multi-scale feature fusion, which improves the accuracy of large-scale motion estimation by introducing a dynamic scaling mechanism. In the present invention, it is used as one of the base methods for generating pseudo-labels.

[0049] 10.FlowFormer++

[0050] An optical flow estimation algorithm based on the Transformer architecture, which models long-range dependencies through self-attention mechanisms and is suitable for complex scene optical flow estimation. In the present invention, it is combined with ScaleFlow++ and GMFlow to generate pseudo-labels.

[0051] 11.GMFlow

[0052] An optical flow estimation algorithm based on the idea of global matching, which balances accuracy and efficiency through a combination of global search and local optimization. In the present invention, the pseudo-labels of GMFlow are combined with other algorithms to generate high-confidence supervision signals.

[0053] The embodiment of the present invention provides a method for self-supervised optimization and adaptive deployment of an optical flow network, comprising the following steps:

[0054] As Figure 1 shown in Figure 1, step 1: multi-scale feature extraction and dual-path matching, specifically including:

[0055] 1.1 Hierarchical feature extraction

[0056] • Input consecutive image pairs (I1, I2), feature extraction is performed through a shallow CNN backbone network (such as Modified ResNet-18);

[0057] • The first branch outputs a feature map F1 with a resolution of 1 / 8 (size W / 8xH / 8x64), which is used for local optimization and feature reuse;

[0058] • The second branch outputs a feature map F2 with a resolution of 1 / 16 (size W / 16xH / 16x128), which is used for global matching.

[0059] 1.2 Global optical flow initialization

[0060] In the 1 / 16 feature layer (F2), a cross-attention mechanism is used, taking F2 of I1 as the query (Query), F2 of I2 as the key (Key) and value (Value), and calculating the attention weight matrix;

[0061] After normalization by Softmax, the initial optical flow field O1 (resolution 1 / 16) is generated, covering a wide range of motion estimation.

[0062] 1.3 Single-layer detail compensation branch

[0063] In the 1 / 8 feature layer (F1), an additional global matching branch is introduced, which uses a lightweight Transformer block (such as 2 layers of self-attention + FFN) to enhance F1;

[0064] Through cross-frame feature similarity calculation, a compensation optical flow field O2 (resolution 1 / 8) is generated to supplement the loss of details at low resolution.

[0065] 1.4 Pyramid iterative detail compensation

[0066] Stacking Refine modules (structure: Conv→ReLU→Deformable Conv→Upsample) at the 1 / 8 resolution scale: upsample the initial optical flow field O1 to 1 / 8 resolution, and fuse it with O2 to generate a coarse optical flow field O coarse .

[0067] Through the feature warping (Warping) operation, align F1 of I2 to the feature space of I1, calculate the residual optical flow ΔO; update the optical flow field: O refined = O coarse + ΔO.

[0068] As Figure 2 shown in FIG. 2, step 2: feature cache and reuse mechanism, specifically includes:

[0069] 2.1 Cross-frame feature cache

[0070] Define a cache queue to store 1 / 8 feature maps F n-1 of the previous frame (I 1prev ), which is empty at initialization, and F 1prev of I1 needs to be extracted completely;

[0071] When processing I n , check the queue status: if not empty, directly reuse F 1prev ; if empty, extract and update the queue.

[0072] 2.2 Dynamic memory management

[0073] When the device memory usage exceeds the threshold (e.g., 80%): serialize the feature maps in the queue into a disk file (format: HDF5);

[0074] Adopt asynchronous I / O scheduling strategy, preferentially load high-frequency access data during feature reuse to reduce latency.

[0075] As Figure 3 shown in FIG. 3, step 3: progressive training strategy, specifically includes:

[0076] 3.1 Train the single-layer detail compensation module to convergence;

[0077] 3.2 Freeze the front-end module parameters, add a Refine module and initialize the weights (e.g., He initialization);

[0078] 3.3 Joint training of the new module, only update its parameters until the convergence condition is met:

[0079] ① EPE drop rate <0.5% (for 3 consecutive epochs);

[0080] ② Signal-to-noise ratio (SNR) improvement amplitude <1 dB.

[0081] 3.4 Multi-level resolution switching, activate the optimization path from bottom to top along the pyramid structure:

[0082] 1 / 8 scale: stack 2 Refine modules (performance limit of experimental calibration);

[0083] 1 / 4 scale: upsample to 1 / 4 resolution through transpose convolution, add 1 Refine module;

[0084] 1 / 2 scale: further upsample and optimize, finally output the full-resolution optical flow field.

[0085] Step 4: Adaptive computing power allocation, specifically including:

[0086] 4.1 Equipment computing power adaptation

[0087] Dynamically configure the optimization path based on the device's computing power threshold (such as the 15W power consumption limit of the Jetson Orin NX):

[0088] Low computing power mode: Only two optimization iterations at the 1 / 8 scale are enabled (EPE=2.1, latency=40ms);

[0089] High-performance computing mode: Enable full pyramid path (1 / 8→1 / 4→1 / 2, EPE=1.7, latency=80ms).

[0090] 4.2 Scene Adaptive Optimization

[0091] Real-time analysis of input scene complexity, and calculation of texture density (the proportion of pixels with gradient magnitude > 0.3):

[0092] If the texture density is >0.6, activate the 1 / 4 and 1 / 2 scale optimization;

[0093] If the texture density is ≤0.6, only retain the 1 / 8 scale optimization.

[0094] Example 1: Validation of a self-supervised optimization method based on reverse optical flow accumulation.

[0095] 1. Dataset and Training Methods

[0096] Data construction: The KITTI dataset is integrated with a self-built real dataset (5 scenes covering high / low light, smooth / bumpy recording, and different frame sampling frequencies). Forward / backward optical flow is generated through ScaleFlow++, FlowFormer++ and GMFlow. Cyclic consistency masks are used to filter 70% of the effective regions to generate pseudo-labels.

[0097] Self-supervised training: Using pseudo-labels as supervision signals, combined with reverse optical flow accumulation and data augmentation strategies (such as illumination perturbation and dynamic blur simulation), the model's ability to generalize to real-world scenes is optimized.

[0098] 2. Core Validation Results

[0099] Accuracy improvement: The EPE in the effective region of KITTI decreased to 0.49 (Table 1), and the reverse accumulation + cyclic consistency reduced the EPE of the self-built dataset to 0.55 (Table 2).

[0100] Table 1 Comparison of optical flow performance in the KITTI dataset

[0101]

[0102] Table 2 Application Tests of Backward Accumulation and Cyclic Consistency

[0103]

[0104] Cross-domain migration: After optimization, the EPE of optical flow on the KITTI dataset was reduced by 7.3% (2.42 vs. 2.61), while the accuracy of the Sintel dataset was slightly reduced due to illumination interference;

[0105] Module effectiveness: Ablation experiments showed that reverse accumulation, cycle consistency and data augmentation contributed to a reduction in EPE of 0.04, 0.08 and 0.07, respectively.

[0106] Example 2: Performance test of adaptive computing power framework.

[0107] 1. Model optimization and training strategies

[0108] Architecture improvements: A global matching branch was added to the 1 / 8 resolution feature layer to mitigate the loss of detail at low resolutions;

[0109] Progressive training: Freeze the parameters of the previous modules in stages, and only update the weights of the newly added Refine modules (learning rate 0.0001) to suppress error accumulation.

[0110] 2. Dynamic computing power allocation and efficiency verification

[0111] Feature reuse: On Jetson Orin NX, inference speed is improved by 14.1% for 1 / 16 resolution models and memory usage is reduced by 42% for 1 / 8 resolution models;

[0112] Accuracy-delay balance:

[0113] At high resolution (320×896), stacking four Refine modules reduced the EPE from 1.63 to 0.93;

[0114] At low resolution (240×320), the EPE of the 1 / 8 resolution model is reduced by 5.9% compared to the 1 / 16 resolution model;

[0115] Adaptive strategy: Dynamically select the number of Refine module stacking times based on the device's computing power (e.g., enable 2 iterations under 15W power consumption) to achieve Pareto optimality in accuracy-latency.

[0116] This invention provides a self-supervised optimization and adaptive deployment method for optical flow networks. Through the collaborative design of a self-supervised optimization method and an adaptive computing power framework, it significantly improves the performance of lightweight optical flow models in real-world scenarios. In terms of accuracy, based on reverse optical flow accumulation and cyclic consistency constraints, the effective endpoint error (EPE) of the KITTI dataset is reduced to 0.49, and the EPE of a self-built complex scene dataset is reduced by 18.3%. In terms of efficiency, the feature reuse mechanism improves the inference speed of embedded devices by 14.1% and reduces memory usage by 42%. The dynamic pyramid iteration module supports flexible balancing of accuracy and latency based on computing power thresholds (e.g., EPE is optimized to 1.7 under 15W power consumption). In addition, the progressive training strategy effectively suppresses error accumulation, improves cross-domain transfer accuracy by 12.4%, and improves the accuracy of optical flow direction estimation by 37% in low-resolution scenes. This provides a solution for optical flow estimation that balances real-time performance (≥25FPS) and high accuracy for mobile platforms such as drones.

[0117] The points to be protected in this embodiment of the invention are as follows:

[0118] 1) Protection scope of self-supervised optimization method: The specific implementation steps of generating pseudo-labels through reverse optical flow accumulation and cyclic consistency masking include multi-algorithm fusion, effective region screening (70% threshold) and data augmentation strategy.

[0119] 2) Protection range of dynamic pyramid iteration module: a strategy to dynamically adjust the number of Refine module stackings based on device computing power thresholds (such as power consumption limits), and a multi-level optimization path design from bottom to top (1 / 8→1 / 2 resolution).

[0120] 3) Protection scope of the feature cache reuse system: The implementation mechanism of cross-frame feature cache (cache queue management, disk serialization storage) and asynchronous I / O scheduling method ensure real-time performance and memory efficiency.

[0121] 4) Scope of protection for progressive training strategy: training process that freezes the parameters of previous modules in stages, updates only the weights of newly added modules, and convergence criteria (EPE decrease rate < 0.5%).

Claims

1. A self-supervised optimization and adaptive deployment method for optical flow networks, characterized in that, Includes the following steps: Feature extraction is performed on the first consecutive image and the second connected image to obtain a 1 / 8 resolution feature map and a 1 / 16 resolution feature map; The attention weight matrix is ​​calculated using the 1 / 16 resolution feature map of the first continuous image as the query and the 1 / 16 resolution feature map of the second connected image as the key and value, and then normalized to obtain the 1 / 16 resolution initial optical flow field. A single-layer detail compensation is adopted, and an additional global matching branch is introduced to enhance the 1 / 8 resolution feature map, which is a single layer. A 1 / 8 resolution compensated optical flow field is generated by cross-frame feature similarity calculation. Pyramid iterative detail compensation is used to sample the initial optical flow field at 1 / 16 resolution to 1 / 8 resolution, and fuse it with the compensated optical flow field at 1 / 8 resolution to generate a coarse optical flow field. The 1 / 8 resolution feature map corresponding to the second consecutive image is aligned to the feature space of the first consecutive image, and the residual optical flow is calculated. The optical flow field is updated according to the residual optical flow and the coarse optical flow field. Define a buffer queue to store the 1 / 8 resolution feature map corresponding to the previous frame of consecutive images. The queue is empty at the time of initialization. At this time, the 1 / 8 resolution feature map of the first consecutive image is completely extracted. When processing the current frame of consecutive images, if the queue is not empty, the 1 / 8 resolution feature map of the first consecutive image is directly reused. If it is empty, the corresponding 1 / 8 resolution feature map of the current frame of consecutive images is extracted again. When the device memory usage exceeds the threshold, the feature maps in the queue are serialized into disk files, and high-frequency access data is loaded first when features are reused. For training with single-layer detail compensation until convergence, freeze the pre-parameters, introduce the Refine module and initialize the weights, and jointly train the Refine module to update the parameters until the convergence condition is met. For multi-level resolution switching with pyramid iterative detail compensation, activate the optimization path from bottom to top along the pyramid structure. The system dynamically configures and optimizes paths based on the device's computing power threshold, analyzes the complexity of the input scene in real time, and calculates texture density to achieve self-supervised optimization and adaptive deployment of the optical flow network.

2. The self-supervised optimization and adaptive deployment method for optical flow networks as described in claim 1, characterized in that, The feature extraction is performed using a shallow CNN backbone network.

3. The self-supervised optimization and adaptive deployment method for optical flow networks as described in claim 1, characterized in that, The attention weight matrix is ​​calculated using a cross-attention mechanism.

4. The self-supervised optimization and adaptive deployment method for optical flow networks as described in claim 1, characterized in that, The normalization is performed using the Softmax function.

5. The self-supervised optimization and adaptive deployment method for optical flow networks as described in claim 1, characterized in that, The 1 / 8 resolution feature map corresponding to the second linked image is aligned to the feature space of the first continuous image by feature warping.

6. The self-supervised optimization and adaptive deployment method for optical flow networks as described in claim 1, characterized in that, The convergence conditions include: for three consecutive epochs, the EPE decrease rate is <0.5%, and the signal-to-noise ratio improvement is <1dB.

Images