Semi-supervised optical flow estimation method based on constructing pseudo labels based on strong and weak transformation differences

Abstract

This invention relates to the field of computer vision technology and discloses a semi-supervised optical flow estimation method that constructs pseudo-labels based on the difference between strong and weak transformations. This invention proposes a semi-supervised learning paradigm that achieves results close to those of supervised learning using only a small amount of data, significantly improving over unsupervised learning. Instead of relying on labels, this invention introduces the difference between strong and weak transformations in the spatial and temporal domains as supervisory signals in the optical flow estimation problem. Furthermore, this method, for the first time, increases the data augmentation strength in this few-shot learning model to a level higher than that of purely supervised learning algorithms.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a semi-supervised optical flow estimation method for constructing pseudo labels based on strong and weak transformation differences. Background Art

[0002] The optical flow estimation problem primarily involves estimating the magnitude and direction of pixel motion on the observation plane for an object moving in space. The input is two images of equal size and a small time interval. The output is an optical flow estimation result of equal size and dimension 2, representing the x- and y-direction displacement of each pixel in the original image. Three fundamental assumptions underlie this task: ① The brightness invariance assumption states that the brightness of the same point does not change over time; ② The small motion assumption states that the position of each pixel does not change drastically over time. Only if this assumption is met can the partial derivative of grayscale with respect to position be approximated using the unit position change between two consecutive frames; and ③ The spatial consistency assumption states that two adjacent points in space are also adjacent when projected into the image, and that adjacent points on a continuous surface move at the same speed. Based on these three assumptions, the Lucas-Kanade algorithm constructs constraint equations based on partial derivatives and uses least squares linear regression to evaluate multiple sets of solutions. FlowNet

[35] was the first work to use an "encoder-decoder" architecture for end-to-end regression estimation of optical flow. It demonstrated that the regression architecture based on convolutional neural networks can directly predict optical flow, but its accuracy at the time still lagged behind the latest energy-based methods. Subsequent research has continuously improved the accuracy of optical flow estimation by designing better network structures, and has continuously achieved new heights in various public evaluations. These achievements are mainly attributed to better network architecture design, for example, by stacking multiple networks to gradually refine previous estimates or constructing feature pyramids to estimate optical flow in a coarse-to-fine manner. Compared with non-deep learning methods, many deep learning methods can run in real time on graphics computing units and have higher accuracy. End-to-end deep network regression has now become the mainstream paradigm in current research on optical flow estimation.

[0003] Currently, the main optical flow estimation methods include FlowNet, IRR-PWC, and ARFlow. The FlowNet method stacks two input images and feeds them through a common network, allowing the network to independently determine how to process the image pair to extract motion information. The network's structure is similar to a Siamese network. Correlation is then performed on the two extracted feature maps, fused into a single feature map, and the final optical flow estimate is output. However, this method is the first to introduce deep learning to the optical flow task. The accuracy of pixel matching is low due to the network's limited receptive field. This narrow receptive field makes it difficult to accurately estimate multi-scale optical flow, and the computational overhead is high. The IRR-PWC method uses a pyramid-structured network to perform flow inference from the initial optical flow in a coarse-to-fine manner. IRR draws inspiration from classic energy minimization methods and residual networks and proposes an iterative residual refinement method based on weight sharing. IRR can be combined with multiple backbone networks, reducing the number of parameters while improving accuracy. It also integrates occlusion prediction and bidirectional flow estimation. IRR can further improve performance, but this method still cannot complete pixel matching at a global level, and matching failures will still occur for pixels with large displacements. The ARFlow method is a state-of-the-art unsupervised learning method. It optimizes the network by performing two forward passes on the original input image after two data augmentations, and then constraining the consistency of the results after the unified transformation. However, since this method constrains the consistency of the two forward passes, some more complex data augmentations will make the optical flow estimation results under the enhancement inaccurate. Therefore, the error will increase when constraining consistency. Therefore, this unsupervised learning method still cannot introduce more complex data augmentation. Based on the above method, it can be concluded that for current supervised learning algorithms, dense optical flow true values ​​are very difficult to obtain. Considering the complex depth data collection and post-processing, the current unsupervised learning algorithm cannot introduce more complex data augmentation to address the problem of difficult data acquisition. This is because the constraints of the unsupervised learning loss function are too weak, and the introduction of relatively strong data augmentation will cause training to not converge. Summary of the Invention

[0004] To address the above problems, the present invention proposes a semi-supervised optical flow estimation method based on constructing pseudo labels based on the difference between strong and weak transformations, and proposes a semi-supervised learning paradigm that can achieve results close to those of supervised learning using only a small amount of data, and the results are significantly improved compared to unsupervised learning.

[0005] In order to achieve the above object, the present invention provides a semi-supervised optical flow estimation method based on constructing pseudo labels based on strong and weak transformation differences, comprising the following steps:

[0006] S1. Input video frames, perform optical flow annotation on two adjacent frames, and output data as two frames of images and corresponding optical flows as input for the supervised part;

[0007] S2, input video frames, combine two adjacent frames and save them as unsupervised input;

[0008] S3, network construction, including feature pyramid encoding network, warping transformation layer, cost loss calculation layer, and decoding layer;

[0009] S4. Perform strong data augmentation on the supervised data, input it into the network for calculation, and calculate the supervised loss together with the data annotation.

[0010] S5. Weakly enhance the unsupervised data, input it into the network for calculation, and calculate the unsupervised part loss and optical flow estimation results;

[0011] S6. Perform strong enhancement (spatial transformation) on the basis of the weakly enhanced data in step 5, input the data into the network for calculation, and obtain the optical flow estimation result;

[0012] S7, performing the same enhancement (spatial transformation) as that of the image in step 6 on the weakly enhanced optical flow estimation result obtained in step 5 to obtain an optical flow transformation result;

[0013] S8. Calculate the error between the two optical flow outputs of step 6 and step 7 as the semi-supervised partial loss function;

[0014] S9, optimize the network model by weighted summing the supervised loss calculated in step 4, the unsupervised loss calculated in step 5, and the semi-supervised loss calculated in step 8;

[0015] S10, forward calculation, input video frame pairs, input the network structure in step 3, and output optical flow results.

[0016] Preferably, the feature pyramid coding network first uses a feature coding network with shared parameters to extract multi-scale features from the two input frames I1 and I2 respectively. In the processing within the feature pyramid, for the input feature map pair of the lth layer, the optical flow from the previous layer is first double-upsampled so that it is consistent with the current spatial scale. The upsampled optical flow is used to perform a warping transformation on the corresponding features of the second frame of the current layer, and the error is calculated between the transformed result and the first frame. The feature extractor is composed of a plurality of convolutional layers stacked together, and a multi-scale feature pyramid is generated for the two input frames I1 and I2 in the form of shared parameters. The feature representation of each scale corresponding to the tth frame is recorded as And the lowest zeroth layer is represented as the input image, that is In order to Generate feature representation and set the stride of the first convolution filter in the l-1 layer to 2 to integrate the features from the previous layer. The feature pyramid extractor performs downsampling 2 times in total 6 times, outputting 6 sets of scale feature representations for each image. The number of feature channels increases in sequence: 16, 32, 64, 96, 128, and 196.

[0017] Preferably, the warping transformation layer performs optical flow estimation in a coarse-to-fine manner after extracting the feature pyramid, that is, the feature pyramid is processed in order from high to low, and optical flow prediction is performed in each layer. Since the correspondence between the features of the two frames of the image in the lth layer may have a large position offset, the features can be aligned using the optical flow prediction of the previous layer, thereby reducing the offset of the corresponding matching points. Specifically, the module performs a warping transformation on the features of the second frame of the image after doubling the optical flow predicted by the l+1 layer to obtain the aligned features:

[0018]

[0019] Where x is the pixel coordinate, ↑ 2× (.) is a double upsampling operation. In addition, when transforming the highest layer, the corresponding optical flow w l+1 Set to zero to perform an identity transformation.

[0020] Preferably, the cost loss calculation layer defines the calculation of the cost volume as calculating the cross-correlation between the features of the first frame and the aligned features of the second frame:

[0021]

[0022] where x t is the pixel position in the tth frame, T is the transpose operator, and N is the column vector For the l-th layer feature with a length and width of H×W, each pixel position can be calculated with all pixels in the other view to obtain a cross-correlation value of H×W size. When the feature scale is large, it will take up a lot of memory and computing power. In order to simplify the calculation, for the feature pyramid with a total of L layers, the cost volume is usually calculated only in a limited range, such as setting the range to |x1-x2|∞≤d. Thanks to the warping transformation layer, the features of the two frames have been aligned, and the feature scale of the high-level layer has been small after multiple downsamplings. Therefore, it is only necessary to search for matching points in a very small range. Therefore, the cost volume layer converts the features of the two frames into a three-dimensional cost volume of H×W×d2.

[0023] Preferably, the decoding layer is formed by stacking convolutional layers and can adapt to input feature maps of different spatial scales.

[0024] Preferably, the supervised partial loss calculation process is: let the decoder network be The input feature of the decoding network is the first frame feature after convolution alignment The error calculated after the warping transformation of the two frames is cv l And the optical flow obtained from the previous level upsampling↑ 2× (U l+1 );

[0025] For the lth layer of the feature pyramid, the decoding process is summarized as follows, that is, the optical flow estimation output is obtained:

[0026]

[0027]

[0028]

[0029]

[0030] After the supervised learning part is calculated for the input image pair, the loss function is used to calculate the loss of supervised learning. The loss of supervised learning is calculated as follows, that is, the L2 distance between the estimated value and the true value:

[0031] L sup =||V est -V gt ||2.

[0032] Preferably, the unsupervised part loss calculation process is as follows: for a pair of input video frames, a random weak image enhancement is first performed on them, and then forward calculation is performed while sharing the weights with the supervised part model to obtain the optical flow estimation result, and the unsupervised loss is calculated, including the photometric loss and the smoothing loss. The calculation formula is as follows:

[0033]

[0034]

[0035] The unsupervised loss is thus:

[0036] L unsup =L ph +λ s L smooth .

[0037] Preferably, the calculation process of the loss function of the semi-supervised part is as follows: after completing the calculation of the unsupervised part, a strong data enhancement is performed on the basis of the input data of the unsupervised part (that is, the data after weak image enhancement), and the image after strong enhancement transformation is input into the network for forward calculation to obtain the estimation result under strong enhancement. The weak enhancement estimation result is used to construct a supervisory signal as a label for the strong enhancement result. Since the optical flow estimation result can be regarded as a 2-dimensional vector diagram, the optical flow estimation result under weak enhancement is transformed. Ideally, the weak enhancement estimation result should be consistent with the strong enhancement estimation result after transformation. However, since the estimation result under strong enhancement has a low accuracy, the result under strong enhancement is constrained to be consistent with the result after weak enhancement transformation, and the loss function of the semi-supervised part is obtained. The calculation formula is as follows:

[0038] L semi =||TV weak -V strong ||2.

[0039] Preferably, the final total loss function is the sum of the three loss functions:

[0040] L final =λ1L sup +λ2L unsup +λ3L semi

[0041] Where λ1 is 1, λ2 is 0.05, and λ3 is 0.05.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] 1. This paper proposes a semi-supervised learning paradigm that can achieve results close to those of supervised learning using only a small amount of data, and the results are significantly improved compared to unsupervised learning.

[0044] 2. This invention does not rely on labels in optical flow estimation, but instead uses the differences in spatial and temporal intensity transformations as supervisory signals. Furthermore, it achieves, for the first time, data augmentation in this few-shot learning model, exceeding that of purely supervised learning algorithms. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 A schematic diagram of the network structure provided by the present invention;

[0046] Figure 2 This is the effect diagram of the weak enhancement of the image of the present invention;

[0047] Figure 3 This is the effect diagram of the image after strong data enhancement of the present invention;

[0048] Figure 4This is the error result diagram of the present invention using labeled data of different scales. DETAILED DESCRIPTION

[0049] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0050] The semi-supervised optical flow estimation method proposed in the present invention for constructing pseudo labels based on the difference between strong and weak transformations includes the following steps:

[0051] S1. Input video frames, perform optical flow annotation on two adjacent frames, and output data as two frames of images and corresponding optical flows as input for the supervised part;

[0052] S2, input video frames, combine two adjacent frames and save them as unsupervised input;

[0053] S3, network construction, including feature pyramid encoding network, warping transformation layer, cost loss calculation layer, and decoding layer;

[0054] S4. Perform strong data augmentation on the supervised data, input it into the network for calculation, and calculate the supervised loss together with the data annotation.

[0055] S5. Perform weak enhancement on the unsupervised data, input it into the network for calculation, and calculate the unsupervised loss and optical flow estimation results;

[0056] S6. Perform strong enhancement (spatial transformation) on the basis of the weakly enhanced data in step 5, input the data into the network for calculation, and obtain the optical flow estimation result;

[0057] S7, performing the same enhancement (spatial transformation) as that of the image in step 6 on the weakly enhanced optical flow estimation result obtained in step 5 to obtain an optical flow transformation result;

[0058] S8. Calculate the error between the two optical flow outputs of step 6 and step 7 as the semi-supervised partial loss function;

[0059] S9, optimize the network model by weighted summing the supervised loss calculated in step 4, the unsupervised loss calculated in step 5, and the semi-supervised loss calculated in step 8;

[0060] S10, forward calculation, input video frame pairs, input the network structure in step 3, and output optical flow results.

[0061] Since optical flow estimation based on deep learning requires labeled data during training, and dense optical flow annotations in natural scenes are very difficult to obtain, semi-supervised learning is introduced to use limited labeled data and relatively sufficient unlabeled data in actual scenarios to enable the model to achieve better results. In the present invention, the main consideration is the case of incomplete supervision, that is, the case where we only have a small amount of labeled data, and these labeled data are not enough to train a good model, but we have a large amount of unlabeled data available for use. Formally expressed, the task of the model is to train from the following training data set, where l samples are labeled and ml samples are unlabeled. If ml = 0, it is converted into an ordinary supervised problem, and if l = 0, it is converted into an ordinary unsupervised problem:

[0062] D={(x 1, y1),…,(x l ,y l ),x l+1 ,…,x m )}

[0063] Common semi-supervised algorithms, such as the PI Model, rely on the principle that regularization techniques (such as data augmentation and dropout) typically do not alter the probability distribution of the model output. The Pi-Model leverages this property of prediction functions in neural networks. For any given input x, it uses different regularizations and then makes two predictions. The goal is to reduce the gap between the two predictions and improve the model's consistency under different perturbations. The Pi-Model uses the Mean Squared Error (MSE) as the loss function between the two probability distributions. The augmentation transformation is random, and the model also uses dropout. Both factors contribute to differences in the results of the two forward passes. Temporal Ensembling is a temporal series combination model that calculates the mean squared error (MSE) by taking the average of the current model's prediction and the average of historical predictions. This effectively preserves historical information, eliminates perturbations, and stabilizes the current value. The self-training algorithm first trains the model with a small amount of labeled data L. The trained model is then used to assign pseudo-labels to unlabeled data points x∈U. The loss between the model predictions and the pseudo-labels is calculated using the cross-entropy loss. Use the trained model to generate proxy labels for the rest of U, and keep looping until the model cannot generate proxy labels.

[0064] The training data sets of the present invention include Sintel-Clean, Sintel-Final, and KITTI data sets. For each data set, part of the data is randomly extracted as the supervised part data (i.e. Figure 1 The lower part of the frame in the figure), and the rest is the unsupervised data (i.e. Figure 1 Since optical flow describes the displacement of pixels between two frames, each input training data for both the supervised and unsupervised parts contains two identical video frames. The input of the supervised part also includes the true optical flow value.

[0065] The supervised part is trained as Figure 1 The lower frame in the figure takes as input an image pair containing the true optical flow value. Depending on the selected pipeline, the overall network model will vary; the Flow Network in the figure can be either IRR-PWC or RAFT. The overall network structure follows the principle of coarse-to-fine progressive optimization. The proposed optical flow estimation network processes each feature group in the feature pyramid from top to bottom. The following describes each layer of the network.

[0066] ① Feature pyramid encoding network. For the two input frames I1 and I2, a feature encoding network with shared parameters is first used to extract multi-scale features from the two images respectively. In the processing within the feature pyramid, for the input feature map pair of the lth layer, the optical flow from the previous layer is first upsampled by two times to make it consistent with the current spatial scale. The upsampled optical flow is used to perform a warping transformation on the corresponding features of the second frame of the current layer, and the error between the transformed result and the first frame is calculated. The feature extractor is composed of a stack of multiple convolutional layers, which generates a multi-scale feature pyramid for the two input frames I1 and I2 in the form of shared parameters. The feature representation of each scale corresponding to the tth frame is denoted as And the lowest zeroth layer is represented as the input image, that is In order to Generate feature representation and set the stride of the first convolution filter in the l-1 layer to 2 to integrate the features from the previous layer. Downsampling by a factor of 2. The feature pyramid extractor performs a total of 6 downsamplings, outputting 6 sets of feature representations at each scale for each image. The number of feature channels increases in sequence: 16, 32, 64, 96, 128, and 196.

[0067] ② Warping transformation layer. The specific role of the warping transformation is: after extracting the feature pyramid, the network performs optical flow estimation in a coarse-to-fine manner, that is, the feature pyramid is processed in order from high to low, and optical flow prediction is performed in each layer. Since the correspondence between the features of the two frames of the l-th layer may have a large position offset, the optical flow prediction of the previous layer can be used to align the features, thereby reducing the offset of the corresponding matching points. Specifically, this module can perform a warping transformation on the features of the second frame of the image after doubling the optical flow predicted by the l+1 layer to obtain the aligned features:

[0068]

[0069] Where x is the pixel coordinate, ↑ 2× (.) is a double upsampling operation. In addition, when transforming the highest layer, the corresponding optical flow w l+1 Setting it to zero means performing an identity transformation. Since the warping transformation process is a linear transformation, it does not affect the differentiability of the network and can be trained in an end-to-end manner.

[0070] ③ Cost loss calculation layer. The cost body is widely used in classic binocular depth methods to measure the correlation between two sets of features. Here, the calculation of the cost body is defined as calculating the cross-correlation between the features of the first frame and the aligned features of the second frame.

[0071]

[0072] where x t is the pixel position in the tth frame, T is the transpose operator, and N is the column vector Length. It is worth noting that for the l-th layer feature with a length and width of H×W, each pixel position can be calculated with all pixels in the other view to obtain a cross-correlation value of H×W size. When the feature scale is large, it will take up a lot of memory and computing power. In order to simplify the calculation, for a feature pyramid with a total of L layers, the cost volume is usually calculated only within a limited range, such as setting the range to |x1-x2|∞≤d. Thanks to the warping transformation layer, the features of the two frames have been aligned, and the feature scale of the high-level layers has been smaller after multiple downsamplings, it is only necessary to search for matching points within a very small range. Therefore, the cost volume layer converts the features of the two frames into a three-dimensional cost volume of H×W×d2.

[0073] ④Decoding layer. The above completes the description of the encoding network. For the decoding network, this article still adopts a shared structure. In order to share the decoder, the decoding network needs to adapt to the input of feature maps of different layers. First of all, the decoder is designed to be completely stacked by convolutional layers, so it can adapt to input feature maps of different spatial scales. But it is worth noting that the number of channels of feature maps on different layers is different, and there may be semantic differences. To this end, the present invention adds an additional set of convolutional layers composed of 1×1 convolution kernels to the feature map corresponding to the first frame for alignment. On the one hand, the convolution layer is used to align the channels of feature maps of various scales. On the other hand, semantic alignment can be achieved by learning the parameters of this layer, so that the decoder can be shared between the layers of the feature pyramid. Note that although this set of convolutions does not share parameters for the various layers of the feature pyramid, since there is only one convolution operation per layer, the additional parameters and computational complexity introduced are almost negligible. Let the decoder network be The input feature of the decoding network is the first frame feature after convolution alignment The error calculated after the warping transformation of the two frames is cv l And the optical flow obtained from the previous level upsampling↑ 2× (U l+1 ).

[0074] For the lth layer of the feature pyramid, the decoding process is summarized as follows, that is, the optical flow estimation output is obtained:

[0075]

[0076]

[0077]

[0078]

[0079] After the supervised learning part is calculated for the input image pair, the loss function is used to calculate the loss of supervised learning. The loss of supervised learning is calculated as follows, which is the L2 distance between the estimated value and the true value:

[0080] L sup =||V est -V gt ||2.

[0081] The unsupervised learning part is training. Figure 1 Since it involves the strong and weak enhancement transformation of the image, the enhancement form is defined first.

[0082] like Figure 2 As shown, weak image enhancement primarily involves color and brightness transformations, as well as noise addition. These transformations only affect the image's appearance. Since these changes are applied simultaneously to the input image pair, they do not alter the original optical flow values, resulting in better estimation for the network and are therefore considered weak enhancements. Strong enhancements, on the other hand, often involve spatial transformations such as flipping, cropping, and affine transformations, which alter the true optical flow value. Spatial transformations primarily introduce new motion patterns, particularly large displacements. Estimating optical flow with large displacements has long been a major challenge, and is particularly challenging for unsupervised learning methods that do not use labels. Under the assumption of photometric consistency, minimizing photometric loss requires only finding pixels with consistent appearance. However, unsupervised learning networks tend to focus on similar pixels near the current pixel, making it difficult to predict correct matching points that are farther away. Introducing spatial transformations into analogical learning creates a supervisory signal for large displacements, encouraging the network to search for matching points over a wider range.

[0083] like Figure 3The following figure shows the effect after strong data enhancement. It can be seen that after strong data enhancement, the overall saturation of the optical flow visualization has been significantly improved, and the saturation of the visualization reflects the magnitude of the optical flow. It can be seen that the overall complexity of the optical flow has also increased after strong data enhancement, making it more difficult for the network to estimate. Next, the general form of this type of strong enhancement is given. Let τ θ Represents the transformation of the pixel coordinate system, which corresponds to the transformation of the image can be expressed as:

[0084]

[0085] The corresponding optical flow transformation cannot be performed on the original optical flow, and an intermediate optical flow field needs to be introduced. The calculation formula for the transformation of optical flow can be:

[0086]

[0087]

[0088] The new occlusion introduced by the spatial transformation can be clearly inferred from the prediction of the bidirectional optical flow through the forward and backward consistency check. Under ideal conditions where the prediction is completely accurate and there is no motion occlusion, the forward and backward estimation results of the optical flow should be consistent in size and opposite in direction. Due to the foreground motion in the image, some pixels in the first frame will not be able to find corresponding points in the second frame. At this time, the difference between the image after the inverse transformation of the optical flow in the occluded area and the original image cannot represent the accuracy of the optical flow. For this reason, it is often necessary to remove the influence of occlusion in unsupervised learning visual motion estimation. The calculation formula for judging the occluded area by the consistency criterion is as follows. The pixel x that meets the following constraints is recorded as unoccluded and marked as 0, otherwise it is forward occluded and marked as 1:

[0089] |U 12 (x)+U 21 (x+U 12 (x))| 2

[0090] <α1(|U 12 (x)| 2 +|U 21 (x+U 12 (x))| 2 )+α2

[0091] After defining the strength of data augmentation, we will now describe the unsupervised training method. For a pair of input video frames, we first perform a random weak image augmentation. Then, we perform a forward pass while sharing the weights with the supervised model to obtain the optical flow estimate. We also calculate the unsupervised loss, which consists of two components: photometric loss and smoothing loss. The calculation formula is as follows:

[0092]

[0093]

[0094] The unsupervised loss is thus:

[0095] L unsup =L ph +λ s L smooth

[0096] After completing the calculation of the unsupervised part, a strong data enhancement is performed on the basis of the input data of the unsupervised part (i.e., the data after weak image enhancement). Since it has been pointed out above that strong image enhancement is a spatial transformation, it can be described by the transformation matrix T. The image after strong enhancement transformation is input into the network for forward calculation to obtain the estimation result under strong enhancement. Since the image under strong enhancement is more complex and the pixel displacement is larger, the accuracy will be lower than the output under weak enhancement, and the network's estimation of occluded and large-displaced pixels will diverge. Therefore, the present invention plans to use the estimation result of weak enhancement to construct a supervisory signal as a label for the strong enhancement result. Since the optical flow estimation result can be regarded as a 2-dimensional vector map, the optical flow estimation result under weak enhancement can be transformed using the transformation formula described above. Ideally, the estimation result of weak enhancement should be consistent with the estimation result under strong enhancement after transformation. However, since the accuracy of the estimation result under strong enhancement is low, the result under strong enhancement is constrained to be consistent with the result after weak enhancement transformation, and the loss function of the semi-supervised part is obtained. The calculation formula is as follows:

[0097] L semi =||TV weak -V strong ||2

[0098] The final total loss function is the sum of the loss functions of the three parts

[0099] L final =λ1L sup +λ2L unsup +λ3L semi

[0100] In summary, the overall network training architecture involves pre-training the baseline network on the FlyingChairs and Flying Things datasets, alternating between supervised and semi-supervised training. The first forward pass calculates the supervised loss, while the second forward pass calculates the semi-supervised loss. The two losses are then summed to update the network weights. The semi-supervised weights are reset to 0 for the first 20k iterations. After 20k iterations, a warmup is applied to the semi-supervised loss. In this paper, λ1 is 1, λ2 is 0.05, and λ3 is 0.05.

[0101] The network forward calculation does not require any data enhancement, etc. It uses the trained model, that is, Figure 1 In the FlowNetwork part, input two consecutive video frames to output the forward optical flow.

[0102] On the Sintel and KITTI datasets, the errors of the proposed method are as follows:

[0103]

[0104]

[0105] The indicators in the table are errors, which are specifically expressed as the L2 error between the true value and the estimated value. Its physical meaning is the number of pixels that differ between the true optical flow value vector and the estimated optical flow value vector:

[0106]

[0107] Ours (40% PWC) means the experimental results of using the semi-supervised method of the present invention with IRR-PWC as the overall network framework and using annotations on 40% of the random input data. Ours (40% RAFT) means the experimental results of using the semi-supervised method of the present invention with RAFT as the overall network framework and using annotations on 40% of the random input data. At the same time, ARFLOW is the experimental result of the unsupervised learning method that also uses IRR-PWC as the overall network framework. It can be seen that the semi-supervised method adopted by the present invention can obtain a smaller error on IRR-PWC using only 40% of the labeled data on both the Sintel dataset and the KITTI dataset. The error is very close to that of the fully supervised baseline (IRR-PWC), and the error is significantly reduced compared to ARFlow, which also uses IRR-PWC as the baseline. At the same time, if the method of the present invention uses RAFT as the baseline, it can achieve performance that exceeds all other supervised learning methods while using only 40% of the labeled data. As Figure 4The figure shows the error results of the present invention using different proportions of labeled data. The vertical axis shows the error value, and the horizontal axis shows the proportion of labeled data to the total training data. The test dataset is the Sintel dataset. It can be seen that before the proportion of labeled data reaches about 40%, the error decreases significantly as the number of labeled data increases.

[0108] Although the present invention is described herein with reference to specific embodiments, it should be understood that these embodiments are merely illustrative of the principles and applications of the invention. It should be understood that many modifications may be made to the illustrative embodiments, and that other arrangements may be devised, without departing from the spirit and scope of the invention as defined by the appended claims. It should be understood that the various dependent claims and features described herein may be combined in ways other than those described in the original claims. It should also be understood that features described in conjunction with individual embodiments may be employed in conjunction with other described embodiments.

Claims

1. A semi-supervised optical flow estimation method based on constructing pseudo labels based on the difference between strong and weak transformations, characterized by: The following steps are involved: S1. Input video frames, perform optical flow annotation on two adjacent frames, and output data as two frames of images and corresponding optical flows as input for the supervised part; S2, input video frames, combine two adjacent frames and save them as unsupervised input; S3, network construction, Includes feature pyramid encoding network, warping transformation layer, cost loss calculation layer, and decoding layer; S4. Perform strong data augmentation on the supervised data, input it into the network for calculation, and calculate the supervised loss together with the data annotation. S5. Weakly enhance the unsupervised data, input it into the network for calculation, and calculate the unsupervised part loss and optical flow estimation results; S6. Perform strong enhancement on the basis of the weakly enhanced data in step 5, input the data into the network for calculation, and obtain the optical flow estimation result; S7, performing the same enhancement as in step 6 on the weakly enhanced optical flow estimation result obtained in step 5 to obtain an optical flow transformation result; S8. Calculate the error between the two optical flow outputs of step 6 and step 7 as the semi-supervised partial loss function; S9, optimize the network model by weighted summing the supervised loss calculated in step 4, the unsupervised loss calculated in step 5, and the semi-supervised loss calculated in step 8; S10, forward calculation, input video frame pairs, input the network structure in step 3, and output optical flow results; The feature pyramid coding network first uses a feature coding network with shared parameters to extract multi-scale features from the two input frames I1 and I2 respectively. In the processing within the feature pyramid, for the input feature map pair of the lth layer, the optical flow from the previous layer is first doubled upsampled to make it consistent with the current spatial scale. The upsampled optical flow is used to perform a warping transformation on the corresponding features of the second frame of the current layer, and the error is calculated between the transformed result and the first frame. The feature extractor is composed of a plurality of convolutional layers stacked together, and a multi-scale feature pyramid is generated for the two input frames I1 and I2 in the form of shared parameters. The feature representation of each scale corresponding to the tth frame is recorded as And the lowest zeroth layer is represented as the input image, that is In order to Generate feature representation and set the stride of the first convolution filter in the l-1 layer to 2 to integrate the features from the previous layer. The feature pyramid extractor performs downsampling 2 times in total 6 times, outputting 6 sets of scale feature representations for each image. The number of feature channels increases in sequence: 16, 32, 64, 96, 128, and 196.

2. The semi-supervised optical flow estimation method based on constructing pseudo labels based on strong and weak transformation differences according to claim 1 is characterized in that After extracting the feature pyramid, the warping transformation layer performs optical flow estimation in a coarse-to-fine manner, that is, the feature pyramid is processed in order from high to low, and optical flow prediction is performed in each layer. Since the correspondence between the features of the two frames of the image in the lth layer may have a large position offset, the features can be aligned using the optical flow prediction of the previous layer, thereby reducing the offset of the corresponding matching points. Specifically, the above step performs a double upsampling on the optical flow predicted by the l+1 layer, and then the features of the second frame of the image can be warped to obtain the aligned features: Where x is the pixel coordinate, ↑ 2× (.) is a double upsampling operation. In addition, when transforming the highest layer, the corresponding optical flow w l+1 Set to zero to perform an identity transformation.

3. The semi-supervised optical flow estimation method based on constructing pseudo labels based on strong and weak transformation differences according to claim 2 is characterized in that The cost loss calculation layer defines the calculation of the cost volume as calculating the cross-correlation between the features of the first frame and the aligned features of the second frame: where x t is the pixel position in the tth frame, T is the transpose operator, and N is the column vector For the l-th layer feature with a length and width of H×W, each pixel position can be calculated with all pixels in the other view to obtain a cross-correlation value of H×W size. When the feature scale is large, it will take up a lot of memory and computing power. In order to simplify the calculation, for the feature pyramid with a total of L layers, the cost volume is usually calculated only in a limited range, such as setting the range to |x1-x2|∞≤d. Thanks to the warping transformation layer, the features of the two frames have been aligned, and the feature scale of the high-level layer has been small after multiple downsamplings. Therefore, it is only necessary to search for matching points in a very small range. Therefore, the cost volume layer converts the features of the two frames into a three-dimensional cost volume of H×W×d2.

4. The semi-supervised optical flow estimation method based on constructing pseudo labels based on strong and weak transformation differences according to claim 3 is characterized in that The decoding layer is formed by stacking convolutional layers and can adapt to input feature maps of different spatial scales.

5. The semi-supervised optical flow estimation method based on constructing pseudo labels based on strong and weak transformation differences according to claim 4 is characterized in that The supervised part loss calculation process is as follows: let the decoder network be The input feature of the decoding network is the first frame feature after convolution alignment The error calculated after the warping transformation of the two frames is cv l And the optical flow obtained from the previous level upsampling↑ 2× (U l+1 ); For the lth layer of the feature pyramid, the decoding process is summarized as follows, that is, the optical flow estimation output is obtained: After the supervised learning part is calculated for the input image pair, the loss function is used to calculate the loss of supervised learning. The loss of supervised learning is calculated as follows, that is, the L2 distance between the estimated value and the true value: L sup =||V est -V gt ||2。 6. The semi-supervised optical flow estimation method based on constructing pseudo labels based on strong and weak transformation differences according to claim 5, characterized in that The unsupervised loss calculation process is as follows: for a pair of input video frames, a random weak image enhancement is first performed on them, and then forward calculation is performed while sharing the weights with the supervised part model to obtain the optical flow estimation result, and the unsupervised loss is calculated, including the photometric loss and the smoothing loss. The calculation formula is as follows: The unsupervised loss is thus: L unsup =L ph +λ s L smooth 。 7. The semi-supervised optical flow estimation method based on constructing pseudo labels based on strong and weak transformation differences according to claim 6, characterized in that The calculation process of the loss function of the semi-supervised part is as follows: after completing the calculation of the unsupervised part, a strong data enhancement is performed on the basis of the input data of the unsupervised part, and the image after the strong enhancement transformation is input into the network for forward calculation to obtain the estimation result under strong enhancement. The weak enhancement estimation result is used to construct a supervision signal as the label of the strong enhancement result. Since the optical flow estimation result can be regarded as a 2D vector diagram, the optical flow estimation result under weak enhancement is transformed. Ideally, the estimation result of weak enhancement should be consistent with the estimation result under strong enhancement after transformation. However, since the estimation result under strong enhancement has a low accuracy, the result under strong enhancement is constrained to be consistent with the result after weak enhancement transformation, and the loss function of the semi-supervised part is obtained. The calculation formula is as follows: IT semi =||TV weak V strong ||2。 8. The semi-supervised optical flow estimation method based on constructing pseudo labels based on strong and weak transformation differences according to claim 7 is characterized in that The final total loss function is the sum of the three loss functions: L final =λ1L sup +λ2L unsup +λ3L semi Where λ1 is 1, λ2 is 0.05, and λ3 is 0.05.

Images