An optical flow estimation method based on improved FlowNetS

Abstract

The present invention provides an optical flow estimation method based on improved FlowNetS, which mainly solves the problem that the optical flow estimation method in the prior art has poor recognition accuracy in a complex motion environment. The method adopted by the present invention is based on the FlowNetS framework of the encoder-decoder architecture, and introduces a superposition fusion module, which uses the superposition fusion module to enhance the feature expression ability of the FlowNetS model, and improves the accuracy and robustness of the method through multi-dataset training and Adam optimization algorithm. Using the method of the present invention, the accuracy error of optical flow estimation is significantly improved compared with the prior art.

Description

Technical Field

[0001] The invention belongs to the field of computer vision and mainly relates to an optical flow estimation method. Background Art

[0002] Optical flow is an ideal dense field of displacement vectors. Optical flow contains rich motion information. In short, optical flow estimates the offset and direction of motion by tracking feature point pixels, and then describes the pixel motion between adjacent frames. The optical flow in the image captures the transformation of the illumination pattern on the plane, effectively representing the movement of pixels between adjacent frames. Optical flow estimation technology can capture the motion trajectory of pixels between adjacent frames and calculate the motion speed and direction of each pixel. It can be applied to pose estimation, motion estimation, etc. Optical flow graphs usually use color coding to represent the motion information of each pixel, where the color represents the direction of motion and the color depth represents the size of the motion. Optical flow estimation can provide pixel point motion information for moving target detection and motion estimation, which can effectively help it identify moving targets, estimate its motion trajectory, and predict its future position. Therefore, optical flow estimation has important value in the field of computer vision. In the existing FlowNetS optical flow estimation method, there is still sensitivity to specific scenes. In the context of complex motion environments, the recognition accuracy is poor. The present invention can effectively solve the problem of model sensitivity to specific scenes and improve its feature recognition ability and accuracy for complex environments. Summary of the invention

[0003] In order to overcome the problem that the optical flow estimation method in the prior art has poor recognition progress in a complex motion environment, the present invention provides an optical flow estimation method based on improved FlowNetS.

[0004] An optical flow estimation method based on improved FlowNetS includes the following steps:

[0005] Step 1: Build data sets, training sets, and validation sets;

[0006] Step 2: Build an optical flow estimation neural network model;

[0007] The optical flow estimation neural network model includes downsampling, upsampling, optical flow prediction operation and superposition fusion modules;

[0008] The training set is used as input to downsample and obtain the optical flow result, and then the optical flow result obtained by downsampling is input to upsampling, the downsampling result and the upsampling result are spliced, and the optical flow prediction operation is performed on the spliced ​​result, and the result of the optical flow prediction operation is input into the superposition fusion module to obtain the final optical flow prediction result;

[0009] The downsampling includes 13 convolutional layers; the first convolutional layer to the thirteenth convolutional layer are connected in sequence, the input of each convolutional layer is a feature map, and the input of the next layer is the output result of the previous layer;

[0010] The optical flow prediction operation is a 3×3 convolution kernel, the step size is 1, and the number of output channels of the optical flow prediction operation is 2;

[0011] The optical flow prediction operation processes the result output by the downsampling layer, changes the number of channels of the output result, and makes the size of the output result consistent with the size of the input feature map;

[0012] The upsampling layer includes 10 neural network layers;

[0013] The superposition fusion module sequentially inputs the optical flow results obtained by the optical flow prediction operation according to different sizes from small to large, and uses a bilinear interpolation method to gradually amplify the optical flow results obtained by performing an optical flow prediction operation on the result of the downsampled thirteenth layer, and respectively adds the optical flow results obtained by upsampling and optical flow prediction operations with the results of the downsampled third, fifth, seventh, ninth, and eleventh layers by pixel, to obtain the final optical flow prediction result;

[0014] Step 3: Initialize the parameters of the optical flow estimation neural network model;

[0015] Initialize the training parameters of the optical flow estimation neural network model;

[0016] Step 4: Train the optical flow estimation neural network model;

[0017] Different types of training sets in the data set are input into the optical flow estimation neural network model respectively, and the training is iterated 300 times. The trained model is saved, and the optical flow estimation accuracy error EPE_multiscale of each model is calculated. The model with the smallest maximum optical flow estimation accuracy error is the best model.

[0018] Furthermore, in step 1, the steps to construct the data set, training set, and validation set are:

[0019] The datasets include FlyingChairs dataset, MPI-Sintel clean dataset, MPI-Sintelfinal dataset, or KITTI Flow 2015 dataset.

[0020] In the data set, the ratio of the number of training sets to the number of validation sets is 8:2;

[0021] When the dataset is the KITTI Flow 2015 dataset, the KITTI Flow 2015 dataset is enhanced;

[0022] The enhancement method is as follows: randomly crop, vertically flip, and horizontally flip the images in the KITTI Flow 2015 dataset in sequence; the size of the randomly cropped images is 320×448;

[0023] When the dataset is the FlyingChairs dataset or the MPI-Sintel dataset, a strong enhancement operation is performed on the FlyingChairs dataset or the MPI-Sintel dataset;

[0024] The method of strong enhancement operation is: randomly translate and randomly rotate the selected data set in turn; the random translation is a random translation of a distance between -10 and 10 pixels; the random rotation is a rotation between -10 and 10 degrees at intervals of 5 degrees.

[0025] Furthermore, the structure of the downsampling is:

[0026] The downsampling includes 13 convolutional layers for extracting image features; the convolution kernel of the first convolutional layer is 7×7, the convolution kernels of the remaining convolutional layers are 5×5, the moving step length stride of each layer is 2, the number of input channels of the first layer is 6, the number of output channels is 32, the number of input channels of the second layer is 32, the number of output channels of the third layer is 64; the input channels and output channels of the fourth convolutional layer are 64 and 128 respectively; the number of input channels and output channels of the fifth layer are both 128; the number of input channels of the sixth layer is 128, the number of output channels is 25 6; the seventh layer has 256 input channels and 512 output channels; the eighth layer has 256 input channels and 512 output channels; the ninth layer has 512 input channels and 1024 output channels; the tenth layer has 512 input channels and 1024 output channels; the eleventh layer has 1024 input channels and 1024 output channels; the twelfth and thirteenth layers are newly added convolutional layers, and the convolution kernels of the twelfth and thirteenth layers are both 5×5; the twelfth layer has 1024 input channels and 2048 output channels, and the thirteenth layer has 2048 input channels and 2048 output channels.

[0027] Furthermore, the optical flow estimation accuracy error EPE_multiscale method is calculated as follows:

[0028] The calculation is implemented as follows:

[0029] ;

[0030] ;

[0031] Where, flow_i_2 is flow_2_2 or flow_3_2 or flow_4_2 or flow_5_2 or flow_6_2 or flow_7_2; EPE_i represents the average endpoint error calculated for each different flow_i_2, that is:

[0032] ;

[0033] ;

[0034] ;

[0035] ;

[0036] ;

[0037] Where target is the true value; ||...|| is the Euclidean distance, mean(...) is the average value, and w_i is the weight;

[0038] The calculation steps of EPE_multiscale are to find the weighted average of EPE_2, EPE_3, EPE_4, EPE_5 and EPE_6 with weights of 0.005, 0.01, 0.02, 0.08 and 0.32 respectively, so as to obtain EPE_multiscale.

[0039] The beneficial effects of the present invention are as follows: due to the introduction of the superposition fusion module, the endpoint error accuracy of the original network structure is improved by 2.86% on the Flying Chairs dataset, 0.70% on the KITTI Flow 2015 dataset, and 7.40% and 2.43% on the MPI-Sintel clean and MPI-Sintel final datasets, respectively, achieving a more competitive advantage. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 The model architecture of FlowNetS with superposition fusion module;

[0041] Figure 2 The model principle of FlowNetS with superposition fusion module;

[0042] Figure 3 This is the schematic diagram of the superposition fusion module;

[0043] Figure 4The EPE change values ​​of the verification results obtained by different models; among them, (a) is the comparison chart of FlowNetS-F and ADD-Block-F; (b) is the comparison chart of FlowNetS-Sc and ADD-Block-Sc; (c) is the comparison chart of FlowNetS-Sf and ADD-Block-Sf; (d) is the comparison chart of FlowNetS-K and ADD-Block-K;

[0044] Figure 5 The results of the Flying Chairs dataset sample test; (a) is the first frame of the original adjacent image pair; (b) is the second frame of the original adjacent image pair; (c) is the optical flow estimation map generated by the prior art; (d) is the optical flow estimation map generated by the present invention;

[0045] Figure 6 The test results are based on the KITTI Flow 2015 dataset; (a) is the first frame of the original adjacent image pair; (b) is the second frame of the original adjacent image pair; (c) is the optical flow estimation image generated by the prior art; (d) is the optical flow estimation image generated by the present invention;

[0046] Figure 7 The test results of the MPI-Sintel clean dataset are shown in Figure 2. (a) is the first frame of the original adjacent image pair; (b) is the second frame of the original adjacent image pair; (c) is the optical flow estimation image generated by the prior art; (d) is the optical flow estimation image generated by the present invention;

[0047] Figure 8 For the flow chart. DETAILED DESCRIPTION

[0048] The steps of the technical solution adopted by the present invention are as follows:

[0049] The flow chart of the present invention is as follows Figure 8 As shown;

[0050] Step 1: Build data sets, training sets, and validation sets;

[0051] The datasets include the FlyingChairs dataset, the MPI-Sintel dataset, or the KITTI Flow 2015 dataset;

[0052] In the data set, the ratio of the number of training sets to the number of validation sets is 8:2;

[0053] When the dataset is the KITTI Flow 2015 dataset, the KITTI Flow 2015 dataset is enhanced; the enhancement method is: randomly cropping, vertically flipping, and horizontally flipping the images in the KITTI Flow 2015 dataset in sequence; the size of the randomly cropped images is 320×448;

[0054] When the dataset is the FlyingChairs dataset or the MPI-Sintel dataset, a strong enhancement operation is performed on the FlyingChairs dataset or the MPI-Sintel dataset. The strong enhancement operation method is: performing random translation and random rotation operations on the selected dataset in sequence; the random translation is a random translation of a distance between -10 and 10 pixels; the random rotation is a rotation between -10 and 10 degrees at intervals of 5 degrees;

[0055] Step 2: Build an optical flow estimation neural network model;

[0056] The optical flow estimation neural network model includes downsampling, upsampling, optical flow prediction operation and superposition fusion modules;

[0057] The training set is used as input to downsample and obtain the optical flow result, and then the optical flow result obtained by downsampling is input to upsampling, the downsampling result and the upsampling result are spliced, and the optical flow prediction operation is performed on the spliced ​​result, and the result of the optical flow prediction operation is input into the superposition fusion module to obtain the final optical flow prediction result;

[0058] The downsampling includes 13 convolutional layers; the first convolutional layer to the thirteenth convolutional layer are connected in sequence, the input of each convolutional layer is a feature map, and the input of the next layer is the output result of the previous layer; the structure of the downsampling is:

[0059]

[0060] The downsampling includes 13 convolutional layers, which are the core components of convolutional neural networks. Image features are extracted by sliding the convolution kernel on the image for calculation. The input of each convolutional layer is a feature map, and the input of the next layer is the output result of the previous layer. The convolution kernel of the first convolutional layer is 7×7, and the convolution kernel of the remaining convolutional layers is 5×5. The moving step length stride of each layer is 2. The number of input channels of the first layer is 6, and the number of output channels is 32. The number of input channels of the second layer is 32, and the number of output channels is 64. The input channels and output channels of the third layer are both 64. It is set after the second layer to further convolve the output features of the second layer, and at the same time perform additional feature extraction and maintain the size of the feature map. The input channels and output channels of the fourth convolutional layer are 64 and 128 respectively. The features of the output of the third layer are downsampled again, and the resolution of the feature map is reduced to achieve downsampling. The fifth layer has the same function as the third layer. The convolution layer of the fifth layer extracts deeper features from the output features of the fourth layer, and its input channels and output channels are both 128. Similarly, since the input of the sixth layer is the output of the fifth layer, the input channel number of the sixth layer is 128, and the output channel number is 256. The seventh layer has the same function as the fifth and third layers, and the input channel number and the output channel number are both 256. The input channel of the eighth layer is the output of the seventh layer, that is, 256 channels, and the output channel number is 512. The input and output results of the ninth layer are both 512. The input channel number of the tenth layer is 512, and the output channel number is 1024. The input and output channel number of the eleventh layer are both 1024. The twelfth and thirteenth layers are newly added convolution layers to realize the addition and addition of core modules in the method of the present invention. Its convolution kernel remains the same as above and is still 5×5; the input of the twelfth layer is the output of the eleventh layer, so its input channel number is 1024, and the output channel number is 2048. The function of the thirteenth layer is still not to extract deeper features, and its input and output channel numbers are both 2048. Finally, the length and width of the optical flow feature map obtained by the 13-layer convolutional neural network layer are 2048×2048;

[0061] The optical flow prediction operation is arranged after the downsampling layer and the upsampling layer, and the optical flow prediction operation is arranged before the superposition fusion module;

[0062] The optical flow prediction operation is a 3×3 convolution kernel, the step size is 1, and the number of output channels of the optical flow prediction operation is 2;

[0063] The optical flow prediction operation processes the output result of the downsampling layer, changes the number of channels of the output result, and makes the size of the output result consistent with the size of the input feature map. The optical flow estimation result of the optical flow prediction operation is recorded as:

[0064] flow[B,C,H,W], where B is the batch size, C is the number of channels, H is the height, and W is the width;

[0065] First, the result out_conv7 obtained by the thirteenth convolutional layer is input into the optical flow prediction operation, and the optical flow prediction operation is performed on out_conv7 to obtain the optical flow estimation result flow7[1,2,3,4], where the four parameters [1,2,3,4] represent that the batch size is 1, the number of channels is 2, the height is 3, and the width is 4; the only difference between flow7[1,2,3,4] and the input out_conv7 is the change in the number of channels, which is unified to 2;

[0066] The upsampling layer includes 10 neural network layers, including deconv2, deconv3, deconv4, deconv5, deconv6, upsampled_flow7_to_6, upsampled_flow6_to_5, upsampled_flow5_to_4, upsampled_flow4_to_3 and upsampled_flow3_to_2, where deconv2, deconv3, deconv4, deconv5 and deconv6 are used to amplify the feature map in the convolution process and restore the size and resolution; And upsampled_flow7_to_6, upsampled_flow6_to_5, upsampled_flow5_to_4, upsampled_flow4_to_3 and upsampled_flow3_to_2 process the optical flow results flow2[2,2,80,112], flow3[2,2,40,56], flow4[2,2,20,28], flow5[2,2,10,14], flow6[2,2,5,7] and flow7[1,2,3,4] respectively, but they are essentially realized through transposed deconvolution operations;

[0067] The result of flow7[1,2,3,4] obtained by upsampling layer upsampled_flow7_to_6, the result of out_conv7 obtained by upsampling layer deconv6 and out_conv6 are concatenated to obtain concat6, and concat6 is subjected to optical flow prediction operation to obtain flow6[2,2,5,7]. This operation is the same as the principle of performing optical flow prediction operation on out_conv7 to obtain flow7[1,2,3,4].

[0068] The result of flow6[2,2,5,7] obtained by upsampling layer upsampled_flow6_to_5, the result of out_conv6 obtained by upsampling layer deconv5 and out_conv5 are concatenated to obtain concat5, and concat5 is subjected to optical flow prediction operation to obtain flow5[2,2,10,14]. This operation is the same as the principle of predicting optical flow of out_conv6 to obtain flow6[2,2,5,7].

[0069] The result of flow5[2,2,10,14] obtained by upsampling layer upsampled_flow5_to_4, the result of concat5 obtained by upsampling layer deconv4 and out_conv4 are concatenated to obtain concat4, and concat4 is subjected to optical flow prediction operation to obtain flow4[2,2,20,28]. This operation is the same as the principle of predicting optical flow of out_conv5 to obtain flow5[2,2,10,14].

[0070] The result of flow4[2,2,20,28] obtained by upsampling layer upsampled_flow4_to_3, the result of concat4 obtained by upsampling layer deconv3 and out_conv3 are concatenated to obtain concat3, and concat3 is input into the optical flow prediction operation to obtain flow3[2,2,40,56]. This operation is the same as the principle of predicting the optical flow of out_conv4 to obtain flow4[2,2,20,28].

[0071] The result of flow3[2,2,40,56] obtained by upsampling layer upsampled_flow3to_2, the result of concat3 obtained by upsampling layer deconv2 and out_conv2 are concatenated to obtain concat2, and concat2 is subjected to optical flow prediction operation to obtain flow2[2,2,80,112]. This operation is the same as the principle of predicting optical flow of out_conv3 to obtain flow3[2,2,40,56].

[0072] The superposition fusion module sequentially inputs the optical flow results obtained by the optical flow prediction operation according to different sizes from small to large, and uses a bilinear interpolation method to gradually amplify the optical flow results obtained by performing an optical flow prediction operation on the result of the downsampled thirteenth layer, and respectively adds the optical flow results obtained by upsampling and optical flow prediction operations with the results of the downsampled third, fifth, seventh, ninth, and eleventh layers by pixel, to obtain the final optical flow prediction result;

[0073] The input of the superposition fusion module is flow2[2,2,80,112], flow3[2,2,40,56], flow4[2,2,20,28], flow5[2,2,10,14], flow6[2,2,5,7], flow7[1,2,3,4]; after superposition fusion, the new optical flow results are flow_2_2, flow_3_2, flow_4_2, flow_5_2, flow_6_2, such as Figure 1 , Figure 2 and Figure 3 As shown, the specific process is:

[0074] (1): Use bilinear interpolation method to obtain upsampled_tensor7_6 for flow7[1,2,3,4], so that the batch size, number of channels, height and width of flow7[1,2,3,4] are consistent with the batch size, number of channels, height and width of flow6[2,2,5,7];

[0075] (2): Directly add the pixels of upsampled_tensor7_6 and the pixels corresponding to flow6[2,2,5,7] to obtain flow6_2, completing the superposition fusion;

[0076] (3): Use bilinear interpolation method for upsampled_tensor7_6 to obtain the result as upsampled_tensor7_5, so that it is consistent with the batch size, number of channels, height and width of flow5[2,2,10,14];

[0077] (4): The pixels corresponding to upsampled_tensor7_5 and flow5[2,2,10,14] are directly added to obtain flow5_2, completing the superposition fusion;

[0078] (5): Use bilinear interpolation method for upsampled_tensor7_5 to obtain the result as upsampled_tensor7_4, so that it is consistent with the batch size, number of channels, height and width of flow4[2,2,20,28];

[0079] (6): The pixels corresponding to upsampled_tensor7_4 and flow4[2,2,20,28] are directly added to obtain flow4_2, completing the superposition fusion;

[0080] (7): Use bilinear interpolation method for upsampled_tensor7_4 to obtain the result as upsampled_tensor7_3, so that it is consistent with the batch size, number of channels, height and width of flow3[2,2,40,56];

[0081] (8): The pixels corresponding to upsampled_tensor7_3 and flow3[2,2,40,56] are directly added to obtain flow3_2, completing the superposition fusion;

[0082] (9): Use bilinear interpolation method for upsampled_tensor7_3 to obtain the result as upsampled_tensor7_2, so that it is consistent with the batch size, number of channels, height and width of flow2[2,2,80,112];

[0083] (10): The pixels corresponding to upsampled_tensor7_2 and flow2[2,2,80,112] are directly added to obtain flow2_2, completing the superposition fusion;

[0084] Step 3: Initialize the parameters of the optical flow estimation neural network model;

[0085] Initialize the training parameters of the optical flow estimation neural network model;

[0086] The training parameters are initialized as follows: optimizer momentum adam is 0.9, weight decay coefficient is 0.0001, bias decay coefficient is 0, number of training rounds is 300, batch_size is set to 1, number of data loading threads is 8, optical flow data scaling ratio is 20, learning rate descent strategy is set to [100, 150, 200], initial learning rate is 0.0001, and multi-scale loss function weights are [0.005, 0.01, 0.02, 0.08, 0.32];

[0087] Step 4: Train the optical flow estimation neural network model;

[0088] The FlyingChairs dataset, MPI-Sintel clean dataset, MPI-Sintel final dataset, and KITTI Flow 2015 dataset were input into the optical flow estimation neural network model respectively, and each dataset was trained for 300 iterations separately. At the same time, the trained models were saved and the optical flow estimation accuracy error EPE_multiscale of each model was calculated.

[0089] The method for calculating the optical flow estimation accuracy error EPE_multiscale is:

[0090] The theoretical endpoint average error EPE is calculated as follows:

[0091] ; (1)

[0092] In the formula, represents the estimated horizontal optical flow, represents the true horizontal optical flow, represents the estimated vertical optical flow, Represents the true vertical optical flow; the endpoint average error measures the Euclidean distance between the calculated optical flow vector and the true optical flow vector, that is, the closer the true and horizontal displacement of the estimated optical flow value is to the true displacement, the smaller the calculated EPE is, indicating that the closer the optical flow estimated by the model is to the true optical flow, the higher the accuracy of the model;

[0093] In the method of the present invention, the optical flow estimation accuracy error EPE_multiscale method is implemented as shown in the following formula:

[0094] ;

[0095] ;

[0096] Where, flow_i_2 is flow_2_2 or flow_3_2 or flow_4_2 or flow_5_2 or flow_6_2 or flow_7_2; EPE_i represents the average endpoint error calculated for each different flow_i_2, that is:

[0097] ;

[0098] ;

[0099] ;

[0100] ;

[0101] ;

[0102] Where target is the true value; ||...|| is the Euclidean distance, mean(...) is the average value, and w_i is the weight;

[0103] The calculation steps of EPE_multiscale are to find the weighted average of EPE_2, EPE_3, EPE_4, EPE_5 and EPE_6 with weights of 0.005, 0.01, 0.02, 0.08 and 0.32 respectively, that is, to obtain EPE_multiscale, which is the indicator of precision error.

[0104] The models obtained by training and verifying the FlyingChairs dataset on the network model of the prior art and the method of the present invention are named FlowNetS-F and ADD-BLOCK-F respectively;

[0105] The models obtained by training and verifying the MPI-Sintel clean data set on the network model of the prior art and the method of the present invention are named FlowNetS-Sc and ADD-BLOCK-Sc respectively;

[0106] The models obtained by training and verifying the MPI-Sintel final data set on the network model of the prior art and the method of the present invention are named FlowNetS-Sf and ADD-BLOCK-Sf respectively;

[0107] The models obtained by training and verifying the KITTI Flow 2015 dataset on the network model of the prior art and the method of the present invention are named FlowNetS-K and ADD-BLOCK-K respectively;

[0108] The method of the present invention selects the model with the smallest maximum value of EPE_multiscale as the best model among the 8 models obtained above;

[0109] This paper innovates the optical flow estimation method based on the FlowNetS model framework and conducts an in-depth discussion on it. At the same time, in order to further enhance the network's expression of optical flow features, the ADD-Block superposition fusion module is introduced. The FlyingChairs dataset, KITTI Flow 2015 dataset and MPI-Sintel (clean and final) dataset were used to train and verify the existing technology and the method with the ADD-Block superposition fusion module under the same conditions. The obtained models are: FlowNetS-F, ADD-Block-F, FlowNetS-Sc, ADD-Block-Sc, FlowNetS-Sf, ADD-Block-Sf, FlowNetS-K, ADD-Block-K. In the above results, the ADD-Block-F model has the highest accuracy, followed by FlowNetS-F. These two models are tested on some samples of the Flying Chairs dataset, KITTI Flow 2015 dataset and MPI-Sintel clean dataset, and the obtained optical flow effect diagrams are compared. The effect of ADD-Block-F is better than that of FlowNetS-F. The superposition fusion module ADD-Block has played a great advantage. Compared with the original network structure, the endpoint error accuracy is in the range of 1.137~1.137. The performance of the Chairs dataset was improved by 2.86%, the KITTI Flow 2015 dataset was improved by 0.70%, and the MPI-Sintel (clean and final) dataset was improved by 7.40% and 2.43% respectively, achieving a competitive advantage. Figure 4 , Figure 5 , Figure 6 and Figure 7 shown.

[0110] The specific implementation idea of ​​the present invention is:

[0111] The technical solution of the present invention is: an optical flow estimation method based on improved FlowNetS, which uses a convolutional neural network (CNN) to estimate the motion of pixels in an image sequence to obtain an optical flow field. Specifically, the method is based on the FlowNetS framework of the encoder-decoder architecture and introduces the ADD-Block module. The method uses the ADD-Block module to enhance the feature expression ability of the FlowNetS model and improves the accuracy and robustness of the method through multi-dataset training and Adam optimization algorithm. It is also applied to other visual sensor fusions to improve the accuracy and robustness of its visual tasks.

[0112] (1) Principle;

[0113] Optical flow estimation is performed using convolutional neural networks (CNNs). CNNs are powerful machine learning models that are good at learning features from data and making predictions. It mainly uses an "encoder-decoder" architecture, which can be summarized as follows: First, the encoder. The encoder part is built by a series of convolutional layers, and the main responsibility of these convolutional layers is to extract features from the image. In each layer of convolution, by sliding the filter (kernel) on the image, the model can capture the local features of the image and generate the corresponding feature map. These feature maps are further used in the subsequent optical flow prediction process. As the number of network layers increases, the filter size decreases and the receptive field increases, which can capture more and more complex global features in the image. Some specific techniques are used in the algorithm, such as convolution with a stride of 2, which can reduce the spatial resolution of the feature map and reduce the computational complexity. Second, the intermediate layers, after the encoder extracts the features, some additional processing will be performed, such as using the deconvolution layer to upsample the feature map and restore the spatial resolution. At the same time, the predicted optical flow layer (predict_flow_layer) is also used to convert the extracted features into optical flow estimates. The third part is the decoder, which cascades the features from the encoder and the intermediate layers to fuse information from different scales. Then, a series of convolutional and deconvolutional layers are used again to gradually restore the spatial resolution and obtain the final optical flow prediction result.

[0114] (2) Network structure;

[0115] The method uses a combination of convolutional layers, transposed convolutions, feature concatenation, and upsampling to efficiently estimate optical flow at different resolutions. First, a series of convolutional layers are used to extract features of the input image. These convolutional layers use filters of different sizes and strides, which can capture spatial and temporal features at different scales. Each convolutional layer is optionally followed by a batch normalization layer to improve training stability. Deconvolutional layers perform the opposite operation to ordinary convolutional layers. Instead of reducing the spatial resolution of the input feature map, they increase its resolution. The method uses deconvolutional layers in the decoder path after each downsampling level in the encoder, and the number of output channels of each deconvolutional layer is the same as the number of channels of the corresponding feature map with the same spatial resolution in the encoder, which allows the upsampled feature maps from the deconvolutional layers to be concatenated with the corresponding feature maps from the encoder, preserving spatial and semantic information. In the method, deconvolution operations and interpolation methods are used to upsample the coarse-grained optical flow predictions generated by the encoder part to the resolution of the original image. This method aims to restore the spatial dimensions lost due to the encoder downsampling step, so that the network can produce optical flow estimation results that match the resolution of the input image.

[0116] The method adopts an encoder-decoder structure. The encoder part gradually downsamples the feature map through strided convolution to capture high-level features. The decoder part gradually upsamples the feature map through transposed convolution to restore the spatial resolution. At each stage of the decoder part, the upsampled feature map is concatenated with the corresponding feature map from the encoder path. Feature concatenation can help the decoder preserve spatial details and incorporate high-level semantic information from the encoder. The structure has a total of 5 prediction layers, one for feature maps of different scales (resolutions). Each prediction layer uses a small convolution filter to predict the optical flow of the corresponding scale, which is represented as a 2-channel vector of horizontal and vertical displacements. The coarser resolution optical flow prediction is upsampled through transposed convolution to match the size of the finer resolution feature map. Cropping is then applied to ensure that the prediction and feature map have the same spatial size for concatenation. During training, the method outputs optical flow predictions for all 5 scales; during inference, only the highest resolution optical flow prediction is returned.

[0117] (3) Core modules;

[0118] The existing FlowNetS optical flow estimation method no longer performs downsampling after the Conv6 layer, and directly performs optical flow prediction, which makes it have certain limitations in processing large-scale motion and detail information. The present invention relies on further deepening the depth of the convolution layer and changing the layered superposition fusion strategy, and proposes an ADD-BLOCK superposition fusion module. Its newly added Conv7 layer can extract deeper features, thereby better capturing global motion information. In addition, in order to more effectively fuse optical flow information at different scales, the present invention adopts a strategy of upsampling the highest resolution optical flow results and superimposing them on other scale results respectively, and upsampling and aligning the newly obtained optical flow and the actually required optical flow flow6, flow5, flow4, flow3, and flow2. After bilinear interpolation, upsampled_tensor7_6, upsampled_tensor7_5, upsampled_tensor7_4, upsampled_tensor7_3, and upsampled_tensor7_2 are obtained, and then the above optical flows are superimposed and fused to obtain the superimposed flow6, flow5, flow4, flow3, and flow2. Instead of fusing layer by layer like the original method, this fusion strategy can effectively integrate information at different scales and improve the accuracy and robustness of the final optical flow estimation.

[0119] (4) Method optimization;

[0120] Internal parameters such as weights (W) and biases (b) play a central role in evaluating the accuracy of the model's predictions of the target values ​​Y in the test set. These parameters are the cornerstone of constructing the loss function E(x), which is used to quantify the degree of deviation between the model's predictions and the actual values. In order to ensure that the model can produce accurate results, these internal parameters play a vital role in the training process. Therefore, various optimization strategies and algorithms are used to update and calculate these network parameters, aiming to make them approach or reach the optimal value, which in turn affects the training effect and final output of the model.

[0121] The method of the present invention adopts an adaptive moment estimation method Adaptive Moment Estimation, Adam. The Adam algorithm only needs to set a learning rate parameter and is simple to use; it can effectively utilize the first-order moment and second-order moment information to accelerate the convergence speed; it can effectively suppress gradient oscillation, thereby improving stability. The calculation steps of the Adam algorithm are: initializing model parameters and learning rates, calculating the first-order moment and second-order moment of each parameter, using the first-order moment and second-order moment to update the learning rate of each parameter, and updating the model parameters in the opposite direction of the gradient. Repeat the latter three steps until the termination condition is reached.

[0122] The Adam method works well, showing faster convergence and more effective learning results than other adaptive learning rate algorithms. In addition, it can successfully correct some common problems in other optimization techniques, such as learning rate disappearance, slow convergence, or large fluctuations in the loss function due to high variance parameter updates.

Claims

1. An optical flow estimation method based on improved FlowNetS, characterized in that: The steps include: Step 1: Build data sets, training sets, and validation sets; Step 2: Build an optical flow estimation neural network model; The optical flow estimation neural network model includes downsampling, upsampling, optical flow prediction operation and superposition fusion modules; The training set is used as input to obtain the optical flow result through downsampling, and then the optical flow result obtained through downsampling is used as input for upsampling. The downsampling result and the upsampling result are spliced, and the spliced ​​result is input into the optical flow prediction operation. The result of the optical flow prediction operation is input into the superposition fusion module to obtain the final optical flow prediction result; The downsampling includes 13 convolutional layers; the first convolutional layer to the thirteenth convolutional layer are connected in sequence, the input of each convolutional layer is a feature map, and the input of the next layer is the output result of the previous layer; The optical flow prediction operation is arranged after the downsampling layer and the upsampling layer, and before the superposition fusion module; the optical flow prediction operation processes the result output by the downsampling layer, changes the number of channels of the output result, and makes the size of the output result consistent with the size of the input feature map; The optical flow prediction operation is a 3×3 convolution kernel, the step size is 1, and the number of output channels of the optical flow prediction operation is 2; The upsampling layer includes 10 neural network layers; The superposition fusion module uses the optical flow results obtained by the optical flow prediction operation as input in order of size from small to large, uses a bilinear interpolation method to gradually amplify the optical flow results obtained by performing an optical flow prediction operation on the result of the downsampled thirteenth layer, and adds the optical flow results obtained by upsampling and optical flow prediction operations with the results of the downsampled third, fifth, seventh, ninth, and eleventh layers, respectively, to obtain the final optical flow prediction result by pixel. Step 3: Initialize the parameters of the optical flow estimation neural network model; Step 4: Train the optical flow estimation neural network model; Different types of training sets in the data set are input into the optical flow estimation neural network model respectively, and the training is iterated 300 times. The trained model is saved, and the optical flow estimation accuracy error EPE_multiscale of each model is calculated. The model with the smallest maximum optical flow estimation accuracy error is the best model.

2. According to claim 1, the optical flow estimation method based on improved FlowNetS is characterized in that: In step 1, the steps to construct the data set, training set, and validation set are: The datasets include the FlyingChairs dataset, the MPI-Sintel clean dataset, the MPI-Sintel final dataset, or the KITTI Flow 2015 dataset. In the data set, the ratio of the number of training sets to the number of validation sets is 8:2; When the dataset is the KITTI Flow 2015 dataset, the KITTI Flow 2015 dataset is enhanced; The enhancement method is as follows: randomly crop, vertically flip, and horizontally flip the images in the KITTI Flow 2015 dataset in sequence; the size of the randomly cropped images is 320×448; When the dataset is the FlyingChairs dataset or the MPI-Sintel dataset, a strong enhancement operation is performed on the FlyingChairs dataset or the MPI-Sintel dataset; The method of strong enhancement operation is: randomly translate and randomly rotate the selected data set in turn; the random translation is a random translation of a distance between -10 and 10 pixels; the random rotation is a rotation between -10 and 10 degrees at intervals of 5 degrees.

3. The optical flow estimation method based on improved FlowNetS according to claim 1, characterized in that: The structure of the downsampling is as follows: the downsampling includes 13 convolutional layers for extracting image features; The convolution kernel of the first convolution layer is 7×7, and the convolution kernels of the remaining convolution layers are 5×5. The stride of each layer is 2. The number of input channels of the first layer is 6, and the number of output channels is 32. The number of input channels of the second layer is 32, and the number of output channels is 64. The number of input channels and output channels of the third layer are both 64; the input channels and output channels of the fourth convolution layer are 64 and 128 respectively; the number of input channels and output channels of the fifth layer are both 128; the number of input channels of the sixth layer is 128, and the number of output channels is 256; the number of input channels and output channels of the seventh layer are both 256; the input channels of the eighth layer are 256, and the output channels are 512; The number of input channels and output channels of the ninth layer are both 512; The number of input channels of the tenth layer is 512, and the number of output channels is 1024; the number of input channels and the number of output channels of the eleventh layer are both 1024; the twelfth and thirteenth layers are newly added convolutional layers, and the convolution kernels of the twelfth and thirteenth layers are both 5×5; the number of input channels of the twelfth layer is 1024, and the number of output channels is 2048, and the number of input channels and the output channels of the thirteenth layer are both 2048.

4. The optical flow estimation method based on improved FlowNetS according to claim 1, characterized in that: The method for calculating the optical flow estimation accuracy error EPE_multiscale is: The calculation is implemented as follows: EPE_i=mean(||flow_i_2-target||); EPE_multiscale=∑(w_i*EPE_i); Where, flow_i_2 is flow_2_2 or flow_3_2 or flow_4_2 or flow_5_2 or flow_6_2; EPE_i represents the average endpoint error calculated for each different flow_i_2, that is: EPE_2=mean(||flow_2_2-target||); EPE_3=mean(||flow_3_2-target||); EPE_4=mean(||flow_4_2-target||); EPE_5=mean(||flow_5_2-target||); EPE_6=mean(||flow_6_2-target||); Where target is the true value; ||…|| is the Euclidean distance, mean(…) is the average value, and w_i is the weight; the calculation steps of EPE_multiscale are to calculate the weighted average of EPE_2, EPE_3, EPE_4, EPE_5 and EPE_6 with weights of 0.005, 0.01, 0.02, 0.08 and 0.32 respectively, that is, to obtain EPE_multiscale.

Images