An unsupervised optical flow estimation method based on improved recurrent generative adversarial network

Abstract

An unsupervised optical flow estimation method based on an improved recurrent generative adversarial network. First, the generator structure is improved. Atrous convolution is introduced to increase the network's receptive field without reducing the size of the feature map. At the same time, the channel-spatial attention module is introduced to make the generator more focused on the moving objects in the input image. Secondly, Spynet is introduced as a teacher network to guide the generator. Finally, optical flow consistency loss and contour consistency loss are introduced. The improvements are put into the recurrent generative adversarial network framework, and the generator is responsible for using the generated image to deceive the discriminator. The generator and the discriminator are trained adversarially in this way. When the discriminator cannot distinguish the source of the image, the training is completed, and then the optical flow estimation can be realized using two consecutive images. This method can be used for autonomous navigation tasks based on optical flow that are easily affected by harsh conditions such as lighting changes and rainy days, such as drones, unmanned vehicles, and underwater unmanned submersibles.

Description

Technical Field

[0001] The present invention relates to the field of photovoltaic power generation system operation and maintenance technology, and specifically to an optical flow unsupervised estimation method based on an improved cyclic generative adversarial network. An arbitrary optical flow result image and two frames of optical flow images to be solved are used to implement unsupervised training of the network, thereby completing the optical flow estimation of an input image. Background Art

[0002] As a key technology for unmanned systems (such as unmanned vehicles, drones, and underwater unmanned submersibles) to perform tasks, navigation has attracted the attention of more and more scholars. At present, the mainstream navigation methods of unmanned systems are the Global Positioning System (GPS) and the Inertial Navigation System (INS). However, GPS is generally unavailable in urban or indoor environments due to signal obstruction, and INS inevitably introduces cumulative errors due to the use of integration to obtain navigation parameters, resulting in reduced accuracy over long periods of time. Due to the limitations of GPS and INS themselves, more and more navigation methods have been developed. Optical flow, as a type of bionic visual navigation, is inspired by nature. At the biological level, insects such as bees use the optical flow information in their compound eyes to complete navigation, obstacle avoidance, and landing. In computer vision, optical flow refers to the relative movement of corresponding object pixels in two frames of images due to motion, so optical flow contains rich motion information and scene structure information. At present, there are two main ways to obtain optical flow, namely classical methods and learning methods. The classical method is dominated by the variational model method, mainly including the HS method and the LK method. Although the variational model method can obtain accurate optical flow estimation, the accuracy of optical flow solution under conditions such as large displacement, motion occlusion and illumination change needs to be further improved. At the same time, due to the existence of a large number of iterative calculations, the variational model method has poor real-time performance, which limits the application of this method. With the rapid development of graphics processing units (GPUs) and computer technology, the optical flow estimation method based on deep learning has attracted widespread attention due to its strong robustness, high real-time performance and strong adaptability. However, most of the optical flow estimation methods based on deep learning currently require the corresponding pixel-by-pixel optical flow true value to participate in the training of the model, and the acquisition of the optical flow true value is extremely cumbersome, which greatly limits the optical flow estimation method based on deep learning. The cyclic generative adversarial network is an unsupervised network based on the idea of ​​zero-sum game. The network realizes the unsupervised mutual conversion of pictures in different domains through the cyclic confrontation between the generator and the discriminator. Based on this, the present invention adopts a cyclic generative adversarial network to realize the cyclic conversion of two frames of pictures and optical flow maps, thereby completing the accurate estimation of optical flow maps.

[0003] In general, optical flow is a bionic visual navigation method. The classical optical flow method is greatly affected by the environment and has high computational complexity. The deep learning method is difficult to obtain the true value of the training sample, which limits its application. The recurrent generative adversarial network can complete the optical flow estimation in the absence of the true value of the sample with its unsupervised training method, but the classical recurrent generative adversarial network does not complete specific optimization for optical flow estimation, which leads to inaccurate optical flow estimation results. Therefore, the present invention makes corresponding improvements to the recurrent generative adversarial network based on the characteristics of the optical flow estimation task. Summary of the invention

[0004] In order to solve the above technical problems, the present invention proposes an unsupervised estimation method of optical flow based on an improved cyclic generative adversarial network to solve the problem that the true value is required to estimate the optical flow field using a deep learning method, and the network is implemented in an end-to-end training mode. Compared with conventional optical flow estimation networks such as Flownet, the optical flow estimation method proposed in the present invention abandons the dependence on the true value of the training sample through a cyclic adversarial method of samples from two different domains. First, the estimation accuracy of the generated optical flow is improved by improving the generator and introducing an attention mechanism. Then, the Spynet network is used as a teacher network to guide the training of the generator, reducing the risk of the generative adversarial network falling into mode collapse, while alleviating the network's dependence on the number of training samples, thereby speeding up network training. Finally, the loss function after optimization is used to further improve the estimation accuracy of the optical flow. Through the above improvements, the estimation effect of the optical flow of moving objects is improved, and an accurate and robust optical flow field is obtained.

[0005] An optical flow unsupervised estimation method based on an improved cyclic generative adversarial network is characterized by comprising the following steps:

[0006] (1) Training phase: The training data includes image I1, image I2, and optical flow visualization image I real There are three parts, among which pictures I1 and I2 are used to calculate the optical flow, I real It is used to guide the optical flow image style finally generated by the network. Specifically, after merging the images I1 and I2 in the channel dimension, a 6-dimensional tensor I is formed. 1+2 , with I real The generator G in the network is trained according to I 1+2 Generate a tensor FLOW with the same size as the original image but with 2 channels, and then use visualization to convert FLOW to a tensor with the same size as I real Images with similar style I fake , and I fake with I realAt the same time, the input is given to the discriminator D for identification. D is committed to distinguishing the source of the input samples, while the generator is committed to deceiving the discriminator with the generated samples. The network forms adversarial training in this way until the discriminator cannot distinguish the source of the image, and the training is completed;

[0007] (2) Prediction stage: After completing the training of the recurrent generative adversarial network, freeze the discriminator and keep only the generator. At this time, for the two images to be calculated for optical flow and After channel merging, the image is obtained Visualizing images with arbitrary optical flow Input to the network, the network will use Complete The predicted optical flow map of the two frames is obtained.

[0008] As a further improvement of the present invention, the specific meaning of improving the cyclic generative adversarial network in step (1) is as follows:

[0009] The improvements of the proposed optical flow calculation method compared to the traditional recurrent generative adversarial network mainly include three aspects. First, the generator structure is improved. The atrous convolution is introduced to increase the receptive field of the network without reducing the size of the feature map. At the same time, the channel-spatial attention module CSAM is introduced to make the generator more focused on the moving objects in the input image, thereby improving the estimation accuracy of the optical flow. Secondly, Spynet is introduced as a teacher network to guide the generator, reducing the network's dependence on large amounts of data and reducing the risk of the recurrent generative adversarial network falling into mode collapse. Finally, optical flow consistency loss and contour consistency loss are introduced to improve the accuracy of the generated optical flow.

[0010] As a further improvement of the present invention, the specific training process of step (1) is as follows:

[0011] Assume that there are two domains in the problem to be solved, domain A and domain B. Domain A represents the 6-dimensional tensor I generated by channel splicing of two optical flow images I1 and I2 to be solved. 1+2 , domain B represents the optical flow graph I real The task of the proposed improved cycle generation network is simplified to the cycle conversion between domain A and domain B, that is, using the generator G A Will I 1+2 Transform to domain B and use generator G B Will I realThe image is converted to domain A. After the conversion, the discriminator is used to identify the source of the image. That is, the image generated by the generator and the real image are input to the discriminator D. The discriminator judges the source of the image through training and guides the optimization of the generator parameters to improve the authenticity of the image generated by the generator. The discriminator and the generator are trained together in this confrontation. During the training process, the optical flow consistency loss improves the accuracy of the generated optical flow by calculating the relationship between the generated optical flow and I1 and I2. The contour consistency loss improves the accuracy of the generated optical flow by comparing the generated optical flow map I fake The difference between the contours of I1 and I2 is used to refine I fake At the same time, Spynet is introduced as the teacher network, and the EPE loss function is used to guide the training of the generator. When the discriminator cannot distinguish whether the image comes from the generator, the Nash equilibrium is reached and the training is completed.

[0012] As a further improvement of the present invention, the specific operation process of step (2) is as follows:

[0013] After completing the training of the improved cyclic generative adversarial network, freeze the discriminator and only use the generator. Similar to the training process, put the two images to be calculated optical flow After channel splicing, we get an image representing domain A. With any real optical flow map As domain B, it is also input into the trained network. At this time, the generator G A and G B It will complete the mutual conversion between domain A and domain B, and then complete the optical flow prediction under unsupervised conditions.

[0014] Beneficial effects:

[0015] (1) The present invention improves the generator structure by introducing dilated convolution to increase the network’s receptive field without reducing the size of the feature map. At the same time, the channel-spatial attention module (CSAM) is introduced to make the generator more focused on moving objects in the input image, thereby improving the estimation accuracy of optical flow.

[0016] (2) The present invention introduces a teacher network to guide the generator, thereby reducing the training cost and the risk of falling into model collapse.

[0017] (3) The present invention improves the optical flow direction and intensity information generated by the network by introducing optical flow consistency loss, and refines the contour of the moving object by contour consistency loss. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is the overall framework of the improved cyclic generative adversarial network of the present invention;

[0019] Figure 2is a color wheel diagram used to determine the direction and magnitude of optical flow;

[0020] Figure 3 This is the network structure of the generator feature extraction part proposed by the present invention;

[0021] Figure 4 The optical flow prediction network structure of the generator proposed in the present invention;

[0022] Figure 5 The channel-spatial attention module proposed in this invention

[0023] Figure 6 The discriminator structure used in the present invention;

[0024] Figure 7 This is an embodiment of the optical flow estimation of the present invention. DETAILED DESCRIPTION

[0025] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments:

[0026] The present invention is aimed at the demand for high robustness and high adaptability of optical flow acquisition under illumination change conditions, and uses deep learning methods to complete the solution of the optical flow of moving objects. Specifically, optical flow is a means of describing the movement of objects pixel by pixel. It is very difficult to obtain the true value of dense optical flow field. The insufficient data set caused by this makes deep network training easy to overfit, reducing the accuracy of optical flow estimation. In order to overcome the disadvantage that deep convolutional networks such as traditional flownet need true values ​​to participate in model training, the present invention introduces an improved cyclic generative adversarial network to perform unsupervised acquisition of the optical flow field of moving objects. First, the present invention improves the network structure of the generator, and introduces a hole convolution to improve the receptive field of the network while maintaining the size of the feature map, and introduces a CSAM attention mechanism to make the network pay more attention to the foreground part of the movement in the input picture, thereby improving the estimation accuracy of the generator for optical flow. Secondly, Spynet is introduced as a teacher network to guide the generator, reducing the risk of the network falling into mode collapse, and further improving the accuracy and robustness of the optical flow map. Finally, the optical flow consistency loss is introduced to improve the optical flow direction and intensity information generated by the network, and the contour consistency loss is introduced to refine the contour of the moving object. This method can be used for autonomous navigation tasks based on optical flow using drones, unmanned vehicles, and underwater unmanned submersibles, which are easily affected by harsh conditions such as lighting changes and rainy days.

[0027] The overall framework of the improved cyclic generative adversarial network of the present invention is as follows: Figure 1 As shown in the figure, two RGB images I1 and I2 whose optical flow is to be solved are concatenated through channels to generate a 6-dimensional tensor, which is convenient for network training.

[0028] In the cyclic generative adversarial network, real_A and real_B are two types of pictures with no corresponding relationship. In the present invention, real_A is two frames of optical flow pictures to be solved, which contain optical flow information. real_B is any real optical flow map, which is mainly responsible for indicating the training direction for the network. The cyclic generative adversarial network consists of two completely symmetrical branches. To simplify the description, the present invention only explains real_A to rec_A. First, real_A passes through the generator G. A Generate a two-dimensional optical flow matrix, and then convert the two-dimensional matrix into a three-dimensional image through optical flow visualization operation, that is, fake_B, that is, generator G A Responsible for converting the original image into an optical flow image. Then fake_B and real_A are simultaneously input to the discriminator D A In the discriminator, the source of the image is determined. If it comes from the generator, it is marked as 0 as much as possible. If it comes from the real sample domain B, it is marked as 1 as much as possible. After that, fake_B is input into the generator G B In the example above, we generate a picture rec_A similar to real_A, i.e., the generator G B Responsible for converting the image from the optical flow map to the original image. Similarly, in the other branch, real_B also undergoes the cyclic conversion of the two generators, which will not be repeated here. This cyclic generation mechanism of the cyclic generative adversarial network enables it to have the ability of unsupervised learning. It does not require a one-to-one correspondence between the true values ​​of the samples, which greatly reduces the difficulty of obtaining training samples.

[0029] The loss function of the generative adversarial network is explained below.

[0030] Since rec_A is the result of real_A undergoing two generator cycles, the network requires rec_A to be as similar as possible to real_A, so as to ensure the semantic correctness of fake_B. Based on this, the cycle consistency loss is introduced.

[0031]

[0032] Among them, x and y represent the two images real_A and real_B respectively, E represents the expected value of the data, and F represents the generator G A , H represents the generator G B .

[0033] Secondly, there is the adversarial loss function in the cyclic generative adversarial network, which guides the game adversarial training between the generator and the discriminator. The adversarial loss is as follows:

[0034]

[0035] It can be seen from the adversarial loss that the discriminator is committed to maximizing the loss, that is, to distinguish the fake images generated by the generator from the real images as much as possible. The generator achieves the purpose of deceiving the discriminator by minimizing the adversarial loss function.

[0036] like Figure 2 As shown, it is a color wheel diagram for representing the size and direction of optical flow. The color wheel is based on the HIS color space. When it rotates tangentially, it represents different directions of optical flow. The saturation increases radially outward from the center of the color wheel, which represents the gradual increase in the size of the optical flow. Therefore, the direction and size of movement in the optical flow diagram can be determined one by one according to the color wheel. The visual conversion of optical flow is based on I1, and the movement of objects in I2 is displayed using the HIS color space. Based on this, the present invention introduces optical flow consistency loss in combination with the definition of optical flow, which is specifically defined as follows:

[0037] L OF =ρ(I2(x2+u,y2+v)-I1(x1,y1))(3)

[0038] Among them, u and v represent the horizontal and vertical optical flows calculated by the network, I1 and I2 represent the two original images of the optical flow to be solved, x and y are the coordinates in the corresponding images, ρ(x) = (x 2 +0.001 2 ) α is the Charbonnier loss function, and α is taken as 0.5 in the present invention. The definition of optical flow is the movement of corresponding pixels from I1 to I2. Therefore, the consistency loss L can be used to calculate the optical flow. OF To optimize the accuracy of optical flow.

[0039] Secondly, the object contour information in the optical flow map is generated based on the position of the moving object in I1. Therefore, the contour loss function is introduced to optimize the contour details of the generated optical flow map compared to I1. The details are as follows:

[0040] L outline =δ(F(x))-δ(I1)(4)

[0041] Among them, δ represents the binary contour extraction operation on the entire image.

[0042] Then, the present invention introduces Spynet as a teacher network to guide the optical flow generated by the generator. Deep learning training requires a large amount of sample data. The introduction of a large amount of training data also slows down the network training. The introduction of the teacher network can reduce the dependence of the cyclic generative adversarial network on training samples, improve the training speed, and also reduce the risk of the network falling into mode collapse. The present invention uses EPE loss as the loss of the teacher network.

[0043]

[0044] in, are the output optical flow values ​​of the teacher network in the horizontal and vertical directions, respectively. and are the optical flow values ​​estimated by the generator in the horizontal and vertical directions respectively. N is the total number of optical flow values.

[0045] Finally, the optimized cyclic generative adversarial network loss function adopted by the present invention is:

[0046] L=L cyc +λ1L GAN +λ2L OF +λ3L outline +λ4L Teacher (6)

[0047] Among them, λ1-λ4 represent weight coefficients.

[0048] In order to generate more accurate optical flow information, the present invention improves the generator in the classic recurrent generative adversarial network by combining the dilated convolution and attention mechanism with the optical flow convolution network. The improved generator is divided into two parts, namely the generator feature extraction network and the generator optical flow prediction network. The specific structures are as follows: Figure 3 and Figure 4 As shown. In the feature extraction part of the generator, the two original images of the optical flow to be solved are spliced ​​in the channel dimension to form a 6-dimensional tensor, and then after three layers of convolution layers (conv1-conv2), the feature map size is reduced to 1 / 8 of the original image, and the dimension is also increased from 6 dimensions to 256 dimensions. Since dense optical flow estimation belongs to pixel-by-pixel prediction, excessive downsampling of the feature map will cause part of the feature information to be lost, which is not conducive to accurate estimation of the optical flow. To address this problem, the present invention uses multi-layer dilated convolution after the feature map is reduced to 1 / 8 of the size, which can obtain a larger receptive field without reducing the size of the feature map. At the same time, since the subsequent optical flow prediction network uses part of the information of the feature extraction network, the channel-spatial attention module (CSAM) is introduced in these feature layers to make the generator more focused on moving objects in the input image, which is beneficial to improving the estimation accuracy of the optical flow. CSAM module Figure 5 shown.

[0049] The CSAM module first calculates the channel attention, compresses the spatial dimension of the feature layer of the input size C*W*H through maximum pooling and average pooling operations, and outputs two matrices of size C*1*1, which have a global receptive field to some extent. Then, a 3*3 learnable convolution is used to extract features from these two matrices and then sum them up. The result of the summation is then activated by sigmoid and mapped to [0,1] to obtain a channel attention of size C*1*1. Finally, the channel attention is multiplied with the C*W*H feature of the original input to obtain the feature map F after the channel attention mechanism operation. CA In the spatial attention module of CSAM, we first convert F CA Perform the maximum and mean operations in the channel dimension, output two matrices of size 1*W*H, and then perform the concatenation operation. Input the resulting matrix of size 2*W*H into a 7*7 convolution layer, and then go through batch normalization and LeakeyReLU activation layer to output a matrix of size 1*W*H. Use sigmoid activation on this matrix to get a spatial attention of size 1*W*H. Finally, combine this spatial attention with F CA Multiply them together to get the final output feature map containing channel and spatial attention.

[0050] In the optical flow prediction part of the generator, the network structure is as follows Figure 4 As shown. The optical flow is predicted layer by layer, and each layer of optical flow prediction is implemented by a layer of convolution, and a total of 5 layers of optical flow are predicted. In predicting the optical flow, the prediction results of the previous layer (except the first layer) need to be combined with the prediction results of the previous layer, which is conducive to enhancing the context information of the optical flow prediction and improving the accuracy of the optical flow prediction. Before predicting the optical flow, a splicing operation is required. Taking the solution of flow5 as an example, before solving flow5, the 1024-dimensional features generated by the generator feature extraction network conv6-1 need to be input into the transposed convolution layer with hole convolution and CSAM to obtain a 512-dimensional feature vector, and then it is spliced ​​with the 512-dimensional features output by conv5-1 and the 2-dimensional flow6 to obtain a total of 1026-dimensional features, which are input into flow5 for optical flow prediction. Then, the update iteration is continuously performed until the flow2 of size H / 4*W / 4*2 is calculated, and then flow2 is subjected to four times upsampling to obtain the final optical flow.

[0051] like Figure 6 This is the discriminator network structure used in the present invention. This network is a two-classification network that is dedicated to separating real images from images generated by the generator. That is, the images generated by the generator are marked as 1 as much as possible, and the real images are marked as 0.

[0052] In the training phase of the cyclic generative adversarial network, two original images of the optical flow to be solved are spliced ​​according to the channel dimension as domain A, and any real optical flow image is domain B. They are input into the improved cyclic generative adversarial network at the same time, and the discriminator and generator are trained alternately. The adversarial and cyclic mechanisms of the network are used to achieve unsupervised training of the network. The optical flow consistency loss is used to improve the prediction accuracy of the optical flow. The contour information of the optical flow is optimized by the contour loss, and the risk of the network falling into mode collapse is reduced by the teacher network. At the same time, the network's demand for large data volume is reduced, and the training speed is improved. The prediction accuracy of the optical flow is further improved by the CSAM attention module with a hollow convolution kernel. Then, the mutual conversion from domain A to domain B and from domain B to domain A is completed, that is, the solution of the original image to the optical flow is completed. Then, the image generated by the data generated by the generator after visual conversion and the real image of the original domain B are input into the discriminator together. The discriminator is trained according to the image source, and is committed to marking the samples generated by the generator as 0 and the real samples as 1 through training. The generator is committed to deceiving the discriminator with the generated data, thereby realizing adversarial training between the discriminator and the generator, and then continuously optimizing its own parameters. When the discriminator cannot distinguish the source of the image, the training of the generative adversarial network is completed.

[0053] In the prediction stage of the cyclic generative adversarial network, all image pairs for which the optical flow is to be solved and any number of arbitrary optical flow maps are simultaneously input into the network. The network can then realize the cyclic transformation of domain A and domain B, and thus realize the prediction of the optical flow.

[0054] According to an example of the present invention, the detailed training device information and parameter information are as follows. In terms of training equipment, the server CPU used in the present invention is 2 Intel Xeon E5-2698 V4, the GPU is 2 Nvidia RTX3090, the video memory is 24GB*2, the total machine memory is 128GB, and the storage is 1TB SSD+4TB HDD. In terms of training parameters, a total of 200 epochs are trained, among which the learning rate is fixed and the size is 0.0002 for the first 100 epochs, and the learning rate of the next 100 epochs continuously decays from 0.0002 until the learning rate decays to 0 at the 200th epoch. At the same time, the teacher network spynet works in the first 100 epochs, and λ4=0 in the next 100 epochs. In the above configuration, the training time for one epoch is approximately 610s, the batch size is set to 4, and the deep learning framework used is Pytroch. Figure 7 It can be seen that the classic cycle-GAN does not make any improvements to the optical flow estimation, resulting in unclear contours of moving objects and large errors. After the improvement of the present invention, the accuracy of optical flow estimation is significantly improved, and the detailed information is also richer.

[0055] The above description is only a preferred embodiment of the present invention and does not constitute any other form of limitation to the present invention. Any modification or equivalent change made based on the technical essence of the present invention still falls within the scope of protection required by the present invention.

Claims

1. An unsupervised optical flow estimation method based on an improved cyclic generative adversarial network, characterized by: The following steps are involved: (1) Training phase: The training data includes image I1, image I2, and optical flow visualization image I real There are three parts, among which pictures I1 and I2 are used to calculate the optical flow, I real It is used to guide the optical flow image style finally generated by the network. Specifically, after merging the images I1 and I2 in the channel dimension, a 6-dimensional tensor I is formed. 1+2 , with I real The generator G in the network is trained according to I 1+2 Generate a tensor FLOW with the same size as the original image but with 2 channels, and then use visualization to convert FLOW to a tensor with the same size as I real Images with similar style I fake , and I fake with I real At the same time, the input is given to the discriminator D for identification. D is committed to distinguishing the source of the input samples, while the generator is committed to deceiving the discriminator with the generated samples. The network forms adversarial training in this way until the discriminator cannot distinguish the source of the image, and the training is completed; The specific meaning of the improved cyclic generative adversarial network in step (1) is as follows: The proposed optical flow calculation method has three main improvements over the traditional recurrent generative adversarial network. First, the generator structure is improved. The hole convolution is introduced to increase the network's receptive field without reducing the size of the feature map. At the same time, the channel-spatial attention module CSAM is introduced to make the generator more focused on the moving objects in the input image, thereby improving the estimation accuracy of the optical flow. Second, Spynet is introduced as a teacher network to guide the generator, reducing the network's dependence on large amounts of data and reducing the risk of the recurrent generative adversarial network falling into mode collapse. Finally, the optical flow consistency loss and contour consistency loss are introduced to improve the accuracy of the generated optical flow. The specific training process of step (1) is: Assume that there are two domains in the problem to be solved, domain A and domain B. Domain A represents the 6-dimensional tensor I generated by channel splicing of two optical flow images I1 and I2 to be solved. 1+2 , domain B represents the optical flow graph I real The task of the proposed improved cycle generation network is simplified to the cycle conversion between domain A and domain B, that is, using the generator G A Will I 1+2 Transform to domain B and use generator G B Will I real The image is converted to domain A. After the conversion, the discriminator is used to identify the source of the image. That is, the image generated by the generator and the real image are input to the discriminator D. The discriminator judges the source of the image through training and guides the optimization of the generator parameters to improve the authenticity of the image generated by the generator. The discriminator and the generator are trained together in this confrontation. During the training process, the optical flow consistency loss improves the accuracy of the generated optical flow by calculating the relationship between the generated optical flow and I1 and I2. The contour consistency loss improves the accuracy of the generated optical flow by comparing the generated optical flow map I fake The difference between the contours of I1 and I2 is used to refine I fake At the same time, Spynet is introduced as the teacher network, and the EPE loss function is used to guide the training of the generator. When the discriminator cannot distinguish whether the image comes from the generator, the Nash equilibrium is reached and the training is completed. (2) Prediction stage: After completing the training of the recurrent generative adversarial network, freeze the discriminator and keep only the generator. At this time, for the two images to be calculated for optical flow and After channel merging, the image is obtained Visualizing images with arbitrary optical flow Input to the network, the network will use Complete The predicted optical flow map of the two frames is obtained; The specific operation process of step (2) is as follows: After completing the training of the improved cyclic generative adversarial network, freeze the discriminator and only use the generator. Similar to the training process, put the two images to be calculated optical flow After channel splicing, we get an image representing domain A. With any real optical flow map As domain B, it is also input into the trained network. At this time, the generator G A and G B It will complete the mutual conversion between domain A and domain B, and then complete the optical flow prediction under unsupervised conditions.

Images