Optical flow estimation method, device and storage medium

Abstract

The present invention provides an optical flow estimation method, device, and storage medium, belonging to the field of image processing. The method comprises: training a training model using multiple original images to obtain an optical flow estimation model; performing image processing on a video to be estimated to obtain multiple video groups to be estimated; performing optical flow image estimation on each video group to be estimated using the optical flow estimation model, obtaining target optical flow images corresponding to each video group to be estimated, and using all target optical flow images as optical flow estimation results. The present invention enables end-to-end training, thereby quickly obtaining an approximate optical flow estimate between two images, saving significant computational resources and time, and improving the accuracy of optical flow estimation.

Description

Technical Field

[0001] The present invention mainly relates to the field of image processing technology, and in particular to an optical flow estimation method, device and storage medium. Background Art

[0002] Optical flow is a concept used in the detection of moving objects in the field of view. It is used to describe the motion of the observed target, surface or edge caused by the motion relative to the observer. The optical flow method is an important method for motion image analysis. Therefore, the estimation of optical flow of images or videos has become a very important research direction in computer vision and other image processing fields. Its main application scenarios include moving target detection, target tracking, action recognition, video interpolation, autonomous driving, etc.

[0003] Early, the most effective optical flow estimation methods used energy minimization. However, for real-time applications, optimizing a complex energy function consumes significant computational resources and time. Taylor approximation is one of the most commonly used methods in numerical approximation, and many tasks in image processing and computer vision can be approximated as first-order Taylor problems. However, approximating higher-order Taylor series expansions has not been widely used in image processing and computer vision, particularly for optical flow estimation. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide an optical flow estimation method, device and storage medium in response to the deficiencies of the existing technology.

[0005] The present invention solves the above technical problems with the following technical solution: A method for optical flow estimation comprises the following steps:

[0006] Importing multiple original images and constructing a training model, training the training model using the multiple original images to obtain an optical flow estimation model;

[0007] Importing a video to be estimated and performing image processing on the video to be estimated to obtain multiple video groups to be estimated;

[0008] Optical flow image estimation is performed on each of the to-be-estimated video groups using the optical flow estimation model to obtain target optical flow images corresponding to each of the to-be-estimated video groups, and all target optical flow images are used as optical flow estimation results.

[0009] Another technical solution of the present invention to solve the above technical problem is as follows: an optical flow estimation device, comprising:

[0010] A model training module is used to import multiple original images and build a training model, and train the training model using the multiple original images to obtain an optical flow estimation model;

[0011] An image processing module is used to import a video to be estimated and perform image processing on the video to be estimated to obtain multiple video groups to be estimated;

[0012] The optical flow estimation result acquisition module is used to perform optical flow image estimation on each of the video groups to be estimated using the optical flow estimation model, obtain target optical flow images corresponding to each of the video groups to be estimated, and use all target optical flow images as optical flow estimation results.

[0013] Another technical solution of the present invention to solve the above technical problems is as follows: an optical flow estimation device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the optical flow estimation method described above is implemented.

[0014] Another technical solution of the present invention to solve the above technical problem is as follows: a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the optical flow estimation method as described above is implemented.

[0015] The beneficial effects of the present invention are: an optical flow estimation model is obtained by training the training model with original images, multiple video groups to be estimated are obtained by image processing of the video to be estimated, and optical flow estimation results are obtained by estimating the optical flow images of the video groups to be estimated through the optical flow estimation model. End-to-end training can be performed to quickly obtain approximate optical flow estimation between two images, which can save a lot of resources and time in calculation and improve the accuracy of optical flow estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A schematic diagram of the process of an optical flow estimation method provided by an embodiment of the present invention;

[0017] Figure 2 A schematic diagram of the model training process provided by an embodiment of the present invention;

[0018] Figure 3 This is a module block diagram of the optical flow estimation device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0019] The principles and features of the present invention are described below with reference to the accompanying drawings. The examples given are only used to explain the present invention and are not used to limit the scope of the present invention.

[0020] Figure 1 A schematic diagram of a flow chart of an optical flow estimation method provided by an embodiment of the present invention.

[0021] like Figure 1 As shown, an optical flow estimation method includes the following steps:

[0022] Importing multiple original images and constructing a training model, training the training model using the multiple original images to obtain an optical flow estimation model;

[0023] Importing a video to be estimated and performing image processing on the video to be estimated to obtain multiple video groups to be estimated;

[0024] Optical flow image estimation is performed on each of the to-be-estimated video groups using the optical flow estimation model to obtain target optical flow images corresponding to each of the to-be-estimated video groups, and all target optical flow images are used as optical flow estimation results.

[0025] In the above embodiment, an optical flow estimation model is obtained by training the training model with original images, and multiple video groups to be estimated are obtained by image processing of the video to be estimated. The optical flow estimation model is used to estimate the optical flow images of the video groups to be estimated to obtain optical flow estimation results. End-to-end training can be performed to quickly obtain approximate optical flow estimation between two images, which can save a lot of resources and time in calculation and improve the accuracy of optical flow estimation.

[0026] Optionally, as an embodiment of the present invention, Figure 1 and 2 As shown, the training model includes a downsampling analysis module, a convolutional neural network group and an upsampling analysis module, and the multiple original images are arranged in time frame order. The process of constructing the training model and training the training model with the multiple original images to obtain the optical flow estimation model includes:

[0027] Grouping two adjacent original images into a group to obtain multiple original image groups;

[0028] Performing downsampling analysis on each of the original image groups by the downsampling analysis module to obtain downsampled image groups corresponding to each of the original image groups;

[0029] Performing downsampled optical flow image analysis on each of the downsampled image groups using the convolutional neural network group to obtain downsampled optical flow images corresponding to each of the original image groups;

[0030] Performing upsampled optical flow image analysis on each of the downsampled optical flow images by the upsampled analysis module to obtain upsampled optical flow images corresponding to each of the original image groups;

[0031] Importing a plurality of real optical flow images corresponding to the original image groups, and performing loss function calculations on each of the upsampled optical flow images and the real optical flow images corresponding to each of the original image groups, to obtain a loss function corresponding to each of the original image groups;

[0032] The parameters of the training model are updated according to all loss functions to obtain an optical flow estimation model.

[0033] It should be understood that the real optical flow image is an optical flow image directly obtained by other means, which is used to minimize the loss between the prediction result (ie, the upsampled optical flow image) and the real optical flow (ie, the real optical flow image).

[0034] It should be understood that according to the construction of a complete network structure (i.e., the training model), the training process is configured and end-to-end training is performed so that the loss between the predicted result (i.e., the upsampled optical flow image) and the true optical flow (i.e., the true optical flow image) is minimized, and the training parameters with the minimum loss are saved. The parameters can be used to directly predict the optical flow results between the two frames of images.

[0035] In the above embodiment, by constructing a training model and training the training model with original images to obtain an optical flow estimation model, end-to-end training can be performed to quickly obtain an approximate optical flow estimation between two images, so that the loss between the predicted result and the true optical flow is minimized, which can save a lot of resources and time in calculation and improve the accuracy of optical flow estimation.

[0036] Optionally, as an embodiment of the present invention, the downsampling analysis module includes a plurality of first convolutional layer groups and a plurality of maximum pooling layers, the first convolutional layer groups are the same in number as the maximum pooling layers and the two are arranged alternately, and the original image group includes a first original image and a second original image;

[0037] The process of performing downsampling analysis on each of the original image groups by the downsampling analysis module to obtain downsampled image groups corresponding to each of the original image groups includes:

[0038] The first convolutional layer group is used to perform feature extraction on each of the first original images and each of the second original images, thereby obtaining a first feature-extracted image corresponding to each of the first original images and a second feature-extracted image corresponding to each of the second original images, specifically:

[0039] Perform feature extraction of the first feature extracted image on each of the first original images using the first formula to obtain a first feature extracted image corresponding to each of the first original images. The first formula is:

[0040] im1=relu(conv(relu(conv(img1)))),

[0041] Among them, im1 is the first feature extracted image, img1 is the first original image, conv is the convolution operation, and relu is the activation function;

[0042] Perform feature extraction of the second feature extracted image on each of the second original images using the second formula to obtain a second feature extracted image corresponding to each of the second original images. The second formula is:

[0043] im2=relu(conv(relu(conv(img2)))),

[0044] Among them, im2 is the second feature extracted image, img2 is the second original image, conv is the convolution operation, and relu is the activation function;

[0045] Downsampling is performed on each of the first feature-extracted images and each of the second feature-extracted images through a first maximum pooling layer to obtain a first downsampled image corresponding to each of the first original images and a second downsampled image corresponding to each of the second original images, specifically:

[0046] The first down-sampled images corresponding to the first original images are obtained by performing down-sampling processing on the first feature-extracted images according to the third formula, where the third formula is:

[0047] img1′=maxpool(im1),

[0048] Among them, im1 is the image after the first feature extraction, img1′ is the image after the first downsampling, and maxpool is the maximum pooling operation;

[0049] The second down-sampled images are down-sampled to each of the second feature-extracted images using the fourth formula to obtain second down-sampled images corresponding to each of the second original images. The fourth formula is:

[0050] img2′=maxpool(im2),

[0051] Among them, im2 is the image after the second feature extraction, img2′ is the image after the second downsampling, and maxpool is the maximum pooling operation;

[0052] The first downsampled images corresponding to each of the first original images and the second downsampled images corresponding to each of the second original images are respectively input into the next first convolutional layer group until the first downsampled images corresponding to each of the first original images and the second downsampled images corresponding to each of the second original images are obtained after passing through the last maximum pooling layer, and a downsampled image group is obtained based on the first downsampled images and the second downsampled images.

[0053] It should be understood that the first convolutional layer group includes two first convolutional layers.

[0054] It should be understood that the convolution layer (i.e., the first convolution layer) is twice the pooling layer (i.e., the maximum pooling layer), and first passes through two convolution layers (i.e., the first convolution layers) and then passes through one maximum pooling layer.

[0055] It should be understood that multiple convolutional layers (i.e., the first convolutional layer group) and pooling layers (i.e., the maximum pooling layer) are used to perform pyramid-type multi-scale feature extraction and downsampling processing on the above two images (i.e., the first original image and the second original image) to obtain image results with feature information of different scales and resolution sizes (i.e., the downsampled image group).

[0056] Specifically, the convolution layer (i.e., the first convolution layer group) is first used to process the two input images (i.e., the first original image and the second original image) to obtain feature information in the images. Generally, a convolution layer with a convolution kernel of 3x3 and a stride of 1 is selected, as follows:

[0057] im1=relu(conv(relu(conv(img1))))

[0058] im2=relu(conv(relu(conv(img2))))

[0059] Where img1 and img2 represent the previous input image (i.e., the first original image) and the next input image (i.e., the second original image), conv represents the convolution operation, relu represents the activation function, and im1 and im2 represent the results of the image after passing through the convolution layer.

[0060] By downsampling the above results (i.e., the first feature extracted image and the second feature extracted image) using the maximum pooling layer with a kernel size of 2x2, images with half the resolution of the original images (i.e., the first downsampled image and the second downsampled image) can be obtained, as follows:

[0061] img1 = maxpool(im1)

[0062] img2 = maxpool(im2)

[0063] Among them, maxpool represents the maximum pooling operation.

[0064] By combining multiple convolutional layers (i.e., the first convolutional layer group) with the maximum pooling layer, feature information of different scales in the image and image results of different resolution sizes (i.e., the downsampled image group) can be obtained.

[0065] In the above embodiment, the downsampling analysis module performs downsampling analysis on the original image group to obtain a downsampled image group, which can obtain feature information in the image, thereby obtaining feature information of different scales in the image and image results of different resolution sizes, providing accurate data for subsequent processing, saving a lot of resources and time in calculation, and improving the accuracy of optical flow estimation.

[0066] Optionally, as an embodiment of the present invention, the convolutional neural network group includes a plurality of sequentially arranged convolutional neural networks, and the process of analyzing the downsampled optical flow images of each of the downsampled image groups by the convolutional neural network group to obtain the downsampled optical flow images corresponding to each of the original image groups includes:

[0067] S131: stitching the first down-sampled images and the second down-sampled images respectively to obtain stitched images corresponding to the original image groups;

[0068] S132: Calculate the image to be processed for each of the second down-sampled images and the spliced ​​images corresponding to each of the original image groups using the fifth formula and the current convolutional neural network to obtain the image to be processed corresponding to each of the original image groups. The fifth formula is:

[0069] g n out =G n (I)+(n-1)g,

[0070] Among them, g n out is the image to be processed corresponding to the nth convolutional neural network, G n is the nth convolutional neural network, I is the spliced ​​image, n is the number of convolutional neural networks, and g is the second down-sampled image;

[0071] S133: Determine whether the current convolutional neural network is the nth convolutional neural network. If not, use the image to be processed as the second down-sampled image, use the next convolutional neural network as the current convolutional neural network, and return to step S131; if so, execute step S134;

[0072] S134: Calculate downsampled optical flow images for each of the to-be-processed images using the sixth formula to obtain downsampled optical flow images corresponding to each of the original image groups. The sixth formula is:

[0073]

[0074] Among them, t out is the downsampled optical flow image, g k out is the image to be processed corresponding to the kth convolutional neural network, and n is the number of convolutional neural networks.

[0075] It should be understood that the high-order Taylor series expansion formula is derived and expanded as a template to construct a new network framework that is consistent with the task process of optical flow estimation. The two sampling results at the bottom of the aforementioned pyramid (i.e., the downsampled image group) are used as the two inputs of the framework. After a series of approximate derivative neural networks and operations (wherein the approximate derivative neural network can be implemented using a common network, such as u-net, residual network, and Transformer network, etc.), a low-resolution optical flow result (i.e., the downsampled optical flow image) is obtained.

[0076] It should be understood that the result (i.e., the downsampled image group) is used as the input of the approximate Taylor formula expansion network framework. After a series of superposition, approximate derivative neural network, summation and other operations, an optical flow result with the same resolution as the input is obtained (i.e., the downsampled optical flow image).

[0077] Specifically, the high-order Taylor series formula is expanded as follows:

[0078]

[0079]

[0080] By transforming the Taylor expansion formula above, we can obtain that the difference between the function values ​​at two points is equal to the sum of the polynomials formed by the derivatives of each order plus an infinitesimal. The optical flow estimation process can also be simply described as calculating the optical flow difference between two images. The two are closely related, so this framework, which approximates Taylor expansion, can be applied to the task of optical flow estimation.

[0081] Derivation of the high-order Taylor expansion, taking the first-order derivative to the second-order derivative as an example, and taking the derivative of x0 based on the first-order derivative, we have:

[0082] (F′(x0)(x-x0))′=F″(x0)(x-x0)-F′(x0)

[0083] Multiply both sides of the above formula by (x-x0) and perform left-right transformation. The result is as follows:

[0084] F″(x0)(x-x0) 2 =(F′(x0)(x-x0))′(x-x0)+F′(x0)(x-x0)

[0085] We can conclude that, excluding the coefficient, the next-order derivative is the derivative of the previous-order derivative multiplied by the deviation plus the previous-order derivative. Therefore, the n-order derivative is calculated as follows:

[0086] F n (x0)(x-x0) n =(F (n-1) (x0)(x-x0) (n-1) )′(x-x0)+(n-1)F (n-1) (x0)(x-x0) (n-1)

[0087] According to the above derivation results, a network can be built to express the relationship between the previous order derivative and the next order derivative. In order to facilitate calculation and recording, the output F of the nth layer is n (x0)(x-x0) n Denoted as g n out Therefore, the calculation formula of the n-order derivative can also be written as:

[0088] g n out =G(g (n-1) out )+(n-1)g (n-1) out

[0089] Where G is the convolutional neural network used. In addition, the approximate n-order Taylor series expansion can be expressed as:

[0090]

[0091] Based on the above derivation, a network can be built to express the relationship between the previous-order derivative and the next-order derivative. In this network, the output of the previous layer is used as part of the input of the next layer. Then, a neural network is used to replace the derivation process, and the output of the neural network is added to the output of the previous layer multiplied by the corresponding coefficient as the output of the layer. Finally, the output is multiplied by the coefficient of the corresponding Taylor expansion to obtain the Taylor derivative of that order. Adding the Taylor derivatives of each order together is an approximate, complete Taylor series expansion, so this new network framework can be used to approximate the Taylor series expansion formula.

[0092] Specifically, the first layer: First, the two input images (i.e., the first down-sampled image and the second down-sampled image) are spliced ​​in dimension. Then, they are passed through a convolutional neural network, which is equivalent to taking the derivative of the input and multiplying it by the bias, i.e., F′(x0)(x-x0) or g out Then multiply the second input by the corresponding coefficient (0) and sum it with the output of the network to get the output g of this layer. out Finally, the output of this layer is multiplied by the Taylor series parameter of the corresponding order The first-order Taylor component is obtained.

[0093] The nth layer: First, the output g of the (n-1)th layer (n-1) out The concatenation operation is performed on the original first input in order to introduce the features of the original image. Then it passes through the convolutional neural network, which is equivalent to taking the derivative of the input and multiplying it by the deviation, that is, G(g (n-1) out ). Then multiply the output of the (n-1)th layer by the corresponding coefficient (n-1), that is, (n-1)g (n-1) out And sum it with the output of the network to get the output g of this layer n out Finally, the output of this layer is multiplied by the Taylor series parameter of the corresponding order The n-order Taylor component is obtained.

[0094] The sum of the Taylor components of all orders obtained from all layers is the approximate n-order Taylor series expansion, so the Taylor series expansion formula can be approximately expressed using this new network framework.

[0095] In the above embodiment, the downsampled optical flow image of the downsampled image group is analyzed by the convolutional neural network group to obtain the downsampled optical flow image, which can introduce the features of the original image, make the network framework consistent with the task process of optical flow estimation, save a lot of resources and time in calculation, and improve the accuracy of optical flow estimation.

[0096] Optionally, as an embodiment of the present invention, the upsampling analysis module includes a plurality of transposed convolutional layers and a plurality of second convolutional layer groups, the number of the transposed convolutional layers and the second convolutional layer groups is the same as the number of the maximum pooling layers, and the transposed convolutional layers and the second convolutional layer groups are arranged alternately;

[0097] The process of performing upsampled optical flow image analysis on each of the downsampled optical flow images by the upsampling analysis module to obtain upsampled optical flow images corresponding to each of the original image groups includes:

[0098] Each of the downsampled optical flow images is upsampled by the first transposed convolution layer to obtain an optical flow image to be extracted corresponding to each of the original image groups, specifically:

[0099] Upsampling is performed on each of the downsampled optical flow images using the seventh formula to obtain optical flow images to be extracted corresponding to each of the original image groups. The seventh formula is:

[0100] out=ConvTranspose2d(t out ),

[0101] Among them, out is the optical flow image to be extracted, t out is the downsampled optical flow image, ConvTranspose2d is the 2D transposed convolution operation;

[0102] The first second convolutional layer group is used to perform feature extraction on each of the optical flow images to be extracted, and obtains feature-extracted optical flow images corresponding to each of the original image groups, specifically:

[0103] The eighth formula is used to perform feature extraction on each of the optical flow images to be extracted, thereby obtaining an optical flow image after feature extraction corresponding to each of the original image groups. The eighth formula is:

[0104] out′=relu(conv(relu(conv(out)))),

[0105] Among them, out is the optical flow image to be extracted, conv is the convolution operation, relu is the activation function, and out′ is the optical flow image after feature extraction;

[0106] Each of the feature-extracted optical flow images is input into the next transposed convolutional layer respectively until a feature-extracted optical flow image corresponding to each of the original image groups is obtained after passing the last second convolutional layer group, and the feature-extracted optical flow image is used as an upsampled optical flow image to obtain an upsampled optical flow image corresponding to each of the original image groups.

[0107] It should be understood that the second convolutional layer group includes two second convolutional layers.

[0108] It should be understood that the image first passes through one transposed convolution layer and then passes through two convolution layers (i.e., the second convolution layers).

[0109] It should be understood that the result (i.e., the downsampled optical flow image) is pyramidally upsampled using a transposed convolution operation (i.e., the transposed convolution layer) and multiple convolution layers (i.e., the second convolution layer group), and finally an optical flow estimation result (i.e., the upsampled optical flow image) of a size comparable to the original image (i.e., the original image group) is obtained.

[0110] Specifically, the transposed convolution layer is used to upsample the result (i.e., the downsampled optical flow image) to obtain a resolution twice that of the original image. Generally, a transposed convolution layer with a convolution kernel of 3x3 and a stride of 1 is selected, as follows:

[0111] out=ConvTranspose2d(t out )

[0112] Among them, t out Represents the output of the approximate Taylor formula expansion network framework, ConvTranspose2d represents the 2D transposed convolution operation, and out represents the output of the transposed convolution operation.

[0113] The output image (i.e., the optical flow image to be extracted) is processed using a convolutional layer (i.e., the second convolutional layer group) to enhance the feature information in the image. Generally, a convolutional layer with a convolution kernel of 3x3 and a stride of 1 is selected, as follows:

[0114] out=relu(conv(relu(conv(out))))

[0115] By combining the same number of transposed convolutional layers as the pooling layers (i.e., the maximum pooling layers) and the same number of convolutional layers (i.e., the second convolutional layer group), an image with the same resolution as the original image is finally obtained. This image is the optical flow map between the two selected frames of images (i.e., the upsampled optical flow image).

[0116] In the above embodiment, the upsampled optical flow image of the downsampled optical flow image is analyzed by the upsampling analysis module to obtain the upsampled optical flow image, which enhances the feature information in the image, can save a lot of resources and time in calculation, and also improves the accuracy of optical flow estimation.

[0117] Optionally, as an embodiment of the present invention, the process of performing loss function calculation on each of the upsampled optical flow images and the real optical flow images corresponding to each of the original image groups to obtain the loss function corresponding to each of the original image groups includes:

[0118] The loss function corresponding to each of the original image groups is obtained by performing loss function calculation on each of the upsampled optical flow images and the real optical flow images corresponding to each of the original image groups using the ninth formula. The ninth formula is:

[0119]

[0120] Among them, EPE is the loss function, gt is the real optical flow image, and out″ is the upsampled optical flow image.

[0121] It should be understood that a complete network structure is built, the training process is configured, and end-to-end training is performed to minimize the loss between the predicted results and the true optical flow. The training parameters with the minimum loss are saved, and the optical flow results between any two frames in the video can be directly predicted using these parameters.

[0122] It should be understood that the loss function uses the endpoint error (EPE) most commonly used in optical flow estimation tasks. It calculates the Euclidean distance between the predicted optical flow (i.e., the upsampled optical flow image) and the true optical flow (i.e., the true optical flow image), as follows:

[0123]

[0124] Among them, pred represents the predicted optical flow and gt represents the real optical flow image.

[0125] In the above embodiment, the loss function is calculated by the ninth formula on the loss function of the upsampled optical flow image and the real optical flow image, which can minimize the loss between the predicted result and the real optical flow, save a lot of resources and time in calculation, and improve the accuracy of optical flow estimation.

[0126] Optionally, as an embodiment of the present invention, the process of performing image processing on the video to be estimated to obtain multiple video groups to be estimated includes:

[0127] Extracting the video to be estimated frame by frame to obtain a plurality of images to be sorted;

[0128] Sort all the images to be sorted according to the output order of frame-by-frame extraction to obtain multiple images to be estimated;

[0129] Pairwise adjacent images to be estimated are grouped together to obtain multiple video groups to be estimated.

[0130] It should be understood that the input video file (i.e., the video to be estimated) is opened and extracted frame by frame, all the extracted frames (i.e., the images to be sorted) are saved in a separate folder in the order of extraction, and any two adjacent frames of images in the extracted frames (i.e., the images to be estimated) are selected and used as the two input images (i.e., the images to be estimated).

[0131] Specifically, the collected video file (i.e., the video to be estimated) is read, the file format is converted to facilitate subsequent related operations, the frame rate and number of frames of the video (i.e., the video to be estimated) are obtained, and a frame reading operation is performed; each frame of the read image (i.e., the image to be sorted) is saved in a designated folder according to the order of extraction and a fixed naming method; the front and back two images in the folder (i.e., the images to be estimated) are arbitrarily selected, and these two images (i.e., the images to be estimated) are used as input for the entire framework.

[0132] In the above embodiment, multiple images to be sorted are obtained by extracting the video to be estimated frame by frame, and multiple images to be estimated are obtained by sorting all the images to be sorted according to the output order of the frame-by-frame extraction. Each pair of adjacent images to be estimated is grouped together to obtain multiple video groups to be estimated. End-to-end training can be performed to quickly obtain an approximate optical flow estimate between two images, which can save a lot of resources and time in calculation and improve the accuracy of optical flow estimation.

[0133] Optionally, as another embodiment of the present invention, the present invention builds a complete network structure, creates a path for saving input, output, and training parameters, and configures the training process, including the training set, test set, number of training rounds, number of images input in each round, learning rate, etc.; performs end-to-end training to minimize the loss between the final prediction result and the true optical flow, and saves the training parameters with the minimum loss; the above-mentioned training parameters can be used to directly predict the optical flow results between any two frames of images in the video. In addition, the optical flow results of all the previous and next frames of the video are integrated together in sequence to obtain the optical flow of the video. The video optical flow can be applied to moving target detection, target tracking, autonomous driving, and other aspects.

[0134] Optionally, as another embodiment of the present invention, the present invention obtains two adjacent frames of images in a video and uses them as input; performs pyramid downsampling processing on the two frames of images to obtain images of different resolution sizes; constructs a new network framework and uses it to approximate Taylor expansion, so that the framework is consistent with the task process of optical flow estimation; uses the two sampling results at the bottom of the pyramid as input to the above framework, and obtains an approximate result after a series of operations; performs pyramid upsampling processing on the result, and finally obtains an estimated optical flow result; performs end-to-end training according to the above steps to minimize the prediction loss, saves the training parameters, and uses the parameters to predict the optical flow between any two frames of images. The present invention uses a deep network architecture to approximate Taylor expansion, and uses this architecture to achieve the task of optical flow estimation between two frames of images; this method can also be applied to the entire video, and by integrating the optical flows between all frames in sequence, a complete video optical flow can be obtained.

[0135] Optionally, as another embodiment of the present invention, in order to solve the specific application problems of the above-mentioned high-order Taylor series, the present invention proposes an optical flow estimation method based on approximate Taylor expansion, and constructs a new framework by utilizing the consistency between the high-order Taylor series expansion formula and the optical flow estimation task process. By performing end-to-end training, the approximate optical flow estimation between two images can be quickly obtained.

[0136] Figure 3This is a module block diagram of an optical flow estimation device provided by an embodiment of the present invention.

[0137] Alternatively, as another embodiment of the present invention, Figure 3 As shown, an optical flow estimation device includes:

[0138] A model training module is used to import multiple original images and build a training model, and train the training model using the multiple original images to obtain an optical flow estimation model;

[0139] An image processing module is used to import a video to be estimated and perform image processing on the video to be estimated to obtain multiple video groups to be estimated;

[0140] The optical flow estimation result acquisition module is used to perform optical flow image estimation on each of the video groups to be estimated using the optical flow estimation model, obtain target optical flow images corresponding to each of the video groups to be estimated, and use all target optical flow images as optical flow estimation results.

[0141] Alternatively, another embodiment of the present invention provides an optical flow estimation device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the optical flow estimation method described above is implemented. The device may be a computer or other device.

[0142] Optionally, another embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the optical flow estimation method as described above is implemented.

[0143] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus.

[0144] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0145] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of units is merely a logical functional division. In actual implementation, other division methods may be used, such as combining or integrating multiple units or components into another system, or ignoring or not implementing certain features.

[0146] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected based on actual needs to achieve the objectives of the embodiments of the present invention.

[0147] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0148] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program code.

[0149] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. An optical flow estimation method, characterized in that: The steps include: Importing multiple original images and constructing a training model, training the training model using the multiple original images to obtain an optical flow estimation model, the training model including a downsampling analysis module, a convolutional neural network group, and an upsampling analysis module, wherein the multiple original images are arranged in time frame order; Importing a video to be estimated and performing image processing on the video to be estimated to obtain multiple video groups to be estimated; Performing optical flow image estimation on each of the to-be-estimated video groups using the optical flow estimation model, obtaining target optical flow images corresponding to each of the to-be-estimated video groups, and taking all target optical flow images as optical flow estimation results; The convolutional neural network group includes a plurality of sequentially arranged convolutional neural networks, and the process of analyzing the downsampled optical flow images of each of the downsampled image groups by the convolutional neural network group to obtain the downsampled optical flow images corresponding to each of the original image groups includes: S131: stitching together first down-sampled images corresponding to respective first original images and second down-sampled images corresponding to respective second original images, to obtain stitched images corresponding to respective original image groups; S132: Calculate the image to be processed for each of the second down-sampled images and the spliced ​​images corresponding to each of the original image groups using the fifth formula and the current convolutional neural network to obtain the image to be processed corresponding to each of the original image groups. The fifth formula is: ， in, is the image to be processed corresponding to the nth convolutional neural network, is the nth convolutional neural network, is the stitched image, is the number of convolutional neural networks, is the second down-sampled image; S133: Determine whether the current convolutional neural network is the nth convolutional neural network. If not, use the image to be processed as the second down-sampled image, use the next convolutional neural network as the current convolutional neural network, and return to step S131; if so, execute step S134; S134: Calculate downsampled optical flow images for each of the to-be-processed images using the sixth formula to obtain downsampled optical flow images corresponding to each of the original image groups. The sixth formula is: ， in, is the downsampled optical flow image, For the The image to be processed corresponding to the convolutional neural network, is the number of convolutional neural networks.

2. The optical flow estimation method according to claim 1, wherein: The process of constructing a training model and training the training model using a plurality of original images to obtain an optical flow estimation model includes: Grouping two adjacent original images into a group to obtain multiple original image groups; Performing downsampling analysis on each of the original image groups by the downsampling analysis module to obtain downsampled image groups corresponding to each of the original image groups; Performing downsampled optical flow image analysis on each of the downsampled image groups using the convolutional neural network group to obtain downsampled optical flow images corresponding to each of the original image groups; Performing upsampled optical flow image analysis on each of the downsampled optical flow images by the upsampled analysis module to obtain upsampled optical flow images corresponding to each of the original image groups; Importing a plurality of real optical flow images corresponding to the original image groups, and performing loss function calculations on each of the upsampled optical flow images and the real optical flow images corresponding to each of the original image groups, to obtain a loss function corresponding to each of the original image groups; The parameters of the training model are updated according to all loss functions to obtain an optical flow estimation model.

3. The optical flow estimation method according to claim 2, wherein: The downsampling analysis module includes a plurality of first convolutional layer groups and a plurality of maximum pooling layers, the first convolutional layer groups are the same in number as the maximum pooling layers and the two are arranged alternately, and the original image group includes a first original image and a second original image; The process of performing downsampling analysis on each of the original image groups by the downsampling analysis module to obtain downsampled image groups corresponding to each of the original image groups includes: The first convolutional layer group is used to perform feature extraction on each of the first original images and each of the second original images, thereby obtaining a first feature-extracted image corresponding to each of the first original images and a second feature-extracted image corresponding to each of the second original images, specifically: Perform feature extraction of the first feature extracted image on each of the first original images using the first formula to obtain a first feature extracted image corresponding to each of the first original images. The first formula is: ， in, is the image after the first feature extraction, is the first original image, is the convolution operation, is the activation function; Perform feature extraction of the second feature extracted image on each of the second original images using the second formula to obtain a second feature extracted image corresponding to each of the second original images. The second formula is: ， in, is the image after the second feature extraction, is the second original image, is the convolution operation, is the activation function; Downsampling is performed on each of the first feature-extracted images and each of the second feature-extracted images through a first maximum pooling layer to obtain a first downsampled image corresponding to each of the first original images and a second downsampled image corresponding to each of the second original images, specifically: The first down-sampled images corresponding to the first original images are obtained by performing down-sampling processing on the first feature-extracted images according to the third formula, where the third formula is: ， in, is the image after the first feature extraction, is the first down-sampled image, It is the maximum pooling operation; The second down-sampled images are down-sampled to each of the second feature-extracted images using the fourth formula to obtain second down-sampled images corresponding to each of the second original images. The fourth formula is: ， in, is the image after the second feature extraction, is the second down-sampled image, It is the maximum pooling operation; The first downsampled images corresponding to each of the first original images and the second downsampled images corresponding to each of the second original images are respectively input into the next first convolutional layer group until the first downsampled images corresponding to each of the first original images and the second downsampled images corresponding to each of the second original images are obtained after passing through the last maximum pooling layer, and a downsampled image group is obtained based on the first downsampled images and the second downsampled images.

4. The optical flow estimation method according to claim 3, wherein: The upsampling analysis module includes a plurality of transposed convolutional layers and a plurality of second convolutional layer groups, the number of the transposed convolutional layers and the second convolutional layer groups is the same as the number of the maximum pooling layers, and the transposed convolutional layers and the second convolutional layer groups are arranged alternately; The process of performing upsampled optical flow image analysis on each of the downsampled optical flow images by the upsampling analysis module to obtain upsampled optical flow images corresponding to each of the original image groups includes: Each of the downsampled optical flow images is upsampled by the first transposed convolution layer to obtain an optical flow image to be extracted corresponding to each of the original image groups, specifically: Upsampling is performed on each of the downsampled optical flow images using the seventh formula to obtain optical flow images to be extracted corresponding to each of the original image groups. The seventh formula is: ， in, is the optical flow image to be extracted, is the downsampled optical flow image, It is a 2D transposed convolution operation; The first second convolutional layer group is used to perform feature extraction on each of the optical flow images to be extracted, and obtains feature-extracted optical flow images corresponding to each of the original image groups, specifically: The eighth formula is used to perform feature extraction on each of the optical flow images to be extracted, thereby obtaining an optical flow image after feature extraction corresponding to each of the original image groups. The eighth formula is: ， in, is the optical flow image to be extracted, is the convolution operation, is the activation function, This is the optical flow image after feature extraction; Each of the feature-extracted optical flow images is input into the next transposed convolutional layer respectively until a feature-extracted optical flow image corresponding to each of the original image groups is obtained after passing the last second convolutional layer group, and the feature-extracted optical flow image is used as an upsampled optical flow image to obtain an upsampled optical flow image corresponding to each of the original image groups.

5. The optical flow estimation method according to claim 2, wherein: The process of respectively calculating the loss function on each of the upsampled optical flow images and the real optical flow images corresponding to each of the original image groups to obtain the loss function corresponding to each of the original image groups includes: The loss function corresponding to each of the original image groups is obtained by performing loss function calculation on each of the upsampled optical flow images and the real optical flow images corresponding to each of the original image groups using the ninth formula. The ninth formula is: ， in, is the loss function, is the real optical flow image, is the upsampled optical flow image.

6. The optical flow estimation method according to claim 1, wherein: The process of performing image processing on the video to be estimated to obtain multiple video groups to be estimated includes: Extracting the video to be estimated frame by frame to obtain a plurality of images to be sorted; Sort all the images to be sorted according to the output order of frame-by-frame extraction to obtain multiple images to be estimated; Pairwise adjacent images to be estimated are grouped together to obtain multiple video groups to be estimated.

7. An optical flow estimation device, characterized in that: include: A model training module, configured to import a plurality of original images and construct a training model, and train the training model using the plurality of original images to obtain an optical flow estimation model, wherein the training model includes a downsampling analysis module, a convolutional neural network group, and an upsampling analysis module, and the plurality of original images are arranged in time frame order; An image processing module is used to import a video to be estimated and perform image processing on the video to be estimated to obtain multiple video groups to be estimated; an optical flow estimation result acquisition module, configured to perform optical flow image estimation on each of the video groups to be estimated using the optical flow estimation model, obtain target optical flow images corresponding to each of the video groups to be estimated, and use all target optical flow images as optical flow estimation results; The convolutional neural network group includes a plurality of sequentially arranged convolutional neural networks. In the model training module, the process of analyzing the downsampled optical flow images of each of the downsampled image groups by the convolutional neural network group to obtain the downsampled optical flow images corresponding to each of the original image groups includes: S131: stitching together first down-sampled images corresponding to respective first original images and second down-sampled images corresponding to respective second original images, to obtain stitched images corresponding to respective original image groups; S132: Calculate the image to be processed for each of the second down-sampled images and the spliced ​​images corresponding to each of the original image groups using the fifth formula and the current convolutional neural network to obtain the image to be processed corresponding to each of the original image groups. The fifth formula is: ， in, is the image to be processed corresponding to the nth convolutional neural network, is the nth convolutional neural network, is the stitched image, is the number of convolutional neural networks, is the second down-sampled image; S133: Determine whether the current convolutional neural network is the nth convolutional neural network. If not, use the image to be processed as the second down-sampled image, use the next convolutional neural network as the current convolutional neural network, and return to step S131; if so, execute step S134; S134: Calculate downsampled optical flow images for each of the to-be-processed images using the sixth formula to obtain downsampled optical flow images corresponding to each of the original image groups. The sixth formula is: ， in, is the downsampled optical flow image, For the The image to be processed corresponding to the convolutional neural network, is the number of convolutional neural networks.

8. An optical flow estimation system comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the optical flow estimation method according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the optical flow estimation method according to any one of claims 1 to 6 is implemented.

Images