Bidirectional optical flow estimation method and device

By constructing an image pyramid in optical flow estimation and calling the optical flow estimation model recursively, combined with forward distortion processing, the problems of low efficiency and poor generalization of the existing optical flow estimation methods are solved, and efficient and accurate bidirectional light is achieved. Flow estimation to meet real-time and robustness requirements.

CN114581493BActive Publication Date: 2024-12-03SAMSUNG ELECTRONICS CHINA R&D CENT +1
View PDF 14 Cites 0 Cited by

Patent Information

Application Number
CN202210207725.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-03-04
Publication Date
2024-12-03
Estimated Expiration
2042-03-04

AI Technical Summary

Technical Problem

The existing optical flow estimation methods have problems such as large computational overhead, low efficiency and poor generalization, especially the estimation efficiency of bidirectional optical flow and cannot meet the real-time requirements.

Method used

Using the bidirectional optical flow estimation method, the image pyramid is constructed by acquiring the target image pair, and the pre-trained optical flow estimation model is used for recursive calls. Combined with forward distortion processing to the middle, the model parameter amount and training overhead are reduced, and the robustness is improved. and generalization.

Benefits of technology

The efficiency, accuracy and generalization of bidirectional optical flow estimation are improved, the overhead of model training and optical flow estimation is reduced, and the robustness of optical flows at different scales is enhanced, and the real-time requirements are met.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114581493B_ABST
    Figure CN114581493B_ABST
Patent Text Reader

Abstract

The present application discloses a bidirectional optical flow estimation method and device, wherein the method comprises: obtaining a target image pair for which optical flow is to be estimated, and constructing an image pyramid for each target image in the target image pair; based on the image pyramid, using a pre-trained optical flow estimation model, performing bidirectional optical flow estimation to obtain a bidirectional optical flow between the target images; wherein, in accordance with a preset order, based on each layer of images in the image pyramid, recursively calling an optical flow estimation module in the optical flow estimation model to perform bidirectional optical flow estimation, and before each call to the optical flow estimation module, performing a forward warping process toward the middle on the corresponding layer image of the image pyramid, and inputting the intermediate frame image obtained by the process into the optical flow estimation module. By adopting the present application, the efficiency and generalization of bidirectional optical flow estimation can be improved, and the overhead of model training and optical flow estimation can be reduced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to computer vision technology, and in particular to a bidirectional optical flow estimation method and device. Background Art

[0002] In computer vision, optical flow is often used to characterize pixel-level motion in an image. This motion can be caused by camera movement or by the motion of the object itself. Optical flow (also called optical flow field) refers to the set of pixel displacements between two adjacent frames: that is, the set of displacement vectors generated in the process of moving each pixel in the previous image to the corresponding pixel position in the next image. Optical flow estimation is a classic problem in computer vision and a key step in many video understanding algorithms. Algorithms such as video interpolation, moving object detection, and video content understanding often rely on accurate optical flow information.

[0003] Depending on whether sparse points in the image are selected for optical flow estimation, optical flow estimation can be divided into sparse optical flow and dense optical flow. Among them, dense optical flow describes the optical flow of each pixel of the image moving to the next frame. Optical flow in a general context refers to dense optical flow, and the present invention is also a technical solution proposed for dense optical flow.

[0004] The optical flow estimation method based on feature pyramid is a commonly used optical flow estimation algorithm. Figure 1 This is a schematic diagram of the optical flow estimation method. Figure 1 As shown in the figure, the method constructs feature pyramids (i.e., feature pyramid 1 and feature pyramid 2 in the figure) based on the original input images of two adjacent frames. As the number of pyramid layers increases, the feature size gradually decreases. Then, the feature data of the 0th layer of the feature pyramid is input into the optical flow estimation model for processing to obtain the optical flow estimation value. The optical flow estimation model is a convolutional neural network (CNN), which consists of a warping layer, a matching cost layer, and an optical flow estimation layer.

[0005] In the process of implementing the present invention, the inventors found that the above-mentioned existing optical flow estimation methods have problems such as high computational overhead, low efficiency, and poor generalization. Through research and analysis, the inventors found that the reasons for the above-mentioned problems are as follows:

[0006] 1. In the above optical flow estimation method, the parameter scale of the optical flow estimation model is large, which makes the training cost of the optical flow estimation model large and the computational efficiency low.

[0007] 2. The robustness of the optical flow estimation model in the above optical flow estimation method is limited by the image scale of the target dataset during model training. When the downstream task connected to the optical flow estimation method requires the scale of the optical flow to be larger than the image scale of the training dataset, the corresponding optical flow estimation cannot be performed based on the optical flow estimation model. Therefore, the robustness of the above optical flow estimation method for optical flows of different scales will be limited by the image scale of the target dataset during training, and often cannot achieve good generalization results in practical applications.

[0008] 3. The above optical flow estimation method can only obtain unidirectional optical flow between adjacent frames by running it once, and needs to be run twice for bidirectional optical flow. Therefore, the estimation of bidirectional optical flow is inefficient and cannot meet the real-time requirements. Summary of the invention

[0009] In view of this, the main purpose of the present invention is to provide a bidirectional optical flow estimation method and device, which can improve the efficiency and generalization of bidirectional optical flow estimation and reduce the model training and optical flow estimation overhead.

[0010] In order to achieve the above object, the technical solution proposed in the embodiment of the present invention is:

[0011] A bidirectional optical flow estimation method, comprising:

[0012] Obtain a target image pair for which optical flow is to be estimated, and construct an image pyramid for each target image in the target image pair;

[0013] Based on the image pyramid, a pre-trained optical flow estimation model is used to perform bidirectional optical flow estimation to obtain the bidirectional optical flow between the target images; wherein, in accordance with a preset order, based on each layer of images in the image pyramid, an optical flow estimation module in the optical flow estimation model is recursively called to perform bidirectional optical flow estimation, and before each call to the optical flow estimation module, a forward warping process toward the middle is performed on the corresponding layer image of the image pyramid, and the intermediate frame image obtained by the processing is input into the optical flow estimation module.

[0014] The embodiment of the present invention further provides a bidirectional optical flow estimation device, comprising:

[0015] An image pyramid construction unit is used to obtain a target image pair for which optical flow is to be estimated, and to construct an image pyramid for each target image in the target image pair;

[0016] The optical flow estimation unit is used to perform bidirectional optical flow estimation based on the image pyramid and a pre-trained optical flow estimation model to obtain the bidirectional optical flow between the target images; wherein, in a preset order, based on each layer of the image in the image pyramid, the optical flow estimation module in the optical flow estimation model is recursively called to perform bidirectional optical flow estimation, and before each call to the optical flow estimation module, the corresponding layer image of the image pyramid is forward-warped toward the middle, and the intermediate frame image obtained by the processing is input into the optical flow estimation module.

[0017] An embodiment of the present invention further provides a bidirectional optical flow estimation device, including a processor and a memory;

[0018] The memory stores an application program executable by the processor, which is used to enable the processor to execute the bidirectional optical flow estimation method as described above.

[0019] An embodiment of the present invention further provides a computer-readable storage medium storing computer-readable instructions for executing the bidirectional optical flow estimation method as described above.

[0020] An embodiment of the present invention further provides a computer program product, including a computer program / instruction, which implements the steps of the bidirectional optical flow estimation method described above when executed by a processor.

[0021] In summary, the bidirectional optical flow estimation scheme proposed in the present invention is based on the image pyramid of the target image pair whose optical flow is to be estimated, and adopts a recursive calling method to perform bidirectional optical flow estimation. In this way, on the one hand, the image pyramid can be used to improve the optical flow estimation speed, and on the other hand, the recursive calling can be used to reduce the number of model parameters, thereby reducing the model training and optical flow estimation overhead. By combining the recursive calling and the image pyramid, the robustness of optical flows of different scales can be improved and the generalization can be enhanced. In addition, before each optical flow estimation, the corresponding layer image of the image pyramid is forward-warped toward the middle, and the optical flow estimation is performed based on the intermediate frame image obtained by the processing. In this way, the accuracy of the optical flow estimation can be improved. Therefore, by adopting the embodiment of the present invention, the efficiency, accuracy and generalization of the bidirectional optical flow estimation can be improved, and the model training and optical flow estimation overhead can be reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 Schematic diagram of existing optical flow estimation methods based on feature pyramid;

[0023] Figure 2 A schematic diagram of a method flow of an embodiment of the present invention;

[0024] Figure 3A schematic diagram of a process of performing optical flow estimation based on an example of an optical flow estimation model according to an embodiment of the present invention;

[0025] Figure 4 Schematic diagram of optical flow estimation of an optical flow estimation module according to an embodiment of the present invention;

[0026] Figures 5 to 8 A schematic diagram of applying the embodiment of the present invention to different application scenarios is provided;

[0027] Fig. 9 Schematic diagram of the device structure of an embodiment of the present invention. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0029] Figure 2 FIG. 1 is a flow chart of a bidirectional optical flow estimation method according to an embodiment of the present invention. Figure 2 As shown, the bidirectional optical flow estimation method implemented in this embodiment mainly includes the following steps 201-202:

[0030] Step 201: Obtain a target image pair for which optical flow is to be estimated, and construct an image pyramid for each target image in the target image pair.

[0031] In this step, in order to improve the efficiency of subsequent optical flow estimation, a corresponding image pyramid is constructed instead of a feature pyramid based on each image in the target image pair (specifically composed of the previous and next two frames of images) of the current optical flow to be estimated. In this way, in the subsequent steps, a forward warping operation (i.e., forward warping operation) can be quickly performed based on the image, so that the result of the intermediate forward warping operation can be used for recursive optical flow estimation, thereby improving the efficiency and accuracy of bidirectional optical flow estimation.

[0032] The construction of the image pyramid in this step can be implemented by using existing methods. The specific number of layers of the image pyramid is related to the scale of the target image. The larger the scale of the target image, the more layers the pyramid has.

[0033] In practical applications, a three-layer image pyramid can be constructed, that is, the original target image is processed into three pictures of different scales, where the width and height of the image at the top layer are 1 / 4 of the original image, the width and height of the middle layer image are 1 / 2 of the original image, and the bottom layer image is the original image.

[0034] Step 202: Based on the image pyramid, a pre-trained optical flow estimation model is used to perform bidirectional optical flow estimation to obtain the bidirectional optical flow between the target images; wherein, in a preset order, based on each layer of the image in the image pyramid, an optical flow estimation module in the optical flow estimation model is recursively called to perform bidirectional optical flow estimation, and before each call to the optical flow estimation module, a forward warping process is performed toward the middle on the corresponding layer image of the image pyramid, and the intermediate frame image obtained by the processing is input into the optical flow estimation module.

[0035] This step is used to perform bidirectional optical flow estimation based on the image pyramids of the two target images obtained in step 201 using a pre-trained optical flow estimation model.

[0036] Here, when performing bidirectional optical flow estimation, it is necessary to recursively call the optical flow estimation module in the optical flow estimation model to perform bidirectional optical flow estimation, that is, each time the optical flow estimation module is called for bidirectional optical flow estimation, it is necessary to use the optical flow estimation result output by the previous call to perform optical flow estimation. In this way, by adopting the model structure of recursively calling the optical flow estimation module, each optical flow estimation can share parameters. In this way, on the one hand, the number of parameters of the model can be greatly compressed, and on the other hand, it is convenient to perform multi-scale optical flow training. By combining the image pyramid and recursive optical flow estimation, the model can be trained or fine-tuned on lower resolution data, but can achieve good generalization results on high-resolution images. It has strong robustness to optical flows of different scales, thereby effectively improving the robustness of the model to optical flows of different scales.

[0037] In addition, in this step, before calling the optical flow estimation module each time, it is necessary to first perform a forward warping process on the corresponding layer image of the image pyramid (i.e., the image in the image pyramid that currently needs to be input into the optical flow estimation module for optical flow estimation) toward the middle, and then, based on the intermediate frame image obtained by the forward warping process toward the middle, call the optical flow estimation module for optical flow estimation, that is, based on the bidirectional optical flow, the front and rear frames are warped toward the intermediate frame, so that the same objects in the front and rear frames are moved to similar positions. In this way, on the one hand, it is beneficial to encode the currently estimated optical flow at the feature level; on the other hand, it is convenient to construct an accurate matching cost (cost volume) in the optical flow estimation module, and the cost volume is a very discriminative feature for optical flow estimation, thereby effectively improving the accuracy of optical flow estimation.

[0038] Figure 3 A schematic diagram of the process of performing optical flow estimation based on an example of an optical flow estimation model according to an embodiment of the present invention is given. Figure 3As shown in the figure, for the two three-layer image pyramids of the target image pair, the optical flow estimation module in the optical flow estimation model is recursively called three times based on each layer of the image to obtain the final bidirectional optical flow, that is, starting from the image with the lowest resolution, iterates three times on the image pyramid, and outputs the final bidirectional optical flow on the image with the highest resolution. Before calling the optical flow estimation module for optical flow estimation each time, it is necessary to perform a forward warping operation on the two frames before and after the corresponding pyramid layer, so that the intermediate frame image obtained by the forward warping operation can be used to perform accurate and fast optical flow estimation.

[0039] Preferably, in order to save computational overhead and better handle larger-scale optical flows, in one implementation, the order of performing optical flow estimation based on an image pyramid can be: the order of image scales from small to large, that is, from the top of the pyramid to the bottom of the pyramid, traversing each layer of images in the pyramid in turn and recursively calling the optical flow estimation module to perform bidirectional optical flow estimation.

[0040] In one implementation, the following method may be used to perform forward warping toward the middle:

[0041] If the first call to the optical flow estimation module is currently being executed, a forward warping process is performed toward the middle based on the topmost image in the image pyramid and the initial bidirectional optical flow to obtain intermediate frame images corresponding to the topmost images; otherwise, a forward warping process is performed toward the middle based on the image corresponding to the current call in the image pyramid and the bidirectional optical flow output by the last call to the optical flow estimation module to obtain intermediate frame images of the corresponding images; the initial bidirectional optical flow is 0.

[0042] In the above-mentioned method of forward warping toward the middle, it is necessary to use the bidirectional optical flow obtained by the previous optical flow estimation to perform forward warping toward the middle. In this way, the currently estimated optical flow is encoded at the feature level, which is beneficial to improve the robustness of the model to optical flows of different scales.

[0043] Figure 4 FIG. 1 is a schematic diagram of a process of performing bidirectional optical flow estimation using an optical flow estimation module in an embodiment of the present invention. Figure 4 As shown, in one implementation, the following steps may be used to perform bidirectional optical flow estimation:

[0044] Step x1: Use a convolutional neural network (CNN) feature extractor to extract features of the intermediate frame image input into the optical flow estimation module to obtain CNN features of the intermediate frame image.

[0045] The specific method of feature extraction in this step is known to those skilled in the art and will not be described in detail here.

[0046] Step x2: Determine the corresponding matching cost based on the CNN features of the intermediate frame image.

[0047] This step is used to construct the corresponding matching cost based on the CNN features of the intermediate frame image, so as to improve the accuracy of optical flow estimation by using the matching cost. The specific method of constructing the matching cost can be implemented by using the existing technology, which will not be described in detail here.

[0048] Step x3: perform channel stacking using the CNN features of the intermediate frame image, the matching cost, the bidirectional optical flow output during the last optical flow estimation, and the CNN features of the bidirectional optical flow.

[0049] Here, the CNN features of the bidirectional optical flow output during the last optical flow estimation are the CNN features of the last layer of the optical flow estimation network in the optical flow estimation module during the last optical flow estimation.

[0050] In this step, through channel stacking, the CNN features of the intermediate frame image obtained in step x1, the matching cost obtained in step x2, the bidirectional optical flow output during the last optical flow estimation, and the CNN features of the bidirectional optical flow output during the last optical flow estimation are fused to achieve recursive bidirectional optical flow estimation.

[0051] In one implementation, the channel stacking is performed using the following method:

[0052] If this is the first time that the optical flow estimation module is called, the CNN features of the intermediate frame image, the matching cost, the initial bidirectional optical flow, and the CNN features of the initial bidirectional optical flow are channel stacked; otherwise, the CNN features of the intermediate frame image, the matching cost, the bidirectional optical flow output by the optical flow estimation module when the optical flow estimation module last performed optical flow estimation, and the CNN features of the bidirectional optical flow are channel stacked; the initial bidirectional optical flow is 0; the CNN features of the initial bidirectional optical flow are 0.

[0053] Step x4: input the channel stacking result into the optical flow estimation network to perform optical flow estimation, upsample the obtained bidirectional optical flow and the CNN features of the bidirectional optical flow, and output the upsampled results.

[0054] Here, considering that the optical flow estimation network needs to downsample the input features when performing optical flow estimation, the output results need to be upsampled after the optical flow estimation so that they can be fused with the pyramid image and corresponding features participating in the next optical flow estimation, or the final bidirectional optical flow that matches the target image scale can be obtained.

[0055] Specifically, in one implementation, if the optical flow estimation module is currently called for the Nth time, where N is the number of layers of the image pyramid, the resolution of the upsampling result matches the resolution of the bottom-level image of the image pyramid; otherwise, the resolution of the upsampling result matches the resolution of the input image of the next optical flow estimation.

[0056] like Figure 4 As shown in the figure, based on the three-layer image pyramid described above, when the optical flow estimation network performs two-fold downsampling, the optical flow estimated at each layer is 1 / 4 of the input image of this layer. Accordingly, the output results of the first two optical flow estimations need to be upsampled twice, and the output results of the third optical flow estimation need to be upsampled four times.

[0057] The optical flow estimation network used for optical flow estimation in the optical flow estimation module is a CNN network, which can be constructed using existing methods and will not be described in detail here.

[0058] It can be seen from the above technical scheme that in the above method embodiment, based on the image pyramid of the target image pair whose optical flow is to be estimated, a recursive call method is used to perform bidirectional optical flow estimation, and before each optical flow estimation, the corresponding layer image of the image pyramid is first forward-warped to the middle, and then, based on the processed intermediate frame image, bidirectional optical flow estimation is performed. In this way, by combining the recursive call optical flow estimation module with the image pyramid, the bidirectional optical flow of the target image pair can be obtained by executing the scheme only once, which effectively improves the efficiency, accuracy and generalization of the bidirectional optical flow estimation, and can reduce the model training and optical flow estimation overhead. Therefore, the optical flow estimation scheme proposed in the present application has strong application potential for various optical flow-based application scenarios under the constraints of high real-time requirements or low computing power consumption.

[0059] The specific application of the above method embodiment is exemplified below in combination with various application scenarios.

[0060] Figure 5 This is a schematic diagram of video frame insertion based on optical flow. Figure 5 As shown in the figure, input frame-0 image and frame-1 image, first calculate the bidirectional optical flow between the two frames, and then perform pixel synthesis based on the optical flow to obtain the intermediate frame. Figure 5 As shown, it can be seen that the method embodiment of the present application can accurately depict the optical flow of small objects, thereby obtaining better synthesis results.

[0061] Figure 6 Schematic diagram of video image completion based on optical flow. Figure 6As shown, image completion refers to completing the lost or low-quality areas in the image. Since the optical flow estimation method used in the embodiment of the present invention can achieve a good balance between efficiency and accuracy, it can well serve the real-time video image completion task.

[0062] Figure 1 Schematic diagram of video salient object detection based on optical flow. Figure 7 As shown in , a salient object in a video refers to the object that attracts the most attention in the video. Figure 7 As shown, moving objects in a video are often the objects that attract people's attention. Optical flow depicts pixel-level motion, so it can be regarded as an important hint about the salient objects in the video. Since the optical flow estimation model used in the embodiment of the present invention is very lightweight and has high efficiency, it has great potential in real-time salient object detection.

[0063] Figure 2 Schematic diagram of video object detection based on optical flow. Figure 8 As shown, objects of interest in a video often move between adjacent frames. Optical flow information can help depict the edges of moving objects to a large extent, and therefore can assist in video object detection. The optical flow estimation method used in the embodiment of the present invention achieves a good balance between speed and accuracy, and therefore has great application potential in real-time video object detection.

[0064] Corresponding to the above method embodiment, the present application embodiment also proposes a bidirectional optical flow estimation device, such as Fig. 9 As shown, the device comprises:

[0065] The image pyramid construction unit 901 is used to obtain a target image pair for which optical flow is to be estimated, and to construct an image pyramid for each target image in the target image pair.

[0066] The optical flow estimation unit 902 is used to perform bidirectional optical flow estimation based on the image pyramid and a pre-trained optical flow estimation model to obtain the bidirectional optical flow between the target images; wherein, in accordance with a preset order, the optical flow estimation module in the optical flow estimation model is recursively called to perform bidirectional optical flow estimation based on each layer of the image in the image pyramid, and before each call to the optical flow estimation module, the corresponding layer image of the image pyramid is forward-warped toward the middle, and the intermediate frame image obtained by the processing is input into the optical flow estimation module.

[0067] It should be noted that the above method and device embodiments are based on the same inventive concept. Since the principles of solving problems by the method and device are similar, the implementation of the device and method can refer to each other, and the repeated parts will not be repeated.

[0068] Based on the above-mentioned bidirectional optical flow estimation method embodiment, the embodiment of the present invention also implements a bidirectional optical flow estimation device, including a processor and a memory; the memory stores an application program that can be executed by the processor, which is used to enable the processor to execute the bidirectional optical flow estimation method as described above. Specifically, a system or device equipped with a storage medium can be provided, on which a software program code that implements the functions of any of the above-mentioned embodiments is stored, and the computer (or CPU or MPU) of the system or device reads out and executes the program code stored in the storage medium. In addition, the operating system operating on the computer can also be used to complete part or all of the actual operations based on the instructions of the program code. The program code read from the storage medium can also be written to a memory provided in an expansion board inserted into the computer or to a memory provided in an expansion unit connected to the computer, and then based on the instructions of the program code, a CPU installed on the expansion board or the expansion unit is made to perform part and all of the actual operations, thereby realizing the functions of any of the above-mentioned bidirectional optical flow estimation method implementations.

[0069] The memory may be implemented as various storage media such as an electrically erasable programmable read-only memory (EEPROM), a flash memory (Flash memory), and a programmable program read-only memory (PROM). The processor may be implemented as including one or more central processing units or one or more field programmable gate arrays, wherein the field programmable gate array integrates one or more central processing unit cores. Specifically, the central processing unit or the central processing unit core may be implemented as a CPU or an MCU.

[0070] The embodiment of the present application implements a computer program product, including a computer program / instruction, which implements the steps of the bidirectional optical flow estimation method described above when executed by a processor.

[0071] It should be noted that not all steps and modules in the above processes and structure diagrams are necessary, and some steps or modules can be ignored according to actual needs. The execution order of each step is not fixed and can be adjusted as needed. The division of each module is only for the convenience of describing the functional division adopted. In actual implementation, a module can be implemented by multiple modules, and the functions of multiple modules can also be implemented by the same module. These modules can be located in the same device or in different devices.

[0072] The hardware modules in each embodiment can be implemented mechanically or electronically. For example, a hardware module may include a specially designed permanent circuit or logic device (such as a dedicated processor, such as an FPGA or ASIC) for performing a specific operation. The hardware module may also include a programmable logic device or circuit (such as a general-purpose processor or other programmable processor) temporarily configured by software to perform a specific operation. As for whether to implement the hardware module mechanically, or using a dedicated permanent circuit, or using a temporarily configured circuit (such as configured by software), it can be decided based on cost and time considerations.

[0073] In this article, "schematic" means "serving as an example, instance or explanation", and any diagram or implementation method described as "schematic" in this article should not be interpreted as a more preferred or more advantageous technical solution. In order to make the drawings concise, only the parts related to the present invention are schematically shown in each figure, and do not represent the actual structure of the product. In addition, in order to make the drawings concise and easy to understand, in some figures, only one of the parts with the same structure or function is schematically drawn, or only one of them is marked. In this article, "one" does not mean that the number of the relevant parts of the present invention is limited to "only one", and "one" does not mean that the number of the relevant parts of the present invention is "more than one". In this article, "upper", "lower", "front", "back", "left", "right", "inside", "outside", etc. are only used to indicate the relative position relationship between the relevant parts, rather than to limit the absolute position of these relevant parts.

[0074] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A bidirectional optical flow estimation method, It is characterized in that include: Obtain a target image pair for which optical flow is to be estimated, and construct an image pyramid for each target image in the target image pair; Based on the image pyramid, a pre-trained optical flow estimation model is used to perform bidirectional optical flow estimation to obtain the bidirectional optical flow between the target images; wherein, in accordance with a preset order, based on each layer of images in the image pyramid, the optical flow estimation module in the optical flow estimation model is recursively called to perform bidirectional optical flow estimation, and before each call to the optical flow estimation module, the corresponding layer image of the image pyramid is subjected to forward warping processing toward the middle, and the intermediate frame image obtained by the processing is input into the optical flow estimation module; the step of performing forward warping processing toward the middle comprises: if the first call to the optical flow estimation module is currently to be executed, then based on the topmost layer image in the image pyramid and the initial bidirectional optical flow, a forward warping processing toward the middle is performed to obtain the intermediate frame image corresponding to each of the topmost layer images; otherwise, based on the image corresponding to the current call in the image pyramid and the bidirectional optical flow output by the last call to the optical flow estimation module, a forward warping processing toward the middle is performed to obtain the intermediate frame image of each of the corresponding images; the initial bidirectional optical flow is 0.

2. The method according to claim 1, It is characterized in that The order is: the order of image scale from small to large.

3. The method according to claim 1, It is characterized in that Each time the optical flow estimation module is called to perform bidirectional optical flow estimation, it includes: Using a convolutional neural network (CNN) feature extractor, feature extraction is performed on the intermediate frame image input into the optical flow estimation module to obtain CNN features of the intermediate frame image; Determining a corresponding matching cost based on the CNN features of the intermediate frame image; Channel stacking is performed using the CNN features of the intermediate frame image, the matching cost, the bidirectional optical flow output during the last optical flow estimation, and the CNN features of the bidirectional optical flow; the CNN features of the bidirectional optical flow are the CNN features of the last layer of the optical flow estimation network in the optical flow estimation module during the last optical flow estimation; The channel stacking result is input into the optical flow estimation network to perform optical flow estimation, and the obtained bidirectional optical flow and the CNN features of the bidirectional optical flow are upsampled respectively and the upsampled results are output.

4. The method according to claim 3, It is characterized in that The channel stacking comprises: If this is the first time that the optical flow estimation module is called, the CNN features of the intermediate frame image, the matching cost, the initial bidirectional optical flow, and the CNN features of the initial bidirectional optical flow are channel stacked; otherwise, the CNN features of the intermediate frame image, the matching cost, the bidirectional optical flow output by the optical flow estimation module when the optical flow estimation module last performed optical flow estimation, and the CNN features of the bidirectional optical flow are channel stacked; the initial bidirectional optical flow is 0; the CNN features of the initial bidirectional optical flow are 0.

5. The method according to claim 3, It is characterized in that The method further comprises: If this is the Nth time that the optical flow estimation module is called, where N is the number of layers of the image pyramid, the resolution of the upsampling result matches the resolution of the bottom-level image of the image pyramid; otherwise, the resolution of the upsampling result matches the resolution of the input image of the next optical flow estimation.

6. A bidirectional optical flow estimation device, It is characterized in that include: An image pyramid construction unit is used to obtain a target image pair for which optical flow is to be estimated, and to construct an image pyramid for each target image in the target image pair; The optical flow estimation unit is used to perform bidirectional optical flow estimation based on the image pyramid and a pre-trained optical flow estimation model to obtain the bidirectional optical flow between the target images; wherein, in accordance with a preset order, the optical flow estimation module in the optical flow estimation model is recursively called to perform bidirectional optical flow estimation based on each layer of the image in the image pyramid, and before each call of the optical flow estimation module, the corresponding layer image of the image pyramid is subjected to forward warping processing toward the middle, and the intermediate frame image obtained by the processing is input into the optical flow estimation module; the step of performing forward warping processing toward the middle comprises: if the first call to the optical flow estimation module is currently to be executed, then based on the topmost layer image in the image pyramid and the initial bidirectional optical flow, a forward warping processing toward the middle is performed to obtain the intermediate frame image corresponding to each of the topmost layer images; otherwise, based on the image corresponding to the current call in the image pyramid and the bidirectional optical flow output by the last call of the optical flow estimation module, a forward warping processing toward the middle is performed to obtain the intermediate frame image of each of the corresponding images; the initial bidirectional optical flow is 0.

7. A bidirectional optical flow estimation device, It is characterized in that including a processor and a memory; The memory stores an application program executable by the processor, which is used to enable the processor to execute the bidirectional optical flow estimation method according to any one of claims 1 to 5.

8. A computer-readable storage medium, It is characterized in that Computer-readable instructions are stored therein, and the computer-readable instructions are used to execute the bidirectional optical flow estimation method as described in any one of claims 1 to 5.

9. A computer program product comprising a computer program / instructions, It is characterized in that When the computer program / instruction is executed by a processor, the steps of the bidirectional optical flow estimation method according to any one of claims 1 to 5 are implemented.

Citation Information

Patent Citations

  • Aurora motion characterization method based on unsupervised deep optical flow network

    CN112785629A

  • Optical flow estimation method, model training method and corresponding device

    CN113538525A

  • Efficient lightweight optical flow estimation method

    CN113538527A

  • Optical flow estimation method, related device, equipment and computer readable storage medium

    CN113673545A

  • System and method for motion estimation

    CN113781517A