An optical flow estimation method, apparatus, device, and storage medium

Abstract

The application discloses an optical flow estimation method and device, equipment and a storage medium. The method comprises the following steps: obtaining a target image pair to be estimated; performing down-sampling on the target image pair to determine a sampling image pair; performing optical flow estimation on the sampling image pair by using a preset network to obtain a first optical flow image, and determining a ghost region of the first optical flow image according to the sampling image pair and the first optical flow image; and performing optical flow calculation on the target image pair, the first optical flow image and the ghost region by using a dense reverse search algorithm to determine a target optical flow image. In this way, the preset network is used to perform preliminary optical flow estimation on the sampling image pair, and then the dense reverse search algorithm is used to perform iterative calculation on the target image pair, the first optical flow image and the ghost region, so that the calculation amount of optical flow estimation can be reduced, and the accuracy of optical flow estimation can be improved.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to an optical flow estimation method, apparatus, device, and storage medium. Background Technology

[0002] Optical flow is the instantaneous velocity of pixels moving on the imaging plane of a spatially moving object. It utilizes the temporal changes of pixels in an image sequence and the correlation between adjacent frames to find the correspondence between the previous and current frames, thereby calculating the motion information of objects between adjacent frames. Currently, optical flow has wide applications in computer vision, such as multi-frame noise reduction, video frame interpolation, and High Dynamic Range Imaging (HDR).

[0003] In related technologies, traditional optical flow estimation algorithms suffer from significant errors in estimating optical flow for images with complex motion. While artificial intelligence (AI) networks can be used to reduce these errors, AI optical flow calculations are computationally intensive and consume excessive power. Summary of the Invention

[0004] This application proposes an optical flow estimation method, apparatus, device, and storage medium, which can reduce the computational load of optical flow estimation and improve the accuracy of optical flow estimation.

[0005] To achieve the above objectives, the technical solution of this application is implemented as follows:

[0006] In a first aspect, embodiments of this application provide an optical flow estimation method, the method comprising:

[0007] Acquire the target image pair whose optical flow needs to be estimated;

[0008] Downsample the target image pair to determine the sampled image pair;

[0009] Optical flow estimation is performed on the sampled image pairs using a preset network to obtain a first optical flow image. Based on the sampled image pairs and the first optical flow image, the ghosting region of the first optical flow image is determined.

[0010] A dense inverse search algorithm is used to calculate the optical flow of the target image pair, the first optical flow image, and the ghosting region to determine the target optical flow image.

[0011] Secondly, embodiments of this application provide an optical flow estimation device, which includes an acquisition unit, a determination unit, and an estimation unit, wherein:

[0012] The acquisition unit is configured to acquire a pair of target images whose optical flow is to be estimated.

[0013] The determination unit is configured to downsample the target image pair and determine the sampled image pair.

[0014] The estimation unit is configured to perform optical flow estimation on the sampled image pair through a preset network to obtain a first optical flow image, and determine the ghosting region of the first optical flow image based on the sampled image pair and the first optical flow image; and to perform optical flow calculation on the target image pair, the first optical flow image and the ghosting region using a dense inverse search algorithm to determine the target optical flow image.

[0015] Thirdly, embodiments of this application provide an electronic device, which includes a memory and a processor, wherein:

[0016] Memory is used to store computer programs that can run on a processor;

[0017] A processor for executing the method described in the first aspect when running a computer program.

[0018] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by at least one processor, implements the method described in the first aspect.

[0019] This application proposes an optical flow estimation method, apparatus, device, and storage medium. The method involves acquiring a pair of target images for which optical flow to be estimated; downsampling the target image pair to determine a sampled image pair; performing optical flow estimation on the sampled image pair using a preset network to obtain a first optical flow image; and determining the ghosting region of the first optical flow image based on the sampled image pair and the first optical flow image. A dense inverse search algorithm is then used to calculate optical flow on the target image pair, the first optical flow image, and the ghosting region to determine the target optical flow image. This method, by using a preset network to perform optical flow estimation on the downsampled image pair, saves significant computational resources and time, thereby improving the processing speed of optical flow estimation. Furthermore, the iterative calculation of the target image pair, the first optical flow image, and the ghosting region using the dense inverse search algorithm further reduces the computational load of optical flow estimation, resulting in a high-precision target optical flow image and improving the accuracy of optical flow estimation. Attached Figure Description

[0020] Figure 1 A flowchart illustrating an optical flow estimation method provided in this application embodiment. Figure 1 ;

[0021] Figure 2 A flowchart illustrating an optical flow estimation method provided in this application embodiment. Figure 2 ;

[0022] Figure 3A flowchart illustrating an optical flow estimation method provided in this application embodiment. Figure 3 ;

[0023] Figure 4 A flowchart illustrating an optical flow estimation method provided in this application embodiment. Figure 4 ;

[0024] Figure 5 A schematic diagram of the application framework for an optical flow estimation method provided in this application embodiment. Figure 1 ;

[0025] Figure 6 A schematic diagram of the composition structure of a RAFT network provided in an embodiment of this application;

[0026] Figure 7 A schematic diagram of the application framework for an optical flow estimation method provided in this application embodiment. Figure 2 ;

[0027] Figure 8 This is a schematic diagram of the composition structure of an optical flow estimation device provided in an embodiment of this application;

[0028] Figure 9 This is a schematic diagram of the specific hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0029] In order to gain a more detailed understanding of the features and technical content of the embodiments of this application, the implementation of the embodiments of this application will be described in detail below with reference to the accompanying drawings. The accompanying drawings are for reference and illustration only and are not intended to limit the embodiments of this application.

[0030] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0031] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0032] It should also be noted that the terms "first, second, and third" used in the embodiments of this application are only used to distinguish similar objects and do not represent a specific order of objects. It is understood that "first, second, and third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0033] Optical flow is the instantaneous velocity of pixels moving on the imaging plane of a spatially moving object. Optical flow estimation is a method that uses the temporal changes of pixels in an image sequence and the correlation between adjacent frames to find the correspondence between the previous frame and the current frame, thereby calculating the motion information of objects between adjacent frames. Optical flow information is helpful for multi-frame noise reduction, video frame interpolation, HDR, etc.

[0034] In related technologies, traditional optical flow estimation algorithms suffer from significant estimation errors, especially for images with complex object motion, particularly in cases of localized motion. While methods using AI networks to reconstruct optical flow have been proposed, and these methods have demonstrated smaller estimation errors, the computational demands of AI optical flow estimation are substantial, leading to excessive power consumption.

[0035] Based on this, embodiments of this application provide an optical flow estimation method, specifically a fast and accurate scheme for calculating image optical flow. The method involves: acquiring a pair of target images whose optical flow needs to be estimated; downsampling the target image pair to determine a sampled image pair; performing optical flow estimation on the sampled image pair using a preset network to obtain a first optical flow image; and determining the ghosting region of the first optical flow image based on the sampled image pair and the first optical flow image; and using a dense inverse search algorithm to calculate optical flow on the target image pair, the first optical flow image, and the ghosting region to determine the target optical flow image. Thus, by using a preset network to perform optical flow estimation on the downsampled image pair, significant resources and time are saved in computation, thereby improving the processing speed of optical flow estimation. Furthermore, using a dense inverse search algorithm to iteratively calculate the target image pair, the first optical flow image, and the ghosting region can further reduce the computational load of optical flow estimation, obtaining a high-precision target optical flow image, and thus improving the accuracy of optical flow estimation.

[0036] The various embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0037] In one embodiment of this application, Figure 1 A flowchart illustrating an optical flow estimation method provided in this application embodiment. Figure 1 .like Figure 1 As shown, the method may include:

[0038] S101: Obtain the target image pair of the optical flow to be estimated.

[0039] It should be noted that, in the embodiments of this application, the method is applied to an optical flow estimation device or an electronic device integrated with such a device. Here, the electronic device may be such as a mobile phone, tablet computer, laptop computer, handheld computer, personal digital assistant (PDA), wearable device, etc., and is not specifically limited here.

[0040] In some embodiments, for obtaining a target image pair of optical flow to be estimated, the method may include: acquiring two consecutive original images; performing grayscale processing on the two original images to obtain two grayscale images; and determining the two grayscale images as the target image pair of optical flow to be estimated.

[0041] It should also be noted that, in this embodiment, optical flow estimation is based on the difference between two images to determine the optical flow between them; therefore, it is necessary to acquire two consecutive original images. These two original images are adjacent frames within the image sequence.

[0042] Specifically, two raw images can be captured by an image sensor (CMOS sensor). At this time, the image data is in the raw image data storage format (RAW). Then, the two raw images are converted into two grayscale images by an image conversion algorithm (such as Demosai) to obtain the target image pair.

[0043] S102: Downsample the target image pair to determine the sampled image pair.

[0044] It should be noted that, due to the high resolution of the target image pairs, directly performing optical flow estimation on the target image pairs would result in a large computational load and excessive power consumption. In the embodiments of this application, the target image pairs are first downsampled to determine the sampled image pairs, and then optical flow estimation is performed on the sampled image pairs. This can save a lot of computational resources and time, and also improve the processing speed of optical flow estimation.

[0045] It should also be noted that, in the embodiments of this application, the target image pair can be downsampled according to a preset downsampling rate to determine the sampled image pair. The preset downsampling rate can be a predefined value, or it can be determined by the resolution of the target image pair, etc., and is not specifically limited here.

[0046] For example, the higher the resolution of the target image pair, the smaller the value of the preset downsampling rate. In one specific embodiment, the preset downsampling rate can be set to 1 / 2. n .

[0047] For example, for an image with a resolution of 2048×1080, the preset downsampling rate is 1 / 8; for an image with a resolution of 1280×720, the preset downsampling rate is 1 / 4; and for an image with a resolution of 680×480, the preset downsampling rate is 1 / 2.

[0048] S103: Optical flow estimation is performed on the sampled image pairs through a preset network to obtain a first optical flow image, and the ghosting region of the first optical flow image is determined based on the sampled image pairs and the first optical flow image.

[0049] It should be noted that, in the embodiments of this application, the preset network includes at least one of the following: Recurrent All-Pairs Field Transforms (RAFT) network, Global Matching Flow (GMFlow) network, FlowNet optical flow neural network, and PWC-Net network. That is to say, the preset network is not limited to a certain optical flow estimation network (such as RAFT network), and other optical flow estimation networks can also be used for optical flow estimation.

[0050] In some embodiments, for determining the ghosting region of the first optical flow image based on the sampled image pair and the first optical flow image, the method may include: performing a difference calculation between the sampled image pair and the first optical flow image to determine the ghosting region.

[0051] It should be noted that the first optical flow image obtained by the preset network will have ghosting regions in the areas of local displacement after optical flow mapping, meaning that the local displacement images before and after optical flow mapping overlap. In this embodiment, a difference algorithm is used to calculate the difference between the sampled image pair and the first optical flow image to obtain the ghosting regions. No limitations are placed on the algorithm used to determine the ghosting regions.

[0052] S104: Use a dense inverse search algorithm to calculate optical flow for the target image pair, the first optical flow image, and the ghosting region to determine the target optical flow image.

[0053] It should be noted that, in the embodiments of this application, the first optical flow image is used as the base optical flow image of the target image pair, and the ghosting region is used as the iteration parameter. The target optical flow image is obtained by iterative calculation through the Dense Inverse Search (DIS) algorithm.

[0054] It should also be noted that, in the embodiments of this application, the DIS algorithm is a method for optical flow calculation that uses backpropagation and gradient descent to seek optimization. Specifically, it is based on block-level correlation backpropagation, which calculates the gradient once and then uses it for multiple backpropagations without needing to re-initialize and calculate the gradient each time. Thus, using the DIS algorithm for optical flow calculation not only reduces the computational load but also improves the accuracy of optical flow estimation.

[0055] This application provides an optical flow estimation method, which involves acquiring a pair of target images for which optical flow to be estimated; downsampling the target image pair to determine a sampled image pair; performing optical flow estimation on the sampled image pair using a preset network to obtain a first optical flow image; and determining the ghosting region of the first optical flow image based on the sampled image pair and the first optical flow image; and using a dense inverse search algorithm to calculate optical flow on the target image pair, the first optical flow image, and the ghosting region to determine the target optical flow image. Thus, by using a preset network to perform optical flow estimation on the downsampled image pair, a significant amount of computational resources and time can be saved, thereby improving the processing speed of optical flow estimation. Furthermore, using a dense inverse search algorithm to iteratively calculate the target image pair, the first optical flow image, and the ghosting region can further reduce the computational load of optical flow estimation, obtaining a high-precision target optical flow image, and thus improving the accuracy of optical flow estimation.

[0056] In another embodiment of this application, based on the optical flow estimation method described in the foregoing embodiments, Figure 2 A flowchart illustrating an optical flow estimation method provided in this application embodiment. Figure 2 .like Figure 2 As shown, the method may include:

[0057] S201: Perform pyramid downsampling on the target image pairs to construct an image pyramid.

[0058] It should be noted that, in the embodiments of this application, a corresponding image pyramid is constructed based on each image in the target image pair. The construction of the image pyramid can be implemented using existing methods. The specific number of layers in the image pyramid is related to the resolution of the target image pair; the higher the resolution of the target image pair, the more layers are required.

[0059] For example, the image pyramid can be a three-layer pyramid, where the width and height of the image at the top layer are both 1 / 4 of the original image, the width and height of the images in the middle layer are both 1 / 2 of the original image, and the bottom layer image is the original image.

[0060] It should also be noted that, in the embodiments of this application, an image pyramid is constructed using target image pairs, so that the resolution of the target optical flow image is consistent with the resolution of the target image pair.

[0061] S202: Based on the current layer image of the image pyramid, upsample the first optical flow image and the ghosting region to obtain the sampled optical flow image and the sampled ghosting region; wherein, the resolution of the sampled optical flow image and the sampled ghosting region is the same as the resolution of the current layer image.

[0062] It should be noted that, in this embodiment, when calculating the second optical flow image of the current layer image, it is first necessary to upsample the first optical flow image and the ghosting region to obtain the sampled optical flow image and the sampled ghosting region. Here, the resolution of the sampled optical flow image and the sampled ghosting region is the same as the resolution of the current layer image, so that the second optical flow image corresponding to the current layer image can be determined based on the sampled optical flow image and the sampled ghosting region.

[0063] S203: Determine the initial optical flow image and the gray value of the ghosting region corresponding to the current layer image, and determine the second optical flow image corresponding to the current layer image based on the gray value of the ghosting region, the initial optical flow image and the sampled optical flow image.

[0064] It should be noted that, in the embodiments of this application, when the current layer image is the top layer image of the image pyramid, the first optical flow image is determined as the initial optical flow image corresponding to the current layer image.

[0065] It should also be noted that in this application, the first optical flow image obtained by the preset network is already quite accurate. Using the first optical flow image as the initial optical flow image of the top layer of the pyramid and then iterating the DIS algorithm to obtain the target optical flow image improves the accuracy of optical flow estimation.

[0066] It should also be noted that, in the embodiments of this application, the grayscale value of the ghosting region after sampling includes a first value and a second value; wherein, the first value is 255 and the second value is 0.

[0067] It should also be noted that, in this embodiment, the current layer image is divided into multiple image blocks. Based on the grayscale values ​​of the sampled ghosting regions corresponding to the image blocks of the current layer image, the initialized optical flow image, and the sampled optical flow image, a second optical flow image corresponding to each image block of the current layer image is obtained. When the corresponding second optical flow image is determined for all image blocks of the current layer image, the second optical flow image corresponding to the current layer image is thus obtained.

[0068] In some embodiments, a second optical flow image corresponding to the current layer image is determined based on the grayscale values ​​of the sampled ghosting region, the initialized optical flow image, and the sampled optical flow image, such as... Figure 3 As shown, the method may include:

[0069] S301: When the gray value of the ghosting region after sampling is the first value, the initial optical flow image is determined as the second optical flow image corresponding to the current layer image.

[0070] It should be noted that in this embodiment, when the grayscale value of the ghosting region after sampling is 255, that region is an occluded region. If the occluded region is used to iteratively obtain the second optical flow image through the DIS algorithm, the second optical flow image will be disordered, resulting in a large error in optical flow estimation. Therefore, if it is an occluded region, the initial optical flow image is determined as the second optical flow image corresponding to the current layer image, eliminating the need for iterative calculation and improving the accuracy of optical flow estimation.

[0071] S302: When the gray value of the ghosting region after sampling is the second value, determine the second optical flow image corresponding to the current layer image based on the initial optical flow image and the sampled optical flow image.

[0072] It should be noted that, in the embodiments of this application, when the gray value of the ghosting region after sampling is 0, that is, the region is not an occluded region, the second optical flow image corresponding to the current layer image is determined based on the initial optical flow image and the sampled optical flow image.

[0073] In some embodiments, when the grayscale value of the ghosting region after sampling is a second value, a second optical flow image corresponding to the current layer image is determined based on the initialized optical flow image and the sampled optical flow image, such as... Figure 4 As shown, the method may include:

[0074] S401: Determine the difference between the current layer image and the sampled optical flow image.

[0075] It should be noted that, in the embodiments of this application, the difference between the current layer image and the sampled optical flow image is calculated using the diff algorithm to obtain the difference value.

[0076] S402: When the difference value is less than the preset difference threshold, determine the second optical flow image corresponding to the current layer image based on the sampled optical flow image.

[0077] It should be noted that, in this embodiment, optical flow estimation of the sampled image pairs using a preset network results in a first optical flow image with relatively small errors. When the difference value is less than a preset difference threshold, this represents a flat region of the first optical flow image. After upsampling the first optical flow image, the optical flow in the flat region will not change significantly; therefore, the second optical flow image corresponding to the current layer image is determined based on the sampled optical flow image.

[0078] It should also be noted that, in the embodiments of this application, the preset difference threshold is set based on experience and is not specifically limited.

[0079] For example, when the difference between the corresponding pixels of the current layer image and the sampled optical flow image is one pixel, the second optical flow image corresponding to the current layer image is determined based on the sampled optical flow image.

[0080] S403: When the difference value is greater than the preset difference threshold, the dense inverse search algorithm is used to iteratively calculate the current layer image, the sampled optical flow image, and the sampled ghosting region to determine the second optical flow image corresponding to the current layer image.

[0081] It should be noted that, in this embodiment, when the difference value is greater than a preset difference threshold, it is considered to be the edge region of the first optical flow image. However, when upsampling the first optical flow image, the sampled optical flow image is obtained by weighted averaging of the current pixel and surrounding pixels. This step can cause blurring in the edge region of the sampled optical flow image, so it is necessary to correct the optical flow image in the edge region.

[0082] It should also be noted that, in the embodiments of this application, when the difference value is greater than the preset difference threshold, the dense inverse search algorithm is used to iteratively calculate the current layer image to determine the second optical flow image corresponding to the current layer image, thereby correcting the optical flow image of the edge region and improving the accuracy of optical flow estimation.

[0083] It should also be noted that, in the embodiments of this application, when the difference value equals a preset difference threshold, the second optical flow image corresponding to the current layer image can be determined based on the sampled optical flow image, or a dense inverse search algorithm can be used to iteratively calculate the current layer image, the sampled optical flow image, and the sampled ghosting region to determine the second optical flow image corresponding to the current layer image. Here, no limitations are placed on the operation where the difference value equals the preset difference threshold.

[0084] In some specific embodiments, the method further includes: in the dense inverse search algorithm, determining that the iteration step size in the iterative calculation satisfies a preset step size range.

[0085] It should be noted that, in this embodiment, the target optical flow image calculated by the DIS algorithm is not very accurate. To prevent the optical flow value of the first optical flow image from being corrupted, the iteration step size in the dense inverse search algorithm is determined to meet a preset step size range. That is, the offset position after each iteration in the DIS algorithm cannot be too large, so the step size needs to be limited. Specifically, the difference in optical flow between the initial optical flow image and the second optical flow image at a given position cannot exceed the preset step size range; in the DIS algorithm, each iteration and the total iteration must not exceed the preset step size range. Here, the preset step size range can be [-2,2] or [-1,1], and the absolute values ​​of the start and end points of the preset step size range must be relatively small.

[0086] S204: Based on a preset order, determine the next layer image of the current layer image, use the second optical flow image as the initial optical flow image corresponding to the next layer image, and use the next layer image as the current layer image. Continue to perform the steps of upsampling the first optical flow image and the ghosting region to obtain the sampled optical flow image and the sampled ghosting region.

[0087] It should be noted that in this embodiment, the preset order is: the image scale in the image pyramid from largest to smallest. Here, image size and image resolution have an inverse relationship, that is, the larger the image size, the smaller the image resolution. In other words, the preset order can also be: the image resolution in the image pyramid from smallest to largest, that is, from the top to the bottom of the image pyramid, optical flow estimation is performed on each layer of the image pyramid sequentially. This saves computational overhead and allows for better handling of the optical flow of larger resolution images.

[0088] S205: After determining the second optical flow image corresponding to the bottom layer image of the image pyramid, the second optical flow image corresponding to the bottom layer image is determined as the target optical flow image.

[0089] It should be noted that, in the embodiments of this application, when it is determined that the current layer image is the bottom layer image of the image pyramid, the second optical flow image corresponding to the current layer image is determined, and at this time the second optical flow image is the target optical flow image.

[0090] This application provides an optical flow estimation method that performs pyramid downsampling on a target image pair to construct an image pyramid. Based on the current layer image of the image pyramid, the first optical flow image and the ghosting region are upsampled to obtain a sampled optical flow image and a sampled ghosting region. The initial optical flow image corresponding to the current layer image and the gray value of the sampled ghosting region are determined, and the second optical flow image corresponding to the current layer image is determined based on the gray value of the sampled ghosting region, the initial optical flow image, and the sampled optical flow image. The next layer image of the current layer image is determined based on a preset order, and the second optical flow image is used as the initial optical flow image corresponding to the next layer image. The next layer image is used as the current layer image, and the upsampling of the first optical flow image and the ghosting region is continued to obtain a sampled optical flow image and a sampled ghosting region. After determining the second optical flow image corresponding to the bottom layer image of the image pyramid, the second optical flow image corresponding to the bottom layer image is determined as the target optical flow image. Thus, by using a dense inverse search algorithm to iteratively calculate the target image pair, the first optical flow image, and the ghosting region, the computational load of optical flow estimation can be further reduced, a high-precision target optical flow image can be obtained, and the accuracy of optical flow estimation can be improved.

[0091] In another embodiment of this application, based on the optical flow estimation method described in the foregoing embodiments, a fast and accurate scheme for calculating image optical flow is proposed. A first optical flow image of a low-resolution target image pair is calculated using a preset network, and then a high-precision target optical flow image is iteratively calculated using a dense inverse search algorithm. This reduces the computational load of optical flow estimation and yields a high-precision target optical flow image with optical flow accuracy comparable to the first optical flow image.

[0092] Figure 5 This diagram illustrates the application framework of an optical flow estimation method provided in an embodiment of this application. In this embodiment, the preset network is a RAFT network. Figure 5 As shown, the process begins by acquiring the original image, converting it to grayscale to obtain target image pairs, and then feeding these target image pairs into a RAFT network to obtain the first optical flow image. The ghosting region is then calculated using the sampled image pairs and the first optical flow image. Next, an image pyramid is constructed using the target image pairs, and the DIS algorithm is used to calculate optical flow for the target image pairs, the first optical flow image, and the ghosting region to determine the target optical flow image of the target image pairs.

[0093] Combination Figure 5 The detailed steps of the method according to the embodiments of this application may include:

[0094] Step 1: Raw image data (i.e., raw data) captured by the CMOS sensor. Here, the raw image data consists of two consecutive frames.

[0095] Step 2: Use Demosai to convert the original image data to grayscale, resulting in two consecutive grayscale images. Use these two grayscale images as the target image pair.

[0096] Step 3: Downsample the target image pairs to obtain low-resolution sampled image pairs. These sampled image pairs are then input into the AI ​​network to determine the first optical flow image; the AI ​​network used is the RAFT network.

[0097] Step 4: Since the RAFT network will have ghosting regions in the local displacement area after optical flow mapping during the optical flow estimation process (i.e., the local displacement images before and after optical flow mapping overlap), the ghosting region is calculated by the diff algorithm.

[0098] It should be noted that, in the embodiments of this application, Figure 6 This is a schematic diagram illustrating the structural composition of a RAFT network provided in an embodiment of this application. Figure 6As shown, the RAFT network consists of three parts: a feature extraction module 601, a feature correlation module 602, and an update operation module 603. In the feature extraction module 601, the feature encoder extracts features from two consecutive frames for subsequent optical flow estimation; the context encoder extracts contextual information from the first frame to ensure that the estimated first optical flow image maintains the same contextual information and positional correspondence as the original image. In the feature correlation module 602, multi-scale correlated image pairs are generated by performing inner product operations on image feature vectors and incorporating multi-scale operations, thus obtaining all image pairs (4D Correlation Volumes). In the update operation module 603, the predicted optical flow is obtained through iterative optimization of all multi-scale image pairs. The core component of the update operation module 603 is a gated activation unit based on a Gate Recurrent Unit (GRU). 0 represents the initial optical flow. In each iteration, the hidden state output by the GRU is superimposed on the initial optical flow by predicting the update direction of the optical flow through two convolutional layers.

[0099] Step 5: Based on the first optical flow image, the target optical flow map of the target image pair is calculated using the ghosting region as the iteration parameter and the DIS algorithm iterative method.

[0100] For example, the code flow of the DIS algorithm is shown in Table 1. As shown in Table 1, U θss+1 As the initial optical flow, then θ ss and θ sf N represents the image pyramid downsampling coefficients. s U represents the number of grid cells. s+1 For the optical flow at the next higher scale, U s The optical flow is at the current scale. The DIS algorithm consists of two steps: (1) Initialization: Construct an image pyramid for the two frames and perform some computational initialization; (2) Iterative loop: Iterate from the top layer to the bottom layer of the image pyramid, calculate the block integral image of the image based on the gradient image of the current layer; then search in reverse to solve the sparse optical flow, calculate the dense optical flow based on the sparse optical flow, and obtain the final target optical flow image.

[0101] Table 1

[0102]

[0103] For example, Figure 7 A schematic diagram of the application framework for an optical flow estimation method provided in this application embodiment. Figure 2 .like Figure 7As shown, a three-layer image pyramid can be used as an example, where the original target image is processed into three images of different scales. The top image (L3) has a width and height that are 1 / 4 of the original image; the middle image (L2) has a width and height that are 1 / 2 of the original image; and the bottom image (L1) is the original image. Figure 7 As shown, the target image is downsampled to obtain an image pyramid. Optical flow estimation is performed from the top layer of the image pyramid to the bottom layer. First, the second optical flow image of the top layer is calculated, and the initial optical flow image of the top layer is initialized using the first optical flow image. The first optical flow image and the ghosting region are upsampled to the same resolution as the top layer image to obtain the sampled optical flow image and the sampled ghosting region. The DIS algorithm is used to estimate the optical flow of the top layer image, the sampled optical flow image, and the sampled ghosting region to obtain the second optical flow image. Second, the second optical flow image of the middle layer is calculated, and the second optical flow image of the top layer is used as the initial optical flow image of the middle layer. The remaining steps are the same as those for calculating the second optical flow image of the top layer, thus obtaining the second optical flow image of the middle layer. Finally, the second optical flow image of the bottom layer is calculated, and the second optical flow image of the middle layer is used as the initial optical flow image of the bottom layer. The remaining steps are the same as those for calculating the second optical flow image of the top layer, thus obtaining the second optical flow image of the bottom layer, which is the target optical flow image.

[0104] It should also be noted that, in the embodiments of this application, the specific processing steps for calculating the target optical flow map of the target image pair using the iterative method of the DIS algorithm include the following steps:

[0105] Step 5.1: Construct an image pyramid from the target image pairs. The resolution of the top layer of the image pyramid is twice that of the first optical flow image.

[0106] Step 5.2: Use the first optical flow image as the initial optical flow image of the top layer of the image pyramid.

[0107] Step 5.3: Determine the initial optical flow image of the current layer image. When calculating the second optical flow image of the current layer image, regions with a grayscale value of 255 in the ghosting area after sampling are not iterated and are considered occluded regions. The initial optical flow image of the current layer image is determined as the second optical flow image. Regions with a grayscale value of 0 in the ghosting area after sampling are not iterated if the difference value is less than a preset difference threshold. These are considered flat regions, and the corresponding sampled optical flow image of the current layer image is used as the second optical flow image. If the difference value is greater than the preset difference threshold, iteration is performed. These are edge regions. The iteration step size is limited, meaning the optical flow difference between the corresponding positions of the initial and second optical flows of the current layer image must not exceed ±2 in each iteration and in the total iterations. This determines the second optical flow image of the current layer image.

[0108] Step 5.4: Use the second optical flow image obtained in Step 5.3 as the initial optical flow image for the next layer image, and repeat the operation of Step 5.3 to obtain a new second optical flow image. After determining the second optical flow image corresponding to the bottom layer image of the image pyramid, determine the second optical flow image corresponding to the bottom layer image as the target optical flow image, thus obtaining the target optical flow image of the lowest size.

[0109] This application provides an optical flow estimation method. The specific implementation of the aforementioned embodiments has been described in detail through the above embodiments. It can be seen that optical flow estimation is performed by combining an AI network and a traditional optical flow estimation network to obtain a target optical flow image. This combines the advantages of both AI networks and traditional optical flow estimation networks, drastically reducing the computational load in obtaining the target optical flow image and improving the processing speed of optical flow estimation; simultaneously, a high-precision target optical flow image is obtained, thereby improving the accuracy of optical flow estimation.

[0110] Based on the same inventive concept as the foregoing embodiments. Figure 8 This is a schematic diagram illustrating the structural composition of an optical flow estimation device provided in an embodiment of this application. Figure 8 As shown, the optical flow estimation device 80 may include an acquisition unit 801, a determination unit 802, and an estimation unit 803, wherein:

[0111] Acquisition unit 801 is configured to acquire a target image pair of optical flow to be estimated;

[0112] The determining unit 802 is configured to downsample the target image pair and determine the sampled image pair;

[0113] The estimation unit 803 is configured to perform optical flow estimation on the sampled image pair through a preset network to obtain a first optical flow image, and determine the ghosting region of the first optical flow image based on the sampled image pair and the first optical flow image; and to perform optical flow calculation on the target image pair, the first optical flow image and the ghosting region using a dense inverse search algorithm to determine the target optical flow image.

[0114] In some embodiments, the acquisition unit 801 is further configured to acquire two consecutive original images; the determination unit 802 is further configured to perform grayscale processing on the two original images to obtain two grayscale images, and determine the two grayscale images as the target image pair for the optical flow to be estimated.

[0115] In some embodiments, the determining unit 802 is further configured to perform a difference calculation between the sampled image pair and the first optical flow image to determine the ghosting region.

[0116] In some embodiments, the determining unit 802 is further configured to perform pyramid downsampling layering on the target image pair to construct an image pyramid; and based on the current layer image of the image pyramid, upsample the first optical flow image and the ghosting region to obtain a sampled optical flow image and a sampled ghosting region; wherein the resolution of the sampled optical flow image and the sampled ghosting region is the same as the resolution of the current layer image; the estimating unit 803 is further configured to determine the initial optical flow image corresponding to the current layer image and the grayscale value of the sampled ghosting region, and based on the grayscale value of the sampled ghosting region and the initial optical flow... The image and the sampled optical flow image are used to determine the second optical flow image corresponding to the current layer image; the determining unit 802 is further configured to determine the next layer image of the current layer image based on a preset order, use the second optical flow image as the initial optical flow image corresponding to the next layer image, and use the next layer image as the current layer image, and continue to perform the steps of upsampling the first optical flow image and the ghosting region to obtain the sampled optical flow image and the sampled ghosting region; and after determining the second optical flow image corresponding to the bottom layer image of the image pyramid, determine the second optical flow image corresponding to the bottom layer image as the target optical flow image.

[0117] In some embodiments, the preset order is: the order of image scale from largest to smallest in the image pyramid.

[0118] In some embodiments, the determining unit 802 is further configured to determine the first optical flow image as the initial optical flow image corresponding to the current layer image when the current layer image is the top layer image of the image pyramid.

[0119] In some embodiments, the determining unit 802 is further configured to determine the initial optical flow image as the second optical flow image corresponding to the current layer image when the gray value of the ghosting region after sampling is a first value; the estimating unit 803 is further configured to determine the second optical flow image corresponding to the current layer image based on the initial optical flow image and the sampled optical flow image when the gray value of the ghosting region after sampling is a second value.

[0120] In some embodiments, the determining unit 802 is further configured to determine the difference value between the current layer image and the sampled optical flow image; the estimating unit 803 is further configured to determine the second optical flow image corresponding to the current layer image based on the sampled optical flow image when the difference value is less than a preset difference threshold; and to perform iterative calculations on the current layer image, the sampled optical flow image and the sampled ghosting region using a dense inverse search algorithm when the difference value is greater than the preset difference threshold, thereby determining the second optical flow image corresponding to the current layer image.

[0121] In some embodiments, the determining unit 802 is further configured to determine, in a dense inverse search algorithm, that the iteration step size in the iterative calculation satisfies a preset step size range.

[0122] In some embodiments, the preset network includes at least one of the following: RAFT network, GMFlow network, Flow network, and PWC-Net network.

[0123] Understandably, in this embodiment, a "unit" can be a portion of a circuit, a portion of a processor, a portion of a program or software, etc., and can also be a module or a non-modular component. Furthermore, the components in this embodiment can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional module.

[0124] If the integrated unit is implemented as a software functional module and not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the method described in this embodiment. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0125] Therefore, this embodiment provides a computer-readable storage medium storing a computer program that, when executed by at least one processor, implements the steps of the method described in any of the foregoing embodiments.

[0126] Based on the composition of the optical flow estimation device 80 and the computer-readable storage medium described above, Figure 9 This is a schematic diagram of the specific hardware structure of an electronic device provided in an embodiment of this application. For example... Figure 9 As shown, electronic device 90 may include: a communication interface 901, a memory 902, and a processor 903; the various components are coupled together via a bus system 904. It is understood that the bus system 904 is used to implement communication between these components. In addition to a data bus, the bus system 904 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 9 The various buses are all labeled as bus system 904. Among them, the communication interface 901 is used for receiving and sending signals during the process of sending and receiving information with other external network elements;

[0127] Memory 902 is used to store computer programs that can run on processor 903;

[0128] Processor 903, when running the computer program, performs the following:

[0129] Obtain a pair of target images for which optical flow to be estimated; downsample the pair of target images to determine the sampled image pair; perform optical flow estimation on the sampled image pair using a preset network to obtain a first optical flow image, and determine the ghosting region of the first optical flow image based on the sampled image pair and the first optical flow image; use a dense inverse search algorithm to calculate optical flow on the pair of target images, the first optical flow image, and the ghosting region to determine the target optical flow image.

[0130] It is understood that the memory 902 in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DRRAM). The memory 902 of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0131] The processor 903 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 903 or by software instructions. The processor 903 can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory 902, and the processor 903 reads the information in memory 902 and, in conjunction with its hardware, completes the steps of the above method.

[0132] It is understood that the embodiments described herein can be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described herein, or combinations thereof.

[0133] For software implementation, the techniques described herein can be achieved through modules (e.g., procedures, functions, etc.) that perform the functions described herein. The software code can be stored in memory and executed by a processor. The memory can be implemented within the processor or externally.

[0134] Alternatively, as another embodiment, the processor 903 is further configured to perform the steps of the method described in any of the foregoing embodiments when running the computer program.

[0135] In some embodiments, this application also provides an electronic device 90, which may include at least the optical flow estimation device 80 described in any of the foregoing embodiments.

[0136] In this embodiment of the application, for the electronic device 90, optical flow estimation of the downsampled image pair through a preset network can save a lot of resources and time in computation, thereby improving the processing speed of optical flow estimation; moreover, by using a dense inverse search algorithm to iteratively calculate the target image pair, the first optical flow image and the ghosting region, the computational load of optical flow estimation can be further reduced, a high-precision target optical flow image can be obtained, thereby improving the accuracy of optical flow estimation.

[0137] It should be noted that, in this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, product, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, product, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, product, or apparatus that includes that element.

[0138] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0139] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0140] The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0141] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0142] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An optical flow estimation method, characterized in that, The method includes: Acquire the target image pair whose optical flow needs to be estimated; The target image pair is downsampled to determine the sampled image pair; Optical flow estimation is performed on the sampled image pairs using a preset network to obtain a first optical flow image, and the ghosting region of the first optical flow image is determined based on the sampled image pairs and the first optical flow image. The target image pairs are subjected to pyramid downsampling and layering to construct an image pyramid; Based on the current layer image of the image pyramid, the first optical flow image and the ghosting region are upsampled to obtain a sampled optical flow image and a sampled ghosting region; wherein the resolution of the sampled optical flow image and the sampled ghosting region is the same as the resolution of the current layer image; Determine the initial optical flow image corresponding to the current layer image and the gray value of the sampled ghosting region, and determine the second optical flow image corresponding to the current layer image based on the gray value of the sampled ghosting region, the initial optical flow image, and the sampled optical flow image; Based on a preset order, determine the next layer image of the current layer image, use the second optical flow image as the initial optical flow image corresponding to the next layer image, and use the next layer image as the current layer image. Continue to perform the steps of upsampling the first optical flow image and the ghosting region to obtain the sampled optical flow image and the sampled ghosting region. After determining the second optical flow image corresponding to the bottom layer image of the image pyramid, the second optical flow image corresponding to the bottom layer image is determined as the target optical flow image.

2. The method according to claim 1, characterized in that, The acquisition of the target image pair of the optical flow to be estimated includes: Acquire two consecutive raw images; The two original images are processed to obtain two grayscale images, which are then identified as the target image pair for the optical flow to be estimated.

3. The method according to claim 1, characterized in that, The step of determining the ghosting region of the first optical flow image based on the sampled image pair and the first optical flow image includes: The difference between the sampled image pair and the first optical flow image is calculated to determine the ghosting region.

4. The method according to claim 1, characterized in that, The preset order is: the order of image scale from largest to smallest in the image pyramid.

5. The method according to claim 1, characterized in that, The method further includes: When the current layer image is the top layer image of the image pyramid, the first optical flow image is determined as the initial optical flow image corresponding to the current layer image.

6. The method according to claim 1, characterized in that, The step of determining the second optical flow image corresponding to the current layer image based on the grayscale value of the sampled ghosting region, the initialized optical flow image, and the sampled optical flow image includes: When the grayscale value of the ghosting region after sampling is a first value, the initialized optical flow image is determined as the second optical flow image corresponding to the current layer image; When the grayscale value of the ghosting region after sampling is the second value, the second optical flow image corresponding to the current layer image is determined based on the initial optical flow image and the sampled optical flow image.

7. The method according to claim 6, characterized in that, When the grayscale value of the ghosting region is the second value, determining the second optical flow image corresponding to the current layer image based on the initialized optical flow image and the sampled optical flow image includes: Determine the difference between the current layer image and the sampled optical flow image; When the difference value is less than a preset difference threshold, a second optical flow image corresponding to the current layer image is determined based on the sampled optical flow image; When the difference value is greater than a preset difference threshold, a dense inverse search algorithm is used to iteratively calculate the current layer image, the sampled optical flow image, and the sampled ghosting region to determine the second optical flow image corresponding to the current layer image.

8. The method according to claim 7, characterized in that, The method further includes: In the dense reverse search algorithm, the iteration step size in the iterative calculation is determined to satisfy a preset step size range.

9. The method according to any one of claims 1 to 8, characterized in that, The preset network includes at least one of the following: RAFT network, GMFlow network, Flow network, and PWC-Net network.

10. An optical flow estimation device, characterized in that, The optical flow estimation device includes an acquisition unit, a determination unit, and an estimation unit, wherein: The acquisition unit is configured to acquire a pair of target images of optical flow to be estimated. The determining unit is configured to downsample the target image pair to determine the sampled image pair; The estimation unit is configured to perform optical flow estimation on the sampled image pair through a preset network to obtain a first optical flow image, and to determine the ghosting region of the first optical flow image based on the sampled image pair and the first optical flow image. The determining unit is further configured to perform pyramid downsampling layering on the target image pair to construct an image pyramid; based on the current layer image of the image pyramid, upsample the first optical flow image and the ghosting region to obtain a sampled optical flow image and a sampled ghosting region; wherein the resolution of the sampled optical flow image and the sampled ghosting region is the same as the resolution of the current layer image; The estimation unit is further configured to determine the initial optical flow image corresponding to the current layer image and the gray value of the sampled ghosting region, and to determine the second optical flow image corresponding to the current layer image based on the gray value of the sampled ghosting region, the initial optical flow image and the sampled optical flow image; The determining unit is further configured to determine the next layer image of the current layer image based on a preset order, use the second optical flow image as the initial optical flow image corresponding to the next layer image, and use the next layer image as the current layer image to continue performing the step of upsampling the first optical flow image and the ghosting region to obtain the sampled optical flow image and the sampled ghosting region; and after determining the second optical flow image corresponding to the bottom layer image of the image pyramid, determine the second optical flow image corresponding to the bottom layer image as the target optical flow image.

11. An electronic device, characterized in that, The electronic device includes a memory and a processor, wherein: The memory is used to store computer programs that can run on the processor; The processor is configured to perform the method as described in any one of claims 1 to 9 when running the computer program.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by at least one processor, implements the method as described in any one of claims 1 to 9.

Images