Optical flow calculation method, device, electronic device and storage medium

Abstract

The present application discloses an optical flow calculation method, comprising: constructing an N-layer first image pyramid of a target frame and an N-layer second image pyramid of a reference frame; configuring an initial optical flow of an N-th layer image of the first image pyramid; the initial optical flow including a set of offsets for each pixel point, wherein each random offset in each set of offsets is used to indicate a pixel position in an N-th layer image of the second image pyramid; performing block matching processing on the N-th layer images of the first and second image pyramids according to the initial optical flow, determining a refined optical flow of the N-th layer image of the first image pyramid according to the block matching processing results, and upsampling the refined optical flow to obtain an initial optical flow of an N-1-th layer image of the first image pyramid; based on the initial optical flow of the N-1-th layer image, calculating the refined optical flow of each layer image below the N-th layer image of the first image pyramid layer by layer until the refined optical flow of the first layer image is obtained, and denoising the refined optical flow to obtain a target optical flow.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and specifically relates to an optical flow calculation method, device, electronic device and storage medium. Background Art

[0002] Optical flow is a key technology in image processing, used to track moving objects within image sequences. It's widely used in fields like photography and gaming. For example, in photography, cameras can automatically stabilize video footage based on the optical flow of the image sequence, reducing camera shake and resulting in smoother, clearer videos. In gaming, compared to the motion vector information inherent in games, optical flow can account for motion in scenes like shadows and translucency, resulting in more coherent interpolation effects in game footage.

[0003] In related art, a method combining a block matching algorithm and an image pyramid algorithm is used to calculate the optical flow of an image. First, an image pyramid of the target frame and an image pyramid of the reference frame are constructed, where the image with the lowest resolution is at the top layer and the image with the original resolution is at the bottom layer. Then, starting from the top layer of the image pyramid, the initial optical flow of the top layer is set to all zeros, and a fine-tuned optical flow of the top layer is calculated using a block matching method. This fine-tuned optical flow of the top layer is transferred to the next layer of the image pyramid to serve as the initial optical flow of the next layer. Next, the initial optical flow of the next layer is fine-tuned using a block matching method to obtain a fine-tuned optical flow of the next layer. This fine-tuned optical flow of the next layer is transferred to the next layer of the image pyramid to serve as the initial optical flow of the next layer. The above calculation process for the next layer of the image pyramid is repeated for the remaining layers until the fine-tuned optical flow of the bottom layer of the image pyramid is calculated. Finally, the fine-tuned optical flow of the bottom layer of the image pyramid is denoised to obtain the denoised optical flow, which is the optical flow of each pixel in the target frame.

[0004] However, for the top layer of the image pyramid, the initial optical flow of the top layer is all zero, that is, there will be no information. Therefore, for each image block of the top layer image of the image pyramid of the target frame, the best matching block within the search window can only be found pixel by pixel in the top layer image of the image pyramid of the reference frame based on the search window. Since the size of the search window is usually set to be relatively small, the position of the best matching block may be wrong for scenes with large motion, and the fine-tuned optical flow of the top layer will be passed to the second top layer as the initial optical flow of the second top layer. Therefore, if the optical flow of the top layer is wrong, it will be more difficult for the subsequent layers to find the best matching block, which ultimately leads to inaccurate optical flow results calculated for scenes with large motion. Summary of the Invention

[0005] The purpose of the embodiments of the present application is to provide an optical flow calculation method, device, electronic device and storage medium, which can improve the accuracy of the optical flow results calculated for scenes with large motion.

[0006] In a first aspect, an embodiment of the present application provides an optical flow calculation method, the method comprising:

[0007] Constructing a first image pyramid of the target frame and a second image pyramid of the reference frame; wherein the number of layers of the first image pyramid and the number of layers of the second image pyramid are both N, where N is an integer greater than 1;

[0008] Configuring an initial optical flow of an N-th layer image of the first image pyramid; wherein the initial optical flow of the N-th layer image of the first image pyramid comprises: a set of offsets for each pixel in the N-th layer image of the first image pyramid, each set of offsets comprising: at least two different random offsets, each of the random offsets being used to indicate a pixel position in the N-th layer image of the second image pyramid;

[0009] performing block matching on the Nth layer image of the first image pyramid and the Nth layer image of the second image pyramid based on the initial optical flow of the Nth layer image of the first image pyramid, determining a fine-tuned optical flow of the Nth layer image of the first image pyramid based on the block matching result, upsampling the fine-tuned optical flow of the Nth layer image of the first image pyramid, and determining the upsampling result as the initial optical flow of the N-1th layer image of the first image pyramid; wherein the fine-tuned optical flow of the Nth layer image of the first image pyramid includes: a fine-tuned optical flow value of each pixel in the Nth layer image of the first image pyramid;

[0010] Based on the initial optical flow of the N-1th layer image of the first image pyramid, calculating the refined optical flow of each layer image below the Nth layer image of the first image pyramid layer by layer until the refined optical flow of the first layer image of the first image pyramid is obtained;

[0011] Denoising is performed on the fine-tuned optical flow of the first layer image of the first image pyramid to obtain a target optical flow of the target frame.

[0012] In a second aspect, an embodiment of the present application provides an optical flow calculation device, the device comprising:

[0013] a construction module, configured to construct a first image pyramid of a target frame and a second image pyramid of a reference frame; wherein the number of layers of the first image pyramid and the number of layers of the second image pyramid are both N, where N is an integer greater than 1;

[0014] a configuration module, configured to configure an initial optical flow of an N-th layer image of the first image pyramid; wherein the initial optical flow of the N-th layer image of the first image pyramid comprises: a set of offsets for each pixel in the N-th layer image of the first image pyramid, each set of offsets comprising: at least two different random offsets, each of the random offsets being used to indicate a pixel position in the N-th layer image of the second image pyramid;

[0015] a first processing module configured to perform block matching on the Nth layer image of the first image pyramid and the Nth layer image of the second image pyramid based on the initial optical flow of the Nth layer image of the first image pyramid, determine a refined optical flow of the Nth layer image of the first image pyramid based on the block matching result, upsample the refined optical flow of the Nth layer image of the first image pyramid, and determine the upsampling result as the initial optical flow of the N-1th layer image of the first image pyramid; wherein the refined optical flow of the Nth layer image of the first image pyramid includes a refined optical flow value of each pixel in the Nth layer image of the first image pyramid;

[0016] a second processing module, configured to calculate, based on the initial optical flow of the image at the N-1th layer of the first image pyramid, a refined optical flow of each layer below the Nth layer of the first image pyramid, layer by layer, until a refined optical flow of the first layer of the first image pyramid is obtained;

[0017] The denoising module is configured to perform denoising on the fine-tuned optical flow of the first layer image of the first image pyramid to obtain the target optical flow of the target frame.

[0018] In a third aspect, an embodiment of the present application provides an electronic device comprising a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the optical flow calculation method as described in the first aspect are implemented.

[0019] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the optical flow calculation method as described in the first aspect are implemented.

[0020] In a fifth aspect, an embodiment of the present application provides a chip, which includes a processor and a communication interface, the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the optical flow calculation method as described in the first aspect.

[0021] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the optical flow calculation method as described in the first aspect.

[0022] In an embodiment of the present application, a first image pyramid of a target frame and a second image pyramid of a reference frame are constructed; wherein the number of layers of the first image pyramid and the number of layers of the second image pyramid are both N, where N is an integer greater than 1; an initial optical flow of an Nth layer image of the first image pyramid is configured; wherein the initial optical flow of the Nth layer image of the first image pyramid includes: a set of offsets for each pixel point in the Nth layer image of the first image pyramid, each set of offsets includes: at least two different random offsets, each random offset is used to indicate a pixel position in the Nth layer image of the second image pyramid; and block matching is performed on the Nth layer image of the first image pyramid and the Nth layer image of the second image pyramid based on the initial optical flow of the Nth layer image of the first image pyramid. The method includes performing block matching processing, determining a fine-tuned optical flow of an N-th layer image of a first image pyramid according to a block matching processing result, upsampling the fine-tuned optical flow of the N-th layer image of the first image pyramid, and determining the upsampling result as the initial optical flow of an N-1-th layer image of the first image pyramid; wherein the fine-tuned optical flow of the N-th layer image of the first image pyramid includes: a fine-tuned optical flow value of each pixel point in the N-th layer image of the first image pyramid; based on the initial optical flow of the N-1-th layer image of the first image pyramid, calculating the fine-tuned optical flow of each layer image below the N-th layer image of the first image pyramid layer by layer until the fine-tuned optical flow of the first layer image of the first image pyramid is obtained; and performing denoising processing on the fine-tuned optical flow of the first layer image of the first image pyramid to obtain a target optical flow of a target frame.

[0023] It can be seen that compared with the related art in which the optical flow of the top layer of the image pyramid is initialized to all zeros, the initial optical flow based on all zeros cannot provide any information for the top layer, and can only find the most matching block in the neighborhood of the search window based on the search window. The small search box cannot cover large motion scenes. In the embodiment of the present application, the optical flow of the top layer of the image pyramid is randomly initialized. Since the randomly initialized optical flow contains a set of random offsets for each pixel point of the top layer image, and there is always a random offset within a certain range that is lucky enough to hit the most matching image block, further fine-tuning in the next layer can better estimate the final optical flow. Therefore, the block matching processing of the top layer image is no longer limited to the size of the search window, and the accuracy of the optical flow results calculated for scenes with large motion can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 is an example diagram of a block matching method provided by some embodiments of the present application;

[0025] Figure 2 is an example diagram of an image pyramid provided by some embodiments of the present application;

[0026] Figure 3A This is one of the example diagrams of the optical flow calculation method in the related art provided by some embodiments of the present application;

[0027] Figure 3B This is the second example diagram of the optical flow calculation method in the related art provided by some embodiments of the present application;

[0028] Figure 3C This is the third example diagram of the optical flow calculation method in the related art provided by some embodiments of the present application;

[0029] Figure 4 is a flowchart of an optical flow calculation method provided by some embodiments of the present application;

[0030] Figure 5 is an example graph of the initial optical flow of the top image of the first image pyramid provided by some embodiments of the present application;

[0031] Figure 6A is a flowchart of an implementation of step 403 provided in some embodiments of the present application;

[0032] Figure 6B This is one of the example diagrams of an implementation of step 403 provided in some embodiments of the present application;

[0033] Figure 6C This is a second example diagram of an implementation of step 403 provided in some embodiments of the present application;

[0034] Figure 7A is a flowchart of an implementation of step 404 provided in some embodiments of the present application;

[0035] Figure 7B is a flowchart of an implementation of step 4041 provided in some embodiments of the present application;

[0036] Figure 7C This is one of the example diagrams of an implementation of step 4041 provided in some embodiments of the present application;

[0037] Figure 7D This is a second example diagram of an implementation of step 4041 provided in some embodiments of the present application;

[0038] Figure 7E This is a third example diagram of an implementation of step 4041 provided in some embodiments of the present application;

[0039] Figure 7FThis is a fourth example diagram of an implementation of step 4041 provided in some embodiments of the present application;

[0040] Figure 7G This is a fifth example diagram of an implementation of step 4041 provided in some embodiments of the present application;

[0041] Figure 8A is a flowchart of a process for generating a target denoising model provided by some embodiments of the present application;

[0042] Figure 8B is an example diagram of an initial denoising model provided by some embodiments of the present application;

[0043] Figure 8C is an example diagram of a target denoising model provided by some embodiments of the present application;

[0044] Figure 9 is a structural block diagram of an optical flow calculation device provided by some embodiments of the present application;

[0045] Figure 10 is a schematic structural diagram of an electronic device provided by some embodiments of the present application;

[0046] Figure 11 It is a schematic diagram of the hardware structure of the electronic device provided in some embodiments of the present application. DETAILED DESCRIPTION

[0047] The following will be combined with the accompanying drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the embodiments described are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field fall within the scope of protection of this application.

[0048] The terms "first," "second," and the like in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of this application can be implemented in an order other than that illustrated or described herein, and that the objects distinguished by "first," "second," and the like are generally of the same type, and do not limit the number of objects; for example, the first object can be one or more. In addition, the term "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0049] To facilitate understanding, some relevant concepts and application scenarios involved in the embodiments of this application are first introduced.

[0050] 1. Related Concepts

[0051] Optical flow is a vector with direction and length. The purpose of optical flow calculation is to solve the motion vector (or offset) of the corresponding pixel based on two consecutive frames.

[0052] Optical flow field: In space, motion can be described using a motion field. On an image plane, the motion of an object is often reflected by the varying grayscale distributions of different images in a sequence. Therefore, the transfer of the motion field from space to the image is represented as an optical flow field. The optical flow field is a two-dimensional vector field that reflects the changing trend of the grayscale at each point in the image. It can be viewed as the instantaneous velocity field generated by the movement of grayscale pixels on the image plane. The information it contains is the instantaneous velocity vector information of each pixel. The purpose of studying the optical flow field is to approximate the motion field, which cannot be directly obtained from a sequence of images. Ideally, the optical flow field corresponds to the motion field.

[0053] Dense optical flow is an image registration method that performs pixel-by-pixel matching on an image or a specified area. It calculates the offset of all pixels in the image to form a dense optical flow field. This dense optical flow field allows for pixel-level image registration.

[0054] In contrast to dense optical flow, sparse optical flow is not calculated point by point for every pixel in the image.

[0055] Block matching is a commonly used method in image denoising and motion estimation. By matching the query block with adjacent image blocks, the K blocks closest to the query block are found from these adjacent blocks. The so-called adjacent blocks do not necessarily mean adjacent in absolute position, which also leads to local search (local) and global search (non-local).

[0056] For the block matching process in optical flow calculation, such as Figure 1As shown, target frame 10 and reference frame 20 are compared. For example, calculating the optical flow value of a pixel 11 in target frame 10 involves first determining an image block 12 centered at pixel 11. Next, a search window 21 corresponding to image block 12 is determined in reference frame 20. Block matching for image block 12 can only be performed within search window 21. Next, image block 12 is moved up, down, left, and right within search window 21 to search for the matching image block with the highest similarity to image block 12. The similarity between two image blocks can be calculated using algorithms such as squared difference (SSD), absolute difference (SAD), and census transform. For example, if a comparison reveals that image block 22 within search window 21 has the highest similarity to image block 12, image block 22 is determined as the matching image block for image block 12. Finally, the offset between image blocks 12 and 22 is calculated, and the calculated offset is used as the optical flow value for the center pixel 11 of image 12. When the size of the search window 21 is smaller than the size of the reference frame 20, the above block matching process is a local search. When the size of the search window 21 is equal to the size of the reference frame 20, the above block matching process is a global search.

[0057] Image pyramid is a kind of multi-scale representation of images. It is an effective but simple structure to interpret images at multiple resolutions. Take the 4-layer image pyramid as an example. Figure 2 As shown in Figure 1, an image pyramid is a collection of images with decreasing resolutions (from bottom to top) arranged in a pyramidal shape, originating from the same original image. It is obtained by sequential downsampling until a certain termination condition is reached. The layers of an image can be likened to a pyramid: the higher the level, the smaller the image size and the lower the resolution.

[0058] Re-parameterization is a technique in machine learning and deep learning that changes the representation of model parameters to enable more efficient or stable optimization. Re-parameterization represents a random variable (such as a latent vector) as a function of another random variable, making it amenable to standard optimization algorithms.

[0059] Structural reparameterization involves first constructing a structure for training and then converting the parameters to another set of equivalent parameters during inference. This allows for a higher overhead during training but a lower overhead during inference. Alternatively, the reparameterized structure adds parameters during training that can be removed during inference. The core concept of structural reparameterization is that convolution is a linear operation with additive properties. To implement structural reparameterization, a specially designed multi-branch module is required, which only performs linear operations, such as convolution and batch normalization (BN).

[0060] 2. Application Scenarios

[0061] Optical flow is an important technology in computer vision and image processing, used to track moving objects in image sequences. Currently, optical flow technology is widely used in photography and gaming, providing users with a more immersive experience.

[0062] In related technologies, a method combining block matching algorithm and image pyramid algorithm is used to calculate the optical flow of an image. Figure 3A As shown, an image pyramid of the target frame and an image pyramid of the reference frame are constructed, wherein the image with the lowest resolution in the image pyramid is at the top layer, and the image with the original resolution is at the bottom layer. For example, the number of layers of the image pyramid is 4.

[0063] Afterwards, starting from the top layer (i.e., the fourth layer) of the image pyramid, the initial optical flow of the fourth layer of the target frame's image pyramid is set to all zero. Since the all-zero initial optical flow cannot provide any information, for each image block of the fourth layer of the target frame's image pyramid, we can only find the best matching block within the search window pixel by pixel in the fourth layer of the reference frame's image pyramid based on the search window. For example, Figure 3B As shown, the fourth layer image 31 of the image pyramid of the target frame and the fourth layer image 32 of the image pyramid of the reference frame are compared. For example, the calculation of the optical flow value of a pixel 311 in image 31 is performed. First, an image block 312 centered at pixel 311 is determined. Next, a search window 321 corresponding to image block 312 is determined in image 32. Image block 312 can only be matched within search window 321. Next, image block 312 is moved pixel by pixel up, down, left, and right within search window 321 to find the matching image block with the highest similarity to image block 312. The similarity between two image blocks can be calculated using algorithms such as squared difference (SSD), absolute difference (SAD), and census transform. For example, if a comparison shows that image block 322 within search window 321 has the highest similarity to image block 312, image block 322 is determined as the matching image block of image block 312. Finally, the offset between image block 312 and image block 322 is calculated and used as the fine-tuned optical flow value for center pixel 311 of image 312. Similarly, the fine-tuned optical flow value for each pixel in the fourth layer of the target frame's image pyramid can be calculated, i.e., the fine-tuned optical flow of the fourth layer. After calculating the fine-tuned optical flow of the fourth layer of the target frame's image pyramid, the fine-tuned optical flow of the fourth layer is upsampled and transferred to the third layer of the target frame's image pyramid to serve as the initial optical flow of the third layer.

[0064] Then, the initial optical flow of the third layer image is fine-tuned according to the block matching method to obtain the fine-tuned optical flow of the third layer image. Figure 3C As shown, the third layer image 41 of the image pyramid of the target frame and the third layer image 42 of the image pyramid of the reference frame are compared. Taking the optical flow value of a pixel 411 in the refined image 41 as an example, an image block 412 centered at pixel 411 is determined. Based on the initial optical flow value of pixel 411 in the initial optical flow of image 41, an image block 422 corresponding to the initial optical flow value of pixel 411 is determined in image 42. A search window 421 corresponding to image block 412 is determined in image 42. Image block 412 is moved pixel by pixel up, down, left, and right within search window 421. A matching image block with the highest similarity to image block 412 is searched within search window 421. The similarity between two image blocks can be calculated using algorithms such as squared difference (SSD), absolute difference (SAD), and census transform. For example, after comparison, it is found that image block 423 within search window 421 has the highest similarity with image block 412. Since the similarity between image block 423 and image block 412 is greater than the similarity between image block 422 and image block 412, the initial optical flow value of pixel 413 in image 41, corresponding to center pixel 424 of image block 423, is more suitable as the fine-tuned optical flow value for pixel 411 than the initial optical flow value of pixel 411. In this case, the initial optical flow value of pixel 413 is determined as the fine-tuned optical flow value for pixel 411. Similarly, the fine-tuned optical flow value of each pixel in the third layer of the image pyramid of the target frame can be calculated, i.e., the fine-tuned optical flow of the third layer. After calculating the fine-tuned optical flow of the third layer of the image pyramid of the target frame, the fine-tuned optical flow of the third layer is upsampled and the upsampled result is transferred to the second layer of the image pyramid of the target frame to serve as the initial optical flow of the second layer.

[0065] The same calculation process as described above for the third layer is performed on the second layer of the target frame's image pyramid to obtain the fine-tuned optical flow of the second layer of the target frame's image pyramid. The fine-tuned optical flow of the second layer is upsampled, and the upsampled result is transferred to the first layer of the target frame's image pyramid to serve as the initial optical flow of the first layer. The same calculation process as described above for the third layer is performed on the first layer of the target frame's image pyramid to obtain the fine-tuned optical flow of the first layer of the target frame's image pyramid.

[0066] Finally, the fine-tuned optical flow of the bottom layer (i.e., the first layer) of the image pyramid is denoised to obtain the denoised optical flow, which is the optical flow of each pixel in the target frame. The optical flow calculation method based on block matching is adopted. Since local similarity is not considered during block search, the best matching block is searched pixel by pixel. Therefore, the optical flow calculated based on the block matching method usually has relatively large noise. In order to solve this problem, the related art uses a spatial iteration method to denoise the optical flow, that is, after calculating the optical flow, the offset of the current pixel point and its upper and left positions is compared, and these two offsets are applied to the image to be matched to determine whether the block matching is more accurate than the offset of the current position. If so, the optical flow value of the current position is directly used for filling. Alternatively, the related art uses an average filtering method or a median filtering method.

[0067] Although related technologies can calculate optical flow to a certain extent, they are limited by the following considerations: 1) The block matching method mentioned above requires searching for the best matching image block within a certain search window. Therefore, the selection of the search window range is very important. In theory, a global search is a better choice to find the best matching image block, that is, the search window size is the same as the image size. However, in practice, if a global search is chosen, the computational complexity is very large and the time consumption is very long. To achieve real-time performance, even with local search, the search window size is limited to a very small size, which greatly reduces the optical flow effect. The optical flow can only estimate scenes with very small motion. The optical flow effect also fluctuates greatly with the number of pyramid levels. 2) Since the block matching method is based on finding the best matching block, the final offset can be anywhere. In other words, the optical flow between adjacent pixels can vary greatly, which is manifested as a noisy optical flow map. However, this phenomenon is inconsistent with actual motion. In real scenes, adjacent pixels usually have similar motion directions and amplitudes. Therefore, the actual optical flow should have local correlation, that is, the optical flow of adjacent pixels should change smoothly or slowly.

[0068] Related art techniques have the following problems: 1) For the top layer of the image pyramid, block matching-based optical flow estimation methods use all-zero initialization. This all-zero initialization typically lacks any information at the current layer, requiring only a search window to find the best matching block within its neighborhood. However, because the search window size is typically set relatively small, the location of the best matching block may be incorrect for scenes with significant motion. The fine-tuned optical flow from the top layer is then transferred to the next top layer as the initial optical flow for that layer. Consequently, an incorrect optical flow at the top layer makes it difficult for subsequent layers to find the best matching block, ultimately leading to inaccurate optical flow results for scenes with significant motion. While the search window size can be increased, this significantly increases computational complexity and the amount of computation required. Another problem with all-zero initialization is that, because the magnitude of motion increases with the number of pyramid layers, the range of motion vectors at each layer is always different. When attempting to reduce computational effort by changing the number of pyramid layers, the resulting optical flow results can vary significantly. 2) For the second-top layer of the image pyramid and layers below it, the block search for optical flow is based on a search window, searching pixel by pixel within the search window. However, since the optical flow values ​​at neighboring locations are generally smooth and the offset relative to the center pixel is very small, updating the optical flow of the center pixel using the neighboring optical flow is a relatively redundant calculation. 3) Spatial iterative denoising methods are widely used in current optical flow calculations. Since each position depends on the offset from the upper and left edges, parallelization poses significant challenges. Furthermore, since optical flow calculations are based on block matching, there is a certain amount of prior information involved, not random positive and negative salt and pepper noise. Therefore, averaging-based filtering methods such as mean filtering and Gaussian filtering cannot solve this problem. Median filtering is also unsuitable for optical flow because, for scenes with straight lines, the inaccurate matching and superposition of median filtering will cause the straight lines in the image to become distorted after the application of optical flow.

[0069] In order to solve the above technical problems, the embodiments of the present application provide an optical flow calculation method, device, electronic device and storage medium, which can cover scenes with various motion amplitudes, reduce the amount of calculation in the optical flow calculation process, and improve the accuracy of the optical flow calculation results.

[0070] The following describes an optical flow calculation method provided by an embodiment of the present application in conjunction with the accompanying drawings.

[0071] It should be noted that the optical flow calculation method provided in the embodiment of the present application is applicable to electronic devices. In practical applications, the electronic devices include but are not limited to: mobile terminals such as mobile phones, tablet computers, laptops, PDAs, and computer devices such as servers and desktops. The embodiment of the present application does not limit this.

[0072] Figure 4is a flowchart of an optical flow calculation method provided by some embodiments of the present application, such as Figure 4 As shown, the method includes at least the following steps: step 401, step 402, step 403, step 404 and step 405;

[0073] In step 401 , a first image pyramid of a target frame and a second image pyramid of a reference frame are constructed; wherein the number of layers of the first image pyramid and the second image pyramid are both N, where N is an integer greater than 1.

[0074] In an embodiment of the present application, the target frame and the reference frame may be two adjacent frames in an image sequence, wherein the image sequence may be a video captured by a camera device, or the image sequence may be a game screen image of a game application.

[0075] In the embodiment of the present application, the first image pyramid and the second image pyramid may both be Gaussian pyramids.

[0076] Exemplarily, taking the construction of the first image pyramid as an example, first, a series of Gaussian smoothing and downsampling operations are performed on the target frame to generate a set of image hierarchies with gradually decreasing resolutions; wherein, Gaussian smoothing is used to apply a Gaussian filter to the original image to generate a smoothed image; the Gaussian filter is a low-pass filter used to reduce high-frequency noise in the image; the downsampling operation is used to downsample the smoothed image, usually by halving the width and height of the image to obtain a lower resolution image. Afterwards, the above steps are repeated: the Gaussian smoothing and downsampling process is repeated on the downsampled image until the predetermined resolution level is reached, and each newly generated image is called a pyramid layer. For example, N=4, the following can be constructed: Figure 2 Image pyramid of the shown structure.

[0077] For example, N=4, and the first image pyramid is a four-layer pyramid. From top to bottom, they are the fourth layer image, the third layer image, the second layer image, and the first layer image. The fourth layer is the top layer (or first layer) of the pyramid, and the first layer is the bottom layer of the pyramid. Similarly, the second image pyramid is also a four-layer pyramid. From top to bottom, they are the fourth layer image, the third layer image, the second layer image, and the first layer image. The fourth layer is the top layer of the pyramid, and the first layer is the bottom layer of the pyramid.

[0078] In step 402, an initial optical flow of an N-th level image of a first image pyramid is configured. The initial optical flow of the N-th level image of the first image pyramid includes a set of offsets for each pixel in the N-th level image of the first image pyramid, each set of offsets including at least two different random offsets, each random offset being used to indicate a pixel position in the N-th level image of a second image pyramid.

[0079] In the embodiment of the present application, the Nth layer image of the first image pyramid refers to the image at the top layer of the first image pyramid, which has the smallest resolution.

[0080] On the one hand, considering that in the related art the optical flow of the Nth layer image of the first image pyramid is initialized to all zeros and cannot provide any information, only pixel-by-pixel search can be performed within the search window, and the search calculation amount is large; on the other hand, considering that although the search calculation amount can be reduced by reducing the size of the search window, for scenes with large motion, a small search window will cause the position of the most matching image block to be incorrect, making it difficult to maintain a balance between the performance (computational amount) and effect (computational results) of the block matching method.

[0081] In an embodiment of the present application, the optical flow of the top image of the image pyramid is initialized to a random value, and the randomized initial optical flow is used to guide the block matching process of the top image of the image pyramid, thereby replacing the block matching process of the top image of the image pyramid based on the search window in the related art. On the one hand, the optical flow calculation of the top image is no longer limited by the size of the search window, and can cover motion scenes of various amplitudes. For example, a large-size search window is no longer required for large motion scenes, and there will be no situation where the amplitude difference of the motion vector at different layers is too large. On the other hand, since the randomly initialized optical flow is random, it can always hit some of the most matching image blocks. Therefore, further fine-tuning of the subsequent layers can eventually calculate the optical flow with higher accuracy.

[0082] In the embodiment of the present application, the random offset is usually in the form of (m, n). Taking a pixel point as an example, if the position indicated by the random offset is on the right side of the pixel point, the value of m is a positive value; if the position indicated by the random offset is on the left side of the pixel point, the value of m is a negative value; if the position indicated by the random offset is above the pixel point, the value of n is a positive value; if the position indicated by the random offset is below the pixel point, the value of n is a negative value.

[0083] For example, Figure 5As shown, the top image 51 of the first image pyramid of the target frame and the top image 52 of the second image pyramid of the reference frame are taken as an example. Taking a pixel point 511 in the image 51 as an example, a set of offsets is randomly initialized for the pixel point 511. For example, the set of offsets includes 9 random offsets, which are recorded as [(5,4), (3,2), (-3,3), (-1,1), (-3,0), (4,0), (-3,-3), (0,-3), (7,-3)], where the random offset (5,4) of the pixel point 511 indicates the pixel position 521 in the image 52, the random offset (3,2) of the pixel point 511 indicates the pixel position 522 in the image 52, and the random offset (5,4) of the pixel point 511 indicates the pixel position 521 in the image 52. The random offset (-3, 3) of pixel 511 indicates pixel position 523 in image 52, the random offset (-1, 1) of pixel 511 indicates pixel position 524 in image 52, the random offset (-3, 0) of pixel 511 indicates pixel position 525 in image 52, the random offset (4, 0) of pixel 511 indicates pixel position 526 in image 52, the random offset (-3, -3) of pixel 511 indicates pixel position 527 in image 52, the random offset (0, -3) of pixel 511 indicates pixel position 528 in image 52, and the random offset (7, -3) of pixel 511 indicates pixel position 529 in image 52. Similarly, the random offsets of other pixels in image 51 are determined in a similar manner to that of pixel 511, but the values ​​are random and will not be repeated here.

[0084] In the embodiment of the present application, when the random offset is in the form of (m, n), the values ​​of m and n can satisfy [-8, 8], that is, m∈[-8, 8], n∈[-8, 8]. For example, if the number of layers of the image pyramid is 4, the motion range that can be covered is [-8*2 3 ,8*2 3 ], which can cover most scenarios.

[0085] In the embodiment of the present application, considering that the motion vector of each pixel position in the top image of the image pyramid should be uniform and should not be dominated by small motion or large motion, when configuring the initial optical flow of the N-th layer image of the first image pyramid, the initial optical flow of the N-th layer image of the first image pyramid obeys a uniform distribution, that is, all random offsets in the initial optical flow are uniformly distributed.

[0086] For example, the random offset is in the form of (m, n), m∈[-8, 8], n∈[-8, 8], and the probability of the 17 values ​​[-8, 8] appearing in the initial optical flow of the Nth layer image of the first image pyramid is the same, that is, uniformly distributed.

[0087] In step 403, block matching is performed on the Nth layer image of the first image pyramid and the Nth layer image of the second image pyramid based on the initial optical flow of the Nth layer image of the first image pyramid. A fine-tuned optical flow of the Nth layer image of the first image pyramid is determined based on the block matching result. The fine-tuned optical flow of the Nth layer image of the first image pyramid is upsampled, and the upsampling result is determined as the initial optical flow of the N-1th layer image of the first image pyramid. The fine-tuned optical flow of the Nth layer image of the first image pyramid includes the fine-tuned optical flow value of each pixel in the Nth layer image of the first image pyramid.

[0088] In the embodiment of the present application, based on the initial optical flow of the Nth layer image of the first image pyramid, in the process of performing block matching processing on the Nth layer image of the first image pyramid and the Nth layer image of the second image pyramid, the search window is no longer used, but the above-mentioned initial optical flow is used.

[0089] In some embodiments, as Figure 6A As shown, the above step 403 may include the following steps: step 4031, step 4032, step 4033, step 4034 and step 4035;

[0090] In step 4031, for each pixel point P in the Nth layer image of the first image pyramid, ij , read P from the initial optical flow of the Nth layer image of the first image pyramid ij The corresponding set of offsets R ij ; Wherein, 1≤i≤L1, 1≤j≤W1, L1 and W1 are the length and width of the Nth layer image of the first image pyramid respectively.

[0091] Exemplarily, N=4, the first image pyramid and the second image pyramid are both 4-layer pyramids. From top to bottom, the pyramids are the fourth layer image, the third layer image, the second layer image and the first layer image, where the fourth layer is the top layer of the pyramid and the first layer is the bottom layer of the pyramid.

[0092] Starting from the top layer (i.e. the fourth layer) of the image pyramid, for example, Figure 5As shown, the top image 51 of the first image pyramid of the target frame and the top image 52 of the second image pyramid of the reference frame, taking a pixel point 511 in the image 51 as an example, the offset of the pixel point 511 is obtained from the initial optical flow of the top image 51 of the first image pyramid, for example, the offset includes 9 random offsets, which are recorded as [(5,4), (3,2), (-3,3), (-1,1), (-3,0), (4,0), (-3,-3), (0,-3), (7,-3)], where the random offset (5,4) of the pixel point 511 indicates the pixel position 521 in the image 52, and the random offset (3,2) of the pixel point 511 indicates the pixel position 521 in the image 52. 22, the random offset of pixel 511 (-3, 3) indicates pixel position 523 in image 52, the random offset of pixel 511 (-1, 1) indicates pixel position 524 in image 52, the random offset of pixel 511 (-3, 0) indicates pixel position 525 in image 52, the random offset of pixel 511 (4, 0) indicates pixel position 526 in image 52, the random offset of pixel 511 (-3, -3) indicates pixel position 527 in image 52, the random offset of pixel 511 (0, -3) indicates pixel position 528 in image 52, and the random offset of pixel 511 (7, -3) indicates pixel position 529 in image 52.

[0093] In step 4032, the image block B in the Nth layer image of the first image pyramid is determined. ij , where B ij P ij The image block with the center pixel as the center pixel.

[0094] For example, Figure 5 For example, a pixel 511 of the image 51 in FIG. Figure 6B As shown, an image block 512 with pixel 511 as the center pixel in the image 51 is determined.

[0095] It should be noted that the size of the image block in the block matching process can be set according to the actual situation. Figure 6B The following description only takes an image block of 3×3 size as an example.

[0096] In step 4033, the candidate image blocks in the Nth layer image of the second image pyramid are determined, wherein the candidate image blocks are R ij Each random offset in the image block is the center pixel.

[0097] For example, Figure 5 For example, a pixel 511 of the image 51 in FIG. Figure 6BAs shown, an image block 512 with pixel 511 as the center pixel in the image 51 is determined. Figure 6C As shown, in image 52, an alternative image block 531 with pixel 521 as the center pixel, an alternative image block 532 with pixel 522 as the center pixel, an alternative image block 533 with pixel 523 as the center pixel, an alternative image block 534 with pixel 524 as the center pixel, an alternative image block 535 with pixel 525 as the center pixel, an alternative image block 536 with pixel 526 as the center pixel, an alternative image block 537 with pixel 527 as the center pixel, an alternative image block 538 with pixel 528 as the center pixel, and an alternative image block 539 with pixel 529 as the center pixel are determined.

[0098] In step 4034, calculate B ij The similarity between each candidate image block and the candidate image block with the highest similarity is determined as B ij matching image patches.

[0099] For example, Figure 5 Take the pixel 511 of the image 51 as an example, Figure 6C As shown, the similarities between image block 512 and candidate image block 531, the similarities between image block 512 and candidate image block 532, the similarities between image block 512 and candidate image block 533, the similarities between image block 512 and candidate image block 534, the similarities between image block 512 and candidate image block 535, the similarities between image block 512 and candidate image block 536, the similarities between image block 512 and candidate image block 537, the similarities between image block 512 and candidate image block 538, and the similarities between image block 512 and candidate image block 539 are calculated. Based on the nine calculated similarities, the image block with the highest similarity is selected. For example, the image block with the highest similarity is candidate image block 536.

[0100] In step 4035, calculate B ij With B ij The offset between the matching image blocks is calculated and the calculated offset is determined as P ij Fine-tuned optical flow value; among them, all P ij The refined optical flow values ​​constitute the refined optical flow of the Nth layer image of the first image pyramid.

[0101] For example, since candidate image block 536 has the highest similarity to image block 512, the offset between candidate image block 536 and image block 512 is calculated, and the calculated offset is determined as the refined optical flow value of pixel 511. Similarly, refined optical flow values ​​for other pixels in image 51 can be calculated.

[0102] It can be seen that in the embodiment of the present application, random initialization is used for the layer with the smallest resolution of the image pyramid. The motivation for doing so is that random initialization is not limited to the size of the search window and can better cover scenes with various motion amplitudes. This initialization can always hit some of the most matching blocks, and further fine-tuning in the next layer can eventually better estimate the final motion vector. Due to the application of random initialization, there is no need to use a relatively large search box for scenes with large motion, and there will be no situation where the amplitude of the motion vector at different layers is too different.

[0103] In step 404 , based on the initial optical flow of the N-1th layer image of the first image pyramid, the refined optical flow of each layer image below the Nth layer image of the first image pyramid is calculated layer by layer until the refined optical flow of the first layer image of the first image pyramid is obtained.

[0104] In the embodiment of the present application, when N=2, the image at level N-1 of the first image pyramid is the first image level. When N>2, it is necessary to first calculate the fine-tuned optical flow of the image at level N-1 based on the initial optical flow of the image at level N-1 of the first image pyramid, and then calculate the fine-tuned optical flow of the image at level N-2, until the fine-tuned optical flow of the image at level one of the first image pyramid is obtained.

[0105] In some embodiments, as Figure 7A As shown, the above step 404 may include the following steps: step 4041, step 4042 and step 4043;

[0106] In step 4041, based on the search window, block matching processing is performed on the N-1th layer image of the first image pyramid and the N-1th layer image of the second image pyramid. According to the block matching processing result, the initial optical flow of the N-1th layer image of the first image pyramid is fine-tuned to obtain the fine-tuned optical flow of the N-1th layer image of the first image pyramid.

[0107] In the embodiment of the present application, when N=2, it is only necessary to perform the above step 4041 to obtain the fine-tuned optical flow of the first layer image of the first image pyramid.

[0108] In step 4042 , when N>2, the refined optical flow of the N-1th layer image of the first image pyramid is upsampled, and the upsampling result is determined as the initial optical flow of the N-2th layer image of the first image pyramid.

[0109] In the embodiment of the present application, since the resolution of the N-1th layer image is lower than the resolution of the N-2th layer image, it is necessary to upsample the fine-tuned optical flow of the N-1th layer image of the first image pyramid to the same size as the N-2th layer image. At this time, the upsampling result is determined as the initial optical flow of the N-2th layer image of the first image pyramid.

[0110] In step 4043 , the same processing operations as those for the N-1st layer image of the first image pyramid are sequentially performed on the N-2th layer image to the first layer image of the first image pyramid until a refined optical flow of the first layer image of the first image pyramid is obtained.

[0111] It can be seen that in the embodiment of the present application, the optical flow of other layers other than the top layer of the first image pyramid can be calculated based on the search window. Since the block matching technology of the search window is relatively mature, and the optical flow result of the top layer always has a random offset within a certain range that is lucky enough to hit the most matching image block, the accurate optical flow value of the target frame can be calculated in the end.

[0112] For layers other than the top layer of the image pyramid, considering that in related technologies, when performing dense pixel-by-pixel searches within the search window, the optical flow values ​​of neighborhood positions are usually smooth, the offset difference relative to the center pixel is very small, and after several layers of pyramid iterations, the optical flow has gradually approached the actual motion. Therefore, updating the optical flow of the center pixel with the neighborhood optical flow is a relatively redundant calculation.

[0113] To address the above issues, in an embodiment of the present application, during block matching processing of the N-1th layer image of the first image pyramid and the N-1th layer image of the second image pyramid, a sparse search is performed within the search window to update the optical flow value of the center pixel with an offset at a farther position. At the same time, since the optical flow of the top image of the image pyramid is randomly initialized, this sparse search can effectively find a more appropriate offset to update the current optical flow value, thereby achieving fine-tuning of the optical flow.

[0114] In some embodiments, as Figure 7B As shown, the above step 4041 may include the following steps: step 40411, step 40412, step 40413, step 40414, step 40415, step 40416 and step 40417;

[0115] In step 40411, for each pixel Q in the N-1th layer image of the first image pyramid st , read Q from the initial optical flow of the N-1th layer image of the first image pyramid stThe initial optical flow value of ; where 1≤s≤L2, 1≤t≤W2, L2, W2 are the length and width values ​​of the N-1th layer image of the first image pyramid respectively.

[0116] For example, N=4, and both the first and second image pyramids have four layers. From top to bottom, the pyramids are, in order, the fourth layer image, the third layer image, the second layer image, and the first layer image. The fourth layer is the top layer of the pyramid, and the first layer is the bottom layer of the pyramid. The N-1th layer image of the first image pyramid is the third layer image of the first image pyramid.

[0117] Exemplarily, the initial optical flow of the third layer image of the first image pyramid is fine-tuned according to the block matching method to obtain the fine-tuned optical flow of the third layer image of the first image pyramid. Figure 7C As shown, the third layer image 61 of the image pyramid of the target frame and the third layer image 62 of the image pyramid of the reference frame, taking the optical flow value of a pixel point 611 in the fine-tuning image 61 as an example, obtain the initial optical flow value of the pixel point 611 in the initial optical flow of the image 61.

[0118] In step 40412, the image block A in the N-1th layer image of the first image pyramid is determined. st , where A st Q st The image block with the center pixel as the center pixel.

[0119] For example, Figure 7C As an example, a pixel 611 in the image 61 is Figure 7D As shown, an image block 612 with pixel 611 as the center pixel in the image 61 is determined.

[0120] In step 40413, the search window Z in the N-1th layer image of the second image pyramid is determined. st , where Z st A st The corresponding search window.

[0121] For example, Figure 7C As an example, a pixel 611 in the image 61 is Figure 7E As shown, a search window 621 corresponding to the image block 612 in the image 62 is determined.

[0122] It should be noted that, since a sparse search method is adopted when performing block matching, the size of the search window 621 can be set to be slightly larger than the size of the search window in the related art.

[0123] In step 40414, according to Q st The initial optical flow value C1 determines Zst The image block G in st , where G st A st The corresponding image block.

[0124] For example, Figure 7C As an example, a pixel 611 in the image 61 is Figure 7F As shown, an image block 622 corresponding to the initial optical flow value of the pixel point 611 in the image 62 is determined.

[0125] In step 40415, configure A st In Z st The search step size H in , where H is an integer greater than 1.

[0126] In the embodiment of the present application, the value of the search step size H is set to an integer greater than 1 to achieve a sparse search within the search window. For example, H=2.

[0127] In step 40416, according to H, in Z st Search for matching image block Y st , where Y st For A st The image blocks with the highest similarity.

[0128] For example, H=2, such as Figure 7G As shown, the image block 612 moves up, down, left, and right in the search window 621 with a step size of 2 to perform block search, and searches for the matching image block with the highest similarity to the image block 612 in the search window 621. For example, the matching image block with the highest similarity is image block 624.

[0129] In step 40417, determine A st With Y st Is the similarity X1 between them greater than A? st With G st The similarity between them is X2; if X1>X2, the initial optical flow value C2 in the initial optical flow of the N-1th layer of the first image pyramid is determined, and C2 is determined as Q st Fine-tuning optical flow value, where C2 is Y st The initial optical flow value corresponding to the central pixel of st The initial optical flow value C1 is determined as Q st Fine-tuning optical flow value; among them, all Q st The refined optical flow values ​​constitute the refined optical flow of the N-1th layer image of the first image pyramid.

[0130] For example, if the similarity between image blocks 612 and 624 is greater than the similarity between image blocks 612 and 622, the initial optical flow value of pixel 613 in image 61, corresponding to central pixel 623 of image block 624, is more suitable than the initial optical flow value of pixel 611 for serving as the refined optical flow value of pixel 611. In this case, the initial optical flow value of pixel 613 is determined as the refined optical flow value of pixel 611. If the similarity between image blocks 612 and 624 is less than the similarity between image blocks 612 and 622, the initial optical flow value of pixel 611 is the most suitable optical flow value. The initial optical flow value of pixel 611 is determined as the refined optical flow value of pixel 611. Similarly, the refined optical flow values ​​of other pixels in the third layer of the image pyramid of the target frame can be calculated, i.e., the refined optical flow of the third layer. After calculating the fine-tuned optical flow of the third layer image of the image pyramid of the target frame, the fine-tuned optical flow of the third layer image is upsampled, and the upsampling result is passed to the second layer of the image pyramid of the target frame to serve as the initial optical flow of the second layer image.

[0131] In this embodiment of the present application, the same calculation process as described above for the third layer is performed on the second layer of the target frame's image pyramid to obtain the refined optical flow of the second layer of the target frame's image pyramid. The refined optical flow of the second layer of the target frame's image pyramid is transferred to the first layer of the target frame's image pyramid to serve as the initial optical flow of the first layer of the target frame's image pyramid. The same calculation process as described above for the third layer is performed on the first layer of the target frame's image pyramid to obtain the refined optical flow of the first layer of the target frame's image pyramid.

[0132] It can be seen that in the embodiment of the present application, for layers other than the top layer of the image pyramid, block matching can be performed in a larger search space through sparse search, thereby reducing the amount of calculation while ensuring the accuracy of block matching.

[0133] In step 405 , the refined optical flow of the first layer image of the first image pyramid is denoised to obtain the target optical flow of the target frame.

[0134] Considering that the optical flow obtained based on the block matching method has large noise, in order to improve the quality of optical flow estimation, in an embodiment of the present application, the fine-tuned optical flow of the first layer image of the first image pyramid is denoised to obtain the target optical flow of the target frame.

[0135] Considering that large convolution kernel filtering has the following advantages: 1) Larger receptive field: Large convolution kernels can cover a larger input area, thereby capturing more global information in one convolution operation. 2) Reduced network depth: Large convolution kernels can obtain more global information in one operation, reducing the number of layers required, thereby reducing network depth, helping to improve training stability and reduce the problem of gradient vanishing. 3) Smooth output: Large convolution kernels contain more adjacent pixel information, and have a more significant smoothing effect on feature maps, which helps to reduce sensitivity to local noise and enhance the capture of global patterns. Therefore, in an embodiment of the present application, a denoising model based on a large convolution kernel can be used to denoise the fine-tuned optical flow of the first layer image of the first image pyramid to obtain the target optical flow of the target frame.

[0136] In some embodiments, to improve denoising efficiency, step 405 may include the following steps:

[0137] Step 4051;

[0138] In step 4051, the fine-tuned optical flow of the first layer image of the first image pyramid and the first layer image of the second image pyramid are input into the re-parameterized target denoising model for denoising, and the target optical flow of the target frame is output; wherein the target denoising model includes a type of convolution kernel and the size of the convolution kernel is greater than a preset size value.

[0139] In the embodiment of the present application, a neural network is used to learn the filter kernel. In order to enable the optical flow to be applied in real time in videos or games, the efficiency requirements will be relatively high. Since a large convolution kernel filter is used, a network model with too many layers cannot be used. In order to ensure a small number of layers and a good denoising effect, a re-parameterization technology is used to design the network. Specifically, the training model is decoupled from the inference model. Since convolutions are all linear operations, multiple parallel convolution kernels can be merged into a single large-kernel convolution. Since the number of layers of the network is relatively small, after the training is completed, the filtering operation can be performed directly by using a lookup table, which can further improve the efficiency of the optical flow calculation. Among them, the basic principle of re-parameterization calculation is that convolution is a linear operation, so the addition of multiple linear operations is still a linear operation.

[0140] In some embodiments, as Figure 8A As shown, the generation process of the above target denoising model can include the following steps:

[0141] Step 801, step 802, step 803 and step 804;

[0142] In step 801, a training set is obtained; wherein the training set includes multiple groups of sample data, each group of sample data includes: a first sample image, a second sample image, a noisy optical flow of the first sample image and a true optical flow of the first sample image, and the second sample image is a reference image of the first sample image.

[0143] In the embodiment of the present application, in order to ensure the training effect of the model, the training set may include a large amount of sample data.

[0144] In the embodiment of the present application, the real optical flow of the first sample image can be manually annotated or derived from a dataset publicly available on the Internet.

[0145] In step 802, a backbone network is constructed; wherein the backbone network includes at least two groups of convolution kernels, and each group of convolution kernels includes multiple convolution kernels of different sizes connected in parallel.

[0146] For example, Figure 8B As shown in the figure, the backbone network includes two groups of convolution kernels and two activation layers. Each group of convolution kernels includes a 7×7 convolution kernel, a 5×5 convolution kernel, a 3×3 convolution kernel and a 1×1 convolution kernel.

[0147] In an embodiment of the present application, the backbone network includes at least two groups of convolution kernels, each group of convolution kernels includes multiple parallel-connected convolution kernels of different sizes, so that more comprehensive prior information of the training data can be learned during the model training process.

[0148] In step 803, the noisy optical flows of the second sample image and the first sample image in the sample data are input into the backbone network for denoising, and the predicted image and the denoised optical flow of the predicted image are output; the loss value is calculated based on the first sample image, the true optical flow of the first sample image, the predicted image and the denoised optical flow of the predicted image, and the network parameters of the backbone network are updated according to the loss value; the above training process is repeated using different groups of sample data until the model converges to obtain an initial denoising model.

[0149] For example, the loss function during model training can be expressed as follows:

[0150] loss=α*L1(flow, flowgt)+β*L2(I2, warp(I1, flow)) (1)

[0151] Where loss represents the loss function, α and β are weight coefficients, for example, α = 0.5, β = 0.5, flow represents the denoised optical flow of the predicted image, flowgt represents the true optical flow of the first sample image, I1 represents the predicted image, I2 represents the first sample image, L1 represents the absolute difference function, and L2 represents the square difference function.

[0152] In step 804, the initial denoising model is re-parameterized to generate a target denoising model.

[0153] For example, Figure 8B As shown in , all convolution kernels in each group of convolution kernels are jump-connected to the 7×7 convolution kernel. Since a smaller convolution kernel can be expanded into a larger convolution kernel by expanding the edge with 0, the sum of multiple convolution kernels of different sizes can be modeled as a large convolution kernel, as shown in Figure 8C As shown. Based on this principle, the model in the training phase and the model in the inference phase can be decoupled. In the training phase, more parallel branches are used. Figure 8B In the training phase, more parallel branches can be used to learn more information; in the inference phase, Figure 8C The network structure after the merger has no parallel branches and is more memory-friendly, thus enabling more efficient inference. Since there are only two layers of 7×7 convolution kernels, a lookup table can be used to further improve the efficiency of the convolution operation.

[0154] It can be seen that in the embodiments of the present application, through the re-parameterized design, the training and reasoning of the model can be decoupled, so that both the training effect and the reasoning speed can be improved.

[0155] As can be seen from the above embodiment, in this embodiment, a first image pyramid of a target frame and a second image pyramid of a reference frame are constructed; wherein the number of layers of the first image pyramid and the number of layers of the second image pyramid are both N, and N is an integer greater than 1; an initial optical flow of the Nth layer image of the first image pyramid is configured; wherein the initial optical flow of the Nth layer image of the first image pyramid includes: a set of offsets for each pixel point in the Nth layer image of the first image pyramid, each set of offsets includes: at least two different random offsets, each random offset is used to indicate a pixel position in the Nth layer image of the second image pyramid; based on the initial optical flow of the Nth layer image of the first image pyramid, the Nth layer image of the first image pyramid and the Nth layer image of the second image pyramid are subjected to optical flow. Performing block matching, determining a fine-tuned optical flow of an N-th layer image of a first image pyramid based on a result of the block matching, upsampling the fine-tuned optical flow of the N-th layer image of the first image pyramid, and determining the upsampling result as the initial optical flow of an N-1-th layer image of the first image pyramid; wherein the fine-tuned optical flow of the N-th layer image of the first image pyramid includes: a fine-tuned optical flow value of each pixel in the N-th layer image of the first image pyramid; calculating the fine-tuned optical flow of each layer image below the N-th layer image of the first image pyramid layer by layer based on the initial optical flow of the N-1-th layer image of the first image pyramid until the fine-tuned optical flow of the first layer image of the first image pyramid is obtained; and performing denoising on the fine-tuned optical flow of the first layer image of the first image pyramid to obtain a target optical flow of a target frame.

[0156] It can be seen that compared with the related art in which the optical flow of the top layer of the image pyramid is initialized to all zeros, the initial optical flow based on all zeros cannot provide any information for the top layer, and can only find the most matching block in the neighborhood of the search window based on the search window. The small search box cannot cover large motion scenes. In the embodiment of the present application, the optical flow of the top layer of the image pyramid is randomly initialized. Since the randomly initialized optical flow contains a set of random offsets for each pixel point of the top layer image, and there is always a random offset within a certain range that is lucky enough to hit the most matching image block, further fine-tuning in the next layer can better estimate the final optical flow. Therefore, the block matching processing of the top layer image is no longer limited to the size of the search window, and the accuracy of the optical flow results calculated for scenes with large motion can be improved.

[0157] In summary, the beneficial effects of the embodiments of the present application are manifested in the following aspects: 1) Better optical flow effect: By adopting random initialization, scenes of various motion amplitudes can be better covered, and through sparse search, block matching can be performed in a larger search space, which can greatly improve the accuracy of block matching. In the denoising stage, by adding more parallel branches in the training stage, it can help the network learn more information. 2) Faster running speed: The use of sparse search can greatly improve the accuracy of block matching without increasing the search space. In the denoising stage, by merging the parallel branches of the network, the speed of inference can be significantly improved, and the convolution operation can be performed by using a lookup table, which can further improve the operation speed. 3) Easier to parallelize programming: Without adopting a solution similar to spatial iteration, it is very convenient to parallelize the program, so there is more room to further improve the optical flow effect. In general, the embodiments of the present application can achieve a better balance between optical flow effect and optical flow operation efficiency, and can enable optical flow to be applied in real time in videos and games, improving the picture effect of videos or games.

[0158] In addition, the optical flow calculation method provided in the embodiments of the present application can also be applied to the following scenarios: 1) Application in more scenarios: Since image alignment is an important step in the multi-frame image alignment task, a similar method can be used to calculate optical flow in multi-frame high dynamic range imaging, multi-frame super-resolution, and multi-frame denoising, and then the optical flow can be used to align the images. 2) Binocular disparity calculation: Since the calculation principle of binocular disparity is basically similar to that of optical flow, the only difference is that disparity calculation is a one-dimensional matching, while optical flow calculation is a two-dimensional matching. Therefore, this optical flow solution can be directly applied to the binocular disparity estimation task.

[0159] The optical flow calculation method provided in the embodiment of the present application can be executed by an optical flow calculation device. In the embodiment of the present application, the optical flow calculation device provided in the embodiment of the present application is described by taking the optical flow calculation device executing the optical flow calculation method as an example.

[0160] Figure 9 This is a structural block diagram of an optical flow calculation device provided in an embodiment of the present application. Figure 9 As shown, the optical flow calculation device 900 may include: a construction module 901, a configuration module 902, a first processing module 903, a second processing module 904 and a denoising module 905;

[0161] The construction module 901 is configured to construct a first image pyramid of a target frame and a second image pyramid of a reference frame; wherein the number of layers of the first image pyramid and the second image pyramid are both N, where N is an integer greater than 1;

[0162] The configuration module 902 is configured to configure an initial optical flow of the Nth layer image of the first image pyramid; wherein the initial optical flow of the Nth layer image of the first image pyramid includes: a set of offsets for each pixel in the Nth layer image of the first image pyramid, each set of offsets including: at least two different random offsets, each of the random offsets being used to indicate a pixel position in the Nth layer image of the second image pyramid;

[0163] The first processing module 903 is configured to perform block matching on the N-th layer image of the first image pyramid and the N-th layer image of the second image pyramid based on the initial optical flow of the N-th layer image of the first image pyramid, determine a refined optical flow of the N-th layer image of the first image pyramid based on the block matching result, upsample the refined optical flow of the N-th layer image of the first image pyramid, and determine the upsampling result as the initial optical flow of the N-1-th layer image of the first image pyramid; wherein the refined optical flow of the N-th layer image of the first image pyramid includes a refined optical flow value of each pixel in the N-th layer image of the first image pyramid;

[0164] The second processing module 904 is configured to calculate, based on the initial optical flow of the image at the N-1th layer of the first image pyramid, a refined optical flow of each layer below the Nth layer of the first image pyramid, layer by layer, until a refined optical flow of the first layer of the first image pyramid is obtained;

[0165] The denoising module 905 is configured to perform denoising on the fine-tuned optical flow of the first layer image of the first image pyramid to obtain the target optical flow of the target frame.

[0166] As can be seen from the above embodiment, in this embodiment, a first image pyramid of a target frame and a second image pyramid of a reference frame are constructed; wherein the number of layers of the first image pyramid and the number of layers of the second image pyramid are both N, and N is an integer greater than 1; an initial optical flow of the Nth layer image of the first image pyramid is configured; wherein the initial optical flow of the Nth layer image of the first image pyramid includes: a set of offsets for each pixel point in the Nth layer image of the first image pyramid, each set of offsets includes: at least two different random offsets, each random offset is used to indicate a pixel position in the Nth layer image of the second image pyramid; based on the initial optical flow of the Nth layer image of the first image pyramid, the Nth layer image of the first image pyramid and the Nth layer image of the second image pyramid are subjected to optical flow. Performing block matching, determining a fine-tuned optical flow of an N-th layer image of a first image pyramid based on a result of the block matching, upsampling the fine-tuned optical flow of the N-th layer image of the first image pyramid, and determining the upsampling result as the initial optical flow of an N-1-th layer image of the first image pyramid; wherein the fine-tuned optical flow of the N-th layer image of the first image pyramid includes: a fine-tuned optical flow value of each pixel in the N-th layer image of the first image pyramid; calculating the fine-tuned optical flow of each layer image below the N-th layer image of the first image pyramid layer by layer based on the initial optical flow of the N-1-th layer image of the first image pyramid until the fine-tuned optical flow of the first layer image of the first image pyramid is obtained; and performing denoising on the fine-tuned optical flow of the first layer image of the first image pyramid to obtain a target optical flow of a target frame.

[0167] It can be seen that compared with the related art in which the optical flow of the top layer of the image pyramid is initialized to all zeros, the initial optical flow based on all zeros cannot provide any information for the top layer, and can only find the most matching block in the neighborhood of the search window based on the search window. The small search box cannot cover large motion scenes. In the embodiment of the present application, the optical flow of the top layer of the image pyramid is randomly initialized. Since the randomly initialized optical flow contains a set of random offsets for each pixel point of the top layer image, and there is always a random offset within a certain range that is lucky enough to hit the most matching image block, further fine-tuning in the next layer can better estimate the final optical flow. Therefore, the block matching processing of the top layer image is no longer limited to the size of the search window, and the accuracy of the optical flow results calculated for scenes with large motion can be improved.

[0168] Optionally, as an embodiment, the first processing module 903 may be specifically configured to process each pixel point P in the Nth layer image of the first image pyramid. ij , read the P from the initial optical flow of the Nth layer image of the first image pyramid ij The corresponding set of offsets R ijWherein, 1≤i≤L1, 1≤j≤W1, L1 and W1 are respectively the length and width of the Nth layer image of the first image pyramid; Determine the image block B in the Nth layer image of the first image pyramid ij , wherein the B ij For the P ij An image block with the R as the center pixel point; determining a candidate image block in the Nth layer image of the second image pyramid, wherein the candidate image block is an image block with the R ij Each random offset in the image block is the center pixel point; calculate the B ij The similarity between each candidate image block and the candidate image block with the highest similarity is determined as the B ij Matching image blocks; Calculate the B ij With the B ij The offset between the matching image blocks, and the calculated offset is determined as the P ij Fine-tuning optical flow value; wherein all the P ij The refined optical flow values ​​constitute the refined optical flow of the Nth layer image of the first image pyramid.

[0169] Optionally, as an embodiment, the initial optical flow of the Nth layer image of the first image pyramid may obey a uniform distribution.

[0170] Optionally, as an embodiment, the second processing module 904 may be specifically configured to perform block matching processing on the N-1th layer image of the first image pyramid and the N-1th layer image of the second image pyramid based on a search window, and fine-tune the initial optical flow of the N-1th layer image of the first image pyramid based on the block matching processing result to obtain the fine-tuned optical flow of the N-1th layer image of the first image pyramid; when N>2, upsample the fine-tuned optical flow of the N-1th layer image of the first image pyramid, and determine the upsampling result as the initial optical flow of the N-2th layer image of the first image pyramid; and sequentially perform the same processing operations as those for the N-1th layer image of the first image pyramid on the N-2th layer image to the first layer image of the first image pyramid until the fine-tuned optical flow of the first layer image of the first image pyramid is obtained.

[0171] Optionally, as an embodiment, the second processing module 904 may be specifically configured to process each pixel Q in the N-1th layer image of the first image pyramid. st , read the Q from the initial optical flow of the N-1th layer image of the first image pyramid stThe initial optical flow value C1; wherein 1≤s≤L2, 1≤t≤W2, L2, W2 are the length value and width value of the N-1th layer image of the first image pyramid respectively; determine the image block A in the N-1th layer image of the first image pyramid st , wherein the A st For the Q st The image block is the center pixel; determine the search window Z in the N-1 layer image of the second image pyramid st , wherein the Z st For the A st The corresponding search window; according to the Q st The initial optical flow value C1 determines the Z st The image block G in st , wherein the G st For the A st The corresponding image block; configure the A st In the Z st The search step length H in Z is H, where H is an integer greater than 1; according to the H, st Search for matching image block Y st , wherein the Y st For the A st The image block with the highest similarity between them; determine the A st With the Y st Is the similarity X1 between them greater than the A st With the G st If X1>X2, then determine the initial optical flow value C2 in the initial optical flow of the N-1th layer of the first image pyramid, and determine C2 as the Q st The fine-tuned optical flow value of st The initial optical flow value corresponding to the central pixel of st The initial optical flow value C1 is determined as the Q st Fine-tuning optical flow value; wherein all the Q st The refined optical flow values ​​constitute the refined optical flow of the N-1th layer image of the first image pyramid.

[0172] Optionally, as an embodiment, the denoising module 905 can be specifically used to input the fine-tuned optical flow of the first layer image of the first image pyramid and the first layer image of the second image pyramid into a re-parameterized target denoising model for denoising, and output the target optical flow of the target frame; wherein, the target denoising model includes a type of convolution kernel and the size of the convolution kernel is greater than a preset size value.

[0173] Optionally, as an embodiment, the optical flow calculation device 900 may further include: a training module;

[0174] The training module is used to obtain a training set; wherein the training set includes multiple groups of sample data, each group of sample data includes: a first sample image, a second sample image, a noisy optical flow of the first sample image and a real optical flow of the first sample image, and the second sample image is a reference image of the first sample image; construct a backbone network; wherein the backbone network includes at least two groups of convolution kernels, and each group of convolution kernels includes multiple parallel-connected convolution kernels of different sizes; the noisy optical flows of the second sample image and the first sample image in the sample data are input into the backbone network for denoising, and a predicted image and a denoised optical flow of the predicted image are output; a loss value is calculated based on the first sample image, the real optical flow of the first sample image, the predicted image and the denoised optical flow of the predicted image, and the network parameters of the backbone network are updated based on the loss value; the above training process is repeated using different groups of sample data until the model converges to obtain an initial denoising model; the initial denoising model is reparameterized to generate the target denoising model.

[0175] The optical flow calculation device in the embodiment of the present application can be an electronic device or a component in an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other device other than a terminal. For example, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, a mobile Internet device (MID), an augmented reality (AR) / virtual reality (VR) device, a robot, a wearable device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA), etc. It can also be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), a teller machine or a self-service machine, etc., and the embodiment of the present application does not specifically limit it.

[0176] The optical flow calculation device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0177] The optical flow calculation device provided in the embodiment of the present application can achieve Figure 4 、 Figure 6A 、 Figure 7A 、 Figure 7B and Figure 8A To avoid repetition, the various processes implemented in any of the method embodiments are not described here.

[0178] Alternatively, as Figure 10 As shown, an embodiment of the present application further provides an electronic device 1000, including a processor 1001 and a memory 1002, wherein the memory 1002 stores a program or instruction that can be run on the processor 1001, and when the program or instruction is executed by the processor 1001, the various steps of the above-mentioned optical flow calculation method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0179] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0180] Figure 11 It is a schematic diagram of the hardware structure of an electronic device provided in each embodiment of the present application.

[0181] The electronic device 1100 includes but is not limited to: a radio frequency unit 1101, a network module 1102, an audio output unit 1103, an input unit 1104, a sensor 1105, a display unit 1106, a user input unit 1107, an interface unit 1108, a memory 1109 and a processor 1110.

[0182] Those skilled in the art will understand that the electronic device 1100 may also include a power source (such as a battery) to power each component, and the power source may be logically connected to the processor 1110 through a power management system, thereby implementing functions such as charging, discharging, and power consumption management through the power management system. Figure 11 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be repeated here.

[0183] In some embodiments, the processor 1110 is configured to construct a first image pyramid for a target frame and a second image pyramid for a reference frame, wherein the number of layers of the first image pyramid and the number of layers of the second image pyramid are both N, where N is an integer greater than 1; configure an initial optical flow for an Nth layer image of the first image pyramid; wherein the initial optical flow for the Nth layer image of the first image pyramid comprises: a set of offsets for each pixel in the Nth layer image of the first image pyramid, each set of offsets comprising: at least two different random offsets, each of the random offsets being used to indicate a pixel position in the Nth layer image of the second image pyramid; and perform optical flow analysis on the Nth layer image of the first image pyramid and the Nth layer image of the second image pyramid based on the initial optical flow of the Nth layer image of the first image pyramid. The method further comprises performing block matching processing on the image, determining a fine-tuned optical flow of an N-th layer image of the first image pyramid based on the block matching processing result, upsampling the fine-tuned optical flow of the N-th layer image of the first image pyramid, and determining the upsampling result as the initial optical flow of an N-1-th layer image of the first image pyramid; wherein the fine-tuned optical flow of the N-th layer image of the first image pyramid includes: a fine-tuned optical flow value of each pixel in the N-th layer image of the first image pyramid; calculating the fine-tuned optical flow of each layer image below the N-th layer image of the first image pyramid layer by layer based on the initial optical flow of the N-1-th layer image of the first image pyramid, until the fine-tuned optical flow of the first layer image of the first image pyramid is obtained; and performing denoising processing on the fine-tuned optical flow of the first layer image of the first image pyramid to obtain a target optical flow of the target frame.

[0184] It can be seen that compared with the related art in which the optical flow of the top layer of the image pyramid is initialized to all zeros, the initial optical flow based on all zeros cannot provide any information for the top layer, and can only find the most matching block in the neighborhood of the search window based on the search window. The small search box cannot cover large motion scenes. In the embodiment of the present application, the optical flow of the top layer of the image pyramid is randomly initialized. Since the randomly initialized optical flow contains a set of random offsets for each pixel point of the top layer image, and there is always a random offset within a certain range that is lucky enough to hit the most matching image block, further fine-tuning in the next layer can better estimate the final optical flow. Therefore, the block matching processing of the top layer image is no longer limited to the size of the search window, and the accuracy of the optical flow results calculated for scenes with large motion can be improved.

[0185] Optionally, as an embodiment, the processor 1110 is specifically configured to, for each pixel point P in the Nth layer image of the first image pyramid, ij , read the P from the initial optical flow of the Nth layer image of the first image pyramid ij The corresponding set of offsets R ijWherein, 1≤i≤L1, 1≤j≤W1, L1 and W1 are respectively the length and width of the Nth layer image of the first image pyramid; Determine the image block B in the Nth layer image of the first image pyramid ij , wherein the B ij For the P ij An image block with the R as the center pixel point; determining a candidate image block in the Nth layer image of the second image pyramid, wherein the candidate image block is an image block with the R ij Each random offset in the image block is the center pixel point; calculate the B ij The similarity between each candidate image block and the candidate image block with the highest similarity is determined as the B ij Matching image blocks; Calculate the B ij With the B ij The offset between the matching image blocks, and the calculated offset is determined as the P ij Fine-tuning optical flow value; wherein all the P ij The refined optical flow values ​​constitute the refined optical flow of the Nth layer image of the first image pyramid.

[0186] Optionally, as an embodiment, the initial optical flow of the Nth layer image of the first image pyramid obeys a uniform distribution.

[0187] Optionally, as an embodiment, the processor 1110 is specifically configured to perform block matching processing on the N-1th layer image of the first image pyramid and the N-1th layer image of the second image pyramid based on a search window, and fine-tune the initial optical flow of the N-1th layer image of the first image pyramid according to the block matching processing result to obtain the fine-tuned optical flow of the N-1th layer image of the first image pyramid; when N>2, upsample the fine-tuned optical flow of the N-1th layer image of the first image pyramid, and determine the upsampling result as the initial optical flow of the N-2th layer image of the first image pyramid; and sequentially perform the same processing operations as those of the N-1th layer image of the first image pyramid on the N-2th layer image to the first layer image of the first image pyramid until the fine-tuned optical flow of the first layer image of the first image pyramid is obtained.

[0188] Optionally, as an embodiment, the processor 1110 is specifically configured to, for each pixel point Q in the N-1th layer image of the first image pyramid, st , read the Q from the initial optical flow of the N-1th layer image of the first image pyramid stThe initial optical flow value C1; wherein 1≤s≤L2, 1≤t≤W2, L2, W2 are the length value and width value of the N-1th layer image of the first image pyramid respectively; determine the image block A in the N-1th layer image of the first image pyramid st , wherein the A st For the Q st The image block is the center pixel; determine the search window Z in the N-1 layer image of the second image pyramid st , wherein the Z st For the A st The corresponding search window; according to the Q st The initial optical flow value C1 determines the Z st The image block G in st , wherein the G st For the A st The corresponding image block; configure the A st In the Z st The search step length H in Z is H, where H is an integer greater than 1; according to the H, st Search for matching image block Y st , wherein the Y st For the A st The image block with the highest similarity between them; determine the A st With the Y st Is the similarity X1 between them greater than the A st With the G st If X1>X2, then determine the initial optical flow value C2 in the initial optical flow of the N-1th layer of the first image pyramid, and determine C2 as the Q st The fine-tuned optical flow value of st The initial optical flow value corresponding to the central pixel of st The initial optical flow value C1 is determined as the Q st Fine-tuning optical flow value; wherein all the Q st The refined optical flow values ​​constitute the refined optical flow of the N-1th layer image of the first image pyramid.

[0189] Optionally, as an embodiment, the processor 1110 is specifically used to input the fine-tuned optical flow of the first layer image of the first image pyramid and the first layer image of the second image pyramid into a re-parameterized target denoising model for denoising, and output the target optical flow of the target frame; wherein, the target denoising model includes a type of convolution kernel and the size of the convolution kernel is greater than a preset size value.

[0190] Optionally, as an embodiment, the processor 1110 is further used to obtain a training set; wherein the training set includes multiple groups of sample data, each group of sample data includes: a first sample image, a second sample image, a noisy optical flow of the first sample image and a real optical flow of the first sample image, and the second sample image is a reference image of the first sample image; constructing a backbone network; wherein the backbone network includes at least two groups of convolution kernels, and each group of convolution kernels includes multiple parallel-connected convolution kernels of different sizes; the noisy optical flows of the second sample image and the first sample image in the sample data are input into the backbone network for denoising, and a predicted image and a denoised optical flow of the predicted image are output; a loss value is calculated based on the first sample image, the real optical flow of the first sample image, the predicted image and the denoised optical flow of the predicted image, and the network parameters of the backbone network are updated based on the loss value; the above training process is repeated using different groups of the sample data until the model converges to obtain an initial denoising model; the initial denoising model is reparameterized to generate the target denoising model.

[0191] It should be understood that in an embodiment of the present application, the input unit 1104 may include a graphics processing unit (GPU) 11041 and a microphone 11042, and the graphics processor 11041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 1106 may include a display panel 11061, and the display panel 11061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 1107 includes a touch panel 11071 and at least one of other input devices 11072. The touch panel 11071 is also called a touch screen. The touch panel 11071 may include two parts: a touch detection device and a touch controller. Other input devices 11072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and an operating stick, which will not be repeated here.

[0192] The memory 1109 can be used to store software programs and various data. The memory 1109 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, applications or instructions required for at least one function (such as a sound playback function, an image playback function, etc.). In addition, the memory 1109 may include a volatile memory or a non-volatile memory, or the memory 1109 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct RAM bus random access memory (DRRAM). The memory 1109 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0193] Processor 1110 may include one or more processing units. Optionally, processor 1110 integrates an application processor and a modem processor. The application processor primarily handles operations related to the operating system, user interface, and application programs, while the modem processor primarily processes wireless communication signals, such as a baseband processor. It is understood that the modem processor may not be integrated into processor 1110.

[0194] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned optical flow calculation method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0195] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0196] The present application also provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned optical flow calculation method embodiment, and can achieve the same technical effect. To avoid repetition, it is not described here. It should be understood that the chip mentioned in the embodiment of the present application can also be called a system-level chip, a system chip, a chip system, or a system-on-chip chip, etc.

[0197] An embodiment of the present application also provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the various processes of the above-mentioned optical flow calculation method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0198] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element. In addition, it should be noted that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the opposite order according to the functions involved. For example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0199] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), including a number of instructions for enabling a terminal (such as a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0200] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

Claims

1. A method for calculating optical flow, characterized in that: The method comprises: Constructing a first image pyramid of the target frame and a second image pyramid of the reference frame; wherein the number of layers of the first image pyramid and the number of layers of the second image pyramid are both N, where N is an integer greater than 1; Configuring an initial optical flow of an N-th layer image of the first image pyramid; wherein the initial optical flow of the N-th layer image of the first image pyramid comprises: a set of offsets for each pixel in the N-th layer image of the first image pyramid, each set of offsets comprising: at least two different random offsets, each of the random offsets being used to indicate a pixel position in the N-th layer image of the second image pyramid; performing block matching on the Nth layer image of the first image pyramid and the Nth layer image of the second image pyramid based on the initial optical flow of the Nth layer image of the first image pyramid, determining a fine-tuned optical flow of the Nth layer image of the first image pyramid based on the block matching result, upsampling the fine-tuned optical flow of the Nth layer image of the first image pyramid, and determining the upsampling result as the initial optical flow of the N-1th layer image of the first image pyramid; wherein the fine-tuned optical flow of the Nth layer image of the first image pyramid includes: a fine-tuned optical flow value of each pixel in the Nth layer image of the first image pyramid; Based on the initial optical flow of the N-1th layer image of the first image pyramid, calculating the refined optical flow of each layer image below the Nth layer image of the first image pyramid layer by layer until the refined optical flow of the first layer image of the first image pyramid is obtained; Denoising is performed on the fine-tuned optical flow of the first layer image of the first image pyramid to obtain a target optical flow of the target frame.

2. The method according to claim 1, characterized in that The performing block matching on the N-th layer image of the first image pyramid and the N-th layer image of the second image pyramid according to the initial optical flow of the N-th layer image of the first image pyramid, and determining the refined optical flow of the N-th layer image of the first image pyramid according to the block matching result, includes: For each pixel point P in the Nth layer image of the first image pyramid ij , read the P from the initial optical flow of the Nth layer image of the first image pyramid ij The corresponding set of offsets R ij ; Wherein, 1≤i≤L1, 1≤j≤W1, L1 and W1 are the length and width of the Nth layer image of the first image pyramid respectively; Determine the image block B in the Nth layer image of the first image pyramid ij , wherein said B ij For the P ij The image block is the center pixel; Determine a candidate image block in the Nth layer image of the second image pyramid, wherein the candidate image block is a ij Each random offset in the image block is the center pixel; Calculate the B ij The similarity between each candidate image block and the candidate image block with the highest similarity is determined as the B ij Matching image patches; Calculate the B ij With the B ij The offset between the matching image blocks, and the calculated offset is determined as the P ij Fine-tuning optical flow value; wherein all the P ij The refined optical flow values ​​constitute the refined optical flow of the Nth layer image of the first image pyramid.

3. The method according to claim 1 or 2, characterized in that The initial optical flow of the Nth layer image of the first image pyramid obeys a uniform distribution.

4. The method according to claim 1, wherein The step of calculating, based on the initial optical flow of the image at the N-1th layer of the first image pyramid, the refined optical flow of each layer of the image below the Nth layer of the first image pyramid layer by layer until the refined optical flow of the first layer of the image of the first image pyramid is obtained includes: performing block matching on the N-1th layer image of the first image pyramid and the N-1th layer image of the second image pyramid based on the search window, and fine-tuning the initial optical flow of the N-1th layer image of the first image pyramid based on the block matching result to obtain a fine-tuned optical flow of the N-1th layer image of the first image pyramid; when N>2, upsampling the fine-tuned optical flow of the N-1th layer image of the first image pyramid, and determining the upsampling result as the initial optical flow of the N-2th layer image of the first image pyramid; The same processing operations as those for the N-1th layer image of the first image pyramid are sequentially performed on the N-2th layer image to the first layer image of the first image pyramid until a refined optical flow of the first layer image of the first image pyramid is obtained.

5. The method according to claim 4, characterized in that The method further includes performing block matching on the N-1th layer image of the first image pyramid and the N-1th layer image of the second image pyramid based on the search window, and fine-tuning the initial optical flow of the N-1th layer image of the first image pyramid according to the block matching result to obtain the fine-tuned optical flow of the N-1th layer image of the first image pyramid. For each pixel Q in the N-1th layer image of the first image pyramid st , read the Q from the initial optical flow of the N-1th layer image of the first image pyramid st The initial optical flow value C1; wherein, 1≤s≤L2, 1≤t≤W2, L2, W2 are the length and width values ​​of the N-1th layer image of the first image pyramid respectively; Determine the image block A in the N-1th layer image of the first image pyramid st , wherein the A st For the Q st The image block is the center pixel; Determine the search window Z in the N-1th layer image of the second image pyramid st , wherein the Z st For the A st The corresponding search window; According to the Q st The initial optical flow value C1 determines the Z st The image block G in st , wherein the G st For the A st The corresponding image block; Configure the A st In the Z st The search step length H in , where H is an integer greater than 1; According to the H, in the Z st Search for matching image block Y st , wherein the Y st For the A st The image blocks with the highest similarity between them; Determine the A st With the Y st Is the similarity X1 between them greater than the A st With the G st The similarity between them is X2; If X1>X2, then determine the initial optical flow value C2 in the initial optical flow of the N-1th layer of the first image pyramid, and determine C2 as the Q st The fine-tuned optical flow value of st The initial optical flow value corresponding to the central pixel of ; If X1≤X2, then the Q st The initial optical flow value C1 is determined as the Q st Fine-tuning optical flow value; wherein all the Q st The refined optical flow values ​​constitute the refined optical flow of the N-1th layer image of the first image pyramid.

6. The method according to claim 1, characterized in that The performing denoising on the fine-tuned optical flow of the first layer image of the first image pyramid to obtain the target optical flow of the target frame includes: The fine-tuned optical flow of the first layer image of the first image pyramid and the first layer image of the second image pyramid are input into a re-parameterized target denoising model for denoising, and the target optical flow of the target frame is output; wherein the target denoising model includes a type of convolution kernel and the size of the convolution kernel is greater than a preset size value.

7. The method according to claim 6, characterized in that Before the step of inputting the fine-tuned optical flow of the first layer image of the first image pyramid and the first layer image of the second image pyramid into the re-parameterized target denoising model for denoising and outputting the target optical flow of the target frame, the method further includes: Obtaining a training set; wherein the training set includes multiple groups of sample data, each group of sample data includes: a first sample image, a second sample image, a noisy optical flow of the first sample image, and a true optical flow of the first sample image, wherein the second sample image is a reference image of the first sample image; Constructing a backbone network; wherein the backbone network includes at least two groups of convolution kernels, each group of convolution kernels including a plurality of convolution kernels of different sizes connected in parallel; Inputting the noisy optical flows of the second sample image and the first sample image in the sample data into the backbone network for denoising, outputting a predicted image and the denoised optical flow of the predicted image; calculating a loss value based on the first sample image, the true optical flow of the first sample image, the predicted image, and the denoised optical flow of the predicted image, and updating the network parameters of the backbone network based on the loss value; repeating the above training process using different groups of the sample data until the model converges to obtain an initial denoising model; The initial denoising model is re-parameterized to generate the target denoising model.

8. An optical flow calculation device, characterized in that: The device comprises: a construction module, configured to construct a first image pyramid of a target frame and a second image pyramid of a reference frame; wherein the number of layers of the first image pyramid and the number of layers of the second image pyramid are both N, where N is an integer greater than 1; a configuration module, configured to configure an initial optical flow of an N-th layer image of the first image pyramid; wherein the initial optical flow of the N-th layer image of the first image pyramid comprises: a set of offsets for each pixel in the N-th layer image of the first image pyramid, each set of offsets comprising: at least two different random offsets, each of the random offsets being used to indicate a pixel position in the N-th layer image of the second image pyramid; a first processing module configured to perform block matching on the Nth layer image of the first image pyramid and the Nth layer image of the second image pyramid based on the initial optical flow of the Nth layer image of the first image pyramid, determine a refined optical flow of the Nth layer image of the first image pyramid based on the block matching result, upsample the refined optical flow of the Nth layer image of the first image pyramid, and determine the upsampling result as the initial optical flow of the N-1th layer image of the first image pyramid; wherein the refined optical flow of the Nth layer image of the first image pyramid includes a refined optical flow value of each pixel in the Nth layer image of the first image pyramid; a second processing module, configured to calculate, based on the initial optical flow of the image at the N-1th layer of the first image pyramid, a refined optical flow of each layer below the Nth layer of the first image pyramid, layer by layer, until a refined optical flow of the first layer of the first image pyramid is obtained; The denoising module is configured to perform denoising on the fine-tuned optical flow of the first layer image of the first image pyramid to obtain the target optical flow of the target frame.

9. An electronic device, characterized in that: The optical flow calculation method comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the optical flow calculation method according to any one of claims 1 to 7 are implemented.

10. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by the processor, the steps of the optical flow calculation method according to any one of claims 1 to 7 are implemented.

Images