Optical flow estimation method, related device, equipment and computer readable storage medium

Abstract

The present application provides an optical flow estimation method, related devices, equipment and computer-readable storage medium, which relate to the field of artificial intelligence, and specifically to the field of computer vision. The method includes: obtaining a first frame image and a second frame image; performing two feature extractions of different scales on the first frame image through a two-layer network to obtain two first optical flow feature maps; performing two feature extractions of different scales on the second frame image through a two-layer network to obtain two second optical flow feature maps; performing feature fusion of the first fusion feature map corresponding to the second layer network and the first first optical flow feature map and the first second optical flow feature map extracted by the first layer network through the first layer network to obtain the optical flow result corresponding to the first layer network; and outputting the optical flow result corresponding to the first layer network. The implementation of the present application can improve the computational efficiency of the deep learning model while keeping the optical flow estimation result of the deep learning model as close to real-time detection as possible.

Description

Technical Field

[0001] The present application relates to the field of computer vision, and in particular to an optical flow estimation method, related devices, equipment and computer-readable storage medium. Background Art

[0002] Optical flow is the apparent movement of the brightness pattern of an image. When the human eye observes a moving object, the scene of the object forms a series of continuously changing images on the retina of the human eye. This series of continuously changing information continuously "flows" through the retina (i.e., the image plane), like a "flow" of light, so it is called optical flow. In essence, optical flow is the obvious visual movement you feel in this moving world. Optical flow expresses the changes in the image, and because it contains information about the movement of the target, it can be used by the observer to determine the movement of the target. The definition of optical flow can be extended to the optical flow field, which refers to a two-dimensional instantaneous velocity field composed of all pixels in the image, in which the two-dimensional velocity vector is the projection of the three-dimensional velocity vector of the visible point in the scene on the imaging surface.

[0003] The study of optical flow has become an important part of computer vision and related research fields. Over the past two decades, a variety of optical flow estimation methods have been proposed, and the accuracy of optical flow estimation has been continuously improved. However, the traditional gradient-based optical flow calculation method has multiple premise assumptions, such as constant brightness between adjacent frames, continuous frame-taking time of adjacent video frames, relatively "small" movement of objects between adjacent frames, and maintaining spatial consistency. These assumptions may affect the calculation of optical flow in real situations, resulting in accuracy degradation in practical applications. At the same time, since the traditional optical flow method adopts an iterative method, the calculation is complex and time-consuming. The traditional optical flow estimation algorithm based on matching calculates sparse optical flow by continuously locating, tracking and matching the main feature areas of the target. Although this type of algorithm can be robust to large displacement motion and brightness changes, it is difficult to estimate the optical flow with sub-pixel accuracy and the calculation is time-consuming. Without special hardware support, these two types of traditional methods are difficult to apply to real-time detection of video sequences.

[0004] Nowadays, with the development of deep learning, more and more research fields have begun to use deep learning methods to obtain more research results, such as image classification, target recognition, etc. Deep learning is a method in machine learning based on data representation learning. Observations can be represented in a variety of ways, such as a vector of intensity values ​​for each pixel, or more abstractly represented as a series of edges, areas of specific shapes, etc. Using certain specific representation methods makes it easier to learn tasks from examples. Deep learning can obtain the ability to extract features by learning a large amount of data, thereby replacing manual feature acquisition. Using deep learning technology for optical flow estimation can achieve higher accuracy than traditional optical flow methods, and greatly improve the speed of optical flow estimation, achieving a near-real-time detection effect. However, the existing deep learning models are large in scale. On the one hand, a large amount of computer resources are required for calculation, and the computational efficiency of deep learning models is low; on the other hand, the optical flow estimation results cannot achieve the effect of real-time detection. Therefore, how to improve the computational efficiency of deep learning models while keeping the optical flow estimation results of deep learning models as close to real-time detection as possible is a technical problem that needs to be solved urgently. Summary of the invention

[0005] The present application provides an optical flow estimation method, related devices, equipment and computer-readable storage medium, which can improve the computational efficiency of the deep learning model while keeping the optical flow estimation results of the deep learning model as close to real-time detection as possible.

[0006] In a first aspect, a method for optical flow estimation is provided, which is applied to a feature pyramid network, the feature pyramid network includes a two-layer network, and the method includes: obtaining a first frame image and a second frame image; wherein the first frame image and the second frame image contain a target object; performing two feature extractions of different scales on the first frame image through the two-layer network to obtain two first optical flow feature maps; performing two feature extractions of different scales on the second frame image through the two-layer network to obtain two second optical flow feature maps; wherein the resolution of the optical flow feature map extracted by the first layer network is greater than that of the second layer network The method comprises the following steps: a) performing feature fusion on the first fused feature map corresponding to the second layer network and the first first optical flow feature map and the first second optical flow feature map extracted by the first layer network through the first layer network to obtain the optical flow result corresponding to the first layer network; wherein the first fused feature map corresponding to the second layer network is obtained by performing feature fusion on the second first optical flow feature map and the second second optical flow feature map extracted by the second layer network; and outputting the optical flow result corresponding to the first layer network; wherein the optical flow result is used to characterize the motion information of the target object.

[0007] In implementing the embodiment of the present application, when determining the optical flow results of the first frame image and the second frame image, the optical flow estimation device performs feature extraction of the first frame image and the second frame image multiple times at different scales to obtain multiple optical flow feature maps of different scales corresponding to the two frames of images, and then performs feature fusion on the multiple optical flow feature maps of different scales corresponding to the two frames of images, thereby obtaining the optical flow result. Since this implementation method fully considers the different semantic information contained in the multiple optical flow feature maps of different scales, it can improve the computational efficiency of the deep learning model while keeping the optical flow estimation result of the deep learning model as close to real-time detection as possible.

[0008] In a possible implementation, the feature pyramid network includes a pyramid feature network and a feature fusion network; the first to second layers of the feature pyramid network are respectively provided with the pyramid feature network; the first layer of the feature pyramid network is provided with the feature fusion network; the pyramid feature network is used to extract features from the first frame image and the second frame image; the feature fusion network is used to fuse the optical flow feature map extracted by the pyramid feature network; the fusion feature map corresponding to the second layer network and the first first optical flow feature map and the first second optical flow feature map extracted by the first layer network are featured by the first layer network The optical flow result corresponding to the first layer network is obtained by feature fusion, including: performing feature fusion on the second first optical flow feature map and the second second optical flow feature map extracted by the second layer pyramid feature network through the first layer feature fusion network in the feature pyramid network, and obtaining the first fused feature map corresponding to the second layer pyramid feature network; performing feature fusion on the first fused feature map corresponding to the second layer pyramid feature network and the first first optical flow feature map and the first second optical flow feature map extracted by the first layer pyramid feature network through the first layer feature fusion network in the feature pyramid network, and obtaining the optical flow result corresponding to the first layer feature fusion network. In implementing the embodiment of the present application, the optical flow estimation device can perform feature fusion on multiple optical flow feature maps of different scales corresponding to the first frame image and the second frame image through a feature fusion network, and obtain the optical flow result. Since this implementation method fully considers the different semantic information contained in multiple optical flow feature maps of different scales, it can improve the computational efficiency of the deep learning model while keeping the optical flow estimation result of the deep learning model as close to real-time detection as possible.

[0009] In a possible implementation, the optical flow feature map extracted by the second-layer pyramid feature network includes the second first optical flow feature map and the second second optical flow feature map; the optical flow feature map extracted by the first-layer pyramid feature network includes the first first optical flow feature map and the first second optical flow feature map; the resolution of the second first optical flow feature map and the resolution of the second second optical flow feature map belong to a first range; the resolution of the first first optical flow feature map and the resolution of the first second optical flow feature map belong to a second range, and the first range and the second range do not overlap; the fusion feature map corresponding to the second-layer pyramid feature network and the first-layer pyramid feature network extracted by the first-layer pyramid feature network are fused through the first-layer feature fusion network in the feature pyramid network. An optical flow feature map and a first second optical flow feature map are feature-fused to obtain an optical flow result corresponding to the first layer of the network, including: fusing the first first optical flow feature map and the first second optical flow feature map through the first layer of feature fusion network to obtain a first fused image; obtaining a second fused image based on the first second optical flow feature map and the first fused feature map through the first layer of feature fusion network, and extracting features of the second fused image to obtain a third fused image; fusing the first fused image and the third fused image through the first layer of feature fusion network to obtain a fourth fused image; and amplifying the fourth fused image through the first layer of feature fusion network to obtain the optical flow result corresponding to the first layer of feature fusion network.

[0010] In a possible implementation, the first layer feature fusion network is used to perform feature fusion on the first first optical flow feature map and the first second optical flow feature map to obtain a first fused image, including: determining N pixels from the first first optical flow feature map as N first feature points, and determining N pixels from the first second optical flow feature map as N second feature points, matching the N first feature points with the N second feature points, and obtaining the correspondence between the N first feature points and the N second feature points; obtaining the correlation between the N first feature points and the N second feature points according to the correspondence between the N first feature points and the N second feature points through the first layer feature fusion network; and performing feature fusion on the N first feature points and the N second feature points according to the correlation through the first layer feature fusion network to obtain the first fused image.

[0011] In a possible implementation, after the first fused image and the third fused image are feature-fused by the first-layer feature fusion network to obtain the fourth fused image, before the fourth fused image is amplified by the first-layer feature fusion network to obtain the optical flow result corresponding to the first-layer feature fusion network, it also includes: removing noise in the fourth fused image and retaining boundary information of the fourth fused image by the first-layer feature fusion network to obtain the fifth fused image; amplifying the fourth fused image by the first-layer feature fusion network to obtain the optical flow result corresponding to the first-layer feature fusion network includes: amplifying the fifth fused image by the first-layer feature fusion network to obtain the optical flow result corresponding to the first-layer feature fusion network. In implementing the embodiment of the present application, the deep learning module can retain boundary information while removing high-frequency noise in the optical flow feature map. This implementation can improve the model accuracy of the deep learning model and improve the robustness of the deep learning model to noise.

[0012] In a second aspect, an embodiment of the present application provides another optical flow estimation method, which is applied to a feature pyramid network, wherein the feature pyramid network includes an n-layer network, wherein n is a positive integer greater than 2, and the method includes: acquiring a first frame image and a second frame image; wherein the first frame image and the second frame image contain a target object; performing feature extraction of the first frame image n times at different scales through the n-layer network to obtain n first optical flow feature maps; performing feature extraction of the second frame image n times at different scales through the n-layer network to obtain n second optical flow feature maps; wherein the resolution of the optical flow feature map extracted by the n-1th layer of the network is greater than the resolution of the optical flow feature map extracted by the n-1th layer of the network; and merging the first fusion feature map corresponding to the n-1th layer of the network and the n-1th first optical flow feature map and the n-1th second optical flow feature map extracted by the n-1th layer of the network through the n-1th layer of the network. The feature map is feature-fused to obtain the optical flow result and feature output result corresponding to the n-1th layer network; wherein, the first fused feature map corresponding to the nth layer network is obtained by feature fusion of the nth first optical flow feature map and the nth second optical flow feature map extracted by the nth layer network; the optical flow result and feature output result corresponding to the i-th layer network, as well as the i-1th first optical flow feature map and the i-1th second optical flow feature map extracted by the i-1th layer network are feature-fused through the i-1th layer network to obtain the optical flow result output by the i-1th layer network; the i is taken from n-1 to 2 in sequence; the feature output result corresponding to the i-th layer network is determined according to the i-th second optical flow feature map and the feature output result corresponding to the i+1th layer network; the optical flow result corresponding to the 1st layer network is output; wherein, the optical flow result is used to characterize the motion information of the target object.

[0013] In implementing the embodiment of the present application, when determining the optical flow results of the first frame image and the second frame image, the optical flow estimation device performs feature extraction of the first frame image and the second frame image multiple times at different scales to obtain multiple optical flow feature maps of different scales corresponding to the two frames of images, and then performs feature fusion on the multiple optical flow feature maps of different scales corresponding to the two frames of images, thereby obtaining the optical flow result. Since this implementation method fully considers the different semantic information contained in the multiple optical flow feature maps of different scales, it can improve the computational efficiency of the deep learning model while keeping the optical flow estimation result of the deep learning model as close to real-time detection as possible.

[0014] In a possible implementation, the feature pyramid network includes a pyramid feature network and a feature fusion network; the 1st to nth layers of the feature pyramid network are respectively provided with the pyramid feature network; the 1st to n-1th layers of the feature pyramid network are respectively provided with the feature fusion network; the first fusion feature map corresponding to the nth layer network and the n-1th first optical flow feature map and the n-1th second optical flow feature map extracted by the n-1th layer network are feature fused through the n-1th layer network to obtain the optical flow map corresponding to the n-1th layer network The first fusion feature map corresponding to the n-th layer network is obtained by feature fusion of the n-th first optical flow feature map and the n-th second optical flow feature map extracted by the n-th layer network; the optical flow result and feature output result corresponding to the i-th layer network, and the i-1-th first optical flow feature map and the i-1-th second optical flow feature map extracted by the i-1-th layer network are feature fused through the i-1-th layer network to obtain the optical flow result output by the i-1-th layer network, including: The first-layer feature fusion network performs feature fusion on the nth first optical flow feature map and the nth second optical flow feature map extracted by the nth layer pyramid feature network to obtain the first fused feature map corresponding to the nth layer pyramid feature network; the first fused feature map corresponding to the nth layer pyramid feature network and the n-1th first optical flow feature map and the n-1th second optical flow feature map extracted by the n-1th layer pyramid feature network are feature fused through the n-1th layer feature fusion network in the feature pyramid network to obtain the first fused feature map corresponding to the n-1th layer pyramid feature network. Optical flow results and feature output results; the optical flow results and feature output results corresponding to the i-th layer network, as well as the i-1th first optical flow feature map and the i-1th second optical flow feature map extracted by the i-1th layer network are feature fused through the i-1th layer feature fusion network in the feature pyramid network to obtain the optical flow results output by the i-1th layer network; the i is taken from n-1 to 2 in sequence; the feature output result corresponding to the i-th layer network is determined based on the i-th second optical flow feature map and the feature output result corresponding to the i+1th layer network. In implementing the embodiment of the present application, the optical flow estimation device can perform feature fusion on multiple optical flow feature maps of different scales corresponding to the first frame image and the second frame image through multiple feature fusion networks to obtain the optical flow result. Compared with existing implementation methods, since the parameter scale of multiple feature fusion networks is small, and multiple feature fusion networks can perform feature fusion on optical flow feature maps of different scales corresponding to two frames of images, the different semantic information contained in multiple optical flow feature maps of different scales is fully considered. While improving the computational efficiency of the deep learning model, the optical flow estimation results of the deep learning model can be kept as close to real-time detection as possible.

[0015] In a possible implementation, the optical flow feature map extracted by the n-th layer pyramid feature network includes the n-th first optical flow feature map and the n-th second optical flow feature map; the optical flow feature map extracted by the n-1-th layer pyramid feature network includes the n-1-th first optical flow feature map and the n-1-th second optical flow feature map; the resolution of the n-th first optical flow feature map and the resolution of the n-1-th second optical flow feature map belong to a first range; the resolution of the n-1-th first optical flow feature map and the resolution of the n-1-th second optical flow feature map belong to a second range, and the first range and the second range do not overlap; the first fusion feature map corresponding to the n-th layer network and the n-1-th first optical flow feature map and the n-1-th second optical flow feature map extracted by the n-1-th layer network are feature fused through the n-1-th layer network to obtain the optical flow result corresponding to the n-1-th layer network. and feature output results, including: performing feature fusion on the n-1th first optical flow feature map and the n-1th second optical flow feature map through the n-1th layer feature fusion network to obtain a first fused image; obtaining a second fused image according to the n-1th second optical flow feature map and the first fused feature map through the n-1th layer feature fusion network, and extracting features of the second fused image to obtain a third fused image; amplifying the third fused image through the n-1th layer feature fusion network to obtain a feature output result corresponding to the n-1th layer feature fusion network; performing feature fusion on the first fused image and the third fused image through the n-1th layer feature fusion network to obtain a fourth fused image; amplifying the fourth fused image through the n-1th layer feature fusion network to obtain an optical flow result corresponding to the n-1th layer feature fusion network.

[0016] In a possible implementation, the feature fusion of the n-1th first optical flow feature map and the n-1th second optical flow feature map through the n-1th layer feature fusion network to obtain a first fused image includes: determining N pixels as N first feature points from the n-1th first optical flow feature map through the n-1th layer feature fusion network, determining N pixels as N second feature points from the n-1th second optical flow feature map, matching the N first feature points with the N second feature points, and obtaining the correspondence between the N first feature points and the N second feature points; obtaining the correlation between the N first feature points and the N second feature points according to the correspondence between the N first feature points and the N second feature points through the n-1th layer feature fusion network; and feature fusion of the N first feature points and the N second feature points according to the correlation through the n-1th layer feature fusion network to obtain the first fused image.

[0017] In a possible implementation, after the feature fusion of the first fused image and the third fused image is performed through the n-1 layer feature fusion network to obtain the fourth fused image, before the amplification of the fourth fused image through the n-1 layer feature fusion network to obtain the optical flow result corresponding to the n-1 layer feature fusion network, it also includes: removing the noise in the fourth fused image through the n-1 layer feature fusion network and retaining the boundary information of the fourth fused image to obtain the fifth fused image; the amplification of the fourth fused image through the n-1 layer feature fusion network to obtain the optical flow result corresponding to the n-1 layer feature fusion network includes: amplifying the fifth fused image through the n-1 layer feature fusion network to obtain the optical flow result corresponding to the n-1 layer feature fusion network. In implementing the embodiment of the present application, the deep learning module can remove high-frequency noise in the optical flow feature map while retaining the boundary information. This implementation can improve the model accuracy of the deep learning model and improve the robustness of the deep learning model to noise.

[0018] In a third aspect, an embodiment of the present application provides an optical flow estimation device, which is applied to a feature pyramid network, and the feature pyramid network includes a two-layer network. The device may include: an image acquisition module, used to acquire a first frame image and a second frame image; wherein the first frame image and the second frame image contain a target object; a pyramid feature module, used to perform two feature extractions of different scales on the first frame image through the two-layer network to obtain two first optical flow feature maps; perform two feature extractions of different scales on the second frame image through the two-layer network to obtain two second optical flow feature maps; wherein the resolution of the optical flow feature map extracted by the first layer network is The rate is greater than the resolution of the optical flow feature map extracted by the second-layer network; a feature fusion module is used to perform feature fusion on the first fused feature map corresponding to the second-layer network and the first first optical flow feature map and the first second optical flow feature map extracted by the first-layer network through the first-layer network to obtain the optical flow result corresponding to the first-layer network; wherein the first fused feature map corresponding to the second-layer network is obtained by performing feature fusion on the second first optical flow feature map and the second second optical flow feature map extracted by the second-layer network; an output module is used to output the optical flow result corresponding to the first-layer network; wherein the optical flow result is used to characterize the motion information of the target object.

[0019] In implementing the embodiment of the present application, when determining the optical flow results of the first frame image and the second frame image, the optical flow estimation device performs feature extraction of the first frame image and the second frame image multiple times at different scales to obtain multiple optical flow feature maps of different scales corresponding to the two frames of images, and then performs feature fusion on the multiple optical flow feature maps of different scales corresponding to the two frames of images, thereby obtaining the optical flow result. Since this implementation method fully considers the different semantic information contained in the multiple optical flow feature maps of different scales, it can improve the computational efficiency of the deep learning model while keeping the optical flow estimation result of the deep learning model as close to real-time detection as possible.

[0020] In a possible implementation, the feature pyramid network includes a pyramid feature network and a feature fusion network; the first to second layers of the feature pyramid network are respectively provided with the pyramid feature networks; the first layer of the feature pyramid network is provided with the feature fusion network; the pyramid feature network is used to extract features from the first frame image and the second frame image; the feature fusion network is used to fuse the optical flow feature map extracted by the pyramid feature network; the feature fusion module includes a first feature fusion module, and the first feature fusion module is used to: perform feature fusion on the second first optical flow feature map and the second second optical flow feature map extracted by the second layer pyramid feature network through the first layer feature fusion network in the feature pyramid network to obtain the first fused feature map corresponding to the second layer pyramid feature network; perform feature fusion on the first fused feature map corresponding to the second layer pyramid feature network and the first first optical flow feature map and the first second optical flow feature map extracted by the first layer pyramid feature network through the first layer feature fusion network in the feature pyramid network to obtain the optical flow result corresponding to the first layer feature fusion network.

[0021] In a possible implementation, the optical flow feature map extracted by the second-layer pyramid feature network includes the second first optical flow feature map and the second second optical flow feature map; the optical flow feature map extracted by the first-layer pyramid feature network includes the first first optical flow feature map and the first second optical flow feature map; the resolution of the second first optical flow feature map and the resolution of the second second optical flow feature map belong to a first range; the resolution of the first first optical flow feature map and the resolution of the first second optical flow feature map belong to a second range, and the first range and the second range do not overlap; the first feature fusion module is specifically used to: The first-layer feature fusion network performs feature fusion on the first first optical flow feature map and the first second optical flow feature map to obtain a first fused image; the first-layer feature fusion network obtains a second fused image based on the first second optical flow feature map and the first fused feature map, and extracts features of the second fused image to obtain a third fused image; the first-layer feature fusion network performs feature fusion on the first fused image and the third fused image to obtain a fourth fused image; the first-layer feature fusion network amplifies the fourth fused image to obtain an optical flow result corresponding to the first-layer feature fusion network.

[0022] In a possible implementation, the first feature fusion module is specifically used to: determine N pixels from the first first optical flow feature map as N first feature points through the first layer feature fusion network, determine N pixels from the first second optical flow feature map as N second feature points, match the N first feature points with the N second feature points, and obtain the correspondence between the N first feature points and the N second feature points; obtain the correlation between the N first feature points and the N second feature points according to the correspondence between the N first feature points and the N second feature points through the first layer feature fusion network; perform feature fusion on the N first feature points and the N second feature points according to the correlation through the first layer feature fusion network to obtain the first fused image.

[0023] In a possible implementation, the first feature fusion module is further used to: remove noise in the fourth fused image and retain boundary information of the fourth fused image through the first-layer feature fusion network to obtain a fifth fused image; the first feature fusion module is specifically used to: amplify the fifth fused image through the first-layer feature fusion network to obtain an optical flow result corresponding to the first-layer feature fusion network.

[0024] In a fourth aspect, an embodiment of the present application further provides another optical flow estimation device, which is applied to a feature pyramid network, and the feature pyramid network includes an n-layer network, where n is a positive integer greater than 2, and the device includes: an image acquisition module, used to acquire a first frame image and a second frame image; wherein the first frame image and the second frame image contain a target object; a pyramid feature module, used to perform n times of feature extraction of different scales on the first frame image through the n-layer network to obtain n first optical flow feature maps; perform n times of feature extraction of different scales on the second frame image through the n-layer network to obtain n second optical flow feature maps; wherein the resolution of the optical flow feature map extracted by the n-1th layer of the network is greater than the resolution of the optical flow feature map extracted by the n-1th layer of the network; a feature fusion module, used to combine the first fused feature map corresponding to the n-1th layer of the network and the n-1th first optical flow feature map extracted by the n-1th layer of the network through the n-1th layer of the network. The feature map and the n-1th second optical flow feature map are feature fused to obtain the optical flow result and feature output result corresponding to the n-1th layer network; wherein the first fused feature map corresponding to the nth layer network is obtained by feature fusion of the nth first optical flow feature map and the nth second optical flow feature map extracted by the nth layer network; the optical flow result and feature output result corresponding to the i-th layer network, as well as the i-1th first optical flow feature map and the i-1th second optical flow feature map extracted by the i-1th layer network are feature fused through the i-1th layer network to obtain the optical flow result output by the i-1th layer network; the i is taken from n-1 to 2 in sequence; the feature output result corresponding to the i-th layer network is determined according to the i-th second optical flow feature map and the feature output result corresponding to the i+1th layer network; an output module is used to output the optical flow result corresponding to the 1st layer network; wherein the optical flow result is used to characterize the motion information of the target object.

[0025] In implementing the embodiment of the present application, when determining the optical flow results of the first frame image and the second frame image, the optical flow estimation device performs feature extraction of the first frame image and the second frame image multiple times at different scales to obtain multiple optical flow feature maps of different scales corresponding to the two frames of images, and then performs feature fusion on the multiple optical flow feature maps of different scales corresponding to the two frames of images, thereby obtaining the optical flow result. Since this implementation method fully considers the different semantic information contained in the multiple optical flow feature maps of different scales, it can improve the computational efficiency of the deep learning model while keeping the optical flow estimation result of the deep learning model as close to real-time detection as possible.

[0026] In a possible implementation, the feature pyramid network includes a pyramid feature network and a feature fusion network; the 1st to nth layers of the feature pyramid network are respectively provided with the pyramid feature networks; the 1st to n-1th layers of the feature pyramid network are respectively provided with the feature fusion network; the feature fusion module includes an n-1th feature fusion module and an i-1th feature fusion module; wherein the n-1th feature fusion module is used to: perform feature fusion on the nth first optical flow feature map and the nth second optical flow feature map extracted by the nth layer pyramid feature network through the n-1th layer feature fusion network in the feature pyramid network to obtain a first fused feature map corresponding to the nth layer pyramid feature network; perform feature fusion on the nth first optical flow feature map and the nth second optical flow feature map extracted by the nth layer pyramid feature network through the n-1th layer feature fusion network in the feature pyramid network A fusion feature map and the n-1th first optical flow feature map and the n-1th second optical flow feature map extracted by the n-1th layer pyramid feature network are feature fused to obtain the optical flow result and feature output result corresponding to the n-1th layer feature fusion network; the i-1th feature fusion module is used to: perform feature fusion on the optical flow result and feature output result corresponding to the i-th layer network, and the i-1th first optical flow feature map and the i-1th second optical flow feature map extracted by the i-1th layer network through the i-1th layer feature fusion network in the feature pyramid network, to obtain the optical flow result output by the i-1th layer network; the i is taken from n-1 to 2 in sequence; the feature output result corresponding to the i-th layer network is determined according to the i-th second optical flow feature map and the feature output result corresponding to the i+1th layer network.

[0027] In a possible implementation, the optical flow feature map extracted by the n-th layer pyramid feature network includes the n-th first optical flow feature map and the n-th second optical flow feature map; the optical flow feature map extracted by the n-1-th layer pyramid feature network includes the n-1-th first optical flow feature map and the n-1-th second optical flow feature map; the resolution of the n-th first optical flow feature map and the resolution of the n-1-th second optical flow feature map belong to a first range; the resolution of the n-1-th first optical flow feature map and the resolution of the n-1-th second optical flow feature map belong to a second range, and the first range and the second range do not overlap; the n-1-th feature fusion module is specifically used to: fusion the n-1-th first optical flow feature map and The n-1th second optical flow feature map is feature fused to obtain a first fused image; the second fused image is obtained according to the n-1th second optical flow feature map and the first fused feature map through the n-1th layer feature fusion network, and the features of the second fused image are extracted to obtain a third fused image; the third fused image is enlarged through the n-1th layer feature fusion network to obtain a feature output result corresponding to the n-1th layer feature fusion network; the first fused image and the third fused image are feature fused through the n-1th layer feature fusion network to obtain a fourth fused image; the fourth fused image is enlarged through the n-1th layer feature fusion network to obtain an optical flow result corresponding to the n-1th layer feature fusion network.

[0028] In a possible implementation, the n-1th feature fusion module is specifically used to: determine N pixels as N first feature points from the n-1th first optical flow feature map through the n-1th layer feature fusion network, determine N pixels as N second feature points from the n-1th second optical flow feature map, match the N first feature points with the N second feature points, and obtain the correspondence between the N first feature points and the N second feature points; obtain the correlation between the N first feature points and the N second feature points according to the correspondence between the N first feature points and the N second feature points through the n-1th layer feature fusion network; perform feature fusion on the N first feature points and the N second feature points according to the correlation through the n-1th layer feature fusion network to obtain the first fused image.

[0029] In a possible implementation, the n-1th feature fusion module is further used to: remove noise in the fourth fused image and retain boundary information of the fourth fused image through the n-1th layer feature fusion network to obtain a fifth fused image; the n-1th feature fusion module is specifically used to: amplify the fifth fused image through the n-1th layer feature fusion network to obtain an optical flow result corresponding to the n-1th layer feature fusion network.

[0030] In the fifth aspect, an embodiment of the present application also provides an optical flow estimation device, which may include a memory and a processor, wherein the memory is used to store a computer program that supports the device to execute the above method, the computer program includes program instructions, and the processor is configured to call the program instructions to execute the method described in any one of the first or second aspects above.

[0031] In a sixth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes the method described in any one of the first aspect or the second aspect above.

[0032] In a seventh aspect, an embodiment of the present application further provides a computer program, wherein the computer program includes computer software instructions, and when the computer software instructions are executed by a computer, the computer executes any one of the optical flow estimation methods described in the first aspect or the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A schematic diagram of a video stabilization scenario provided by an embodiment of the present application;

[0034] Figure 2 A schematic diagram of a video completion scenario provided in an embodiment of the present application;

[0035] Figure 3a A schematic diagram of the structure of a convolutional neural network 300 provided in an embodiment of the present application;

[0036] Figure 3b A schematic diagram of the structure of another convolutional neural network 300 provided in an embodiment of the present application;

[0037] Figure 3c A schematic diagram of the structure of another convolutional neural network 300 provided in an embodiment of the present application;

[0038] Figure 3d A schematic diagram of the structure of another convolutional neural network 300 provided in an embodiment of the present application;

[0039] Figure 3eA schematic diagram of a chip hardware structure provided in an embodiment of the present application;

[0040] Figure 4a A schematic diagram of the structure of a feature pyramid network 40 provided in an embodiment of the present application;

[0041] Figure 4b A schematic diagram of the structure of a 2-layer feature pyramid network 40 provided in an embodiment of the present application;

[0042] Figure 4c A schematic diagram of the structure of a three-layer feature pyramid network 40 provided in an embodiment of the present application;

[0043] Figure 5a A schematic diagram of the structure of a feature fusion network 500 provided in an embodiment of the present application;

[0044] Figure 5b A schematic diagram of the structure of another feature fusion network 500 provided in an embodiment of the present application;

[0045] Figure 5c A schematic diagram of the structure of another feature fusion network 500 provided in an embodiment of the present application;

[0046] Figure 6a A schematic diagram of a flow chart of an optical flow estimation method provided in an embodiment of the present application;

[0047] Figure 6b A schematic diagram of a flow chart of another optical flow estimation method provided in an embodiment of the present application;

[0048] Figure 7a A schematic diagram of a first frame image provided in an embodiment of the present application;

[0049] Figure 7b A schematic diagram of a second frame image provided in an embodiment of the present application;

[0050] Figure 8a A comparison effect diagram of optical flow results provided in an embodiment of the present application;

[0051] Figure 8b A schematic diagram of visualizing an optical flow result obtained by an optical flow estimation method according to an embodiment of the present application;

[0052] Figure 8c Another optical flow result comparison effect diagram provided in an embodiment of the present application;

[0053] Figure 9a A schematic diagram of the structure of a video completion architecture provided in an embodiment of the present application;

[0054] Figure 9bA schematic diagram of the structure of a video completion network 902 provided in an embodiment of the present application;

[0055] Fig.9c A schematic diagram of obtaining a video completion result of a frame n image provided in an embodiment of the present application;

[0056] Fig.10a A schematic block diagram of an optical flow estimation device provided in an embodiment of the present application;

[0057] Fig.10b A schematic block diagram of an optical flow estimation device provided in an embodiment of the present application;

[0058] Fig.11 A schematic diagram of the structure of an optical flow estimation device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0059] The technical solutions in the embodiments of the present application are clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments.

[0060] The terms "first" and "second" in the specification and drawings of the present application are used to distinguish different objects, or to distinguish different treatments of the same object, rather than to describe the specific order of objects. In addition, the terms "including" and "having" and any variations thereof mentioned in the description of the present application are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes other steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices. It should be noted that in the embodiments of the present application, words such as "exemplarily" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design method described as "exemplarily" or "for example" in the embodiments of the present application should not be interpreted as being superior or more advantageous than other embodiments or design schemes. Specifically, the use of words such as "exemplarily" or "for example" is intended to present related concepts in a specific way. In the embodiments of the present application, "A and / or B" means A and B, A or B. "A, and / or B, and / or C" means any one of A, B, and C, or any two of A, B, and C, or A, B, and C. The technical solution in the present application is described below in conjunction with the accompanying drawings.

[0061] The embodiment of the present application provides an optical flow estimation method, which can estimate the optical flow result (also known as: optical flow field) between two images. The optical flow estimation method described in this application can be applied to many fields such as video editing, motion detection, target tracking, scene analysis, motion capture and recognition, video stabilization, video completion, visual SLAM, augmented reality (AR), etc. to achieve specific functions.

[0062] The first application scenario: video stabilization scenario

[0063] In the embodiment of the present application, the video anti-shake scenario refers to: the video capture device may cause the captured image to be unstable or out of focus due to shaking during the shooting process. For example, the video capture device can be a mobile terminal with a camera function, a video surveillance, a handheld camera, a digital camera, etc. Figure 1 As shown in FIG. 1 , a user obtains a video sequence by shooting a landmark (e.g., office building A) through a mobile terminal. Since the user shakes a little when shooting the video of office building A with the mobile terminal device, some of the captured images appear unstable and out of focus. For example, Figure 1 (a) and Figure 1 In this case, the mobile terminal device can use the optical flow estimation method described in this application to estimate the optical flow through the pyramid feature network. Figure 1 (a) and Figure 1 In (b), multiple feature extractions of different scales are performed respectively to obtain multiple optical flow feature maps of different scales corresponding to the two frames of images. Here, the resolution of the optical flow feature map extracted by the upper pyramid feature network is greater than the resolution of the optical flow feature map extracted by the lower pyramid feature network. Then, the multiple optical flow feature maps of different scales corresponding to the two frames of images are fused through the feature fusion network to obtain the optical flow result. After that, motion compensation can be performed on each frame of the picture according to the motion vector obtained by optical flow estimation. For example, the motion compensation can include global motion compensation, block motion compensation, etc., so that a video with jitter eliminated can be obtained.

[0064] Second application scenario: video completion scenario

[0065] In the embodiment of the present application, the video completion function means that since the shooting time interval between the two frames of the video sequence is relatively fixed, for a fast-moving object, the pixel position of the object in the two frames of the video sequence is quite different. The optical flow estimation method described in the present application can find the corresponding pixel points of adjacent image frames, thereby realizing the video completion function and removing the specified object from the video. Figure 2As shown in (a), a user uses a mobile terminal to shoot a moving object (for example, Xiao Ming walks toward the window) to obtain a video sequence. In order to remove the image of "Xiao Ming walking" in the video, the mobile terminal device can use the optical flow estimation method described in this application, and use the pyramid feature network to perform multiple feature extractions of different scales on multiple consecutive frames of "Xiao Ming walking" to obtain multiple optical flow feature maps of different scales corresponding to each of the consecutive multiple frames. Here, the resolution of the optical flow feature map extracted by the pyramid feature network of the previous layer is greater than the resolution of the optical flow feature map extracted by the pyramid feature network of the next layer; then, the feature fusion network is used to perform feature fusion on the multiple optical flow feature maps of different scales corresponding to each of the consecutive multiple frames, so as to obtain the optical flow result. Afterwards, the corresponding pixel points of each of the two adjacent image frames are found through the optical flow results, so as to realize the video completion function and remove the "image of Xiao Ming walking" (that is, the target object) in the video. For example, the video image can be as follows. Figure 2 As shown in (b).

[0066] It should be noted that the several application scenarios described above are only used as examples, and the optical flow results obtained by the optical flow estimation method described in this application are not limited to the above application scenarios.

[0067] Since the embodiments of the present application involve the application of a large number of neural networks, in order to facilitate understanding, the relevant terms and related concepts such as neural networks involved in the embodiments of the present application are first introduced below.

[0068] (1) Optical flow

[0069] Optical flow is the instantaneous speed of the pixel movement of a moving object in space on the observation imaging plane. It is a method of finding the correspondence between the previous frame image and the current frame image by using the change of pixels in the time domain in the image sequence and the correlation between adjacent frame images, thereby calculating the motion information of the object between adjacent frame images. Generally speaking, the optical flow is caused by the movement of the foreground target itself in the scene, or the movement of the camera, or the joint movement of the two. It should be noted that the optical flow estimation method of the embodiment of the present application can be applied to video editing, motion detection, target tracking, scene analysis, motion capture and recognition, video stabilization, video completion, visual SLAM, augmented reality (AR) and other fields by estimating the optical flow field between two images.

[0070] (2) Image upsampling

[0071] Image upsampling, also known as image enlargement or image interpolation, is a method of upsampling an image so that the upsampled image can be displayed on a display device with a higher resolution. For example, an interpolation method can be used to insert new elements between pixels in the original image using a suitable interpolation algorithm, thereby increasing the resolution of the image.

[0072] (3) Image downsampling

[0073] Image downsampling, also known as downsampling or image reduction, is a method of downsampling an image so that the image after downsampling can fit the size of the display area corresponding to the display device. For example, assuming that a frame of image (e.g., image a) is of size M*N, by downsampling image a by S times, image a can be reduced by S times, that is, the length and width of image a are reduced by S times at the same time. Then, the size of the image after downsampling is (M / S)*(N / S)

[0074] (4) Neural Network

[0075] A neural network can be composed of neural units. A neural unit can be a s and intercept b as inputs, the output of the operation unit can be:

[0076]

[0077] Where, s = 1, 2, ... n, n is a natural number greater than 1, w s For x s The weight of the neural unit, b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolutional layer. The activation function can be a sigmoid function. A neural network is a network formed by connecting many of the above-mentioned single neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be an area composed of several neural units.

[0078] (5) Deep Neural Networks

[0079] A deep neural network (DNN), also known as a multi-layer neural network, can be understood as a neural network with many hidden layers. There is no special metric for "many" here. From the position of different layers of DNN, the neural network inside DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are all hidden layers. The layers are fully connected, that is, any neuron in the i-th layer must be connected to any neuron in the i+1-th layer. Although DNN looks complicated, the work of each layer is actually not complicated. Simply put, it is the following linear relationship expression: in, is the input vector, is the output vector, b is the offset vector, w is the weight matrix (also called coefficient), and α() is the activation function. Each layer is just an input vector. After such a simple operation, the output vector Since there are many DNN layers, the number of coefficients w and offset vectors b is also large. The definitions of these parameters in DNN are as follows: Taking coefficient w as an example: Assume that in a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as The superscript 3 represents the layer number of the coefficient W, while the subscripts correspond to the output third layer index 2 and the input second layer index 4. In summary, the coefficients from the kth neuron in the L-1th layer to the jth neuron in the Lth layer are defined as It should be noted that the input layer does not have a w parameter. In a deep neural network, more hidden layers allow the network to better describe complex situations in the real world. Theoretically, the more parameters a model has, the higher its complexity and the greater its "capacity", which means it can complete more complex learning tasks. Training a deep neural network is the process of learning the weight matrix, and its ultimate goal is to obtain the weight matrix of all layers of the trained deep neural network (the weight matrix formed by many layers of vector w).

[0080] (6) Convolutional Neural Networks

[0081] Convolutional neural network (CNN) is a deep neural network with a convolutional structure. Convolutional neural network contains a feature extractor consisting of a convolution layer and a subsampling layer. The feature extractor can be regarded as a filter, and the convolution process can be regarded as using a trainable filter to convolve with an input data (such as image data, described by taking image data as an example) or a convolution feature plane (feature map). Convolution layer refers to the neuron layer in the convolutional neural network that performs convolution processing on the input signal. In the convolution layer of the convolutional neural network, a neuron can only be connected to some neurons in the adjacent layer. A convolution layer usually contains several feature planes, and each feature plane can be composed of some rectangularly arranged neural units. The neural units in the same feature plane share weights, and the shared weights here are the convolution kernels. Shared weights can be understood as the way to extract image information is independent of position. The implicit principle is that the statistical information of a part of the image is the same as that of other parts. This means that the image information learned in a certain part can also be used in another part. Therefore, the same learned image information can be used for all positions on the image. In the same convolution layer, multiple convolution kernels can be used to extract different image information. Generally speaking, the more convolution kernels there are, the richer the image information reflected by the convolution operation.

[0082] The convolution kernel can be initialized in the form of a matrix of random size, and the convolution kernel can obtain reasonable weights through learning during the training process of the convolutional neural network. In addition, the direct benefit of shared weights is to reduce the connections between the layers of the convolutional neural network, while reducing the risk of overfitting.

[0083] As mentioned in the previous basic concept introduction, a convolutional neural network is a deep neural network with a convolution structure, and is a deep learning architecture. A deep learning architecture refers to multiple levels of learning at different abstract levels through machine learning algorithms. As a deep learning architecture, CNN is a feed-forward artificial neural network, in which each neuron can respond to the image input into it. In order to better understand the optical flow estimation method described in this application, the structure of the convolutional neural network that may be involved is specifically introduced below:

[0084] In some possible implementations, such as Figure 3a As shown, the convolutional neural network (CNN) 300 may include an input layer 310 , a convolutional layer / pooling layer 320 (wherein the pooling layer is optional), and a neural network layer 330 .

[0085] Convolutional layer / pooling layer 320:

[0086] Convolutional Layer:

[0087] like Figure 3a The convolution layer / pooling layer 320 shown may include layers 321-326 as shown in the examples. For example, in one implementation, layer 321 is a convolution layer, layer 322 is a pooling layer, layer 323 is a convolution layer, layer 324 is a pooling layer, layer 325 is a convolution layer, and layer 326 is a pooling layer; in another implementation, layers 321 and 322 are convolution layers, layer 323 is a pooling layer, layers 324 and 325 are convolution layers, and layer 326 is a pooling layer. That is, the output of a convolution layer can be used as the input of a subsequent pooling layer, or as the input of another convolution layer to continue the convolution operation.

[0088] The following will take convolutional layer 321 as an example to introduce the internal working principle of a convolutional layer.

[0089] The convolution layer 321 may include a plurality of convolution operators, which are also called kernels. The convolution operator is equivalent to a filter that extracts specific information from the input image matrix in image processing. The convolution operator can be essentially a weight matrix, which is usually predefined. In the process of performing convolution operations on the image, the weight matrix is ​​usually processed one pixel after another (or two pixels after two pixels, depending on the value of the stride) in the horizontal direction on the input image, thereby completing the work of extracting specific features from the image. The size of the weight matrix should be related to the size of the image. It should be noted that the depth dimension of the weight matrix is ​​the same as the depth dimension of the input image. In the process of performing convolution operations, the weight matrix will extend to the entire depth of the input image. Therefore, convolution with a single weight matrix will produce a convolution output with a single depth dimension, but in most cases, a single weight matrix is ​​not used, but multiple weight matrices of the same size (row × column), that is, multiple isotype matrices, are applied. The output of each weight matrix is ​​stacked to form the depth dimension of the convolution image, and the dimension here can be understood as being determined by the "multiple" mentioned above. Different weight matrices can be used to extract different features in the image, for example, one weight matrix is ​​used to extract image edge information, another weight matrix is ​​used to extract specific colors of the image, and another weight matrix is ​​used to blur unwanted noise in the image, etc. The multiple weight matrices have the same size (rows × columns), and the feature maps extracted by the multiple weight matrices of the same size are also the same size. The extracted feature maps of the same size are then merged to form the output of the convolution operation.

[0090] The weight values ​​in these weight matrices need to be obtained through a lot of training in practical applications. The weight matrices formed by the weight values ​​obtained through training can be used to extract information from the input image, so that the convolutional neural network 300 can make correct predictions.

[0091] When the convolutional neural network 300 has multiple convolutional layers, the initial convolutional layer (for example, 321) often extracts more general features, which can also be called low-level features. As the depth of the convolutional neural network 300 increases, the features extracted by the later convolutional layers (for example, 326) become more and more complex, such as high-level semantic features. Features with higher semantics are more suitable for the problem to be solved.

[0092] Pooling layer:

[0093] Since it is often necessary to reduce the number of training parameters, it is often necessary to periodically introduce a pooling layer after the convolution layer. Figure 3a The layers 321-326 illustrated in 320 may be a convolution layer followed by a pooling layer, or may be multiple convolution layers followed by one or more pooling layers. Specifically, the pooling layer is used to sample the data and reduce the amount of data. For example, taking the data as image data, in the image processing process, the spatial size of the image can be reduced by the pooling layer. Generally, the pooling layer may include an average pooling operator and / or a maximum pooling operator to sample the input image to obtain an image of smaller size. The average pooling operator can calculate the pixel values ​​in the image within a specific range to generate an average value as the result of average pooling. The maximum pooling operator can take the pixel with the largest value in the range within a specific range as the result of maximum pooling. In addition, just as the size of the weight matrix used in the convolution layer should be related to the image size, the operator in the pooling layer should also be related to the image size. The size of the image output after processing by the pooling layer can be smaller than the size of the image input to the pooling layer, and each pixel in the image output by the pooling layer represents the average value or maximum value of the corresponding sub-region of the image input to the pooling layer.

[0094] Neural network layer 330:

[0095] After being processed by the convolution layer / pooling layer 320, the convolution neural network 300 is not sufficient to output the required output information. As mentioned above, the convolution layer / pooling layer 320 only extracts features and reduces the parameters brought by the input image. However, in order to generate the final output information (the required class information or other related information), the convolution neural network 300 needs to use the neural network layer 330 to generate one or a group of outputs of the required number of classes. Therefore, the neural network layer 330 may include multiple hidden layers (such as Figure 3aAs shown in 331, 332 to 33n) and the output layer 340, the parameters contained in the multi-layer hidden layers can be pre-trained according to relevant training data of a specific task type. For example, the task type may include image recognition, image classification, image super-resolution reconstruction, etc.

[0096] After the multiple hidden layers in the neural network layer 330, that is, the last layer of the entire convolutional neural network 300 is the output layer 340, which has a loss function similar to the classification cross entropy, specifically used to calculate the prediction error. Once the forward propagation of the entire convolutional neural network 300 (such as Figure 3a The propagation from 310 to 340 is forward propagation), and the reverse propagation (such as Figure 3a The propagation from 340 to 310 is called back propagation) and then starts to update the weight values ​​and biases of the aforementioned layers to reduce the loss of the convolutional neural network 300 and the error between the result output by the convolutional neural network 300 through the output layer and the ideal result.

[0097] It should be noted that if Figure 3a The convolutional neural network 300 shown is only an example of a convolutional neural network. In specific applications, the convolutional neural network may also exist in the form of other network models. Figure 3b The multiple convolutional layers / pooling layers shown are operated in parallel, and the features extracted respectively are input to the neural network layer 330 for processing.

[0098] In some possible implementations, such as Figure 3c As shown, the convolutional neural network (CNN) 300 may include an input layer 310, a convolution layer / pooling layer 320 (where the pooling layer is optional), a deconvolution layer 350, and an output layer 340. Here, deconvolution is also called transposed convolution. In essence, deconvolution is the inverse process of convolution. It should be noted that the deconvolution operation cannot restore the input value of the convolution operation. It only reverses the steps in the convolution transformation process once, by transposing the convolution kernel and performing a convolution operation again with the result after convolution. Specifically, the convolution layer 320 is used to extract the features of the input data. For example, when the input data is an image, the convolution layer 320 is used to extract the features of the input image and obtain a feature image to reduce the parameters brought by the input image; the deconvolution layer 350 is used to map the feature image to the input image space corresponding to the convolution layer 320. Here, the size of the mapped image is the same as the size of the input image of the input convolution layer 320. Afterwards, the processed image is output through the output layer 340.

[0099] In some possible implementations, such as Figure 3dAs shown, the convolutional neural network (CNN) 300 may include an input layer 310, a convolution layer / pooling layer 320 (where the pooling layer is optional), a hole convolution layer 360, and an output layer 340. Here, hole convolution, also known as dilated convolution, refers to a convolution layer with interval sampling. Hollow convolution can effectively increase the receptive field of the convolution kernel without increasing model parameters or calculation amount. Specifically, the convolution layer 320 is used to extract the features of the input data. For example, when the input data is an image, the convolution layer 320 is used to extract the features of the input image and obtain a feature image to reduce the parameters brought by the input image; the hole convolution layer 360 is used to increase the receptive field of the feature image. Afterwards, the processed image is output through the output layer 340.

[0100] The following introduces a chip hardware structure provided by an embodiment of the present application.

[0101] Figure 3e A chip hardware structure is provided for an embodiment of the present application, and the chip includes an artificial intelligence processor 30.

[0102] Specifically, the artificial intelligence processor 30 can be a neural network processor (Network Processing Unit, NPU), a tensor processor (Tensor Processing Unit, TPU), or a graphics processor (Graphics Processing Unit, GPU) and any other processor suitable for large-scale XOR operation processing. Take NPU as an example: NPU can be mounted on the host CPU (Host CPU) as a coprocessor, and the host CPU assigns tasks to it. The core part of NPU is the operation circuit 303, which is controlled by the controller 304 to extract the matrix data in the memory (301 and 302) and perform multiplication and addition operations.

[0103] In some implementations, the operation circuit 303 includes multiple processing units (Process Engine, PE) inside. In some implementations, the operation circuit 303 is a two-dimensional systolic array. The operation circuit 303 can also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the operation circuit 303 is a general-purpose matrix processor.

[0104] For example, assume there is an input matrix A, a weight matrix B, and an output matrix C. The operation circuit 303 takes the weight data of the matrix B from the weight memory 302 and caches it on each PE in the operation circuit 303. The operation circuit 303 takes the input data of the matrix A from the input memory 301, performs matrix operations based on the input data of the matrix A and the weight data of the matrix B, and stores the partial results or the final results of the matrix in the accumulator 308.

[0105] The unified memory 306 is used to store input data and output data. The weight data is directly transferred to the weight memory 302 through the memory unit access controller (DMAC, Direct Memory Access Controller) 305. The input data is also transferred to the unified memory 306 through the DMAC.

[0106] The bus interface unit (BIU) 310 is used for the interaction between the DMAC and the instruction fetch buffer 309; the bus interface unit 301 is also used for the instruction fetch buffer 309 to obtain instructions from the external memory; the bus interface unit 301 is also used for the storage unit access controller 305 to obtain the original data of the input matrix A or the weight matrix B from the external memory.

[0107] DMAC is mainly used to transfer input data in the external memory DDR to the unified memory 306, or to transfer weight data to the weight memory 302, or to transfer input data to the input memory 301.

[0108] The vector calculation unit 307 may include multiple operation processing units, and if necessary, further process the output of the operation circuit 303, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. The vector calculation unit 307 is mainly used for the calculation of non-convolutional layers or fully connected layers (FC) in neural networks, and can specifically process the calculation of Pooling, Normalization, etc. For example, the vector calculation unit 307 can apply a nonlinear function to the output of the operation circuit 303, such as a vector of accumulated values, to generate an activation value. In some implementations, the vector calculation unit 307 generates a normalized value, a merged value, or both.

[0109] In some implementations, the vector calculation unit 307 stores the processed vectors in the unified memory 306. In some implementations, the vectors processed by the vector calculation unit 307 can be used as activation inputs of the operation circuit 303, for example, for use in subsequent layers in a neural network, such as Figure 3a As shown, if the current processing layer is hidden layer 1 (231), the vector processed by the vector calculation unit 307 can also be used in the calculation of hidden layer 2 (232).

[0110] An instruction fetch buffer 309 connected to the controller 304 for storing instructions used by the controller 304;

[0111] The unified memory 306, the input memory 301, the weight memory 302 and the instruction fetch memory 309 are all on-chip memories. The external memory is independent of the NPU hardware architecture.

[0112] in, Figure 3a , Figure 3b , Figure 3c , Figure 3d The operations of each layer in the convolutional neural network shown can be performed by the operation circuit 303 or the vector calculation unit 307.

[0113] The following is a detailed description of the Feature Pyramid Networks (FPN) involved in this application. Figure 4a As shown, the feature pyramid network 40 includes a pyramid feature network 410 and a feature fusion network 420. The pyramid feature network is used to extract the features of the image; the feature fusion network is used to fuse the features of the image. For example, the feature fusion method may include but is not limited to point-by-point addition, vector concatenation, etc. Specifically, the feature pyramid network 40 includes n layers, wherein the first to n layers of the feature pyramid network are respectively provided with a pyramid feature network 410, which may be: the first layer pyramid feature network, the second layer pyramid feature network, ..., the nth layer pyramid feature network; the first to n-1 layers of the feature pyramid network are respectively provided with a feature fusion network 420, which may be the first layer feature fusion network, the second layer feature fusion network, and the n-1th layer feature fusion network. The pyramid feature network 410 is used to extract features from the first frame image and the second frame image respectively, and obtain a plurality of optical flow feature maps of different scales corresponding to the first frame image and the second frame image; the feature fusion network 420 is used to perform feature fusion on a plurality of optical flow feature maps of different scales corresponding to the first frame image and the second frame image, and obtain an optical flow result.

[0114] In one example, n is equal to 2; in another example, n is a positive integer greater than 2, for example, n=3, etc. The specific structure of the feature pyramid network 40 is described below using n=2 and n>2 as examples:

[0115] When n = 2, such as Figure 4b As shown, the feature pyramid network 40 includes two layers, wherein the first layer and the second layer of the feature pyramid network are provided with pyramid feature networks 411 and 412 respectively, and the first layer of the feature pyramid network is provided with a feature fusion network 420. At this time, the pyramid feature network 411 is used to extract features from the first frame image to obtain the first first optical flow feature Figure 1 A; the pyramid feature network 411 is also used to extract features from the second frame image to obtain the first second optical flow feature Figure 1 B; then, the first optical flow feature obtained by downsampling Figure 1 A and the first second optical flow feature map are input into the pyramid feature network 412. In this case, the pyramid feature network 412 is used to Figure 1 A performs feature extraction to obtain the second first optical flow feature Figure 2 A; pyramid feature network 412, also used for the first second optical flow feature Figure 1 B performs feature extraction to obtain the second optical flow feature Figure 2 B. Among them, the first optical flow feature Figure 1 The resolution of A and the first and second optical flow features Figure 1 B has the same resolution and belongs to the second range; the second first optical flow feature Figure 2 The resolution of A and the second optical flow feature Figure 2 B has the same resolution and belongs to the first range; here, the first range does not overlap with the second range. For example, the size of the first frame image is 100*100, and the first frame image is extracted through the pyramid feature network 411 to obtain the first first optical flow feature Figure 1 A, then, the first optical flow feature obtained after downsampling Figure 1 The size of A is 50*50. The first optical flow feature is processed by the pyramid feature network 412 Figure 1 A performs feature extraction to obtain the second first optical flow feature Figure 2 A. The size of the second frame image is 100*100. The second frame image is feature extracted through the pyramid feature network 411 to obtain the first second optical flow feature Figure 1 B, then, the first second optical flow feature obtained after downsampling Figure 1 The size of B is 50*50. The first and second optical flow features are processed by the pyramid feature network 412 Figure 1B performs feature extraction to obtain the second optical flow feature Figure 2 B. Then, in this case, the feature fusion network 420 is used to extract the second first optical flow feature from the second layer pyramid feature network. Figure 2 A. Second optical flow feature Figure 2 B performs feature fusion to obtain the first fused feature map, and then the first fused feature map corresponding to the second layer pyramid feature network and the first optical flow feature extracted by the first layer pyramid feature network are combined. Figure 1 A and the first second optical flow feature Figure 1 B performs feature fusion to obtain the optical flow result corresponding to the first-layer feature fusion network 420. Here, the 2-fold downsampling method is only used as an example and should not constitute a limitation.

[0116] When n = 3, Figure 4c As shown, the feature pyramid network 40 includes 3 layers, wherein the first layer, the second layer and the third layer of the feature pyramid network are provided with pyramid feature networks 411, 412 and 413 respectively, and the first layer and the second layer of the feature pyramid network are provided with feature fusion network 421 and feature fusion network 422 respectively. At this time, the first layer of the pyramid feature network 411 is used to extract features from the first frame image to obtain the first first optical flow feature Figure 1 A, then the first optical flow feature obtained by downsampling Figure 1 A inputs the second layer pyramid feature network 412; the pyramid feature network 411 is also used to extract features from the second frame image to obtain the first second optical flow feature Figure 1 B, then the first second optical flow feature obtained by downsampling Figure 1 B inputs the second layer of pyramid feature network 412; the pyramid feature network 412 is used to obtain the first optical flow feature after downsampling Figure 1 A performs feature extraction to obtain the second first optical flow feature Figure 2 A, then the second first optical flow feature obtained by downsampling Figure 2 A inputs the third layer pyramid feature network 413; the pyramid feature network 412 is also used to obtain the first second optical flow feature after downsampling Figure 1 B performs feature extraction to obtain the second optical flow feature Figure 2 B, then the second optical flow feature obtained by downsampling Figure 2 B inputs the third layer pyramid feature network 413; the pyramid feature network 413 is used to obtain the second first optical flow feature after downsampling Figure 2 A is used to extract features to obtain the third first optical flow feature map 3A; the pyramid feature network 413 is also used to extract features of the second second optical flow obtained by downsampling. Figure 2 B performs feature extraction to obtain the third second optical flow feature map 3B. Among them, the first first optical flow feature Figure 1 The resolution of A and the first and second optical flow features Figure 1 The resolution of B is the same and belongs to the third range; the second first optical flow feature Figure 2 The resolution of A and the second optical flow feature Figure 2 The resolution of the third first optical flow feature map 3A is the same as the resolution of the third second optical flow feature map 3B, and belongs to the first range. Here, the first range does not overlap with the second range or the third range. For example, the size of the first frame image is 100*100, and the first frame image is extracted through the pyramid feature network 411 to obtain the first first optical flow feature. Figure 1 A, then, the first optical flow feature obtained after downsampling Figure 1 The size of A is 50*50. The first optical flow feature is processed by the pyramid feature network 412 Figure 1 A performs feature extraction to obtain the second first optical flow feature Figure 2 A, then, the second first optical flow feature obtained after downsampling Figure 2 The size of A is 25*25. The second first optical flow feature is processed by the pyramid feature network 413 Figure 2 A is used to extract features, and the third first optical flow feature map 3A is obtained. The size of the second frame image is 100*100, and the second frame image is extracted through the pyramid feature network 411 to obtain the first second optical flow feature map Figure 1 B, then the first second optical flow feature obtained by downsampling Figure 1 The size of B is 50*50. The first second optical flow feature obtained by downsampling through the pyramid feature network 412 Figure 1 B performs feature extraction to obtain the second optical flow feature Figure 2 B, then the second optical flow feature obtained by downsampling Figure 2 The size of B is 25*25. The second optical flow feature obtained by downsampling through the pyramid feature network 413 Figure 2 B performs feature extraction to obtain the third second optical flow feature map 3B. The feature fusion network 422 is used to perform feature fusion on the third first optical flow feature map 3A and the third second optical flow feature map 3B extracted by the third layer of the pyramid feature network to obtain a first fused feature map, and then the first fused feature map corresponding to the third layer of the pyramid feature network and the second first optical flow feature map extracted by the second layer of the pyramid feature network are fused. Figure 2 A and the second optical flow feature Figure 2B performs feature fusion to obtain the optical flow result and feature output result corresponding to the second-layer feature fusion network 422. After that, the feature fusion network 421 combines the optical flow result and feature output result corresponding to the second-layer feature fusion network 422 and the first optical flow feature extracted by the first-layer pyramid network. Figure 1 A and the first second optical flow feature Figure 1 B performs feature fusion to obtain the optical flow result corresponding to the first-layer feature fusion network 421. Here, the 2-fold downsampling method is only used as an example and should not constitute a limitation.

[0117] It should be noted that Figure 4a-4c The feature fusion network 420 shown can be constructed based on a convolutional neural network CNN. The following is a detailed description of the possible structure of the feature fusion network n-1 (n is a positive integer greater than or equal to 2):

[0118] In one possible implementation, Figure 5a As shown, the specific structure of the feature fusion network n-1 500 may include an input layer 510, a series layer 520, a convolutional neural network (a) 530, a distortion layer 540, a cross-correlation layer 550, a convolutional neural network (b) 560, an upsampling layer 570, and an output layer 580.

[0119] Among them, the input layer 510 is used to receive input data. Specifically, the input data may include the n-1th layer pyramid feature network and the optical flow feature map extracted by each of the n-1th layer pyramid feature network. For example, the first frame image is feature extracted by the n-1th layer pyramid feature network to obtain the n-1th first optical flow feature map; the second frame image is feature extracted by the n-1th layer pyramid feature network to obtain the n-1th second optical flow feature map; then, the n-1th first optical flow feature map and the n-1th second optical flow feature map obtained by downsampling are input into the n-1th layer pyramid feature network, and the n-1th first optical flow feature map is feature extracted by the n-1th layer pyramid feature network to obtain the nth first optical flow feature map; the n-1th second optical flow feature map is feature extracted by the n-1th layer pyramid feature network to obtain the nth second optical flow feature map;

[0120] A series connection layer 520 is used to perform feature fusion on the nth first optical flow feature map and the nth second optical flow feature map to obtain a first fused feature map;

[0121] Convolutional neural network (a) 530, used to obtain a feature output result corresponding to feature fusion network n-1 according to the first feature fusion image;

[0122] The distortion layer 540 is used to obtain the correspondence between the N first feature points in the n-1th first optical flow feature map and the N second feature points in the n-1th second optical flow feature map, and obtain the correlation between the N first feature points and the N second feature points according to the correspondence between the N first feature points and the N second feature points;

[0123] The cross-correlation layer 550 is used to perform feature fusion according to the correlation between the N first feature points and the N second feature points to obtain a fused image;

[0124] Convolutional neural network (b) 560, used to generate images after multiple fusions;

[0125] The upsampling layer 570 is used to enlarge the image after multiple fusions to obtain the optical flow result corresponding to the feature fusion network n-1. Specifically, the upsampling layer 570 can enlarge the image by 2 times upsampling, 4 times upsampling or other methods.

[0126] For the feature fusion network n-2, the input layer 510 is used to receive input data. Specifically, the input data may include the n-2th first optical flow feature map and the n-2th second optical flow feature map extracted by the n-2th layer pyramid feature network, the optical flow result obtained by the n-1th layer feature fusion network, and the feature output result; then, in this case, the series layer 520-upsampling layer 570 is used to obtain the optical flow result based on the n-2th first optical flow feature map and the n-2th second optical flow feature map extracted by the n-2th layer pyramid feature network, the optical flow result obtained by the n-1th layer feature fusion network, and the feature output result.

[0127] In one possible implementation, Figure 5b As shown, the convolutional neural network (a) 530 may include a convolutional layer 531 and a deconvolutional layer 532; the convolutional neural network (b) 560 may include a convolutional layer 561, a series layer 562, a convolutional layer 563 (one or more convolutional layers), a hole convolutional layer 564, a series layer 565, and a convolutional layer 566. Here, the series layer is used to connect the depths corresponding to the two data when the length and width of the data are the same. Taking the data as image data as an example, the length and width of the data represent the size of the image, and the depth of the data represents the color channel of the image. It can be understood that after the two images pass through the series layer, the feature information in the generated image can be richer.

[0128] In one possible implementation, Figure 5cAs shown, the convolutional neural network (a) 530 may include a convolutional layer 531 and an upsampling layer 533; the convolutional neural network (b) 560 may include a convolutional layer 561, a cascade layer 562, a convolutional layer 563 (one or more convolutional layers), a hole convolution layer 564, a cascade layer 565, a convolutional layer 566, and a bilateral filter layer 567. Here, the bilateral filter layer is used to remove high-frequency noise in the image and retain the boundary information of the image.

[0129] It should be noted that if Figure 5a-5c The feature fusion network 500 shown is only an example of a neural network. In specific applications, the feature fusion network may also exist in the form of other network models.

[0130] The method involved in the embodiment of the present application is described in detail below. The method is described around two situations. In one example, the feature pyramid network includes a 2-layer network; in another example, the feature pyramid network includes an n-layer network, where n is a positive integer greater than 2. Figure 4b Take the feature pyramid network shown as an example, Figure 6a An optical flow estimation method is provided in an embodiment of the present application. The method describes how to process a first frame image and a second frame image to obtain an optical flow result when the number of layers of the feature pyramid network is 2 (that is, the feature pyramid network only contains one feature fusion network).

[0131] In the above embodiment Figure 4b Taking the feature pyramid shown as an example, the feature pyramid network 40 includes a 2-layer network, wherein the first layer and the second layer of the feature pyramid network are respectively provided with a pyramid feature network 410, which are the first layer pyramid feature network and the second layer pyramid feature network. In one example, the first layer pyramid feature network and the second layer pyramid feature network can be connected in a cascade manner. In addition, the first layer of the feature pyramid network is provided with a feature fusion network 420, which is used to fuse multiple optical flow feature maps of different scales extracted by the pyramid feature network 410.

[0132] In one example, the method can be processed by a central processing unit (CPU), or by a CPU and a GPU, or by using other processors suitable for neural network calculations instead of a GPU, such as Figure 3e The artificial intelligence processor 30 shown is not limited in this application. The method may include some or all of the following steps:

[0133] Step S600a: Acquire a first frame image and a second frame image, wherein the first frame image and the second frame image contain a target object.

[0134] In the embodiment of the present application, the first frame image and the second frame image can be two consecutive frame images in a video sequence, or two frame images separated by a preset number of frames in a video sequence, which is not specifically limited in the embodiment of the present application. For example, the first frame image can also be called a target frame image, and the second frame image can also be called a reference frame image. Figure 7a and Figure 7b The following are two images captured by the video acquisition device during its motion. Assume that: Figure 7a The first frame is shown. Figure 7b The second frame image is shown.

[0135] In the embodiment of the present application, the first frame image and the second frame image have the same resolution. Here, resolution can be used to describe the ability to resolve image details. Generally, "resolution" can be expressed as the number of pixels in each direction, such as 640*480.

[0136] In an embodiment of the present application, the first frame image and the second frame image may be images containing one or more objects. In the case where the first frame image and the second frame image are images containing multiple objects, the target object may be any one of the multiple objects, or may be a specific object (e.g., a moving person, etc.) among the multiple objects. In one example, in the case where the first frame image and the second frame image contain one object, the object is the target object.

[0137] Step S602a, performing feature extraction of different scales on the first frame image twice through a two-layer network to obtain two first optical flow feature maps; performing feature extraction of different scales on the second frame image twice through a two-layer network to obtain two second optical flow feature maps; wherein the resolution of the optical flow feature map extracted by the first layer of the network is greater than the resolution of the optical flow feature map extracted by the second layer of the network.

[0138] In the embodiment of the present application, two layers of feature pyramid networks can be used to perform two feature extractions of different scales on the first frame image to obtain two first optical flow feature maps; two layers of feature pyramid networks can be used to perform two feature extractions of different scales on the second frame image to obtain two second optical flow feature maps. Here, the resolution of the optical flow feature map extracted by the first layer of pyramid feature network is greater than the resolution of the optical flow feature map extracted by the second layer of pyramid feature network.

[0139] The implementation process of performing two feature extractions on the first frame image through a two-layer pyramid feature network can be described as: performing feature extraction on the first frame image through the first-layer pyramid feature network to obtain the first first optical flow feature map, and then inputting the first first optical flow feature map obtained after downsampling into the second-layer pyramid feature network; thereby, the first first optical flow feature map obtained after downsampling can be feature extracted through the second-layer pyramid feature network to obtain the second first optical flow feature map.

[0140] The implementation process of performing two feature extractions on the second frame image through a two-layer pyramid feature network can be described as: performing feature extraction on the second frame image through the first layer of the pyramid feature network to obtain the first second optical flow feature map, and then inputting the first second optical flow feature map obtained after downsampling into the second layer of the pyramid feature network; thereby, the first second optical flow feature map obtained after downsampling can be subjected to feature extraction through the second layer of the pyramid feature network to obtain the second second optical flow feature map.

[0141] Here, the first optical flow feature Figure 1 The resolution of A and the first and second optical flow features Figure 1 The resolution of B is the same and belongs to range 1. The second first optical flow feature Figure 2 The resolution of A and the second optical flow feature Figure 2 The resolution of B is the same. Taking range 1 as an example, in one example, the range that is greater than or equal to threshold 1 is range 1, and the range that is less than threshold 1 is range 2. In one example, the range that is greater than threshold 1 is range 1, and the range that is less than or equal to threshold 1 is range 2.

[0142] In the embodiment of the present application, the image can be processed by 2 times downsampling, or by 4 times downsampling, which is not specifically limited in the embodiment of the present application.

[0143] Step S604a, performing feature fusion on the first fused feature map corresponding to the second layer network and the first first optical flow feature map and the first second optical flow feature map extracted by the first layer network through the first layer network to obtain the optical flow result corresponding to the first layer network; wherein the first fused feature map corresponding to the second layer network is obtained by performing feature fusion on the second first optical flow feature map and the second second optical flow feature map extracted by the second layer network.

[0144] In an embodiment of the present application, the first fusion feature map corresponding to the second layer of the network and the first first optical flow feature map and the first second optical flow feature map extracted by the first layer of the network are feature-fused through the first layer of the network to obtain the optical flow result corresponding to the first layer of the network. The implementation process may include: the second first optical flow feature map and the second second optical flow feature map extracted by the second layer of the pyramid feature network are feature-fused through the first layer of the feature fusion network in the feature pyramid network to obtain the first fusion feature map corresponding to the second layer of the pyramid feature network; then, the first fusion feature map corresponding to the second layer of the pyramid feature network and the first first optical flow feature map and the first second optical flow feature map extracted by the first layer of the pyramid feature network are feature-fused through the first layer of the feature fusion network to obtain the optical flow result corresponding to the first layer of the feature fusion network. Here, the optical flow result corresponding to the first layer of the feature fusion network is also the result finally output by the entire feature pyramid network.

[0145] The following specifically describes how the first-layer feature fusion network fuses the first fused feature map corresponding to the second-layer pyramid feature network and the first first optical flow feature map and the first second optical flow feature map extracted by the first-layer pyramid feature network. The specific implementation process can be described as:

[0146] Step A1: feature-fusing the first optical flow feature map and the first second optical flow feature map through the first-layer feature fusion network to obtain a first fused image.

[0147] Exemplarily, the implementation of feature fusion may include, but is not limited to: adding pixel points bit by bit and performing vector splicing.

[0148] Specifically, N pixels can be determined as N first feature points from the first first optical flow feature map through the first layer feature fusion network, and N pixels can be determined as N second feature points from the first second optical flow feature map, and the N first feature points and the N second feature points can be matched to obtain the correspondence between the N first feature points and the N second feature points; the correlation between the N first feature points and the N second feature points can be obtained according to the correspondence between the N first feature points and the N second feature points through the first layer feature fusion network; the N first feature points and the N second feature points can be feature fused according to the correlation through the first layer feature fusion network to obtain a first fused image.

[0149] In an embodiment of the present application, the correlation is used to characterize the degree of association between the N first feature points and the N second feature points. Exemplarily, the correlation between the N first feature points and the N second feature points can be obtained through the covariance matrix. The covariance matrix is ​​an N*N matrix, and each element in the matrix is ​​the covariance between each feature point. Each element is used to characterize the covariance between different feature points in the N*N matrix. For example, the covariance can be a positive value, a negative value, or 0. Specifically, when the covariance between two feature points in the N*N matrix is ​​a positive value, it indicates that the two feature points are positively correlated. When the covariance between two feature points in the N*N matrix is ​​a negative value, it indicates that the two feature points are negatively correlated. When the covariance between two feature points in the N*N matrix is ​​0, it indicates that the two feature points are two independent feature points.

[0150] Step A2: obtaining a second fused image according to the first second optical flow feature map and the first fused feature map through the first layer feature fusion network, and extracting features of the second fused image to obtain a third fused image.

[0151] Step A3: perform feature fusion on the first fused image and the third fused image through the first-layer feature fusion network to obtain a fourth fused image.

[0152] Step A4: amplify the fourth fused image through the first-layer feature fusion network to obtain the optical flow result corresponding to the first-layer feature fusion network. For example, in the first-layer feature fusion network, the fourth fused image can be amplified by 2 times upsampling, or by 4 times upsampling, which is not specifically limited in the embodiment of the present application.

[0153] It can be understood that in this implementation, the first-layer feature fusion network can be processed according to three input data to obtain the optical flow result corresponding to the first-layer feature fusion network, wherein the three input data are: the first fusion feature map corresponding to the second-layer pyramid feature network (the first fusion feature map corresponding to the second-layer network is obtained by feature fusion of the second first optical flow feature map and the second second optical flow feature map extracted by the second-layer network), the first first optical flow feature map and the first second optical flow feature map extracted by the first-layer pyramid feature network. This implementation fully considers the different semantic information contained in multiple optical flow feature maps of different scales, and can improve the computational efficiency of the deep learning model while keeping the optical flow estimation results of the deep learning model as close to real-time detection as possible.

[0154] It should be noted that, in one example, the noise in the fourth fused image can be removed by the bilateral filtering layer in the first-layer feature fusion network, and the boundary information of the fourth fused image can be retained to obtain the fifth fused image. Then, in this case, the fifth fused image is enlarged by the first-layer feature fusion network (for example, the fifth fused image is enlarged by 2 times upsampling) to obtain the optical flow result. Since the high-frequency noise in the optical flow feature map can be removed while retaining the boundary information, this implementation method can improve the model accuracy of the deep learning model and improve the robustness of the deep learning model to noise.

[0155] It should also be noted that for steps A1-A4, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should know that the present disclosure is not limited to the order of the actions described, because according to the present application, certain steps can be performed in other orders or simultaneously, which should not be construed as a limitation here.

[0156] Step S606a, outputting the optical flow result corresponding to the first layer of the network; wherein the optical flow result is used to characterize the motion information of the target object.

[0157] In the embodiment of the present application, the output method includes but is not limited to output through text, image, voice, video, etc.

[0158] It is understandable that the optical flow estimation device can perform feature fusion on multiple optical flow feature maps of different scales corresponding to the first frame image and the second frame image through a feature fusion network to obtain an optical flow result. Compared with the existing implementation method, since the parameter scale of the feature fusion network is small, and the feature fusion network can perform feature fusion on the optical flow feature maps of different scales corresponding to the two frames of images, the different semantic information contained in the optical flow feature maps of different scales is fully considered, and the computational efficiency of the deep learning model can be improved while keeping the optical flow estimation result of the deep learning model as close to real-time detection as possible.

[0159] In implementing the embodiment of the present application, when determining the optical flow results of the first frame image and the second frame image, the optical flow estimation device performs feature extraction of multiple scales on the first frame image and the second frame image respectively to obtain multiple optical flow feature maps of different scales corresponding to the two frames of images, and then performs feature fusion on the multiple optical flow feature maps of different scales corresponding to the two frames of images, thereby obtaining the optical flow result. Since this implementation method fully considers the different semantic information contained in the multiple optical flow feature maps of different scales, it can improve the computational efficiency of the deep learning model while keeping the optical flow estimation result of the deep learning model as close to real-time detection as possible.

[0160] by Figure 4aTake the feature pyramid network shown as an example, Figure 6b An optical flow estimation method is provided in an embodiment of the present application. The method describes how to process a first frame image and a second frame image to obtain an optical flow result when the number of layers of a feature pyramid network is greater than 2 (that is, the feature pyramid network contains multiple feature fusion networks).

[0161] In the above embodiment Figure 4a Taking the feature pyramid network shown as an example, the feature pyramid network 40 may include n layers, wherein the 1st to nth layers of the feature pyramid network are respectively provided with pyramid feature networks 410, namely: the 1st layer pyramid feature network, the 2nd layer pyramid feature network, ..., the nth layer pyramid feature network. In one example, the 1st layer pyramid feature network, the 2nd layer pyramid feature network, ..., the nth layer pyramid feature network can be connected in a cascade manner. In addition, the 1st to n-1 layers of the feature pyramid network are respectively provided with feature fusion networks 420, namely the 1st layer feature fusion network, the 2nd layer feature fusion network and the n-1th layer feature fusion network.

[0162] In one example, the method can be processed by a central processing unit (CPU), or by a CPU and a GPU, or by using other processors suitable for neural network calculations instead of a GPU, such as Figure 3e The artificial intelligence processor 30 shown is not limited in this application. The method may include some or all of the following steps:

[0163] Step S600b: acquiring a first frame image and a second frame image, wherein the first frame image and the second frame image contain the target object.

[0164] In the embodiment of the present application, the first frame image and the second frame image can be two consecutive frame images in a video sequence, or two frame images separated by a preset number of frames in a video sequence, which is not specifically limited in the embodiment of the present application. For example, the first frame image can also be called a target frame image, and the second frame image can also be called a reference frame image. Figure 7a and Figure 7b The following are two images captured by the video acquisition device during its motion. Assume that: Figure 7a The first frame is shown. Figure 7b The second frame image is shown.

[0165] In the embodiment of the present application, the first frame image and the second frame image have the same resolution. Here, resolution can be used to describe the ability to resolve image details. Generally, "resolution" can be expressed as the number of pixels in each direction, such as 640*480.

[0166] In an embodiment of the present application, the first frame image and the second frame image may be images containing one or more objects. In the case where the first frame image and the second frame image are images containing multiple objects, the target object may be any one of the multiple objects, or may be a specific object (e.g., a moving person, etc.) among the multiple objects. In one example, in the case where the first frame image and the second frame image contain one object, the object is the target object.

[0167] Step S602b, performing feature extraction of different scales on the first frame image n times through the n-layer network to obtain n first optical flow feature maps; performing feature extraction of different scales on the second frame image n times through the n-layer network to obtain n second optical flow feature maps; wherein the resolution of the optical flow feature map extracted by the n-1th layer of the network is greater than the resolution of the optical flow feature map extracted by the nth layer of the network.

[0168] In the embodiment of the present application, two layers of feature pyramid networks can be used to perform two feature extractions of different scales on the first frame image to obtain two first optical flow feature maps; two layers of feature pyramid networks can be used to perform two feature extractions of different scales on the second frame image to obtain two second optical flow feature maps. Here, the resolution of the optical flow feature map extracted by the upper layer of pyramid feature network is greater than the resolution of the optical flow feature map extracted by the lower layer of pyramid feature network.

[0169] The implementation process of performing n feature extraction on the first frame image through n (n is greater than 2) layers of pyramid feature networks can be described as: performing feature extraction on the first frame image through the first layer of pyramid feature network to obtain the first first optical flow feature map, and then inputting the first first optical flow feature map obtained after downsampling into the second layer of pyramid feature network; the first first optical flow feature map obtained after downsampling can be subjected to feature extraction through the second layer of pyramid feature network to obtain the second first optical flow feature map, and so on, the n-1th first optical flow feature map can be subjected to feature extraction through the nth layer of pyramid feature network to obtain the nth first optical flow feature map.

[0170] The implementation process of performing n feature extractions on the second frame image through n (n is greater than 2) layers of pyramid feature networks can be described as: performing feature extraction on the second frame image through the first layer of pyramid feature network to obtain the first second optical flow feature map, and then inputting the first second optical flow feature map obtained after downsampling into the second layer of pyramid feature network; the first second optical flow feature map obtained after downsampling can be subjected to feature extraction through the second layer of pyramid feature network to obtain the second second optical flow feature map, and so on. The n-1th first optical flow feature map can be subjected to feature extraction through the nth layer of pyramid feature network to obtain the nth first optical flow feature map.

[0171] Here, the resolution of the first optical flow feature map is the same as the resolution of the first second optical flow feature map, and belongs to range 1; the resolution of the second first optical flow feature map is the same as the resolution of the second second optical flow feature map, and belongs to range 2; the resolution of the nth first optical flow feature map is the same as the resolution of the nth second optical flow feature map, and belongs to range n. Taking range 1 as an example, in one example, range 1 is greater than or equal to threshold 1, and range 2 is less than threshold 1. In one example, range 1 is greater than threshold 1, and range 2 is less than or equal to threshold 1.

[0172] In the embodiment of the present application, the image can be processed by 2 times downsampling, or by 4 times downsampling, which is not specifically limited in the embodiment of the present application.

[0173] Step S604b, feature fusion is performed on the first fused feature map corresponding to the n-1 layer network and the n-1th first optical flow feature map and the n-1th second optical flow feature map extracted by the n-1 layer network through the n-1 layer network to obtain the optical flow result and feature output result corresponding to the n-1 layer network; wherein, the first fused feature map corresponding to the n-1 layer network is obtained by feature fusion of the n-th first optical flow feature map and the n-th second optical flow feature map extracted by the n-1 layer network.

[0174] In an embodiment of the present application, the first fused feature map corresponding to the n-1 layer network and the n-1th first optical flow feature map and the n-1th second optical flow feature map extracted by the n-1 layer network are feature-fused through the n-1 layer network to obtain the optical flow result and feature output result corresponding to the n-1 layer network. The implementation process may include: performing feature fusion on the n-1th first optical flow feature map and the n-2nd optical flow feature map extracted by the n-1 layer pyramid feature network through the n-1th layer feature fusion network in the feature pyramid network to obtain the first fused feature map corresponding to the n-1 layer pyramid feature network; performing feature fusion on the first fused feature map corresponding to the n-1 layer pyramid feature network and the n-1th first optical flow feature map and the n-1th second optical flow feature map extracted by the n-1 layer pyramid feature network through the n-1th layer feature fusion network in the feature pyramid network to obtain the optical flow result and feature output result corresponding to the n-1th layer feature fusion network.

[0175] The following specifically describes how the n-1th layer feature fusion network fuses the first fusion feature map corresponding to the nth layer pyramid feature network and the nth first optical flow feature map and the nth second optical flow feature map extracted by the nth layer pyramid feature network. The specific implementation process can be described as:

[0176] Step B1: feature-fusing the n-1th first optical flow feature map and the n-1th second optical flow feature map through the n-1th layer feature fusion network to obtain a first fused image.

[0177] Exemplarily, the implementation of feature fusion may include, but is not limited to: adding pixel points bit by bit and performing vector splicing.

[0178] Specifically, N pixels can be determined as N first feature points from the n-1th first optical flow feature map through the n-1th layer feature fusion network, and N pixels can be determined as N second feature points from the n-1th second optical flow feature map, and the N first feature points and the N second feature points can be matched to obtain the correspondence between the N first feature points and the N second feature points; then, the correlation between the N first feature points and the N second feature points can be obtained according to the correspondence between the N first feature points and the N second feature points through the n-1th layer feature fusion network; thus, the N first feature points and the N second feature points can be feature fused according to the correlation through the n-1th layer feature fusion network to obtain a first fused image.

[0179] Step B2: obtaining a second fused image according to the n-1th second optical flow feature map and the first fused feature map through the n-1th layer feature fusion network, and extracting features of the second fused image to obtain a third fused image.

[0180] Step B3, amplify the third fused image through the n-1 layer feature fusion network to obtain the feature output result corresponding to the n-1 layer feature fusion network. For example, in the n-1 layer feature fusion network, the third fused image can be amplified by 2 times upsampling, or by 4 times upsampling, which is not specifically limited in the embodiment of the present application.

[0181] Step B4: perform feature fusion on the first fused image and the third fused image through the n-1th layer feature fusion network to obtain a fourth fused image.

[0182] Step B5, amplify the fourth fused image through the n-1 layer feature fusion network to obtain the optical flow result corresponding to the n-1 layer feature fusion network. For example, in the n-1 layer feature fusion network, the fourth fused image can be amplified by 2 times upsampling, or by 4 times upsampling, which is not specifically limited in the embodiment of the present application.

[0183] It can be understood that in this implementation, the n-1th layer feature fusion network can be processed according to three input data to obtain the optical flow result corresponding to the n-1th layer feature fusion network, wherein the three input data are: the first fused feature map corresponding to the n-th layer pyramid feature network (the first fused feature map corresponding to the n-th layer network is obtained by feature fusion of the n-th first optical flow feature map and the n-th second optical flow feature map extracted from the n-th layer network), the n-1th first optical flow feature map and the n-1th second optical flow feature map extracted from the n-1th layer pyramid feature network.

[0184] It should be noted that, in one example, the noise in the fourth fused image can be removed by the bilateral filtering layer in the n-1th layer feature fusion network and the boundary information of the fourth fused image can be retained to obtain the fifth fused image. Then, in this case, the fifth fused image is enlarged by the n-1th layer feature fusion network (for example, the fifth fused image is enlarged by 2 times upsampling in the n-1th layer feature fusion network) to obtain the optical flow result. Figure 8a , Figure 8bAs shown, when a bilateral filter layer is added to the n-1th feature fusion network, the smoothness and consistency of the optical flow estimation result obtained by the optical flow estimation method described in this application are greatly improved. For example, when a bilateral filter layer is added to the optical flow estimation method, the average pixel error (Estimated Pixel Error, EPE) of the obtained optical flow result compared with the true reference data (ground truth) is 0.361; when the bilateral filter layer is not added to the optical flow estimation method, the average pixel error (Estimated Pixel Error, EPE) of the obtained optical flow result compared with the true reference data is 0.361. By implementing the embodiment of the present application, since high-frequency noise in the optical flow feature map can be removed while retaining boundary information, this implementation method can improve the model accuracy of the deep learning model and improve the robustness of the deep learning model to noise.

[0185] It should also be noted that for steps B1-B5, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should know that the present disclosure is not limited to the order of the actions described, because according to the present application, certain steps can be performed in other orders or simultaneously, which should not constitute a limitation here.

[0186] Step S606b, perform feature fusion on the optical flow result and feature output result corresponding to the i-th layer network, as well as the i-1th first optical flow feature map and the i-1th second optical flow feature map extracted by the i-1th layer network through the i-1th layer network, to obtain the optical flow result output by the i-1th layer network; i takes values ​​from n-1 to 2 in sequence; the feature output result corresponding to the i-th layer network is determined based on the i-th second optical flow feature map and the feature output result corresponding to the i+1th layer network.

[0187] Taking the n-2th layer (that is, when i=n-1) feature fusion network as an example, the optical flow results and feature output results corresponding to the n-1th layer feature fusion network, and the n-2th first optical flow feature map and the n-2th second optical flow feature map extracted by the n-2th layer pyramid feature network are feature fused through the n-2th layer feature fusion network to obtain the optical flow results output by the n-2th layer feature fusion network. Here, the optical flow results and feature output results corresponding to the n-1th layer feature fusion network can be obtained through step S604b.

[0188] Taking the n-3th layer (that is, when i=n-3) feature fusion network as an example, the optical flow results and feature output results corresponding to the n-2th layer feature fusion network, as well as the n-3th first optical flow feature map and the n-3th second optical flow feature map extracted by the n-3th layer pyramid feature network are feature fused through the n-3th layer feature fusion network to obtain the optical flow results output by the n-3th layer feature fusion network. Here, for the specific implementation process of how to obtain the optical flow results and feature output results corresponding to the n-2th layer feature fusion network, please refer to the above description, and no further details will be given here.

[0189] Taking the first layer (that is, when i=2) feature fusion network as an example, the optical flow results and feature output results corresponding to the second layer feature fusion network, as well as the first optical flow feature map and the first second optical flow feature map extracted by the first layer pyramid feature network are feature fused through the first layer feature fusion network to obtain the optical flow results output by the first layer feature fusion network. Here, for the specific implementation process of how to obtain the optical flow results and feature output results corresponding to the second layer feature fusion network, please refer to the above description, and no further details will be given here.

[0190] It can be understood that in this implementation, for feature fusion networks other than the n-1th layer feature fusion network, taking the n-2th layer feature fusion network as an example, the n-2th layer feature fusion network can be processed according to 4 input data to obtain the optical flow result corresponding to the n-2th layer feature fusion network, wherein the 4 input data are: the optical flow result and feature output result corresponding to the n-1th layer network, and the n-2th first optical flow feature map and the n-2th second optical flow feature map extracted by the n-2th layer pyramid feature network.

[0191] Thus, the optical flow estimation device can obtain the optical flow result through the feature fusion network of the n-1th layer, the feature fusion network of the n-2th layer, ..., and the feature fusion network of the 1st layer in the pyramid feature network.

[0192] It should be noted that, when the pyramid feature network includes a feature fusion network of the n-1th layer, a feature fusion network of the n-2th layer, ..., and a feature fusion network of the 1st layer, for one of the feature fusion networks, the noise in the fourth fused image can be removed by the bilateral filtering layer in the network and the boundary information of the fourth fused image can be retained to obtain a fifth fused image; for multiple feature fusion networks, each feature fusion network can remove the noise in the fourth fused image and retain the boundary information of the fourth fused image by the bilateral filtering layer in the network to obtain a fifth fused image, which is not specifically limited in the embodiments of the present application.

[0193] It is understandable that the optical flow estimation device can perform feature fusion on the optical flow feature maps of different scales corresponding to the first frame image and the second frame image through multiple feature fusion networks to obtain the optical flow result. Compared with the existing implementation method, since the parameter scale of the multiple feature fusion networks is small, and the multiple feature fusion networks can perform feature fusion on the optical flow feature maps of different scales corresponding to the two frames of images, the different semantic information contained in the optical flow feature maps of different scales is fully considered, and the optical flow estimation result of the deep learning model can be kept as close to real-time detection as possible while improving the computational efficiency of the deep learning model.

[0194] Step S608b, outputting the optical flow result corresponding to the first layer of the network; wherein the optical flow result is used to characterize the motion information of the target object.

[0195] In the embodiment of the present application, the output method includes but is not limited to output through text, image, voice, video, etc.

[0196] Figure 8c This is a visualization diagram of the optical flow result obtained by the optical flow estimation method according to an embodiment of the present application. After testing, Figure 8c The average pixel error EPE of image (1) is 13.7%, that is, the motion of 13.7% of the pixels in the entire image is inconsistent with the value of the true reference data; Figure 8c The average pixel error EPE of the middle image (2) is 5.54%, which means that the motion of 5.54% of the pixels in the whole image is inconsistent with the value of the real reference data; Figure 8c The average pixel error EPE of the middle image (3) is 4.92%, that is, the motion of 4.92% of the pixels in the whole image is inconsistent with the value of the real reference data. It can be understood that the optical flow result obtained by the optical flow estimation method described in this application has high accuracy.

[0197] It should be noted that in order to achieve the goal of near real-time optical flow estimation, the entire process can be accelerated by a graphics processing unit (GPU) in the embodiment of the present application. After acceleration, the performance can be improved by about 20 times, thereby achieving real-time optical flow estimation.

[0198] In implementing the embodiment of the present application, when determining the optical flow results of the first frame image and the second frame image, the optical flow estimation device performs feature extraction of multiple scales on the first frame image and the second frame image respectively to obtain multiple optical flow feature maps of different scales corresponding to the two frames of images, and then performs feature fusion on the multiple optical flow feature maps of different scales corresponding to the two frames of images, thereby obtaining the optical flow result. Since this implementation method fully considers the different semantic information contained in the multiple optical flow feature maps of different scales, it can improve the computational efficiency of the deep learning model while keeping the optical flow estimation result of the deep learning model as close to real-time detection as possible.

[0199] The above embodiments focus on how to obtain the optical flow results of two frames of images. The following specifically describes how to obtain the video completion results based on the optical flow results in conjunction with a specific example of video completion. Figure 9a As shown, the video completion architecture 90 includes a feature pyramid network 900 and a video completion network 902, wherein the pyramid feature network 900 is used to obtain the optical flow results corresponding to each of the multiple image pairs in the video sequence; the video completion network 902 is used to obtain the video completion results of the video sequence based on the optical flow results of the multiple image pairs and the area to be completed.

[0200] In the embodiment of the present application, please refer to the above description for the specific implementation of the feature pyramid network 900. The video completion network 902 is constructed based on a convolutional neural network. For example, the specific structure of the video completion network 902 can be as follows: Figure 9b As shown, the RF completion network may include an input layer 910, a convolution layer 920, a conv-LSTM layer 930, a fully connected layer 940, and an output layer 950. Specifically, the input layer 910 is used to receive input data, and specifically, the input data may include an input image and annotation information of the area to be completed in the input image; the convolution layer 920 is used to extract features of the input data, for example, when the input data is an image, the convolution layer 920 is used to extract features of the input image to reduce the parameters brought by the input image; the conv-LSTM layer 930 is used to obtain features with a temporal relationship; the fully connected layer 940 is used to integrate local information with category distinction in the conv-LSTM layer 930 (or pooling layer), for example, the fully connected layer 940 can connect the features with a temporal relationship extracted by the conv-LSTM layer 930. In practical applications, in order to improve the network performance of the video completion network 902, the excitation function of each neuron in the fully connected layer 940 generally adopts a ReLU function. The output value of the last fully connected layer 940 is passed to an output, so that the processing result can be output through the output layer 950.

[0201] Specifically, a user may use a computer device (for example, the computer device includes a video acquisition device) to shoot a moving object (for example, Xiao Ming walks toward a window, that is, a target object) to obtain a video sequence. In order to remove the image of "Xiao Ming walking" in the video, the computer device may use the optical flow estimation method described in the present application to obtain the optical flow results corresponding to two adjacent frames of video frame images. For example, the video frame images include frame n-2 images, frame n-1 images, frame n images, frame n+1 images, and frame n+2 images (here, frame n-2 images, frame n-1 images, frame n images, frame n+1 images, and frame n+2 images are images of "Xiao Ming walking" extracted by the computer device in the video sequence). Fig.9c As shown, the computer device can sequentially obtain the optical flow results corresponding to two adjacent frames of images through the feature pyramid network 900. For example, after processing the frame n-2 image and the frame n-1 image through the feature pyramid network 900, the optical flow result n-1 can be obtained; after processing the frame n-1 image and the frame n image through the feature pyramid network 900, the optical flow result n can be obtained; after processing the frame n image and the frame n+1 image through the feature pyramid network 900, the optical flow result n+1 can be obtained; after processing the frame n+1 image and the frame n+2 image through the feature pyramid network 900, the optical flow result n+2 can be obtained. Afterwards, the computer device can obtain the video completion result of the frame n image according to the to-be-completed area in the frame n image and the optical flow results corresponding to each two adjacent frames of images (optical flow result n-1, optical flow result n, optical flow result n+1 and optical flow result n+2) through the video completion network 902. For example, the video completion result of the frame n image is a screen image with "Xiao Ming in a walking state" removed. It can be understood that, through this implementation, specific screens can be removed to meet the personalized needs of users.

[0202] Combination of the above Figure 1-9c The optical flow estimation method involved in the embodiment of the present application is described in detail, and the device involved in the embodiment of the present application is introduced below in conjunction with the accompanying drawings.

[0203] Fig.10a Schematic block diagram of an optical flow estimation device 100a in an embodiment of the present application. Fig.10a The optical flow estimation device 100a shown may include:

[0204] An image acquisition module 1000a is used to acquire a first frame image and a second frame image; wherein the first frame image and the second frame image contain a target object;

[0205] The pyramid feature module 1002a is used to perform two feature extractions of different scales on the first frame image through the two-layer network to obtain two first optical flow feature maps; perform two feature extractions of different scales on the second frame image through the two-layer network to obtain two second optical flow feature maps; wherein the resolution of the optical flow feature map extracted by the first layer of the network is greater than the resolution of the optical flow feature map extracted by the second layer of the network;

[0206] The feature fusion module 1004a is used to perform feature fusion on the first fusion feature map corresponding to the second layer network and the first first optical flow feature map and the first second optical flow feature map extracted by the first layer network through the first layer network to obtain the optical flow result corresponding to the first layer network; wherein the first fusion feature map corresponding to the second layer network is obtained by performing feature fusion on the second first optical flow feature map and the second second optical flow feature map extracted by the second layer network;

[0207] The output module 1006a is used to output the optical flow result corresponding to the first layer network; wherein the optical flow result is used to characterize the motion information of the target object.

[0208] In some possible implementations, the feature pyramid network includes a pyramid feature network and a feature fusion network; the first and second layers of the feature pyramid network are respectively provided with the pyramid feature networks; the first layer of the feature pyramid network is provided with the feature fusion network; the pyramid feature network is used to extract features from the first frame image and the second frame image; the feature fusion network is used to fuse the optical flow feature map extracted by the pyramid feature network; the feature fusion module 1004a includes a first feature fusion module, and the first feature fusion module is used to:

[0209] The first layer feature fusion network in the feature pyramid network performs feature fusion on the second first optical flow feature map extracted by the second layer pyramid feature network and the second second optical flow feature map to obtain a first fused feature map corresponding to the second layer pyramid feature network;

[0210] The first fusion feature map corresponding to the second layer pyramid feature network and the first first optical flow feature map and the first second optical flow feature map extracted by the first layer pyramid feature network are feature fused through the first layer feature fusion network in the feature pyramid network to obtain the optical flow result corresponding to the first layer feature fusion network.

[0211] In a possible implementation, the optical flow feature map extracted by the second-layer pyramid feature network includes the second first optical flow feature map and the second second optical flow feature map; the optical flow feature map extracted by the first-layer pyramid feature network includes the first first optical flow feature map and the first second optical flow feature map; the resolution of the second first optical flow feature map and the resolution of the second second optical flow feature map belong to a first range; the resolution of the first first optical flow feature map and the resolution of the first second optical flow feature map belong to a second range, and the first range and the second range do not overlap; the first feature fusion module is specifically used to:

[0212] Performing feature fusion on the first optical flow feature map and the first second optical flow feature map through the first layer feature fusion network to obtain a first fused image;

[0213] Obtaining a second fused image according to the first second optical flow feature map and the first fused feature map through the first layer feature fusion network, and extracting features of the second fused image to obtain a third fused image;

[0214] Performing feature fusion on the first fused image and the third fused image through the first-layer feature fusion network to obtain a fourth fused image;

[0215] The fourth fused image is enlarged by the first-layer feature fusion network to obtain an optical flow result corresponding to the first-layer feature fusion network.

[0216] In a possible implementation, the first feature fusion module is specifically configured to:

[0217] Determine, by the first layer feature fusion network, N pixels from the first first optical flow feature map as N first feature points, determine N pixels from the first second optical flow feature map as N second feature points, match the N first feature points with the N second feature points, and obtain a correspondence between the N first feature points and the N second feature points;

[0218] Obtaining, by the first-layer feature fusion network, the correlation between the N first feature points and the N second feature points according to the corresponding relationship between the N first feature points and the N second feature points;

[0219] The first fused image is obtained by performing feature fusion on the N first feature points and the N second feature points according to the correlation through the first-layer feature fusion network.

[0220] In a possible implementation, the first feature fusion module is further configured to:

[0221] Removing noise in the fourth fused image and retaining boundary information of the fourth fused image through the first-layer feature fusion network to obtain a fifth fused image;

[0222] The first feature fusion module is specifically used for:

[0223] The fifth fused image is enlarged through the first-layer feature fusion network to obtain an optical flow result corresponding to the first-layer feature fusion network.

[0224] In the embodiments of the present application, the specific implementation of each unit can refer to the relevant description in the above embodiments, which will not be repeated here.

[0225] In implementing the embodiment of the present application, when determining the optical flow results of the first frame image and the second frame image, the optical flow estimation device performs feature extraction of the first frame image and the second frame image multiple times at different scales to obtain multiple optical flow feature maps of different scales corresponding to the two frames of images, and then performs feature fusion on the multiple optical flow feature maps of different scales corresponding to the two frames of images, thereby obtaining the optical flow result. Since this implementation method fully considers the different semantic information contained in the multiple optical flow feature maps of different scales, it can improve the computational efficiency of the deep learning model while keeping the optical flow estimation result of the deep learning model as close to real-time detection as possible.

[0226] Fig.10b Schematic block diagram of an optical flow estimation device 100b in an embodiment of the present application. Fig.10b The optical flow estimation device 100b shown may include:

[0227] An image acquisition module 1000b is used to acquire a first frame image and a second frame image; wherein the first frame image and the second frame image contain a target object;

[0228] The pyramid feature module 1002b is used to perform feature extraction of different scales on the first frame image n times through the n-layer network to obtain n first optical flow feature maps; perform feature extraction of different scales on the second frame image n times through the n-layer network to obtain n second optical flow feature maps; wherein the resolution of the optical flow feature map extracted by the n-1-th layer network is greater than the resolution of the optical flow feature map extracted by the n-th layer network;

[0229] The feature fusion module 1004b is used to perform feature fusion on the first fusion feature map corresponding to the n-th layer network and the n-1th first optical flow feature map and the n-1th second optical flow feature map extracted by the n-1th layer network through the n-1th layer network, so as to obtain the optical flow result and feature output result corresponding to the n-1th layer network; wherein, the first fusion feature map corresponding to the n-th layer network is obtained by performing feature fusion on the n-th first optical flow feature map and the n-th second optical flow feature map extracted by the n-th layer network; perform feature fusion on the optical flow result and feature output result corresponding to the i-th layer network through the i-1th layer network, and the i-1th first optical flow feature map and the i-1th second optical flow feature map extracted by the i-1th layer network, so as to obtain the optical flow result output by the i-1th layer network; the i is taken from n-1 to 2 in sequence; the feature output result corresponding to the i-th layer network is determined according to the i-th second optical flow feature map and the feature output result corresponding to the i+1th layer network;

[0230] The output module 1006b is used to output the optical flow result corresponding to the first layer of the network; wherein the optical flow result is used to characterize the motion information of the target object.

[0231] In a possible implementation, the feature pyramid network includes a pyramid feature network and a feature fusion network; the 1st to nth layers of the feature pyramid network are respectively provided with the pyramid feature network; the 1st to n-1th layers of the feature pyramid network are respectively provided with the feature fusion network; the feature fusion module 1004b includes the n-1th feature fusion module and the i-1th feature fusion module; wherein,

[0232] The n-1th feature fusion module is used to: perform feature fusion on the nth first optical flow feature map and the nth second optical flow feature map extracted by the nth layer pyramid feature network through the n-1th layer feature fusion network in the feature pyramid network, so as to obtain a first fused feature map corresponding to the nth layer pyramid feature network;

[0233] The first fusion feature map corresponding to the n-th layer pyramid feature network and the n-1th first optical flow feature map and the n-1th second optical flow feature map extracted by the n-1th layer pyramid feature network are subjected to feature fusion through the n-1th layer feature fusion network in the feature pyramid network to obtain an optical flow result and a feature output result corresponding to the n-1th layer feature fusion network;

[0234] The i-1th feature fusion module is used to: perform feature fusion on the optical flow result and feature output result corresponding to the i-th layer network, as well as the i-1th first optical flow feature map and the i-1th second optical flow feature map extracted by the i-1th layer network through the i-1th layer feature fusion network in the feature pyramid network, to obtain the optical flow result output by the i-1th layer network; the i is taken from n-1 to 2 in sequence; the feature output result corresponding to the i-th layer network is determined based on the i-th second optical flow feature map and the feature output result corresponding to the i+1th layer network.

[0235] In a possible implementation, the optical flow feature map extracted by the n-th layer pyramid feature network includes the n-th first optical flow feature map and the n-th second optical flow feature map; the optical flow feature map extracted by the n-1-th layer pyramid feature network includes the n-1-th first optical flow feature map and the n-1-th second optical flow feature map; the resolution of the n-th first optical flow feature map and the resolution of the n-th second optical flow feature map belong to a first range; the resolution of the n-1-th first optical flow feature map and the resolution of the n-1-th second optical flow feature map belong to a second range, and the first range and the second range do not overlap; the n-1-th feature fusion module is specifically used to:

[0236] Performing feature fusion on the n-1th first optical flow feature map and the n-1th second optical flow feature map through the n-1th layer feature fusion network to obtain a first fused image;

[0237] Obtaining a second fused image according to the n-1th second optical flow feature map and the first fused feature map through the n-1th layer feature fusion network, and extracting features of the second fused image to obtain a third fused image;

[0238] Amplifying the third fused image through the n-1th layer feature fusion network to obtain a feature output result corresponding to the n-1th layer feature fusion network;

[0239] Performing feature fusion on the first fused image and the third fused image through the n-1th layer feature fusion network to obtain a fourth fused image;

[0240] The fourth fused image is enlarged by the n-1th layer feature fusion network to obtain an optical flow result corresponding to the n-1th layer feature fusion network.

[0241] In a possible implementation, the n-1th feature fusion module is specifically used to:

[0242] Determining N pixels from the n-1th first optical flow feature map as N first feature points through the n-1th layer feature fusion network, determining N pixels from the n-1th second optical flow feature map as N second feature points, matching the N first feature points with the N second feature points, and obtaining a corresponding relationship between the N first feature points and the N second feature points;

[0243] Obtaining, through the feature fusion network of the n-1th layer, the correlation between the N first feature points and the N second feature points according to the corresponding relationship between the N first feature points and the N second feature points;

[0244] The first fused image is obtained by performing feature fusion on the N first feature points and the N second feature points according to the correlation through the n-1th layer feature fusion network.

[0245] In a possible implementation, the n-1th feature fusion module is further used to:

[0246] Removing noise from the fourth fused image and retaining boundary information of the fourth fused image through the n-1th layer feature fusion network to obtain a fifth fused image;

[0247] The n-1th feature fusion module is specifically used for:

[0248] The fifth fused image is enlarged by the n-1th layer feature fusion network to obtain an optical flow result corresponding to the n-1th layer feature fusion network.

[0249] In the embodiments of the present application, the specific implementation of each unit can refer to the relevant description in the above embodiments, which will not be repeated here.

[0250] In implementing the embodiment of the present application, when determining the optical flow results of the first frame image and the second frame image, the optical flow estimation device performs feature extraction of the first frame image and the second frame image multiple times at different scales to obtain multiple optical flow feature maps of different scales corresponding to the two frames of images, and then performs feature fusion on the multiple optical flow feature maps of different scales corresponding to the two frames of images, thereby obtaining the optical flow result. Since this implementation method fully considers the different semantic information contained in the multiple optical flow feature maps of different scales, it can improve the computational efficiency of the deep learning model while keeping the optical flow estimation result of the deep learning model as close to real-time detection as possible.

[0251] like Fig.11As shown, an optical flow estimation device 110 provided in an embodiment of the present application may include a processor 1101, a memory 1102, a communication bus 1103 and a communication interface 1104, and the processor 1101 is connected to the memory 1102 and the communication interface 1103 via the communication bus.

[0252] Processor 1101 can adopt a general central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), a graphics processing unit (GPU), a neural network processor (NPU) or one or more integrated circuits to execute relevant programs to execute the optical flow estimation method described in the method embodiment of the present application.

[0253] The processor 1101 may also be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the optical flow estimation method of the present application may be completed by an integrated logic circuit of hardware or software instructions in the processor 1101. The above-mentioned processor 1101 may also be a general-purpose processor, a digital signal processor (DigitalSignal Processing, DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (Field ProgrammableGate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present application may be directly embodied as being executed by a hardware decoding processor, or may be executed by a combination of hardware and software modules in a decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 1102 , and the processor 1101 reads the information in the memory 1102 and executes the optical flow estimation method of the method embodiment of the present application in combination with its hardware.

[0254] The memory 1102 may be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1102 may store programs and data, such as the program of the optical flow estimation method in the embodiment of the present application. When the program stored in the memory 1102 is executed by the processor 1101, the processor 1101 and the communication interface 1104 are used to execute the various steps of the optical flow estimation method in the embodiment of the present application.

[0255] For example, a program used to implement the optical flow estimation method in an embodiment of the present application, etc.

[0256] The communication interface 1104 uses a transceiver such as, but not limited to, a transceiver to implement communication between the optical flow estimation device 110 and other devices or communication networks. For example, a trained neural network can be obtained through the communication interface 1104 to implement information interaction with an execution device, a client device, a user device, or a terminal device.

[0257] Optionally, the optical flow estimation device may also include an artificial intelligence processor 1105, which may be a neural network processor (Network Processing Unit, NPU), a tensor processor (TensorProcessing Unit, TPU), or a graphics processor (Graphics Processing Unit, GPU) and other processors suitable for large-scale XOR operation processing. The artificial intelligence processor 1105 can be mounted on the host CPU (Host CPU) as a coprocessor, and the host CPU assigns tasks to it. The artificial intelligence processor 1105 can implement one or more operations involved in the above-mentioned optical flow estimation method. For example, taking NPU as an example, the core part of the NPU is the operation circuit, which controls the operation circuit through the controller to extract the matrix data in the memory 1102 and perform multiplication and addition operations.

[0258] The processor 1101 is used to call the data and program code in the memory and execute:

[0259] Acquire a first frame image and a second frame image; wherein the first frame image and the second frame image contain a target object;

[0260] Performing feature extraction of different scales twice on the first frame image through the two-layer network to obtain two first optical flow feature maps; performing feature extraction of different scales twice on the second frame image through the two-layer network to obtain two second optical flow feature maps; wherein the resolution of the optical flow feature map extracted by the first layer of the network is greater than the resolution of the optical flow feature map extracted by the second layer of the network;

[0261] The first fusion feature map corresponding to the second layer network and the first first optical flow feature map and the first second optical flow feature map extracted by the first layer network are subjected to feature fusion through the first layer network to obtain the optical flow result corresponding to the first layer network; wherein the first fusion feature map corresponding to the second layer network is obtained by performing feature fusion on the second first optical flow feature map and the second second optical flow feature map extracted by the second layer network;

[0262] Output the optical flow result corresponding to the first layer of the network; wherein the optical flow result is used to characterize the motion information of the target object.

[0263] The feature pyramid network includes a pyramid feature network and a feature fusion network; the first to second layers of the feature pyramid network are respectively provided with the pyramid feature network; the first layer of the feature pyramid network is provided with the feature fusion network; the pyramid feature network is used to extract features from the first frame image and the second frame image; the feature fusion network is used to fuse the optical flow feature map extracted by the pyramid feature network; the processor 1101 performs feature fusion on the fusion feature map corresponding to the second layer network and the first first optical flow feature map and the first second optical flow feature map extracted by the first layer network through the first layer network to obtain the optical flow result corresponding to the first layer network, which may include:

[0264] The first layer feature fusion network in the feature pyramid network performs feature fusion on the second first optical flow feature map extracted by the second layer pyramid feature network and the second second optical flow feature map to obtain a first fused feature map corresponding to the second layer pyramid feature network;

[0265] The first fusion feature map corresponding to the second layer pyramid feature network and the first first optical flow feature map and the first second optical flow feature map extracted by the first layer pyramid feature network are feature fused through the first layer feature fusion network in the feature pyramid network to obtain the optical flow result corresponding to the first layer feature fusion network.

[0266] Among them, the optical flow feature map extracted by the second-layer pyramid feature network includes the second first optical flow feature map and the second second optical flow feature map; the optical flow feature map extracted by the first-layer pyramid feature network includes the first first optical flow feature map and the first second optical flow feature map; the resolution of the second first optical flow feature map and the resolution of the second second optical flow feature map belong to the first range; the resolution of the first first optical flow feature map and the resolution of the first second optical flow feature map belong to the second range, and the first range and the second range do not overlap; the processor 1101 performs feature fusion on the fusion feature map corresponding to the second-layer pyramid feature network and the first first optical flow feature map and the first second optical flow feature map extracted by the first-layer pyramid feature network through the first-layer feature fusion network in the feature pyramid network to obtain the optical flow result corresponding to the first-layer network, which may include:

[0267] Performing feature fusion on the first optical flow feature map and the first second optical flow feature map through the first layer feature fusion network to obtain a first fused image;

[0268] Obtaining a second fused image according to the first second optical flow feature map and the first fused feature map through the first layer feature fusion network, and extracting features of the second fused image to obtain a third fused image;

[0269] Performing feature fusion on the first fused image and the third fused image through the first-layer feature fusion network to obtain a fourth fused image;

[0270] The fourth fused image is enlarged by the first-layer feature fusion network to obtain an optical flow result corresponding to the first-layer feature fusion network.

[0271] The processor 1101 performs feature fusion on the first optical flow feature map and the first second optical flow feature map through the first layer feature fusion network to obtain a first fused image, which may include:

[0272] Determine, by the first layer feature fusion network, N pixels from the first first optical flow feature map as N first feature points, determine N pixels from the first second optical flow feature map as N second feature points, match the N first feature points with the N second feature points, and obtain a correspondence between the N first feature points and the N second feature points;

[0273] Obtaining, by the first-layer feature fusion network, the correlation between the N first feature points and the N second feature points according to the corresponding relationship between the N first feature points and the N second feature points;

[0274] The first fused image is obtained by performing feature fusion on the N first feature points and the N second feature points according to the correlation through the first-layer feature fusion network.

[0275] Wherein, after the processor 1101 performs feature fusion on the first fused image and the third fused image through the first-layer feature fusion network to obtain a fourth fused image, the processor 1101 amplifies the fourth fused image through the first-layer feature fusion network to obtain the optical flow result corresponding to the first-layer feature fusion network, and may further include:

[0276] Removing noise in the fourth fused image and retaining boundary information of the fourth fused image through the first-layer feature fusion network to obtain a fifth fused image;

[0277] The processor 1101 amplifies the fourth fused image through the first-layer feature fusion network to obtain an optical flow result corresponding to the first-layer feature fusion network, which may include:

[0278] The fifth fused image is enlarged through the first-layer feature fusion network to obtain an optical flow result corresponding to the first-layer feature fusion network.

[0279] It should be understood that the implementation of each device may also refer to the corresponding description in the above optical flow estimation method embodiment, and the embodiments of the present application will not be repeated.

[0280] In an optional embodiment, Fig.11 The processor 1101 in the optical flow estimation device shown can also be used to read the computer program code stored in the memory 1102, so that the optical flow estimation device implements the method described in any one of the second aspects above.

[0281] The embodiment of the present invention further provides a computer storage medium, in which instructions are stored, and when the instructions are executed on a computer or a processor, the computer or the processor executes one or more steps of the method described in any of the above embodiments. If the components of the above device are implemented in the form of software functional units and sold or used as independent products, they can be stored in the computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer product is stored in a computer-readable storage medium.

[0282] The computer-readable storage medium may be an internal storage unit of the device described in the foregoing embodiments, such as a hard disk or a memory. The computer-readable storage medium may also be an external storage device of the device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Further, the computer-readable storage medium may also include both an internal storage unit of the device and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by the device. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.

[0283] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk and other media that can store program codes.

[0284] The steps in the method of the embodiment of the present application can be adjusted in order, combined and deleted according to actual needs.

[0285] The modules in the device of the embodiment of the present application can be merged, divided and deleted according to actual needs.

[0286] It is to be understood that those of ordinary skill in the art can appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the various embodiments of the present application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0287] Those skilled in the art will appreciate that the functions described in the various illustrative logic blocks, modules, and algorithm steps disclosed in the various embodiments of the present application can be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions described in the various illustrative logic blocks, modules, and steps can be stored or transmitted as one or more instructions or codes on a computer-readable medium and executed by a hardware-based processing unit. Computer-readable media may include computer-readable storage media, which corresponds to tangible media, such as data storage media, or includes any communication media that facilitates the transfer of computer programs from one place to another (e.g., according to a communication protocol). In this way, computer-readable media can generally correspond to (1) non-temporary tangible computer-readable storage media, or (2) communication media, such as signals or carrier waves. Data storage media can be any available media that can be accessed by one or more computers or one or more processors to retrieve instructions, codes, and / or data structures for implementing the technology described in the present application. A computer program product may include a computer-readable medium.

[0288] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0289] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0290] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0291] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0292] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0293] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An optical flow estimation method, characterized in that: The method is applied to a feature pyramid network, the feature pyramid network includes a 2-layer network, and the method includes: Acquire a first frame image and a second frame image; wherein the first frame image and the second frame image contain a target object; Performing feature extraction of different scales twice on the first frame image through the two-layer network to obtain two first optical flow feature maps; performing feature extraction of different scales twice on the second frame image through the two-layer network to obtain two second optical flow feature maps; wherein the resolution of the optical flow feature map extracted by the first layer of the network is greater than the resolution of the optical flow feature map extracted by the second layer of the network; The first fusion feature map corresponding to the second layer network and the first first optical flow feature map and the first second optical flow feature map extracted by the first layer network are subjected to feature fusion through the first layer network to obtain the optical flow result corresponding to the first layer network; wherein the first fusion feature map corresponding to the second layer network is obtained by performing feature fusion on the second first optical flow feature map and the second second optical flow feature map extracted by the second layer network; Outputting the optical flow result corresponding to the first layer network; wherein the optical flow result is used to characterize the motion information of the target object; The feature pyramid network includes a pyramid feature network and a feature fusion network; the first to second layers of the feature pyramid network are respectively provided with the pyramid feature network; the first layer of the feature pyramid network is provided with the feature fusion network; the pyramid feature network is used to extract features from the first frame image and the second frame image; the feature fusion network is used to fuse the optical flow feature map extracted by the pyramid feature network; the fusion feature map corresponding to the second layer network and the first first optical flow feature map and the first second optical flow feature map extracted by the first layer network are feature-fused through the first layer network to obtain the optical flow result corresponding to the first layer network, including: The first layer feature fusion network in the feature pyramid network performs feature fusion on the second first optical flow feature map extracted by the second layer pyramid feature network and the second second optical flow feature map to obtain a first fused feature map corresponding to the second layer pyramid feature network; The first fusion feature map corresponding to the second layer pyramid feature network and the first first optical flow feature map and the first second optical flow feature map extracted by the first layer pyramid feature network are feature fused through the first layer feature fusion network in the feature pyramid network to obtain the optical flow result corresponding to the first layer feature fusion network.

2. The method according to claim 1, characterized in that The optical flow feature map extracted by the second-layer pyramid feature network includes the second first optical flow feature map and the second second optical flow feature map; the optical flow feature map extracted by the first-layer pyramid feature network includes the first first optical flow feature map and the first second optical flow feature map; the resolution of the second first optical flow feature map and the resolution of the second second optical flow feature map belong to the first range; the resolution of the first first optical flow feature map and the resolution of the first second optical flow feature map belong to the second range, and the first range and the second range do not overlap; the first-layer feature fusion network in the feature pyramid network performs feature fusion on the fusion feature map corresponding to the second-layer pyramid feature network and the first first optical flow feature map and the first second optical flow feature map extracted by the first-layer pyramid feature network to obtain the optical flow result corresponding to the first-layer network, including: Performing feature fusion on the first optical flow feature map and the first second optical flow feature map through the first layer feature fusion network to obtain a first fused image; Obtaining a second fused image according to the first second optical flow feature map and the first fused feature map through the first layer feature fusion network, and extracting features of the second fused image to obtain a third fused image; Performing feature fusion on the first fused image and the third fused image through the first-layer feature fusion network to obtain a fourth fused image; The fourth fused image is enlarged by the first-layer feature fusion network to obtain an optical flow result corresponding to the first-layer feature fusion network.

3. The method according to claim 2, characterized in that The step of fusing the first optical flow feature map and the first second optical flow feature map through the first layer feature fusion network to obtain a first fused image includes: Determine, by the first layer feature fusion network, N pixels from the first first optical flow feature map as N first feature points, determine N pixels from the first second optical flow feature map as N second feature points, match the N first feature points with the N second feature points, and obtain a correspondence between the N first feature points and the N second feature points; Obtaining, by the first-layer feature fusion network, the correlation between the N first feature points and the N second feature points according to the corresponding relationship between the N first feature points and the N second feature points; The first fused image is obtained by performing feature fusion on the N first feature points and the N second feature points according to the correlation through the first-layer feature fusion network.

4. The method according to claim 2, characterized in that After the first-layer feature fusion network is used to perform feature fusion on the first fused image and the third fused image to obtain a fourth fused image, and before the first-layer feature fusion network is used to amplify the fourth fused image to obtain an optical flow result corresponding to the first-layer feature fusion network, the method further includes: Removing noise in the fourth fused image and retaining boundary information of the fourth fused image through the first-layer feature fusion network to obtain a fifth fused image; The enlarging the fourth fused image by the first-layer feature fusion network to obtain an optical flow result corresponding to the first-layer feature fusion network includes: The fifth fused image is enlarged through the first-layer feature fusion network to obtain an optical flow result corresponding to the first-layer feature fusion network.

5. An optical flow estimation method, characterized in that: The method is applied to a feature pyramid network, the feature pyramid network includes an n-layer network, n is a positive integer greater than 2, and the method includes: Acquire a first frame image and a second frame image; wherein the first frame image and the second frame image contain a target object; Performing feature extraction of different scales on the first frame image n times through the n-layer network to obtain n first optical flow feature maps; performing feature extraction of different scales on the second frame image n times through the n-layer network to obtain n second optical flow feature maps; wherein the resolution of the optical flow feature map extracted by the n-1-th layer network is greater than the resolution of the optical flow feature map extracted by the n-th layer network; The first fused feature map corresponding to the n-th layer network and the n-1th first optical flow feature map and the n-1th second optical flow feature map extracted by the n-1th layer network are subjected to feature fusion through the n-1th layer network to obtain the optical flow result and feature output result corresponding to the n-1th layer network; wherein the first fused feature map corresponding to the n-th layer network is obtained by performing feature fusion on the n-th first optical flow feature map and the n-th second optical flow feature map extracted by the n-th layer network; The optical flow result and feature output result corresponding to the i-th layer network, as well as the i-1th first optical flow feature map and the i-1th second optical flow feature map extracted by the i-1th layer network are feature-fused through the i-1th layer network to obtain the optical flow result output by the i-1th layer network; the i is sequentially taken from n-1 to 2; the feature output result corresponding to the i-th layer network is determined according to the i-th second optical flow feature map and the feature output result corresponding to the i+1th layer network; Outputting the optical flow result corresponding to the first layer of the network; wherein the optical flow result is used to characterize the motion information of the target object; The feature pyramid network includes a pyramid feature network and a feature fusion network; the 1st to nth layers of the feature pyramid network are respectively provided with the pyramid feature network; the 1st to n-1th layers of the feature pyramid network are respectively provided with the feature fusion network; the first fusion feature map corresponding to the nth layer network and the n-1th first optical flow feature map and the n-1th second optical flow feature map extracted by the n-1th layer network are feature fused through the n-1th layer network to obtain the optical flow result and feature output result corresponding to the n-1th layer network; wherein the first fusion feature map corresponding to the nth layer network is obtained by feature fusion of the nth first optical flow feature map and the nth second optical flow feature map extracted by the nth layer network; the optical flow result and feature output result corresponding to the i-th layer network and the i-1th first optical flow feature map and the i-1th second optical flow feature map extracted by the i-1th layer network are feature fused through the i-1th layer network to obtain the optical flow result output by the i-1th layer network, including: The n-1th layer feature fusion network in the feature pyramid network performs feature fusion on the n-th first optical flow feature map extracted by the n-th layer pyramid feature network and the n-th second optical flow feature map to obtain a first fused feature map corresponding to the n-th layer pyramid feature network; The first fusion feature map corresponding to the n-th layer pyramid feature network and the n-1th first optical flow feature map and the n-1th second optical flow feature map extracted by the n-1th layer pyramid feature network are subjected to feature fusion through the n-1th layer feature fusion network in the feature pyramid network to obtain an optical flow result and a feature output result corresponding to the n-1th layer feature fusion network; The optical flow result and feature output result corresponding to the i-th layer network, as well as the i-1th first optical flow feature map and the i-1th second optical flow feature map extracted by the i-1th layer network are feature fused through the i-1th layer feature fusion network in the feature pyramid network to obtain the optical flow result output by the i-1th layer network; the i is taken from n-1 to 2 in sequence; the feature output result corresponding to the i-th layer network is determined according to the i-th second optical flow feature map and the feature output result corresponding to the i+1th layer network.

6. The method according to claim 5, characterized in that The optical flow feature map extracted by the n-th layer pyramid feature network includes the n-th first optical flow feature map and the n-th second optical flow feature map; the optical flow feature map extracted by the n-1-th layer pyramid feature network includes the n-1-th first optical flow feature map and the n-1-th second optical flow feature map; the resolution of the n-th first optical flow feature map and the resolution of the n-1-th second optical flow feature map belong to the first range; the resolution of the n-1-th first optical flow feature map and the resolution of the n-1-th second optical flow feature map belong to the second range, and the first range and the second range do not overlap; the first fusion feature map corresponding to the n-th layer network and the n-1-th first optical flow feature map and the n-1-th second optical flow feature map extracted by the n-1-th layer network are feature fused through the n-1-th layer network to obtain the optical flow result and feature output result corresponding to the n-1-th layer network, including: Performing feature fusion on the n-1th first optical flow feature map and the n-1th second optical flow feature map through the n-1th layer feature fusion network to obtain a first fused image; Obtaining a second fused image according to the n-1th second optical flow feature map and the first fused feature map through the n-1th layer feature fusion network, and extracting features of the second fused image to obtain a third fused image; Amplifying the third fused image through the n-1th layer feature fusion network to obtain a feature output result corresponding to the n-1th layer feature fusion network; Performing feature fusion on the first fused image and the third fused image through the n-1th layer feature fusion network to obtain a fourth fused image; The fourth fused image is enlarged by the n-1th layer feature fusion network to obtain an optical flow result corresponding to the n-1th layer feature fusion network.

7. The method according to claim 6, characterized in that The step of fusing the n-1th first optical flow feature map and the n-1th second optical flow feature map through the n-1th layer feature fusion network to obtain a first fused image includes: Determining N pixels from the n-1th first optical flow feature map as N first feature points through the n-1th layer feature fusion network, determining N pixels from the n-1th second optical flow feature map as N second feature points, matching the N first feature points with the N second feature points, and obtaining a corresponding relationship between the N first feature points and the N second feature points; Obtaining, through the feature fusion network of the n-1th layer, the correlation between the N first feature points and the N second feature points according to the corresponding relationship between the N first feature points and the N second feature points; The first fused image is obtained by performing feature fusion on the N first feature points and the N second feature points according to the correlation through the n-1th layer feature fusion network.

8. The method according to claim 6, characterized in that After the first fused image and the third fused image are feature-fused by the n-1th layer feature fusion network to obtain a fourth fused image, and before the fourth fused image is amplified by the n-1th layer feature fusion network to obtain an optical flow result corresponding to the n-1th layer feature fusion network, the method further includes: Removing noise from the fourth fused image and retaining boundary information of the fourth fused image through the n-1th layer feature fusion network to obtain a fifth fused image; The enlarging the fourth fused image by the n-1th layer feature fusion network to obtain the optical flow result corresponding to the n-1th layer feature fusion network includes: The fifth fused image is enlarged by the n-1th layer feature fusion network to obtain an optical flow result corresponding to the n-1th layer feature fusion network.

9. An optical flow estimation device, characterized in that: The device is applied to a feature pyramid network, the feature pyramid network includes a 2-layer network, and the device includes: An image acquisition module, used to acquire a first frame image and a second frame image; wherein the first frame image and the second frame image contain a target object; A pyramid feature module, configured to perform two feature extractions of different scales on the first frame image through the two-layer network to obtain two first optical flow feature maps; perform two feature extractions of different scales on the second frame image through the two-layer network to obtain two second optical flow feature maps; wherein the resolution of the optical flow feature map extracted by the first layer of the network is greater than the resolution of the optical flow feature map extracted by the second layer of the network; A feature fusion module, used for performing feature fusion on the first fusion feature map corresponding to the second layer network and the first first optical flow feature map and the first second optical flow feature map extracted by the first layer network through the first layer network, so as to obtain the optical flow result corresponding to the first layer network; wherein the first fusion feature map corresponding to the second layer network is obtained by performing feature fusion on the second first optical flow feature map and the second second optical flow feature map extracted by the second layer network; An output module, used to output the optical flow result corresponding to the first layer network; wherein the optical flow result is used to characterize the motion information of the target object; The feature pyramid network includes a pyramid feature network and a feature fusion network; the first and second layers of the feature pyramid network are respectively provided with the pyramid feature network; the first layer of the feature pyramid network is provided with the feature fusion network; the pyramid feature network is used to extract features from the first frame image and the second frame image; the feature fusion network is used to fuse the optical flow feature map extracted by the pyramid feature network; the feature fusion module includes a first feature fusion module, and the first feature fusion module is used to: The first layer feature fusion network in the feature pyramid network performs feature fusion on the second first optical flow feature map extracted by the second layer pyramid feature network and the second second optical flow feature map to obtain a first fused feature map corresponding to the second layer pyramid feature network; The first fusion feature map corresponding to the second layer pyramid feature network and the first first optical flow feature map and the first second optical flow feature map extracted by the first layer pyramid feature network are feature fused through the first layer feature fusion network in the feature pyramid network to obtain the optical flow result corresponding to the first layer feature fusion network.

10. The device according to claim 9, characterized in that The optical flow feature map extracted by the second-layer pyramid feature network includes the second first optical flow feature map and the second second optical flow feature map; the optical flow feature map extracted by the first-layer pyramid feature network includes the first first optical flow feature map and the first second optical flow feature map; the resolution of the second first optical flow feature map and the resolution of the second second optical flow feature map belong to a first range; the resolution of the first first optical flow feature map and the resolution of the first second optical flow feature map belong to a second range, and the first range and the second range do not overlap; the first feature fusion module is specifically used to: Performing feature fusion on the first optical flow feature map and the first second optical flow feature map through the first layer feature fusion network to obtain a first fused image; Obtaining a second fused image according to the first second optical flow feature map and the first fused feature map through the first layer feature fusion network, and extracting features of the second fused image to obtain a third fused image; Performing feature fusion on the first fused image and the third fused image through the first-layer feature fusion network to obtain a fourth fused image; The fourth fused image is enlarged by the first-layer feature fusion network to obtain an optical flow result corresponding to the first-layer feature fusion network.

11. The device according to claim 10, characterized in that The first feature fusion module is specifically used for: Determine, by the first layer feature fusion network, N pixels from the first first optical flow feature map as N first feature points, determine N pixels from the first second optical flow feature map as N second feature points, match the N first feature points with the N second feature points, and obtain a correspondence between the N first feature points and the N second feature points; Obtaining, by the first-layer feature fusion network, the correlation between the N first feature points and the N second feature points according to the corresponding relationship between the N first feature points and the N second feature points; The first fused image is obtained by performing feature fusion on the N first feature points and the N second feature points according to the correlation through the first-layer feature fusion network.

12. The device according to claim 10, characterized in that The first feature fusion module is further used for: Removing noise in the fourth fused image and retaining boundary information of the fourth fused image through the first-layer feature fusion network to obtain a fifth fused image; The first feature fusion module is specifically used for: The fifth fused image is enlarged through the first-layer feature fusion network to obtain an optical flow result corresponding to the first-layer feature fusion network.

13. An optical flow estimation device, characterized in that: The device is applied to a feature pyramid network, the feature pyramid network includes an n-layer network, n is a positive integer greater than 2, and the device includes: An image acquisition module, used to acquire a first frame image and a second frame image; wherein the first frame image and the second frame image contain a target object; A pyramid feature module, configured to perform feature extraction of the first frame image at different scales n times through the n-layer network to obtain n first optical flow feature maps; perform feature extraction of the second frame image at different scales n times through the n-layer network to obtain n second optical flow feature maps; wherein the resolution of the optical flow feature map extracted by the n-1-th layer network is greater than the resolution of the optical flow feature map extracted by the n-th layer network; A feature fusion module, used for performing feature fusion on the first fusion feature map corresponding to the n-th layer network and the n-1th first optical flow feature map and the n-1th second optical flow feature map extracted by the n-1th layer network through the n-1th layer network, so as to obtain the optical flow result and feature output result corresponding to the n-1th layer network; wherein the first fusion feature map corresponding to the n-th layer network is obtained by performing feature fusion on the n-th first optical flow feature map and the n-th second optical flow feature map extracted by the n-th layer network; performing feature fusion on the optical flow result and feature output result corresponding to the i-th layer network and the i-1th first optical flow feature map and the i-1th second optical flow feature map extracted by the i-1th layer network through the i-1th layer network, so as to obtain the optical flow result output by the i-1th layer network; the i is taken from n-1 to 2 in sequence; the feature output result corresponding to the i-th layer network is determined according to the i-th second optical flow feature map and the feature output result corresponding to the i+1th layer network; An output module, used to output the optical flow result corresponding to the first layer of the network; wherein the optical flow result is used to characterize the motion information of the target object; The feature pyramid network includes a pyramid feature network and a feature fusion network; the 1st to nth layers of the feature pyramid network are respectively provided with the pyramid feature network; the 1st to n-1th layers of the feature pyramid network are respectively provided with the feature fusion network; the feature fusion module includes the n-1th feature fusion module and the i-1th feature fusion module; wherein, The n-1th feature fusion module is used to: perform feature fusion on the nth first optical flow feature map and the nth second optical flow feature map extracted by the nth layer pyramid feature network through the n-1th layer feature fusion network in the feature pyramid network, so as to obtain a first fused feature map corresponding to the nth layer pyramid feature network; The first fusion feature map corresponding to the n-th layer pyramid feature network and the n-1th first optical flow feature map and the n-1th second optical flow feature map extracted by the n-1th layer pyramid feature network are subjected to feature fusion through the n-1th layer feature fusion network in the feature pyramid network to obtain an optical flow result and a feature output result corresponding to the n-1th layer feature fusion network; The i-1th feature fusion module is used to: perform feature fusion on the optical flow result and feature output result corresponding to the i-th layer network, as well as the i-1th first optical flow feature map and the i-1th second optical flow feature map extracted by the i-1th layer network through the i-1th layer feature fusion network in the feature pyramid network, to obtain the optical flow result output by the i-1th layer network; the i is taken from n-1 to 2 in sequence; the feature output result corresponding to the i-th layer network is determined based on the i-th second optical flow feature map and the feature output result corresponding to the i+1th layer network.

14. The device according to claim 13, characterized in that The optical flow feature map extracted by the n-th layer pyramid feature network includes the n-th first optical flow feature map and the n-th second optical flow feature map; the optical flow feature map extracted by the n-1-th layer pyramid feature network includes the n-1-th first optical flow feature map and the n-1-th second optical flow feature map; the resolution of the n-th first optical flow feature map and the resolution of the n-th second optical flow feature map belong to a first range; the resolution of the n-1-th first optical flow feature map and the resolution of the n-1-th second optical flow feature map belong to a second range, and the first range and the second range do not overlap; the n-1-th feature fusion module is specifically used to: Performing feature fusion on the n-1th first optical flow feature map and the n-1th second optical flow feature map through the n-1th layer feature fusion network to obtain a first fused image; Obtaining a second fused image according to the n-1th second optical flow feature map and the first fused feature map through the n-1th layer feature fusion network, and extracting features of the second fused image to obtain a third fused image; Amplifying the third fused image through the n-1th layer feature fusion network to obtain a feature output result corresponding to the n-1th layer feature fusion network; Performing feature fusion on the first fused image and the third fused image through the n-1th layer feature fusion network to obtain a fourth fused image; The fourth fused image is enlarged by the n-1th layer feature fusion network to obtain an optical flow result corresponding to the n-1th layer feature fusion network.

15. The device according to claim 14, characterized in that The n-1th feature fusion module is specifically used for: Determining N pixels from the n-1th first optical flow feature map as N first feature points through the n-1th layer feature fusion network, determining N pixels from the n-1th second optical flow feature map as N second feature points, matching the N first feature points with the N second feature points, and obtaining a corresponding relationship between the N first feature points and the N second feature points; Obtaining, through the feature fusion network of the n-1th layer, the correlation between the N first feature points and the N second feature points according to the corresponding relationship between the N first feature points and the N second feature points; The first fused image is obtained by performing feature fusion on the N first feature points and the N second feature points according to the correlation through the n-1th layer feature fusion network.

16. The device according to claim 14, characterized in that The n-1th feature fusion module is further used for: Removing noise from the fourth fused image and retaining boundary information of the fourth fused image through the n-1th layer feature fusion network to obtain a fifth fused image; The n-1th feature fusion module is specifically used for: The fifth fused image is enlarged by the n-1th layer feature fusion network to obtain an optical flow result corresponding to the n-1th layer feature fusion network.

17. An optical flow estimation device, characterized in that: It includes a processor and a memory, which are connected to each other, wherein the memory is used to store a computer program, the computer program includes program instructions, and the processor is configured to call the program instructions to execute the method according to any one of claims 1-4 or 5-8.

18. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes the method according to any one of claims 1-4 or 5-8.

Images