Optical flow estimation method and computer device

Abstract

The present invention provides an optical flow estimation method and computer device. The optical flow estimation method includes: obtaining two image frames and determining a plurality of feature maps corresponding to each image frame; determining a feature map pair corresponding to the image frame based on the feature map and a feature map of the target image size in the plurality of feature maps corresponding to the image frame; and determining a plurality of fused feature maps corresponding to the image frame based on all the feature map pairs corresponding to the image frame; and determining a target optical flow corresponding to the two image frames based on all the fused feature maps corresponding to each image frame. In the present invention, each fused feature map incorporates a feature map of the target image size, and the feature map of the target image size includes more image information. Optical flow estimation is then performed based on each fused map, thereby obtaining a more accurate optical flow estimation result.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an optical flow estimation method and computer device. Background Art

[0002] Optical flow is an important method for motion image analysis. It is the instantaneous speed of pixels of a moving object in space on the observation imaging plane. Optical flow is widely used in the field of computer vision.

[0003] At present, with the development of deep learning, neural networks can be used for optical flow estimation. Existing optical flow estimation methods usually first determine the features of the extracted image and then estimate the optical flow based on the extracted features. However, the features of the image extracted in the existing technology are limited, resulting in inaccurate results of the optical flow estimation.

[0004] Therefore, the existing technology needs to be improved. Summary of the Invention

[0005] The present invention provides an optical flow estimation method and computer device. The present invention fuses the extracted feature maps to obtain a fused map corresponding to each feature map. Compared with its corresponding feature map, the fused map has more layers of features. Optical flow estimation is then performed based on each fused map, which can obtain a more accurate optical flow estimation result.

[0006] In a first aspect, an embodiment of the present invention provides an optical flow estimation method, including:

[0007] Acquire two image frames, and determine a plurality of feature maps corresponding to each image frame, wherein image sizes of the plurality of feature maps of each image frame are different from each other;

[0008] For each feature map of each image frame, determine a feature map pair corresponding to the image frame based on the feature map and feature maps of the target image size in several feature maps corresponding to the image frame, and determine several fused feature maps corresponding to the image frame based on all feature map pairs corresponding to the image frame;

[0009] The target optical flow corresponding to the two image frames is determined based on all the fused feature maps corresponding to each image frame.

[0010] In one implementation, determining the plurality of feature maps corresponding to each image frame specifically includes:

[0011] For each image frame, the image frame is input into a feature pyramid network to obtain a plurality of pyramid feature maps, and some pyramid feature maps are selected from the plurality of pyramid feature maps to obtain a plurality of feature maps corresponding to the image frame.

[0012] In one implementation, determining a plurality of fused feature maps corresponding to the image frame based on all feature map pairs corresponding to the image frame specifically includes:

[0013] For each feature map pair of the image frame, input the feature map pair into the hole module to obtain a hole feature map corresponding to the feature map pair;

[0014] For each feature map pair of the image frame, the hole feature maps corresponding to each feature map pair are input into the addition module to obtain the fused feature map corresponding to each feature map pair.

[0015] In one implementation, each feature map pair of the image frame includes a first feature map pair and a plurality of second feature map pairs;

[0016] The hole module includes a first hole submodule and a plurality of second hole submodules;

[0017] An input item of the first hole submodule is the first feature map pair;

[0018] The plurality of second feature map pairs correspond one-to-one to the plurality of second hole sub-modules, and an input item of each second hole sub-module is a second feature map pair.

[0019] In one implementation, the first feature map pair includes two first feature maps of maximum image size, and each second feature map pair includes a second feature map and the first feature map; the first hole submodule includes a first hole spatial pyramid pooling unit and a splicing unit, and each second hole submodule includes an addition unit and a second hole spatial pyramid pooling unit;

[0020] For each feature map pair of the image frame, inputting the feature map pair into the hole module to obtain a hole feature map corresponding to the feature map pair specifically includes:

[0021] For a first feature map pair, one first feature map in the first feature map pair is input into the first atrous spatial pyramid pooling unit to obtain a first output result, and the first output result and the other first feature map in the first feature map pair are input into the splicing unit to obtain a atrous feature map corresponding to the first feature map pair.

[0022] In one implementation, for each image frame, a hole feature map sequence is determined based on all hole feature maps corresponding to the image frame, wherein the hole feature map sequence includes: a first hole feature map, a plurality of second hole feature maps, and a third hole feature map, with image sizes decreasing in sequence; the addition module includes a first addition unit, a plurality of second addition units, and a third addition unit; the plurality of second addition units and the plurality of second hole feature maps are added to the first hole feature map sequence; Figure 1 One to one correspondence;

[0023] For each feature map pair of the image frame, the hole feature maps corresponding to each feature map pair are input into the addition module to obtain a fused feature map corresponding to each feature map pair, specifically including:

[0024] For the third hole feature map, input the first hole feature map and the third hole feature map into a first adding unit to obtain a third fused feature map corresponding to the third hole feature map;

[0025] For each second hole feature map, obtain the second hole feature map in the hole feature map sequence. Figure 1 A candidate hole feature map of the candidate hole feature map is obtained, and a candidate fused feature map corresponding to the candidate hole feature map is obtained. The second hole feature map, the third hole feature map, and the candidate fused feature map are input into a second addition module corresponding to the second hole feature map to obtain a second fused feature map corresponding to the second hole feature map;

[0026] For the first hole feature map, obtain a reference hole feature map that is arranged one position after the first hole feature map in the hole feature map sequence, obtain a reference fusion feature map corresponding to the reference hole feature map, and input the first hole feature map and the reference fusion feature map into a third addition module to obtain a first fusion feature map corresponding to the first hole feature map.

[0027] In one implementation, determining the target optical flow corresponding to the two image frames based on all the fused feature maps corresponding to each of the determined image frames specifically includes:

[0028] Determining a plurality of fused feature map pairs based on all the determined fused feature maps corresponding to each image frame, wherein two fused feature maps in the fused feature map pair correspond to two image frames in a one-to-one manner;

[0029] The target optical flow corresponding to the two image frames is determined based on the obtained fusion feature map pairs.

[0030] In one implementation, determining the target optical flow corresponding to the two image frames based on the acquired fusion feature map pairs specifically includes:

[0031] Determining a fusion sequence according to the plurality of fusion feature map pairs, the fusion sequence comprising a first fusion feature map pair and a plurality of second fusion feature map pairs with successively decreasing image sizes;

[0032] Inputting the first fused feature map into an optical flow estimation model, and determining a first predicted optical flow corresponding to the first fused feature map through the optical flow estimation model;

[0033] For a second fused feature map pair, obtaining a candidate fused feature map pair arranged before the second fused feature map pair in the fusion sequence, obtaining a candidate predicted optical flow corresponding to the candidate fused feature map pair, and inputting the candidate predicted optical flow and the second fused feature map pair into an optical flow estimation model to obtain a second predicted optical flow corresponding to the second fused feature map pair;

[0034] Continue to perform the step of obtaining, for a second fused feature map pair, a candidate fused feature map pair arranged before the second fused feature map pair in the fusion sequence until the second predicted optical flow corresponding to the second fused feature map pair with the smallest image size is obtained, and use the second predicted optical flow corresponding to the second fused feature map pair with the smallest image size as the target optical flow.

[0035] In a second aspect, an embodiment of the present invention provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0036] Acquire two image frames, and determine a plurality of feature maps corresponding to each image frame, wherein image sizes of the plurality of feature maps of each image frame are different from each other;

[0037] For each feature map of each image frame, determine a feature map pair corresponding to the image frame based on the feature map and feature maps of the target image size in several feature maps corresponding to the image frame, and determine several fused feature maps corresponding to the image frame based on all feature map pairs corresponding to the image frame;

[0038] The target optical flow corresponding to the two image frames is determined based on all the fused feature maps corresponding to each image frame.

[0039] In a third aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0040] Acquire two image frames, and determine a plurality of feature maps corresponding to each image frame, wherein image sizes of the plurality of feature maps of each image frame are different from each other;

[0041] For each feature map of each image frame, determine a feature map pair corresponding to the image frame based on the feature map and feature maps of the target image size in several feature maps corresponding to the image frame, and determine several fused feature maps corresponding to the image frame based on all feature map pairs corresponding to the image frame;

[0042] The target optical flow corresponding to the two image frames is determined based on all the fused feature maps corresponding to each image frame.

[0043] Compared with the prior art, the embodiments of the present invention have the following advantages:

[0044] In an embodiment of the present invention, two image frames are obtained, and several feature maps corresponding to each image frame are determined, wherein the image sizes of the feature maps in the several feature maps of each image frame are different from each other; for each feature map of each image frame, based on the feature map and the feature maps of the target image size in the several feature maps corresponding to the image frame, the feature map pair corresponding to the image frame is determined, and based on all the feature map pairs corresponding to the image frame, the several fused feature maps corresponding to the image frame are determined; based on all the fused feature maps corresponding to each image frame, the target optical flow corresponding to the two image frames is determined. In the present invention, based on the feature map and the feature maps of the target image size in the several feature maps corresponding to the image frame, the feature map pair corresponding to the image frame is determined, and based on all the feature map pairs corresponding to the image frame, the several fused feature maps corresponding to the image frame are determined, that is, each fused feature map fuses the feature map of the target image size, and the feature map of the target image size includes more image information. Optical flow estimation is then performed based on each fused map, and a more accurate optical flow estimation result can be obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0046] Figure 1 Schematic diagram of a flow chart of an optical flow estimation method according to an embodiment of the present invention;

[0047] Figure 2 A schematic diagram of a process for determining a fusion feature map during a specific implementation of an embodiment of the present invention;

[0048] Figure 3 Schematic diagram of a process for determining target optical flows corresponding to two image frames based on a plurality of acquired fusion feature maps in an embodiment of the present invention;

[0049] Figure 4 2 is a diagram showing the internal structure of a computer device in an embodiment of the present invention. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0051] It will be understood by those skilled in the art that, unless expressly stated otherwise, the singular forms "a", "an", "said" and "the" used herein may also include the plural forms. It should be further understood that the term "comprising" used in the description of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, "connected" or "coupled" as used herein may include wireless connections or wireless couplings. The term "and / or" used herein includes all or any units and all combinations of one or more associated listed items.

[0052] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art in the art to which the present invention pertains. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with their meanings in the context of the prior art and, unless specifically defined as herein, will not be interpreted in an idealized or overly formal sense.

[0053] After research, the inventors found that optical flow is an important method for motion image analysis. It is the instantaneous speed of pixels of a spatial moving object on the observation imaging plane; optical flow has a wide range of applications in the field of computer vision.

[0054] At present, with the development of deep learning, neural networks can be used for optical flow estimation. Existing optical flow estimation methods usually first determine the features of the extracted image and then estimate the optical flow based on the extracted features. However, the features of the image extracted in the existing technology are limited, resulting in inaccurate results of the optical flow estimation.

[0055] In order to solve the above problem, in an embodiment of the present invention, two image frames are obtained, and several feature maps corresponding to each image frame are determined, wherein the image size of each feature map in the several feature maps of each image frame is different from each other; for each feature map of each image frame, based on the feature map and the feature maps of the target image size in the several feature maps corresponding to the image frame, the feature map pair corresponding to the image frame is determined, and based on all the feature map pairs corresponding to the image frame, several fused feature maps corresponding to the image frame are determined; based on all the fused feature maps corresponding to each image frame, the target optical flow corresponding to the two image frames is determined. In the present invention, based on the feature map and the feature maps of the target image size in the several feature maps corresponding to the image frame, the feature map pair corresponding to the image frame is determined, and based on all the feature map pairs corresponding to the image frame, several fused feature maps corresponding to the image frame are determined. That is, each fused feature map fuses the feature map of the target image size, and the feature map of the target image size includes more image information. Then, optical flow estimation is performed based on each fused map, and a more accurate optical flow estimation result can be obtained.

[0056] This embodiment provides an optical flow estimation method that can be applied to electronic devices, such as PCs, servers, mobile phones, tablet computers, PDAs, and personal digital assistants (PDAs), to determine the optical flow corresponding to any two adjacent image frames in a video.

[0057] The invention will be further explained below through description of embodiments in conjunction with the accompanying drawings.

[0058] See Figure 1 , this embodiment provides an optical flow estimation method, including:

[0059] S1. Obtain two image frames and determine several feature maps corresponding to each image frame.

[0060] In an embodiment of the present invention, the two image frames are two adjacent image frames in a video, and the two image frames are used to predict the target optical flow corresponding to the two image frames. For example, if a video includes 100 consecutive image frames, the two adjacent image frames may be the first image frame and the second image frame in the 100 consecutive image frames.

[0061] In the embodiment of the present invention, the image sizes of the feature maps of each image frame are different from each other, and the feature maps corresponding to each image frame can be determined by a feature pyramid network. Specifically, step S1 includes:

[0062] S11. For each image frame, input the image frame into a feature pyramid network to obtain a plurality of pyramid feature maps, and select some pyramid feature maps from the plurality of pyramid feature maps to obtain a plurality of feature maps corresponding to the image frame.

[0063] In this embodiment of the present invention, for ease of description, the two adjacent image frames are referred to as a first image frame and a second image frame, respectively. For the first image frame, the first image frame is input into a feature pyramid network, and the feature pyramid network determines several feature maps corresponding to the first image frame. Similarly, for the second image frame, the second image frame is input into a feature pyramid network, and the feature pyramid network determines several feature maps corresponding to the second image frame.

[0064] In an embodiment of the present invention, the first image frame is used as an example to illustrate determining the multiple feature maps corresponding to each image frame. The first image frame is input into a feature pyramid network, which outputs pyramid feature maps of different resolutions (the image sizes of the pyramid feature maps are different). To reduce the amount of computation, some pyramid feature maps are selected from the multiple pyramid feature maps to obtain the multiple feature maps corresponding to the first image frame. In other words, the number of feature maps corresponding to the first image frame is less than the number of pyramid feature maps corresponding to the first image frame. Since the feature pyramid network outputs pyramid feature maps of different resolutions, the image sizes of the multiple feature maps corresponding to the first image frame are different.

[0065] For example, the number of feature maps corresponding to the first image frame output by the feature pyramid network is 5, that is, there are 5 pyramid feature maps, and 3 of the 5 pyramid feature maps are selected as the feature maps corresponding to the first image frame.

[0066] S2. For each feature map of each image frame, determine the feature map pair corresponding to the image frame based on the feature map and the feature maps of the target image size in the feature maps corresponding to the image frame, and determine the multiple fused feature maps corresponding to the image frame based on all the feature map pairs corresponding to the image frame.

[0067] In an embodiment of the present invention, the target image size may be the maximum image size. That is, for the multiple feature maps of each image frame, the feature map with the largest image size is used as the feature map of the target image size, and the feature map with the largest image size is the shallowest feature map. For each feature map of each image frame, each feature map forms a feature map pair with the shallowest feature map, and a fused feature map corresponding to the feature map is obtained based on all feature map pairs corresponding to the image frame.

[0068] For example, the several feature maps corresponding to the first image frame P1 are T11, T12 and T13, where T11 is the feature map of the target image size. The feature map pair corresponding to T11 includes: T11 and T11; the feature map pair corresponding to T12 includes: T11 and T12; the feature map pair corresponding to T13 includes: T13 and T11.

[0069] Specifically, step S2 includes:

[0070] S21. For each feature map pair of the image frame, input the feature map pair into a hole module to obtain a hole feature map corresponding to the feature map pair.

[0071] In an embodiment of the present invention, each feature map pair of the image frame includes a first feature map pair and multiple second feature map pairs. The feature map with the largest size is recorded as the first feature map, the feature map pair corresponding to the first feature map is recorded as the first feature map pair, all feature maps except the first feature map among the multiple feature maps are recorded as second feature maps, and all feature map pairs except the first feature map pair among the multiple feature map pairs are recorded as second feature map pairs.

[0072] The hole module includes a first hole submodule and several second hole submodules. The first hole submodule receives the first feature map pair as input; the hole feature map corresponding to the first feature map pair is obtained through the first hole submodule. The several second feature map pairs correspond one-to-one to the several second hole submodules, and each second hole submodule receives a second feature map pair as input; for each second feature map pair, the second hole submodule obtains the hole feature map corresponding to the second feature map pair.

[0073] In the embodiment of the present invention, the first dilated sub-module includes a first dilated spatial pyramid pooling unit and a concatenation unit, and each second dilated sub-module includes an addition unit and a second dilated spatial pyramid pooling unit.

[0074] In an embodiment of the present invention, the first and second atrous spatial pyramid pooling units have the same structure, both being conventional atrous spatial pyramid pooling (ASPP). ASPP samples a given input using atrous convolutions at different sampling rates in parallel. The input and output items of ASPP are the same size, and ASPP maintains consistent resolution while extracting features. Compared to conventional feature extraction networks, the atrous feature maps obtained using ASPP have a larger receptive field.

[0075] Specifically, step S21 includes:

[0076] S211. For a first feature map pair, input one first feature map in the first feature map pair into the first atrous spatial pyramid pooling unit to obtain a first output result, and input the first output result and the other first feature map in the first feature map pair into the splicing unit to obtain a atrous feature map corresponding to the first feature map pair.

[0077] In an embodiment of the present invention, the first output result is obtained by the first dilated space pyramid pooling unit, the first output result has the same image size as the first feature map, and the dilated rate of the first dilated space in the pyramid pooling unit is 1, 6, 12, and 18.

[0078] The first output result is concatenated with the other first feature map in the first feature map pair to obtain the hole feature map corresponding to the first feature map pair. The so-called concatenation is based on the number of channels, and the number of channels corresponding to the hole feature map is equal to the sum of the number of channels of the first feature map and the number of channels of the first output result.

[0079] For example, the first feature map T11 corresponding to the first image frame is input into ASPP, and the first output result L11 corresponding to T11 is obtained through ASPP. L11 and T11 are input into the splicing unit to obtain the hole feature map F1 corresponding to T1.

[0080] S212. For each second feature map pair, input the first feature map and the second feature map in the second feature map pair into the addition unit corresponding to the second feature map pair to obtain a second output result, and input the second output result into the second pyramid pooling unit corresponding to the second feature map pair to obtain the hole feature map corresponding to the second feature map pair.

[0081] In an embodiment of the present invention, the first feature map and the second feature map in the second feature map pair are added together. Since the image sizes of the first feature map and the second feature map are different, the first feature map needs to be processed so that the image sizes of the first feature map and the second feature map are the same. The process of processing the first feature map includes: downsampling the first feature map so that the image resolution of the downsampled first feature map and the second feature map are the same, and then using a 1*1 convolution kernel to process the downsampled first feature map so that the dimension of the first feature map processed by the 1*1 convolution kernel is the same as the dimension of the second feature map, and the image sizes of the processed first feature map and the second feature map are the same. Then, the processed first feature map and the second feature map are added together to obtain a second output result. The second output result is input into the second pyramid pooling unit to obtain a hole feature map corresponding to the second feature map pair.

[0082] S22. For each feature map pair of the image frame, input the hole feature maps corresponding to each feature map pair into the addition module to obtain a fused feature map corresponding to each feature map pair.

[0083] In this embodiment of the present invention, for each image frame, a hole feature map sequence is determined based on all hole feature maps corresponding to that image frame. The hole feature map sequence includes, in descending order of image size, a first hole feature map, a plurality of second hole feature maps, and a third hole feature map. That is, for all hole feature maps corresponding to each image frame, the hole feature map with the largest image size is used as the first hole feature map, the hole feature map with the smallest image size is used as the third hole feature map, and the remaining hole feature maps are the plurality of second hole feature maps.

[0084] The adding module includes a first adding unit, a plurality of second adding units and a third adding unit, wherein the plurality of second adding units and the plurality of second hole features are Figure 1 One to one correspondence.

[0085] Specifically, step S22 includes:

[0086] S221. For the third hole feature map, input the first hole feature map and the third hole feature map into a first adding unit to obtain a third fused feature map corresponding to the third hole feature map.

[0087] In an embodiment of the present invention, the function of the first adding unit is the same as that of the add function. The image sizes of the first hole feature map and the third hole feature map are different. The first hole feature map needs to be processed so that the image size of the first hole feature map is equal to the image size of the third hole feature map. The process of processing the image size of the first hole feature map includes: downsampling the first hole feature map so that the resolution of the first hole feature map is the same as that of the third hole feature map, and then using a 1*1 convolution kernel to process the downsampled first hole feature map so that the dimension of the processed first hole feature map is the same as that of the third hole feature map. At this point, the image size of the processed first hole feature map is the same as that of the third hole feature map, and then the processed first hole feature map is added to the third hole feature map. The third fused feature map is obtained.

[0088] S222. For each second hole feature map, obtain a candidate hole feature map that is arranged one position after the second hole feature map in the hole feature map sequence, obtain a candidate fused feature map corresponding to the candidate hole feature map, and input the second hole feature map, the third hole feature map and the candidate fused feature map into the second addition module corresponding to the second hole feature map to obtain a second fused feature map corresponding to the second hole feature map.

[0089] In an embodiment of the present invention, for example, the hole feature map sequence includes the following images, arranged from large to small: R1, R2, R3, and R4, where R1 is the first hole feature map, R2 and R3 are the second hole feature maps, and R4 is the third hole feature map. For the second hole feature map R3, the candidate hole feature map arranged before the second hole feature map in the hole feature map sequence is obtained. That is, R4 is the candidate hole feature map corresponding to R3. The candidate fusion map Fu4 corresponding to R4 is obtained. Fu4, R3, and R1 are input into the second addition module to obtain the second fusion feature map corresponding to R3. For R2, the candidate hole feature map arranged before the second hole feature map in the hole feature map sequence is obtained. That is, R3 is the candidate hole feature map corresponding to R2. The candidate fusion map Fu3 corresponding to R3 is obtained. Fu3, R2, and R1 are input into the second addition module to obtain the second fusion feature map Fu2 corresponding to R2.

[0090] S223. For the first hole feature map, obtain a reference hole feature map that is arranged one position after the first hole feature map in the hole feature map sequence, obtain a reference fused feature map corresponding to the reference hole feature map, and input the first hole feature map and the reference fused feature map into a third addition module to obtain a first fused feature map corresponding to the first hole feature map.

[0091] In the above example, R1, R2, R3 and R4, among which R1 is the first hole feature map, and the one arranged after R1 is R2, that is, R2 is the reference hole feature map corresponding to R1, and the fused feature map Fu2 corresponding to R2 is obtained. Fu2 and R1 are input into the first addition module to obtain the first fused feature map corresponding to R1.

[0092] Here is an example of a specific implementation, see Figure 2 .

[0093] For each image frame, the feature pyramid network generates five corresponding pyramid feature maps. Three of these maps are selected as the corresponding feature maps for that image frame. These feature maps are labeled T1, T2, and T3, where T1 has the largest image size and T3 has the smallest. The first feature map pair consists of two T1s, the second feature map pair consists of T1 and T2, and the third feature map pair consists of T1 and T3.

[0094] The first feature map pair is input into the first hole submodule 100 to obtain the hole feature map R1 corresponding to T1. The second feature map pair is input into the second hole submodule 200 to obtain the hole feature map R2 corresponding to T2. The third feature map pair is input into the third hole submodule 300 to obtain the hole feature map R3 corresponding to T3. R1 and R3 are added together to obtain the fused feature map Fu3 corresponding to R3. R1, R2, and Fu3 are added together to obtain the fused feature map Fu2 corresponding to R2. R1 and Fu2 are added together to obtain the fused feature map Fu1 corresponding to R1.

[0095] Specifically, after inputting T1 into ASPP, the output of ASPP is concatenated with T1 to obtain R1. T1 is downsampled to the same resolution as T3, and then convolved with a kernel of 1 on the downsampled T1. Then, it is added with T3 and convolved with a kernel of 3. The result is then input into ASPP to obtain R3. R3 and R1 are added to obtain the fused feature map Fu3 corresponding to R3. T1 is downsampled again to the same resolution as T2, and then convolved with a kernel of 1 on the downsampled T1. Then, it is added with T2 and convolved with a kernel of 3. The result is then input into ASPP to obtain R2. R2, R1, and Fu3 are added to obtain the fused feature map Fu2 corresponding to R2. R1 and Fu2 are added to obtain the fused feature map Fu1 corresponding to R1.

[0096] S3. Determine the target optical flow corresponding to the two image frames based on all the fusion feature maps corresponding to each of the determined image frames.

[0097] In an embodiment of the present invention, all fused feature maps corresponding to each image frame include: a plurality of fused feature maps corresponding to the first image frame, and a plurality of fused feature maps corresponding to the second image frame; the image sizes of the plurality of fused feature maps corresponding to the first image frame are different from each other, and the image sizes of the plurality of fused feature maps corresponding to the second image frame are different from each other. A plurality of fused feature map pairs are determined based on the plurality of fused feature map pairs corresponding to the first image frame and the plurality of fused feature map pairs corresponding to the second image frame. Target optical flow is then determined based on the plurality of fused feature map pairs.

[0098] Specifically, step S3 includes:

[0099] S31. Determine a plurality of fused feature map pairs based on all the fused feature maps corresponding to each image frame, wherein two fused feature maps in the fused feature map pair correspond to two image frames one-to-one.

[0100] In an embodiment of the present invention, each fused feature map pair includes two fused feature maps, and the image size of the two fused feature maps is the same. That is, all the fused feature maps corresponding to each image frame are divided into several fused feature map pairs according to the image size, wherein each fused feature map pair includes two fused feature maps with the same image size, and the two fused feature maps correspond one-to-one to the two image frames.

[0101] For example, all fused feature maps corresponding to the first image frame include the following: Fu11, Fu12, and Fu13, with image sizes from large to small; all fused feature maps corresponding to the second image frame include the following: Fu21, Fu22, and Fu23, with image sizes from large to small. Then, a fused feature map pair including Fu11 and Fu21, a fused feature map pair including Fu12 and Fu22, and a fused feature map pair including Fu13 and Fu23 can be obtained.

[0102] S32. Determine the target optical flow corresponding to the two image frames based on the obtained fusion feature map pairs.

[0103] In an embodiment of the present invention, multiple fused feature map pairs are input into an optical flow estimation model, which determines the target optical flow. The optical flow estimation model is obtained by training a preset network based on multiple training image groups. The optical flow estimation model can be a PWC-Net. The process of training the preset network will be described later.

[0104] Specifically, step S32 includes:

[0105] S321. Determine a fusion sequence based on the plurality of fused feature map pairs, where the fusion sequence includes a first fused feature map pair and a plurality of second fused feature map pairs whose image sizes decrease in sequence.

[0106] In an embodiment of the present invention, the image size of the fused feature map in the first fused feature map pair is larger than the image size of the fused feature map in any one of the second fused feature map pairs.

[0107] For example, the first fused feature map pair includes Fu11 and Fu21, one of the plurality of second fused feature map pairs includes Fu12 and Fu22, and another of the plurality of second fused feature map pairs includes Fu13 and Fu23. The image size of Fu11 is equal to the image size of Fu21, the image size of Fu12 is equal to the image size of Fu22, and the image size of Fu13 is equal to the image size of Fu23.

[0108] The fused feature maps corresponding to the first image frame include the following: Fu11, Fu12 and Fu13, with image sizes from large to small; the fused feature maps corresponding to the second image frame include the following: Fu21, Fu22 and Fu23, with image sizes from large to small. Then, a first fused feature map pair including Fu11 and Fu21, a second fused feature map pair including Fu12 and Fu22, and another second fused feature map pair including Fu13 and Fu23 can be obtained.

[0109] S322: Input the first fused feature map into an optical flow estimation model, and determine a first predicted optical flow corresponding to the first fused feature map through the optical flow estimation model.

[0110] In an embodiment of the present invention, the image size corresponding to the first fused feature map pair is the largest, the first fused feature map pair is input into the optical flow estimation model, and the first predicted optical flow corresponding to the first fused feature map is determined by the optical flow estimation model.

[0111] S323. For a second fused feature map pair, obtain a candidate fused feature map pair arranged before the second fused feature map pair in the fusion sequence, obtain a candidate predicted optical flow corresponding to the candidate fused feature map pair, and input the candidate predicted optical flow and the second fused feature map pair into the optical flow estimation model to obtain a second predicted optical flow corresponding to the second fused feature map pair.

[0112] In the embodiment of the present invention, step S322 and step S323 are sequential, and step S322 needs to be performed before step S323. The image size corresponding to the second fused feature map pair is smaller than that of the first fused feature map pair.

[0113] During specific implementation, the fused feature map pair includes a first fused feature map pair K1, and a second fused feature map pair K2 and K3 with successively smaller image sizes. K1 is input into the optical flow estimation model to obtain the first predicted optical flow OF1 corresponding to K1. For the second fused feature map pair K2, the candidate fused feature map K1 arranged before K2 in the fusion sequence is obtained, the predicted optical flow OF1 corresponding to K1 is obtained, OF1 and K2 are input into the optical flow estimation model to obtain the second predicted optical flow OF2 corresponding to K2; for the second fused feature map pair K3, the candidate fused feature map K2 arranged before K3 in the fusion sequence is obtained, the predicted optical flow OF2 corresponding to K2 is obtained, OF2 and K3 are input into the optical flow estimation model to obtain the second predicted optical flow OF3 corresponding to K3.

[0114] S324. Continue to execute the step of obtaining, for a second fused feature map pair, a candidate fused feature map pair arranged before the second fused feature map pair in the fusion sequence, until the second predicted optical flow corresponding to the second fused feature map pair with the smallest image size is obtained, and use the second predicted optical flow corresponding to the second fused feature map pair with the smallest image size as the target optical flow.

[0115] In an embodiment of the present invention, the operation of step S323 is performed on each second fusion feature map pair, and the predicted optical flow corresponding to the second fusion feature map pair arranged last in the fusion sequence is used as the target optical flow. The image size corresponding to the second fusion feature map pair arranged last in the fusion sequence is the smallest.

[0116] See also Figure 3 In the specific implementation, the process of determining the target optical flow corresponding to two image frames based on the obtained fusion feature maps is as follows:

[0117] A fused feature map pair Ki is obtained based on the first image frame and the second image frame. The value of i can be: 1, 2, ... In this embodiment, i can be 1, 2, and 3. For K1 with the largest image size, K1 is input into the optical flow estimation model to obtain the predicted optical flow OF1 corresponding to K1. For K2, the upsampled optical flow corresponding to K2 is OF1. OF1 and K2 are input into the optical flow estimation model to obtain the predicted optical flow OF2 corresponding to K2. For K3, the upsampled optical flow corresponding to K3 is OF2. OF2 and K3 are input into the optical flow estimation model to obtain the predicted optical flow OF3 corresponding to K3. Since the maximum value of i is 3, K3 is the fused feature map pair with the smallest image size, and OF3 is the target optical flow.

[0118] The optical flow estimation model includes: Warping, Cost volume, optical flow estimator and context network. Specifically, OF1 and K2 are input into the optical flow estimation model, and the predicted optical flow OF2 corresponding to K2 is obtained.

[0119] Fu22 and OF2 are input into Warping, the output result of Warping and Fu12 are input into Cost volume, the output result of Cost volume, Fu12 and OF2 are input into the optical flow estimator, and the output result of the optical flow estimator is input into the context network. The predicted optical flow OF2 corresponding to K2 is obtained through the context network.

[0120] In an embodiment of the present invention, two image frames are acquired, and a number of feature maps corresponding to each image frame are determined, wherein the image size of each feature map in the number of feature maps is different from each other; for each feature map of each image frame, a feature map pair is determined based on the feature map and the feature map with the largest image size among the number of feature maps, and a number of fused feature maps corresponding to the image frame are determined based on all the determined feature map pairs; and a target optical flow corresponding to the two image frames is determined based on all the determined fused feature maps. In the present invention, a feature map pair is determined based on the feature map and the feature map with the largest image size among the number of feature maps, and a number of fused feature maps corresponding to the image frame are determined based on all the determined feature map pairs. That is, each fused feature map fuses the feature map with the largest image size. The feature map with the largest image size is the shallowest feature and includes more image information. Optical flow estimation is then performed based on each fused map, and a more accurate optical flow estimation result can be obtained.

[0121] Next, we introduce the process of training the preset network to obtain the optical flow estimation network.

[0122] L1. Obtain several training image groups, each of which includes two consecutive image frames in a video, namely a first image X and a second image Y.

[0123] L2. For a training image group X and a second image Y, determine several training feature maps corresponding to X, with image sizes from 1 to 2. The process of determining the several training feature maps corresponding to X is the same as the process of determining the several feature maps corresponding to each image frame in step S11. Therefore, the process of determining the several training feature maps corresponding to X can refer to the description of step S11. Similarly, determine several training feature maps y1, y2, and y3 corresponding to Y, with image sizes from 1 to 2. Here, x1 and y1 have the same image size, x2 and y2 have the same image size, and x3 and y3 have the same image size.

[0124] L3. Based on x1, x2, and x3, determine the training fusion feature map Fux1 corresponding to x1, the training fusion feature map Fux2 corresponding to x2, and the training fusion feature map Fux3 corresponding to x3; the process of determining Fux1, Fux2, and Fux3 is the same as the process of determining the several fusion feature maps corresponding to the image frame in step S2. Therefore, for the process of determining Fux1, Fux2, and Fux3, please refer to the description of step S2. Similarly, determine the training fusion feature map Fuy1 corresponding to y1, the training fusion feature map Fuy2 corresponding to y2, and the training fusion feature map Fuy3 corresponding to y3. Among them, the image size of Fux1 and Fuy1 is the same, the image size of Fux2 and Fuy2 is the same, and the image size of Fux3 and Fuy3 is the same.

[0125] L4. Input Fux1 and Fuy1 into the preset network model to obtain the training optical flow OFx1-t corresponding to x1 and y1, obtain the real optical flow OFx1-r corresponding to x1 and y1, and train the preset network model according to the training optical flow OFx1-t and the real optical flow OFx1-r;

[0126] Input Fux2 and Fuy2 into the preset network model, obtain the training optical flow OFx2-t corresponding to x2 and y2, obtain the real optical flow OFx2-r corresponding to x2 and y2, and train the preset network model according to the training optical flow OFx2-t and the real optical flow OFx2-t;

[0127] Input Fux3 and Fuy3 into the preset network model, obtain the training optical flow OFx3-t corresponding to x3 and y3, obtain the real optical flow OFx3-r corresponding to x3 and y3, and train the preset network model according to the training optical flow OFx3-t and the real optical flow OFx3-t.

[0128] L5. Continue to execute step L2 until the training conditions of the preset network model are met to obtain the optical flow estimation model.

[0129] In step L4, the preset network model is trained according to the training optical flow OFx1-t and the real optical flow OFx1-r. Specifically, the loss function value is calculated by formula (1) according to the training optical flow OFx1-t and the real optical flow OFx1-r, and the parameters of the preset network model are adjusted according to the loss function value to train the preset network model.

[0130]

[0131] Among them, L(Θ) is the loss function value, Θ is the training parameter of the preset network model, Is to train optical flow, l can be: x1, x2 and x3, l = x1, Represents the predicted optical flow corresponding to x1 and y1; is the real optical flow, the value of l can be: x1, x2 and x3, when l = x1, Represents the true optical flow corresponding to x1 and y1; α and γ are hyperparameters, and the values ​​of α and γ can be set according to experience.

[0132] In an embodiment of the present invention, the preset training conditions include the loss function value meeting the preset requirements or the number of training times reaching the preset number. The preset requirements can be determined based on the accuracy of the optical flow estimation model, which will not be explained in detail here. The preset number of times can be the maximum number of training times of the preset network model, for example, 4000 times. Therefore, after calculating the loss function value, it is determined whether the loss function value meets the preset requirements. If the loss function value meets the preset requirements, the training is terminated; if the loss function value does not meet the preset requirements, it is determined whether the number of training times of the preset network model reaches the prediction number. If the preset number is not reached, the network parameters of the preset network model are corrected according to the loss function value; if the preset number is reached, the training is terminated. In this way, by judging whether the training of the preset network model is terminated based on the loss function value and the number of training times, it is possible to avoid the training of the preset network model entering an infinite loop due to the failure of the loss function value to meet the preset requirements.

[0133] Furthermore, since the adjustment of the parameters of the preset network model is performed when the training situation of the preset network model does not meet the preset conditions (for example, the loss function value does not meet the preset requirements and the number of training times does not reach the preset number of times), after the parameters of the preset network model are adjusted according to the loss function value, it is necessary to continue training the preset network model.

[0134] In an embodiment of the present invention, two image frames are acquired, and a number of feature maps corresponding to each image frame are determined, wherein the image size of each feature map in the number of feature maps is different from each other; for each feature map of each image frame, a feature map pair is determined based on the feature map and the feature map with the largest image size among the number of feature maps, and a number of fused feature maps corresponding to the image frame are determined based on all the determined feature map pairs; and a target optical flow corresponding to the two image frames is determined based on all the determined fused feature maps. In the present invention, a feature map pair is determined based on the feature map and the feature map with the largest image size among the number of feature maps, and a number of fused feature maps corresponding to the image frame are determined based on all the determined feature map pairs. That is, each fused feature map fuses the feature map with the largest image size. The feature map with the largest image size is the shallowest feature and includes more image information. Optical flow estimation is then performed based on each fused map, and a more accurate optical flow estimation result can be obtained.

[0135] In one embodiment, the present invention provides a computer device, which may be a terminal, and the internal structure is as follows: Figure 4As shown. The computer device includes a processor, a memory, a network model interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network model interface of the computer device is used to communicate with an external terminal via a network model connection. When the computer program is executed by the processor, an optical flow estimation method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad provided on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0136] Those skilled in the art will understand that Figure 4 What is shown is only a block diagram of a portion of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0137] An embodiment of the present invention provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and wherein the processor performs the following steps when executing the computer program:

[0138] Acquire two image frames, and determine a plurality of feature maps corresponding to each image frame, wherein image sizes of the plurality of feature maps of each image frame are different from each other;

[0139] For each feature map of each image frame, determine a feature map pair corresponding to the image frame based on the feature map and feature maps of the target image size in several feature maps corresponding to the image frame, and determine several fused feature maps corresponding to the image frame based on all feature map pairs corresponding to the image frame;

[0140] The target optical flow corresponding to the two image frames is determined based on all the fused feature maps corresponding to each image frame.

[0141] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0142] Acquire two image frames, and determine a plurality of feature maps corresponding to each image frame, wherein image sizes of the plurality of feature maps of each image frame are different from each other;

[0143] For each feature map of each image frame, determine a feature map pair corresponding to the image frame based on the feature map and feature maps of the target image size in several feature maps corresponding to the image frame, and determine several fused feature maps corresponding to the image frame based on all feature map pairs corresponding to the image frame;

[0144] The target optical flow corresponding to the two image frames is determined based on all the fused feature maps corresponding to each image frame.

[0145] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0146] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. An optical flow estimation method, characterized in that: include: Acquire two image frames, and determine a plurality of feature maps corresponding to each image frame, wherein image sizes of the plurality of feature maps of each image frame are different from each other; For each feature map of each image frame, based on the feature map and the feature maps of the target image size in several feature maps corresponding to the image frame, a feature map pair corresponding to the image frame is determined, and the feature map pair is input into a hole module to obtain a hole feature map corresponding to the feature map pair; based on the hole feature maps corresponding to each feature map pair, a fused feature map corresponding to each feature map pair is obtained; the hole module includes a first hole submodule and several second hole submodules, the first hole submodule includes a splicing unit and first hole spatial pyramid pooling units with different hole rates, and each second hole submodule includes an addition unit and a second hole spatial pyramid pooling unit; The target optical flow corresponding to the two image frames is determined based on all the fused feature maps corresponding to each image frame.

2. The optical flow estimation method according to claim 1, wherein: The determining of the plurality of feature maps corresponding to each image frame specifically includes: For each image frame, the image frame is input into a feature pyramid network to obtain a plurality of pyramid feature maps, and some pyramid feature maps are selected from the plurality of pyramid feature maps to obtain a plurality of feature maps corresponding to the image frame.

3. The optical flow estimation method according to claim 1, wherein: The step of obtaining a fused feature map corresponding to each feature map pair based on the hole feature maps corresponding to each feature map pair specifically includes: For each feature map pair of the image frame, the hole feature maps corresponding to each feature map pair are input into the addition module to obtain the fused feature map corresponding to each feature map pair.

4. The optical flow estimation method according to claim 3, wherein: Each feature map pair of the image frame includes a first feature map pair and a plurality of second feature map pairs; An input item of the first hole submodule is the first feature map pair; The plurality of second feature map pairs correspond one-to-one to the plurality of second hole sub-modules, and an input item of each second hole sub-module is a second feature map pair.

5. The optical flow estimation method according to claim 4, characterized in that The first feature map pair includes two first feature maps of the maximum image size, and each second feature map pair includes a second feature map and the first feature map; for each feature map pair of the image frame, inputting the feature map pair into the hole module to obtain a hole feature map corresponding to the feature map pair, specifically including: For a first feature map pair, inputting one first feature map in the first feature map pair into the first atrous spatial pyramid pooling unit to obtain a first output result, and inputting the first output result and the other first feature map in the first feature map pair into the splicing unit to obtain a atrous feature map corresponding to the first feature map pair; For each second feature map pair, the first feature map and the second feature map in the second feature map pair are input into the addition unit corresponding to the second feature map pair to obtain a second output result, and the second output result is input into the second pyramid pooling unit corresponding to the second feature map pair to obtain the hole feature map corresponding to the second feature map pair.

6. The optical flow estimation method according to claim 3, wherein: For each image frame, a hole feature map sequence is determined based on all hole feature maps corresponding to the image frame, wherein the hole feature map sequence includes: a first hole feature map, a plurality of second hole feature maps, and a third hole feature map in decreasing order of image size; the addition module includes a first addition unit, a plurality of second addition units, and a third addition unit; the plurality of second addition units and the plurality of second hole feature maps are in one-to-one correspondence; For each feature map pair of the image frame, the hole feature maps corresponding to each feature map pair are input into the addition module to obtain a fused feature map corresponding to each feature map pair, specifically including: For the third hole feature map, input the first hole feature map and the third hole feature map into a first adding unit to obtain a third fused feature map corresponding to the third hole feature map; For each second hole feature map, obtaining a candidate hole feature map arranged one position after the second hole feature map in the hole feature map sequence, obtaining a candidate fused feature map corresponding to the candidate hole feature map, and inputting the second hole feature map, the third hole feature map, and the candidate fused feature map into the second addition module corresponding to the second hole feature map to obtain a second fused feature map corresponding to the second hole feature map; For the first hole feature map, obtain a reference hole feature map that is arranged one position after the first hole feature map in the hole feature map sequence, obtain a reference fusion feature map corresponding to the reference hole feature map, and input the first hole feature map and the reference fusion feature map into a third addition module to obtain a first fusion feature map corresponding to the first hole feature map.

7. The optical flow estimation method according to claim 1, wherein: The determining of the target optical flow corresponding to the two image frames based on all the fused feature maps corresponding to each of the determined image frames specifically includes: Determining a plurality of fused feature map pairs based on all the determined fused feature maps corresponding to each image frame, wherein two fused feature maps in the fused feature map pair correspond to two image frames in a one-to-one manner; The target optical flow corresponding to the two image frames is determined based on the obtained fusion feature map pairs.

8. The optical flow estimation method according to claim 7, wherein: The determining of the target optical flow corresponding to the two image frames based on the obtained plurality of fusion feature map pairs specifically includes: Determining a fusion sequence according to the plurality of fusion feature map pairs, the fusion sequence comprising a first fusion feature map pair and a plurality of second fusion feature map pairs with successively decreasing image sizes; Inputting the first fused feature map into an optical flow estimation model, and determining a first predicted optical flow corresponding to the first fused feature map through the optical flow estimation model; For a second fused feature map pair, obtaining a candidate fused feature map pair arranged before the second fused feature map pair in the fusion sequence, obtaining a candidate predicted optical flow corresponding to the candidate fused feature map pair, and inputting the candidate predicted optical flow and the second fused feature map pair into an optical flow estimation model to obtain a second predicted optical flow corresponding to the second fused feature map pair; Continue to perform the step of obtaining, for a second fused feature map pair, a candidate fused feature map pair arranged before the second fused feature map pair in the fusion sequence until the second predicted optical flow corresponding to the second fused feature map pair with the smallest image size is obtained, and use the second predicted optical flow corresponding to the second fused feature map pair with the smallest image size as the target optical flow.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the optical flow estimation method according to any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the optical flow estimation method according to any one of claims 1 to 8 are implemented.

Images