Optical flow estimation method and device, storage medium and electronic equipment

Abstract

The invention provides an optical flow estimation method and device, a storage medium and electronic equipment, and relates to the technical field of motion tracking. The method comprises the following steps: acquiring a first original feature map and a second original feature map, wherein the first original feature map and the second original feature map are obtained by respectively performing feature extraction on two adjacent frames of original images; estimating an initial optical flow based on the first original feature map and the second original feature map; selecting a plurality of reference coordinate anchor points based on the initial optical flow, and generating candidate optical flows corresponding to the reference coordinate anchor points; and performing fusion processing on the candidate optical streams to obtain a target optical stream. According to the invention, the precision performance of optical flow estimation under different displacement amplitudes can be improved, and the adaptability to complex motion scenes is enhanced.

Description

Technical Field

[0001] This disclosure relates to the field of motion tracking technology, and more specifically, to an optical flow estimation method, apparatus, storage medium, and electronic device. Background Technology

[0002] Optical flow estimation, as a key technology for obtaining pixel-level motion information between image sequences, is widely used in visual tasks such as target tracking, 3D reconstruction, and behavior recognition.

[0003] Currently, optical flow estimation is typically based on deep neural network structures, such as convolutional neural networks or Transformer architectures, to extract features from input image pairs and then regress the spatial displacement vector of each pixel. To improve estimation accuracy, a multi-scale pyramid mechanism is often introduced, which optimizes the optical flow step-by-step at different resolution scales, achieving a coarse-to-fine estimation process.

[0004] However, for scenes with large displacements or cross-scale motion, the optical flow obtained by this type of method at low-resolution levels is prone to large deviations, and these deviations will continue to accumulate in subsequent high-resolution levels, thereby further weakening the accuracy and robustness of the overall optical flow estimation.

[0005] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0006] The purpose of this disclosure is to provide an optical flow estimation method, apparatus, storage medium, and electronic device that can improve the adaptability to large displacement and complex motion scenarios, thereby enhancing the accuracy and robustness of optical flow estimation results.

[0007] According to a first aspect of this disclosure, an optical flow estimation method is provided, comprising:

[0008] A first original feature map and a second original feature map are obtained, wherein the first original feature map and the second original feature map are obtained by extracting features from two adjacent original images respectively.

[0009] The initial optical flow is estimated based on the first and second original feature maps;

[0010] Based on the initial optical flow, multiple reference coordinate anchor points are selected, and candidate optical flows corresponding to each reference coordinate anchor point are generated;

[0011] The candidate optical flows are fused to obtain the target optical flow.

[0012] In one exemplary embodiment of this disclosure, estimating the initial optical flow based on the first original feature map and the second original feature map includes:

[0013] The current optical flow is initialized based on the first and second original feature maps at the first resolution.

[0014] Based on the current optical flow, the first original feature map is spatially transformed to obtain the first aligned feature map;

[0015] The initial optical flow is calculated based on the first aligned feature map and the second original feature map.

[0016] In one exemplary embodiment of this disclosure, calculating the initial optical flow based on the first aligned feature map and the second original feature map includes:

[0017] The first aligned feature map and the second original feature map are concatenated to obtain a fused feature map.

[0018] The initial optical flow is obtained by performing a convolution operation on the fused feature map.

[0019] In one exemplary embodiment of this disclosure, the initial optical flow is a polar coordinate optical flow, including an initial amplitude component and an initial direction component.

[0020] In one exemplary embodiment of this disclosure, selecting multiple reference coordinate anchor points based on the initial optical flow includes:

[0021] Based on the initial amplitude component and initial direction component in the initial optical flow, multiple reference amplitude components and multiple reference direction components are set respectively;

[0022] The reference amplitude components and reference direction components are combined to generate multiple reference coordinate anchor points.

[0023] In one exemplary embodiment of this disclosure, generating candidate optical flows corresponding to each of the reference coordinate anchor points includes:

[0024] Predicted optical flow corresponding to each of the aforementioned reference coordinate anchor points is generated through a first neural network;

[0025] The predicted optical flow corresponding to each of the reference coordinate anchor points is converted into the candidate optical flow corresponding to each of the reference coordinate anchor points.

[0026] In one exemplary embodiment of this disclosure, generating the predicted optical flow corresponding to each of the reference coordinate anchor points via a first neural network includes:

[0027] Each of the reference coordinate anchor points is input into the first neural network to obtain the predicted amplitude component and the predicted direction component corresponding to each of the reference coordinate anchor points.

[0028] The predicted optical flow corresponding to each reference coordinate anchor point is determined based on the predicted amplitude component and the predicted direction component corresponding to each reference coordinate anchor point.

[0029] In an exemplary embodiment of this disclosure, converting the predicted optical flow corresponding to each of the reference coordinate anchor points into a candidate optical flow corresponding to each of the reference coordinate anchor points includes:

[0030] Based on each of the reference coordinate anchor points and the predicted optical flow corresponding to each of the reference coordinate anchor points, calculate the first intermediate optical flow corresponding to each of the reference coordinate anchor points;

[0031] The first intermediate optical flow is transformed to obtain the second intermediate optical flow, and the initialized current optical flow is superimposed to calculate the candidate optical flow corresponding to each of the reference coordinate anchor points.

[0032] In one exemplary embodiment of this disclosure, the first intermediate optical flow is an optical flow in polar coordinates, and the second intermediate optical flow, the current optical flow, and the candidate optical flow are optical flows in spatial coordinates.

[0033] In an exemplary embodiment of this disclosure, calculating the first intermediate optical flow corresponding to each of the reference coordinate anchor points based on each of the reference coordinate anchor points and the corresponding predicted optical flow includes:

[0034] The intermediate amplitude component is obtained by multiplying the reference amplitude component in each of the reference coordinate anchor points with the predicted amplitude component in the corresponding predicted optical flow.

[0035] The reference direction component in each of the reference coordinate anchor points is added to the prediction direction component in the corresponding prediction optical flow to obtain the intermediate direction component;

[0036] The first intermediate optical flow is determined based on the intermediate amplitude component and the intermediate direction component.

[0037] In one exemplary embodiment of this disclosure, the step of fusing the candidate optical flows to obtain the target optical flow includes:

[0038] The first original feature map and the second original feature map are input into the second neural network to obtain the first deep feature map and the second deep feature map;

[0039] Based on each of the candidate optical flows, the first deep feature map is spatially transformed to obtain the second aligned feature map corresponding to each of the candidate optical flows;

[0040] Each of the second aligned feature maps is matched with the second deep feature map to obtain the matching result;

[0041] The candidate optical flows are fused according to the matching results to obtain the target optical flow.

[0042] In an exemplary embodiment of this disclosure, the step of performing spatial transformation on the first deep feature map based on each of the candidate optical flows to obtain a second aligned feature map corresponding to each of the candidate optical flows includes:

[0043] Based on the displacement vector in each candidate optical flow, the pixel positions in the first deep feature map are transformed to obtain the second aligned feature map corresponding to each candidate optical flow.

[0044] In one exemplary embodiment of this disclosure, the matching result includes a similarity score;

[0045] The step of matching each of the second aligned feature maps with the second deep feature map to obtain the matching result includes:

[0046] Calculate the similarity between each of the second aligned feature maps and the second deep feature map to obtain multiple similarity scores.

[0047] In one exemplary embodiment of this disclosure, the step of fusing the candidate optical flows according to the matching result to obtain the target optical flow includes:

[0048] The similarity scores are normalized to obtain the similarity coefficients corresponding to each similarity score;

[0049] Using the similarity coefficients as weights, the corresponding candidate optical flows are weighted and summed to obtain the target optical flow.

[0050] In one exemplary embodiment of this disclosure, after obtaining the target optical flow, the method further includes:

[0051] Based on the target optical flow, a spatial transformation is performed on the first deep feature map to obtain a third aligned feature map;

[0052] The third aligned feature map and the second deep feature map are input into the third neural network to obtain the optical flow increment;

[0053] The target optical flow is adjusted according to the optical flow increment to obtain the adjusted target optical flow.

[0054] In one exemplary embodiment of this disclosure, the method further includes:

[0055] Based on the first deep feature map or the second deep feature map, the adjusted target optical flow is regularized to obtain the final target optical flow.

[0056] In one exemplary embodiment of this disclosure, the method further includes:

[0057] Each of the original images is input into the fourth neural network to obtain multiple first original feature maps and multiple second original feature maps with different resolutions.

[0058] In one exemplary embodiment of this disclosure, the target optical flow is an output optical flow estimated based on a first original feature map and a second original feature map with a first resolution, and the method further includes:

[0059] Based on the first and second original feature maps at the current resolution, and the target optical flow estimated at the previous resolution, the target optical flow at the current resolution is estimated; wherein the current resolution is higher than the previous resolution.

[0060] The target optical flow is estimated sequentially in order of increasing resolution until the target optical flow at the second resolution is estimated.

[0061] According to a second aspect of this disclosure, an optical flow estimation apparatus is provided, comprising:

[0062] The image feature extraction module is used to obtain a first original feature map and a second original feature map, which are obtained by extracting features from two adjacent original images respectively.

[0063] An initial optical flow generation module is used to estimate the initial optical flow based on the first and second original feature maps;

[0064] A candidate optical flow generation module is used to select multiple reference coordinate anchor points based on the initial optical flow and generate candidate optical flows corresponding to each reference coordinate anchor point;

[0065] The target optical flow generation module is used to fuse the candidate optical flows to obtain the target optical flow.

[0066] According to a third aspect of this disclosure, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processing unit, implements the above-described optical flow estimation method.

[0067] According to a fourth aspect of this disclosure, an electronic device is provided, comprising:

[0068] The processing unit; and the storage unit for storing executable instructions of the processing unit; wherein the processing unit is configured to execute the above optical flow estimation method by executing the executable instructions.

[0069] The exemplary embodiments disclosed herein may have some or all of the following beneficial effects:

[0070] The optical flow estimation method provided in this exemplary embodiment of the present disclosure, by performing initial optical flow estimation based on a first and second original feature maps, helps to utilize semantic consistency features as a basis for motion estimation, thereby giving the initial optical flow a stronger overall directional guidance capability. This effectively alleviates the problem of large-scale offset errors caused by pixel-level regression error accumulation in traditional methods, and is particularly suitable for constructing the global contour of the initial motion field in low-resolution feature layers. Specifically, the first and second original feature maps are obtained by extracting features from two adjacent original images, which can transform the high-dimensional pixel data of the original images into feature representations with abstract semantics. This compresses the dimensionality of the input information while improving the comparability between images, enhancing the consolidation in subsequent processing. The ability to capture structural and contextual information provides a more stable and robust input foundation for the optical flow estimation process. Furthermore, multiple reference coordinate anchor points are selected based on the initial optical flow, and candidate optical flows corresponding to each reference coordinate anchor point are generated. This is equivalent to introducing multiple sets of motion modes with different assumptions into the feature space. This step not only expands the model's coverage of complex displacement modes, but also overcomes the limitation of traditional methods in dealing with large-scale motions due to the limited range of displacement assumptions. It still has strong predictive adaptability in cross-scale scenarios. Finally, by fusing the candidate optical flows, the effective information of multiple motion assumptions can be fully integrated, thereby generating a target optical flow with stronger coherence and more complete detail preservation, improving the stability and accuracy of the overall optical flow estimation.

[0071] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0072] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0073] Figure 1 A system architecture diagram is shown that can be applied to the optical flow estimation method in the embodiments of this disclosure.

[0074] Figure 2 A flowchart illustrating an optical flow estimation method according to an embodiment of this disclosure is shown.

[0075] Figure 3A schematic diagram of a process for generating candidate optical flows corresponding to each reference coordinate anchor point is shown in an embodiment of this disclosure.

[0076] Figure 4 A schematic diagram of a process for converting a predicted optical flow into a candidate optical flow is shown in an embodiment of this disclosure.

[0077] Figure 5 A schematic diagram of a process for fusing candidate optical flows to obtain a target optical flow is shown in an embodiment of this disclosure.

[0078] Figure 6 A schematic diagram of a refined processing flow for a target optical flow is shown in an embodiment of this disclosure.

[0079] Figure 7 A schematic diagram illustrating the principle of an optical flow estimation method according to an embodiment of this disclosure is shown.

[0080] Figure 8 A block diagram of an optical flow estimation apparatus according to an embodiment of the present disclosure is shown.

[0081] Figure 9 A schematic diagram of the structure of an electronic device suitable for implementing embodiments of the present disclosure is shown.

[0082] In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts. Detailed Implementation

[0083] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this disclosure more comprehensive and complete, and to fully convey the concept of the example embodiments to those skilled in the art. The described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a full understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced with one or more of the specific details omitted, or other methods, components, apparatus, steps, etc., can be employed. In other instances, well-known technical solutions are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0084] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0085] Figure 1 A system architecture diagram is shown that can be applied to the optical flow estimation method in the embodiments of this disclosure.

[0086] like Figure 1 As shown, system architecture 100 may include one or more terminal devices such as smartphone 101, portable computer 102, and desktop computer 103, network 104, and server 105. Network 104 is used as a medium to provide a communication link between the terminal devices and server 105. Network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables. Terminal devices may be various electronic devices with image processing and display functions, and may have a display screen for displaying two adjacent frames of original images, estimated initial optical flow, candidate optical flow, target optical flow, etc.

[0087] It is understood that the terminal device can also be any electronic device with image acquisition capabilities. For example, the two adjacent original images used for optical flow estimation can be acquired in real time by an electronic device. Of course, the two adjacent original images can also be pre-acquired by other image acquisition devices and stored in the terminal device or server 105, and then called up when optical flow estimation is performed. This disclosure does not limit this.

[0088] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, there can be any number of terminal devices, networks, and servers. For example, server 105 could be a server cluster composed of multiple servers.

[0089] This disclosure provides an example implementation of an optical flow estimation method, referencing... Figure 2 As shown, the method may include the following steps S210 to S240:

[0090] Step S210: Obtain the first original feature map and the second original feature map. The first original feature map and the second original feature map are obtained by extracting features from two adjacent original images respectively.

[0091] Step S220: Estimate the initial optical flow based on the first and second original feature maps;

[0092] Step S230: Select multiple reference coordinate anchor points based on the initial optical flow, and generate candidate optical flows corresponding to each reference coordinate anchor point;

[0093] Step S240: The candidate optical flows are fused to obtain the target optical flow.

[0094] The optical flow estimation method provided by the exemplary embodiments of this disclosure helps to form a motion prior with overall directional guidance by estimating the initial optical flow based on the original feature map, thus mitigating the large displacement estimation bias caused by the accumulation of pixel-level errors in traditional methods. Moreover, the original feature map can transform high-dimensional image information into a stable feature representation with semantic expression, enhancing structural and context-aware capabilities and providing robust input for optical flow estimation. Furthermore, by selecting multiple reference coordinate anchor points to generate candidate optical flows and introducing multiple sets of motion hypotheses to cover a larger displacement range, the adaptability of the model in complex or cross-scale scenes can be improved. Finally, by fusing the candidate optical flows, multiple motion path information can be integrated to generate a more coherent and accurate target optical flow, thereby improving the overall estimation effect.

[0095] The optical flow estimation method in this example embodiment will now be described in detail.

[0096] In step S210, a first original feature map and a second original feature map are obtained. The first original feature map and the second original feature map are obtained by extracting features from two adjacent original images respectively.

[0097] In this context, two adjacent original images refer to two consecutively acquired images, such as the first original image and the second original image. They can come from the same video sequence or a continuously sampled image stream, and are used to reflect the positional changes of objects at adjacent time points. Feature extraction from the two original images involves encoding the input images using a pre-defined feature extraction network, such as a convolutional neural network or a Transformer, to extract intermediate feature representations containing spatial structural and semantic information. The image feature extraction process compresses and transforms information from the original images, making subsequent tasks easier to model and compute. Correspondingly, the first and second original feature maps correspond to the output results of the first and second original images after processing by the feature extraction network, respectively. Each original feature map can be a single-scale feature map or a multi-scale feature map, preserving key information such as edges, textures, and semantics of the input image, serving as the input basis for subsequent optical flow estimation.

[0098] Transforming two consecutive original images into discriminative feature representations, which serve as the semantic and structural basis for subsequent optical flow modeling, not only reduces the computational burden of redundant data but also enhances the model's responsiveness to contextual structure, local alignment, and large motion recognition.

[0099] In some example implementations, each original image can be input into a fourth neural network to obtain multiple first original feature maps and multiple second original feature maps with different resolutions, that is, to obtain the corresponding multi-scale feature map sequence.

[0100] For example, the fourth neural network can be a deep neural network with multi-scale feature extraction capabilities, such as a ResNet (Residual Network) network or a MobileNet (Mobile Neural Network) network with a pyramid structure, an Inception (deep embedding) network with a multi-branch structure, or a neural network with a dilated convolution stacking structure. It should be understood that this disclosure does not limit the specific type of the fourth neural network; other network structures with similar multi-scale feature extraction capabilities can be selected according to actual application requirements.

[0101] Taking the ResNet network as an example, during the feature extraction process, the network processes each original input image step by step through stacked convolutional modules and downsampling operations (such as convolution or pooling with a stride of 2), thereby forming a series of multi-scale feature maps with gradually decreasing resolution and gradually increasing semantic level, which constitutes a typical pyramid-shaped feature map sequence.

[0102] Specifically, each original image is first preprocessed through an initial convolutional layer, and then sequentially fed into multiple residual modules. Each residual module contains multiple convolutional units, and downsampling operations are introduced in some residual modules to proportionally reduce the spatial dimension of the feature map while extracting higher-level semantic information. In this way, the neural network can construct multi-scale feature representations with different levels of semantic abstraction layer by layer.

[0103] Taking the initial resolution of each original image as H×W as an example, the feature extraction process can output feature maps at multiple scales, such as: the first layer outputs a feature map with a resolution of H×W, denoted as feat1; the second layer outputs a feature map with a resolution of H / 2×W / 2, denoted as feat2; the third layer outputs a feature map with a resolution of H / 4×W / 4, denoted as feat3; the fourth layer outputs a feature map with a resolution of H / 8×W / 8, denoted as feat4, and so on, until the last layer outputs a high-order feature map with the smallest resolution but the richest semantics.

[0104] In the exemplary embodiment of this disclosure, after inputting the first frame of the original image into the fourth neural network, a set of feature maps with different resolutions can be obtained, i.e., multiple first original feature maps are obtained. The same processing is applied to the second frame of the original image to obtain corresponding multiple second original feature maps. Since both are processed using a neural network with a shared structure, the output multiple first original feature maps and multiple second original feature maps have consistent spatial structure and resolution characteristics at corresponding scales, thus providing a structural matching basis for subsequent feature alignment, optical flow estimation, and fusion operations.

[0105] Multi-scale feature extraction of images can provide coarse-to-fine contextual information support for subsequent optical flow estimation modules, facilitating not only the identification of local details but also the modeling of large-scale motion. Simultaneously, multi-scale features can be used to construct a pyramid-structured decoder, progressively improving the accuracy of optical flow estimation.

[0106] In step S220, the initial optical flow is estimated based on the first original feature map and the second original feature map.

[0107] The initial optical flow refers to the displacement information of pixel movement between two original images estimated based on the feature maps extracted from the two original images at the current resolution scale. It is used to provide a motion direction reference and an initial value for the search range in the subsequent optical flow thinning process.

[0108] For example, the first and second original feature maps can be input into a neural network for optical flow estimation. This network can calculate the displacement vector of each pixel in the time dimension based on the feature differences and spatial correspondence between the two maps, and then output the initial optical flow. For example, each pixel position in the initial optical flow corresponds to a two-dimensional vector, representing the displacement experienced by the pixel at that position in the first frame original feature map to the corresponding position in the second frame original feature map.

[0109] Understandably, the neural network used for optical flow estimation can adopt any deep neural network structure suitable for feature matching modeling, such as convolutional neural networks. The specific network structure can be flexibly configured according to the resolution, number of channels, and task requirements of the selected feature map.

[0110] By performing matching learning on two original feature maps in the feature space, the initial optical flow can reflect the global motion trend while taking into account certain local details, thus providing an initial displacement estimation result with prior guidance for subsequent operations such as candidate optical flow generation and target optical flow fusion.

[0111] In some example implementations, when estimating the initial optical flow based on the first and second original feature maps, the current optical flow can be initialized based on the first and second original feature maps at a first resolution. Then, the first original feature map is spatially transformed according to the current optical flow to obtain a first aligned feature map. Finally, the initial optical flow is calculated based on the first aligned feature map and the second original feature map. Here, the current optical flow is in spatial coordinate form. It should be noted that this initial optical flow is in polar coordinate form, that is, the optical flow is represented in polar coordinates, including an initial amplitude component and an initial direction component. For example, it can be denoted as (r0, θ0), where r0 represents the initial amplitude component and θ0 represents the initial direction component.

[0112] For example, the first and second original feature maps can be the lowest-resolution feature maps output by the fourth neural network, such as the bottom-level feature maps in a pyramid structure network. The lowest-resolution feature maps have a large receptive field and strong global semantic representation ability, which can be used to make a preliminary estimate of the overall optical flow field, thus providing an initial reference for the optical flow refinement of subsequent high-resolution feature layers.

[0113] For the first and second original feature maps at the first resolution, the initial optical flow can be set to zero field, indicating that no prior motion information is introduced at this stage, ensuring the universality and stability of the estimation process. Next, a spatial transformation is performed on the first original feature map based on the current optical flow. This spatial transformation can be implemented using a reverse mapping method based on bilinear interpolation. That is, for each pixel position, the pixel value at the corresponding position in the first original feature map is mapped to the target position according to the displacement vector in the current optical flow, thereby obtaining a feature representation aligned spatially with the second original feature map, i.e., obtaining the first aligned feature map.

[0114] Understandably, when estimating the target optical flow at the second lowest resolution, the current optical flow used for spatial transformation of the first original feature map should be the target optical flow at the lowest resolution, not the initialized current optical flow. Similarly, when estimating the target optical flow at the highest resolution, the current optical flow used for spatial transformation should be the target optical flow at the second highest resolution.

[0115] Finally, when calculating the initial optical flow based on the first aligned feature map and the second original feature map, a fused feature map can be obtained by splicing the first aligned feature map and the second original feature map, and feature extraction and displacement regression operations can be performed on the fused feature map to obtain the initial optical flow.

[0116] For example, the initial optical flow can be obtained by performing convolution operations on the fused feature map. Specifically, the fused feature map can be input into a lightweight network containing multiple convolutional layers, and the optical flow displacement information of each pixel can be calculated and output through continuous convolution, which is the initial optical flow.

[0117] Since this process is performed at the lowest resolution level, it can effectively capture global motion trends and provide coarse-grained displacement estimates with low computational cost, laying the foundation for subsequent multi-scale refinement processes. Furthermore, using polar coordinates to regress the direction and magnitude of pixel offsets not only better aligns with the physical nature of optical flow but also explicitly introduces the coupling relationship between the horizontal and vertical components. This alleviates the decoupling error problem caused by independent component regression in traditional Cartesian coordinates, improving overall fitting accuracy and estimation stability.

[0118] In step S230, multiple reference coordinate anchor points are selected based on the initial optical flow, and candidate optical flows corresponding to each reference coordinate anchor point are generated.

[0119] As shown in step S220, the initial optical flow is a coarse optical flow information estimated at the first resolution feature layer, which can reflect the overall motion trend between image sequences. The candidate optical flow is a set of diverse and comprehensive optical flow estimation results corresponding to the image feature structure, used to characterize different motion distributions that may exist at the current pixel. The initial optical flow provides a central reference or offset basis for the generation of candidate optical flows, while the candidate optical flow, based on the initial optical flow, introduces multiple sets of directional and amplitude perturbations to construct a richer set of motion hypotheses, used to cover local pixel motions of a wider range or more complex forms, thereby improving the accuracy and robustness of subsequent optical flow estimation.

[0120] Based on this initial optical flow, multiple reference coordinate anchor points can be constructed in the feature space to establish the candidate optical flow search range in subsequent steps. In the exemplary embodiment of this disclosure, the reference coordinate anchor points can be regarded as a set of representative motion assumptions preset in the optical flow estimation space. By introducing a perturbation combination of amplitude scale and direction angle on the basis of the initial optical flow, a set of reference vectors covering various possible pixel displacement scenarios is constructed. Simply put, the reference coordinate anchor points can be understood as a set of preset motion reference points, which have a certain degree of discreteness and representativeness in direction or amplitude.

[0121] In some example implementations, when selecting multiple reference coordinate anchor points based on the initial optical flow, specifically: based on the initial amplitude component and the initial direction component in the initial optical flow, multiple reference amplitude components and multiple reference direction components are set respectively, and multiple reference coordinate anchor points are generated by combining each reference amplitude component with each reference direction component.

[0122] Specifically, for the initial optical flow (r0, θ0), it can be discretely segmented or sampled at fixed intervals, and multiple representative reference amplitude components and reference direction components can be set based on the initial optical flow (r0, θ0).

[0123] For example, for the initial direction component θ0, a set of normalized reference direction components anchor_θ can be set to cover possible motion directions from 0 to 2π. For example, the value of anchor_θ can be [0, 1], which represents the direction angle within the unit circle. It can also be further refined into multiple equally spaced direction anchor points, such as [0, 0.25, 0.5, 0.75, 1.0], which means that the direction angle is divided on the unit circle in an equally spaced manner. Each value represents the proportional relationship with the normalized circumference (2π radians) to improve the coverage density and direction resolution of different motion directions.

[0124] For the initial amplitude component r0, multiple reference amplitude components anchor_r can be set, such as [1, 2, 3, 4, 6]. These represent multi-scale assumptions about optical flow intensity (i.e., the amplitude of pixel displacement) in polar coordinate space, adapting to various motion intensity requirements such as small displacement, normal displacement, and large-scale displacement. For example, anchor_r=1 represents a unit pixel-level displacement assumption, suitable for subtle motion detection; anchor_r=2 represents a medium-amplitude displacement assumption, used for modeling normal object or background motion; and anchor_r=6 represents a large-scale pixel displacement, suitable for scenarios such as fast-moving targets or drastic camera changes. Selecting multiple reference amplitude components can cover different displacement scale ranges, improving the adaptability of the candidate optical flow generation module to complex motion distributions.

[0125] Next, by pairwise combining multiple reference amplitude components with multiple reference direction components, that is, pairing each reference amplitude with each reference direction, a set of reference coordinate anchor points with different combinations of motion amplitude and direction can be generated, which can be denoted as (ar, aθ). It can be seen that the process of combining reference amplitude components and reference direction components essentially constructs a motion hypothesis space, and each reference coordinate anchor point corresponds to a possible motion trend, thus providing a parameter basis for the subsequent generation of candidate optical flows corresponding to that anchor point.

[0126] After selecting multiple reference anchor points, the reference Figure 3 As shown, the process of generating candidate optical flows corresponding to each reference coordinate anchor point may include the following steps S310 and S320:

[0127] Step S310: Generate the predicted optical flow corresponding to each reference coordinate anchor point through the first neural network.

[0128] The predicted optical flow refers to the estimated optical flow offset output by the first neural network given reference coordinate anchor points, which is used to fine-tune the motion assumptions represented by each reference coordinate anchor point. The first neural network can be an MLP (Multilayer Perceptron) network, a lightweight Transformer structure, a convolutional neural network, etc., as long as it can perform local refinement based on the initial optical flow. This disclosure does not limit it in this regard.

[0129] For example, each reference coordinate anchor point can be input into a first neural network to obtain the predicted amplitude component and predicted direction component corresponding to each reference coordinate anchor point. Optionally, the first neural network can be a neural network with a two-branch regression structure, used to output the predicted amplitude component and predicted direction component, respectively, such as denoted as pr and pθ. The first neural network can learn the optimal local motion correction at the current reference anchor point by inputting feature information related to the reference coordinate anchor point, such as the polar coordinate position of the anchor point and the contextual representation of the surrounding feature map region.

[0130] Then, the predicted optical flow corresponding to each reference coordinate anchor point is determined based on the predicted amplitude component and the predicted direction component. For example, the predicted optical flow can be formed by combining the predicted amplitude component pr and the predicted direction component pθ, which can be denoted as (pr, pθ), that is, the displacement estimation result corresponding to the current reference anchor point is represented in polar coordinates.

[0131] This disclosure avoids direct regression on two-dimensional optical flow components, which facilitates the model's explicit learning of the coupling relationship between motion direction and motion amplitude, and improves estimation performance in regions of large motion or drastic direction changes. Furthermore, the predicted optical flow can be further used to form a candidate optical flow set with reference coordinate anchor points, participating in optical flow fusion and refinement operations.

[0132] Step S320: Convert the predicted optical flow corresponding to each reference coordinate anchor point into the candidate optical flow corresponding to each reference coordinate anchor point.

[0133] For example, refer to Figure 4 As shown, the process of converting the predicted optical flow into a candidate optical flow may include the following steps S410 and S420:

[0134] Step S410: Calculate the first intermediate optical flow corresponding to each reference coordinate anchor point based on each reference coordinate anchor point and the predicted optical flow corresponding to each reference coordinate anchor point.

[0135] Specifically, the intermediate amplitude component can be obtained by multiplying the reference amplitude component in each reference coordinate anchor point and the predicted amplitude component in the predicted optical flow corresponding to each reference coordinate anchor point, as follows:

[0136] r1 = ar × pr

[0137] Where r1 is the intermediate amplitude component, ar is the reference amplitude component, and pr is the predicted amplitude component;

[0138] Adding the reference direction components in each reference coordinate anchor point and the predicted direction components in the corresponding predicted optical flow at each reference coordinate anchor point yields the intermediate direction component:

[0139] θ1=aθ+pθ

[0140] Where θ1 is the intermediate direction component, aθ is the reference direction component, and pθ is the predicted direction component;

[0141] Finally, the first intermediate optical flow, denoted as (r1, θ1), is determined based on the intermediate amplitude component and the intermediate direction component. The resulting first intermediate optical flow is still in polar coordinate form and is used to represent the corrected motion estimation result relative to the reference anchor point position.

[0142] Step S420: Perform coordinate transformation on the first intermediate optical flow to obtain the second intermediate optical flow, and superimpose the initialized current optical flow to calculate the candidate optical flow corresponding to each reference coordinate anchor point.

[0143] For example, the second intermediate optical flow, the current optical flow, and the candidate optical flow are all optical flows in spatial coordinate form. Accordingly, the first intermediate optical flow is transformed in coordinates to obtain the second intermediate optical flow, specifically:

[0144] x1 = r1cos(θ1 * 2π)

[0145] y1 = r1sin(θ1 * 2π)

[0146] Where r1 is the intermediate amplitude component, θ1 is the intermediate direction component, x1 is the first horizontal component, and y1 is the first vertical component. The second intermediate optical flow is determined by the first horizontal component x1 and the first vertical component y1, and is denoted as (x1, y1).

[0147] After obtaining the second intermediate optical flow, the corresponding candidate optical flow is calculated by superimposing it with the initialized current optical flow. Specifically, the second intermediate optical flow and the initialized current optical flow can be superimposed at the pixel level to obtain the corresponding candidate optical flow.

[0148] For example, if the current optical flow is denoted as (fx_p, fy_p), the corresponding candidate optical flow is:

[0149] x2=x1+fx_p

[0150] y2=y1+fy_p

[0151] Where x2 is the second horizontal component, y2 is the second vertical component, x1 is the first horizontal component, y1 is the first vertical component, fx_p is the third horizontal component in the current optical flow, and fy_p is the third vertical component in the current optical flow. Finally, the candidate optical flow can be determined by the second horizontal component x2 and the second vertical component y2, denoted as (x2, y2).

[0152] Specifically, when estimating the target optical flow at the first resolution (e.g., the lowest resolution), if the initialized current optical flow is zero field, then both fx_p and fy_p are 0. It is understandable that when estimating the target optical flow at the second lowest resolution, the current optical flow to be superimposed should be the target optical flow at the lowest resolution, not the initialized current optical flow. Similarly, when estimating the target optical flow at the second resolution (e.g., the highest resolution), the current optical flow to be superimposed should be the target optical flow at the second highest resolution. This disclosure will not elaborate further on this point.

[0153] This disclosure generates candidate optical flows based on multiple reference coordinate anchor points. Guided by the initial optical flow, it constructs a set of motion hypothesis solutions covering different combinations of amplitude and direction for each pixel position, thus significantly expanding the model's modeling space for motion displacement distribution. Compared to traditional methods that predict based on only a single optical flow regression path, this disclosure constructs multiple candidate directions and amplitudes by setting multiple reference coordinate anchor points, improving the model's ability to express complex and non-uniform motion regions. Especially in scenarios such as fast-moving targets and long-distance cross-frame motion, it exhibits stronger matching robustness and adaptability.

[0154] In addition, each reference coordinate anchor point corresponds to a candidate optical flow, which enables the subsequent fusion module to select a better combination of components from multiple candidate optical flows when facing estimation uncertainties such as occlusion, blurring, and texture loss. This improves the continuity and accuracy of the final optical flow estimation results and enhances the global consistency and local detail preservation capabilities of the overall estimation process.

[0155] In step S240, the candidate optical flows are fused to obtain the target optical flow.

[0156] The fusion process refers to the joint analysis of all candidate optical flows based on a pre-defined fusion strategy or a trained neural network model. This analysis may involve weighted averaging, attention-based fusion, feature stacking regression, or confidence evaluation filtering. The goal is to integrate the different directional, amplitude, and contextual features carried by multiple candidate optical flows to generate a more accurate and consistent optical flow output. Correspondingly, the target optical flow is the optical flow estimation result obtained after fusion processing. It represents the spatial displacement vector of each pixel between two adjacent frames on the time axis, exhibiting higher prediction stability, accuracy, and robustness. It can be directly used in subsequent visual tasks such as image registration, video analysis, and 3D reconstruction.

[0157] In some example implementations, references Figure 5 As shown, the process of fusing candidate optical flows to obtain the target optical flow may include the following steps S510 to S540:

[0158] Step S510: Input the first original feature map and the second original feature map into the second neural network to obtain the first deep feature map and the second deep feature map.

[0159] The second neural network can be a ResNet network, a U-Net (U-shaped network), a feature enhancement network with an attention mechanism, a convolutional neural network, etc., which can further mine the structural information and contextual relationships hidden in the original feature map.

[0160] For example, during feature processing, the second neural network can perform deep modeling on the input first and second original feature maps through a series of convolutions, non-linear activations, normalization, and upsampling operations, extracting semantic features with stronger discriminative power layer by layer. The output first and second deep feature maps will maintain consistency in spatial structure and have a higher level of semantic abstraction compared to the original feature maps, thus providing a rich and stable feature foundation for subsequent fusion processing.

[0161] Step S520: Based on each candidate optical flow, perform spatial transformation on the first deep feature map to obtain the second aligned feature map corresponding to each candidate optical flow.

[0162] By aligning the first deep feature map with the displacement information of different candidate optical flows, a feature representation that is highly consistent with the second deep feature map in spatial location is constructed, providing a registration basis for subsequent feature fusion and optical flow estimation.

[0163] For example, the pixel positions in the first deep feature map can be transformed according to the displacement vector in each candidate optical flow to obtain the second aligned feature map corresponding to each candidate optical flow.

[0164] Each candidate optical flow can be viewed as a pixel-level spatial displacement vector, where each component represents the motion displacement of the current pixel within the image sequence. For example, a reverse mapping strategy can be used, guided by the candidate optical flows, to map and transform the pixel coordinates in the first deep feature map, thereby generating the corresponding second aligned feature map.

[0165] Furthermore, the mapping transformation process can use bilinear interpolation to resample pixel values. That is, for each pixel in the first deep feature map, the new position to which the pixel should be mapped is calculated based on the displacement vector provided by the candidate optical flow, and the feature value of the target position is obtained by interpolation, thereby constructing the second aligned feature map under the action of the candidate optical flow.

[0166] The resulting multiple second-aligned feature maps correspond to the pixel motion patterns assumed by each candidate optical flow. Moreover, the second-aligned feature maps will fit the second deep feature maps as closely as possible in terms of spatial structure, providing a better matching basis and facilitating subsequent similarity analysis or fusion processing, thereby improving the accuracy and robustness of the final optical flow estimation.

[0167] Step S530: Match each second aligned feature map with the second deep feature map to obtain the matching result.

[0168] By measuring the similarity between the aligned second feature map and the second deep feature map, we can determine the fitting ability of each candidate optical flow to the current motion pattern, thus providing a basic reference for subsequent fusion operations.

[0169] For example, the matching results can be represented in the form of similarity scores. Specifically, the similarity between each second aligned feature map and the second deep feature map can be calculated to obtain multiple similarity scores. Similarity calculation methods may include, but are not limited to, cosine similarity, negative Euclidean distance, and dot product responses of feature points.

[0170] Step S540: Based on the matching results, the candidate optical flows are fused to obtain the target optical flow.

[0171] Taking the example of obtaining multiple similarity scores as the matching result, in order to achieve effective fusion, the similarity scores can be normalized to obtain the similarity coefficients corresponding to each similarity score. The normalization process can include the Softmax function, linear normalization, or maximum value normalization, etc. After normalization, multiple corresponding similarity coefficients can be obtained, and each similarity coefficient can reflect the fitting reliability of the current candidate optical flow to the real motion.

[0172] Then, using the similarity coefficients as weights, the corresponding candidate optical flows are weighted and summed to obtain the target optical flow, denoted as (fx, fy), where fx is the fourth horizontal component and fy is the fourth vertical component in the target optical flow. This target optical flow integrates effective information from multiple motion hypotheses at the pixel level, preserving accurate predictions for regions with high confidence while balancing the differences between various possible motion paths, thus improving the continuity and consistency of the overall optical flow estimation.

[0173] In the process of fusing multiple candidate optical flows, this disclosure can make full use of the complementarity of each candidate optical flow in terms of spatial scale, direction of motion and semantic level, effectively alleviate the local error or information loss problem that may be caused by a single estimation path, thereby improving the performance of the final optical flow estimation in terms of boundary preservation, detail representation and global coherence.

[0174] Furthermore, after obtaining the target optical flow, to further improve the accuracy of the optical flow estimation results, the target optical flow can be refined to reduce the prediction error in the optical flow estimation. (Reference) Figure 6 As shown, the refined processing flow of the target optical flow may include the following steps S610 to S630:

[0175] Step S610: Based on the target optical flow, perform spatial transformation on the first deep feature map to obtain the third aligned feature map.

[0176] Specifically, the pixel displacement vector in the target optical flow can be used as a guide to perform coordinate transformation on the position of each pixel in the first deep feature map, mapping it to a position that is aligned with the second deep feature map, thus forming a third aligned feature map.

[0177] For example, spatial transformation operations of feature maps can be implemented using methods such as inverse bilinear interpolation to ensure the effective transmission and alignment of pixel information in continuous space.

[0178] Step S620: Input the third aligned feature map and the second deep feature map into the third neural network to obtain the optical flow increment.

[0179] The third neural network can be a lightweight convolutional network containing multiple convolutional layers, or a refined module containing an attention mechanism. Based on the difference between the third aligned feature map and the second deep feature map, the third neural network can learn and output a fine-grained optical flow compensation amount, i.e., optical flow increment, which can be denoted as (Δx, Δy) to correct local errors in the target optical flow.

[0180] Step S630: Adjust the target optical flow according to the optical flow increment to obtain the adjusted target optical flow.

[0181] For example, there are:

[0182] Fx=Δx+fx

[0183] Fy=Δy+fy

[0184] Where fx is the fourth horizontal component of the target optical flow, fy is the fourth vertical component of the target optical flow, Δx is the fifth horizontal component of the optical flow increment, Δy is the fifth vertical component of the optical flow increment, Fx is the fifth horizontal component of the adjusted target optical flow, and Fy is the fifth vertical component of the adjusted target optical flow.

[0185] This example implementation, by introducing refined compensation, can effectively repair underfitting or error regions in the initial estimation, and improve the prediction accuracy and stability of the entire optical flow estimation model in key regions such as structural edges and motion details.

[0186] In some example implementations, the adjusted target optical flow can also be regularized based on the first deep feature map or the second deep feature map to obtain the final target optical flow.

[0187] Regularization can be implemented through a pre-defined regularization module. For example, convolutional neural networks, residual block structures, attention mechanisms, or other context-aware neural network structures can be used to suppress and correct abnormal displacements, boundary blurring, or noise responses that may exist in the optical flow. Another example is the use of bilateral filtering to smooth the adjusted target optical flow by combining the spatial distance and color differences between pixels, effectively avoiding the loss of edge information due to over-smoothing. Other regularization methods are also possible, and this disclosure does not limit them.

[0188] By combining the semantic and structural information provided by the deep feature map, the adjusted target optical flow is subjected to edge sharpening, smoothing, or confidence enhancement, resulting in a smoother, more continuous, and clearly defined final target optical flow, thereby further improving the stability and accuracy of the overall optical flow estimation.

[0189] It should be noted that, taking the first resolution as the lowest resolution as an example, after estimating the target optical flow based on the first and second original feature maps at the lowest resolution, the optical flow can be further refined step by step based on the image feature information at a higher resolution to improve the overall spatial accuracy and structural reconstruction capability of the estimation.

[0190] For example, the target optical flow at the current resolution can be estimated based on the first and second original feature maps at the current resolution, as well as the target optical flow estimated at the previous resolution. Since the current resolution is higher than the previous resolution, the target optical flow obtained at the previous resolution can be used as initial optical flow guidance information for estimation at the current resolution, thereby reducing initial matching errors and providing a reference for the global motion trend.

[0191] To achieve progressively refined optical flow estimation, the target optical flow estimation process can be executed sequentially from low to high resolution, i.e., steps S220 to S240, until the target optical flow at the second resolution, i.e. the highest resolution, is estimated. This completes the entire multi-scale optical flow estimation process, ensuring a smooth transition and accurate modeling from rough outlines to local details.

[0192] refer to Figure 7 The diagram shown illustrates the principle of an optical flow estimation method. Figure 7 In the process, for two adjacent original images, namely the first original image 701 and the second original image 702, when estimating the optical flow between the first original image 701 and the second original image 702, the first original image 701 and the second original image 702 can be input into the feature extraction network 703 respectively to obtain a set of multi-scale feature images, which are used for subsequent optical flow estimation processes at different resolution levels.

[0193] Of course, in some example implementations, pre-generated multi-scale feature images can be directly obtained, and optical flow estimation can be performed based on these images. By directly calling pre-generated feature images, the real-time computational burden on the feature extraction network can be reduced, improving the overall efficiency of the optical flow estimation process. Simultaneously, the feature hierarchy structure can be adaptively optimized during the feature generation stage according to different scenarios or data sources, thereby further improving the accuracy and robustness of optical flow estimation.

[0194] Specifically, the feature extraction network 703 can be a deep neural network with a pyramid structure. Through hierarchical convolution and downsampling operations, it sequentially extracts semantic feature maps of different scales. The resulting feature maps include the lowest resolution original feature map 7031, the second low resolution original feature map 7032, ..., the second high resolution original feature map 7033 and the highest resolution original feature map 7034.

[0195] It should be noted that a set of corresponding first original feature map 704 and second original feature map 705 can be obtained at each resolution, which are used to describe the structural feature information of the two original images at that scale.

[0196] Throughout the optical flow estimation process, a progressive optical flow optimization is performed using a multi-scale hierarchical order from low to high. That is, a coarse optical flow is first estimated at the lowest resolution level, and then refined layer by layer to a high resolution level.

[0197] in, Figure 7 In this context, flow(i-1) represents the target optical flow estimated from the feature map of layer (i-1). It is used as the initial reference for the optical flow estimation of layer i. If the current layer is the lowest resolution layer, assuming i=1, then flow(i-1) is flow(0), which is set as the initial optical flow field, such as a zero field.

[0198] Taking the lowest resolution first original feature map 704 and second original feature map 705 as examples, the corresponding optical flow estimation process includes the following steps:

[0199] Initial optical flow estimation steps: When estimating the initial optical flow 707 based on the lowest resolution first original feature map 704 and second original feature map 705, the initialized flow(0) is first combined with the first original feature map 704, and a spatial transformation operation is performed to obtain the aligned feature map 706. This spatial transformation can use the inverse bilinear interpolation method to map the pixels in the original feature map to the target position according to the optical flow information, thereby achieving spatial alignment. Next, the aligned feature map 706 and the second original feature map 705 are concatenated in the channel dimension to form a fused feature input, and feature fusion and regression are performed through several convolutional layers. The final output is represented as the initial optical flow 707 in polar coordinates, including the initial direction component and the initial magnitude component.

[0200] Predicted optical flow generation steps: Based on the direction and amplitude information in the initial optical flow 707, a set of reference coordinate anchor points (multiple anchors) is constructed. This set of anchor points is representative and comprehensive in both direction and amplitude dimensions, and can capture a variety of possible motion patterns.

[0201] Subsequently, each reference anchor point is input into the optical flow prediction network to obtain the corresponding predicted amplitude component and predicted direction component, which are then combined to form multiple predicted optical flows 708. These predicted optical flows are represented in polar coordinates and are used to simulate pixel motion trends under different assumptions.

[0202] Candidate optical flow generation steps: Convert each predicted optical flow 708 into an optical flow in spatial coordinate form, including horizontal and vertical components, and combine it with flow(0), which is also in spatial coordinate form, for coordinate correction to obtain multiple candidate optical flows 709;

[0203] Target optical flow generation steps: A fusion operation is performed on all candidate optical flows 709, which can be weighted and integrated based on similarity, confidence, or neural network strategies. By integrating multiple motion hypothesis information, a target optical flow 710 with stronger structural continuity and smaller error is obtained;

[0204] Target optical flow adjustment steps: To further improve the estimation accuracy, the target optical flow 710 can be refined based on the first deep feature map and the second deep feature map to obtain the adjusted target optical flow 711; then, optional regularization can be performed to enhance the spatial consistency and smoothness of the flow field, and finally the final target optical flow flow (1) at this resolution level can be obtained.

[0205] Furthermore, after completing the optical flow estimation of the lowest resolution layer, the current target optical flow flow(i) can be upsampled sequentially and input together with the first and second original feature maps of the current higher resolution into the optical flow estimation module of the corresponding layer, and the above optical flow estimation steps can be repeated.

[0206] Finally, after the target optical flow estimation is completed at the highest resolution level, the pixel-level complete optical flow between the first original image 701 and the second original image 702 can be obtained, which can be used for subsequent visual tasks such as target tracking, 3D modeling, and motion analysis.

[0207] In an exemplary embodiment of this disclosure, an optical flow estimation apparatus is also provided. (See reference...) Figure 8 As shown, the optical flow estimation device 800 includes an image feature extraction module 801, an initial optical flow generation module 802, a candidate optical flow generation module 803, and a target optical flow generation module 804, wherein:

[0208] The image feature extraction module 801 is used to obtain a first original feature map and a second original feature map, which are obtained by extracting features from two adjacent original images respectively.

[0209] The initial optical flow generation module 802 is used to estimate the initial optical flow based on the first original feature map and the second original feature map;

[0210] The candidate optical flow generation module 803 is used to select multiple reference coordinate anchor points based on the initial optical flow and generate candidate optical flows corresponding to each reference coordinate anchor point.

[0211] The target optical flow generation module 804 is used to fuse the candidate optical flows to obtain the target optical flow.

[0212] In one alternative implementation, the initial optical flow generation module 802 includes:

[0213] The optical flow initialization submodule is used to initialize the current optical flow based on the first and second original feature maps with a first resolution.

[0214] The first feature map processing submodule is used to perform spatial transformation on the first original feature map according to the current optical flow to obtain the first aligned feature map;

[0215] The initial optical flow determination submodule is used to calculate the initial optical flow based on the first aligned feature map and the second original feature map.

[0216] In one alternative implementation, the initial optical flow determination submodule includes:

[0217] The feature map fusion unit is used to stitch together the first aligned feature map and the second original feature map to obtain a fused feature map.

[0218] The initial optical flow determination unit is used to perform convolution operations on the fused feature map to obtain the initial optical flow.

[0219] In one alternative implementation, the initial optical flow is a polar coordinate optical flow, including an initial amplitude component and an initial direction component.

[0220] In one optional implementation, the candidate optical flow generation module 803 includes:

[0221] The optical flow parameter setting submodule is used to set multiple reference amplitude components and multiple reference direction components based on the initial amplitude component and initial direction component in the initial optical flow.

[0222] The coordinate anchor point generation submodule is used to combine each reference amplitude component with each reference direction component to generate multiple reference coordinate anchor points.

[0223] In an optional implementation, the candidate optical flow generation module 803 further includes:

[0224] The predicted optical flow generation submodule is used to generate the predicted optical flow corresponding to each reference coordinate anchor point through the first neural network;

[0225] The candidate optical flow generation submodule is used to convert the predicted optical flow corresponding to each reference coordinate anchor point into the candidate optical flow corresponding to each reference coordinate anchor point.

[0226] In one alternative implementation, the predicted optical flow generation submodule includes:

[0227] The optical flow parameter prediction unit is used to input each reference coordinate anchor point into the first neural network to obtain the predicted amplitude component and predicted direction component corresponding to each reference coordinate anchor point.

[0228] The predicted optical flow determination unit is used to determine the predicted optical flow corresponding to each reference coordinate anchor point based on the predicted amplitude component and the predicted direction component corresponding to each reference coordinate anchor point.

[0229] In one optional implementation, the candidate optical flow generation submodule includes:

[0230] The first intermediate optical flow calculation unit is used to calculate the first intermediate optical flow corresponding to each reference coordinate anchor point based on each reference coordinate anchor point and the predicted optical flow corresponding to each reference coordinate anchor point.

[0231] The candidate optical flow generation unit is used to perform coordinate transformation on the first intermediate optical flow to obtain the second intermediate optical flow, and superimpose the initialized current optical flow to calculate the candidate optical flow corresponding to each reference coordinate anchor point.

[0232] In one alternative implementation, the first intermediate optical flow is an optical flow in polar coordinates, while the second intermediate optical flow, the current optical flow, and the candidate optical flow are optical flows in spatial coordinates.

[0233] In one alternative implementation, the first intermediate optical flow calculation unit includes:

[0234] The intermediate amplitude component calculation subunit is used to multiply the reference amplitude component in each reference coordinate anchor point with the prediction amplitude component in the corresponding prediction optical flow to obtain the intermediate amplitude component.

[0235] The intermediate direction component calculation subunit is used to add the reference direction component in each reference coordinate anchor point to the prediction direction component in the corresponding prediction optical flow to obtain the intermediate direction component.

[0236] The first intermediate optical flow calculation subunit is used to determine the first intermediate optical flow based on the intermediate amplitude component and the intermediate direction component.

[0237] In one alternative implementation, the target optical flow generation module 804 includes:

[0238] The deep feature extraction submodule is used to input the first original feature map and the second original feature map into the second neural network to obtain the first deep feature map and the second deep feature map;

[0239] The second feature map processing submodule is used to perform spatial transformation on the first deep feature map based on each candidate optical flow to obtain the second aligned feature map corresponding to each candidate optical flow.

[0240] The deep feature matching submodule is used to match each second aligned feature map with the second deep feature map to obtain the matching result;

[0241] The candidate optical flow fusion submodule is used to fuse the candidate optical flows according to the matching results to obtain the target optical flow.

[0242] In one alternative implementation, the deep feature processing submodule is configured to perform position transformation processing on the pixel positions in the first deep feature map according to the displacement vector in each candidate optical flow, so as to obtain the second aligned feature map corresponding to each candidate optical flow.

[0243] In one optional implementation, the matching result includes a similarity score; the deep feature matching submodule includes:

[0244] The deep feature matching unit is used to match each second aligned feature map with the second deep feature map to obtain matching results, including:

[0245] The similarity calculation unit is used to calculate the similarity between each second aligned feature map and the second deep feature map, and obtain multiple similarity scores.

[0246] In one alternative implementation, the candidate optical flow fusion submodule includes:

[0247] The similarity coefficient calculation unit is used to normalize each similarity score to obtain the similarity coefficient corresponding to each similarity score.

[0248] The candidate optical flow fusion unit is used to perform weighted summation of each candidate optical flow using each similarity coefficient as a weight, to obtain the target optical flow.

[0249] In one optional embodiment, the optical flow estimation device further includes a target optical flow adjustment module, which includes:

[0250] The third feature map processing submodule is used to perform spatial transformation on the first deep feature map based on the target optical flow to obtain the third aligned feature map;

[0251] The optical flow increment determination submodule is used to input the third aligned feature map and the second deep feature map into the third neural network to obtain the optical flow increment;

[0252] The target optical flow adjustment submodule is used to adjust the target optical flow according to the optical flow increment to obtain the adjusted target optical flow.

[0253] In one alternative implementation, the optical flow estimation device further includes a target optical flow processing module configured to perform regularization processing on the adjusted target optical flow based on a first deep feature map or a second deep feature map to obtain the final target optical flow.

[0254] In one alternative implementation, the optical flow estimation device further includes a feature extraction module configured to input each original image into a fourth neural network to obtain multiple first original feature maps and multiple second original feature maps with different resolutions.

[0255] In one optional implementation, the target optical flow is the output optical flow estimated based on a first original feature map and a second original feature map at a first resolution. The optical flow estimation device further includes an optical flow optimization module, which is configured to estimate the target optical flow at the current resolution based on the first original feature map and the second original feature map at the current resolution, and the target optical flow estimated at the previous resolution. The current resolution is higher than the previous resolution. The target optical flow is estimated sequentially in order of resolution from low to high until the target optical flow at the second resolution is estimated.

[0256] The specific details of each module in the aforementioned optical flow estimation device have been described in detail in the corresponding optical flow estimation methods, so they will not be repeated here.

[0257] Exemplary embodiments of this disclosure also provide a computer-readable storage medium having a program product stored thereon capable of implementing the methods described above in this specification. In some possible embodiments, various aspects of this disclosure may also be implemented as a program product including program code that, when run on an electronic device, causes the electronic device to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure. This program product may be a portable compact disc read-only memory (CD-ROM) including program code and may run on an electronic device, such as a personal computer. However, the program product of this disclosure is not limited thereto. In this document, the readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0258] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0259] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0260] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0261] Program code for performing the operations of this disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C#, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0262] Exemplary embodiments of this disclosure also provide an electronic device capable of implementing the above-described method. Referring below... Figure 9 To describe an electronic device 900 according to such an exemplary embodiment of the present disclosure. Figure 9 The electronic device 900 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0263] like Figure 9 As shown, the electronic device 900 can be represented as a general-purpose computing device. The components of the electronic device 900 may include, but are not limited to: at least one processing unit 910, at least one storage unit 920, a bus 930 connecting different system components (including the storage unit 920 and the processing unit 910), and a display unit 940.

[0264] The storage unit 920 stores program code that can be executed by the processing unit 910, causing the processing unit 910 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure. For example, the processing unit 910 can execute... Figures 2 to 7 The methods and steps in the text.

[0265] Storage unit 920 may include readable media in the form of volatile storage units, such as random access memory (RAM) 921 and / or cache memory (Cache) 922, and may further include read-only memory (ROM) 923.

[0266] The storage unit 920 may also include a program / utility 924 having a set (at least one) of program modules 925, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0267] Bus 930 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0268] Electronic device 900 can also communicate with one or more external devices 1000 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 900, and / or with any device that enables electronic device 900 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 950. Furthermore, electronic device 900 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 960. As shown, network adapter 960 communicates with other modules of electronic device 900 via bus 930. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 900, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0269] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the method according to the exemplary embodiments of this disclosure.

[0270] Furthermore, the above figures are merely illustrative representations of the processes included in the methods according to exemplary embodiments of this disclosure, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0271] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0272] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0273] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. An optical flow estimation method, characterized in that, include: A first original feature map and a second original feature map are obtained, wherein the first original feature map and the second original feature map are obtained by extracting features from two adjacent original images respectively. The initial optical flow is estimated based on the first and second original feature maps; Based on the initial optical flow, multiple reference coordinate anchor points are selected, and candidate optical flows corresponding to each reference coordinate anchor point are generated; The candidate optical flows are fused to obtain the target optical flow.

2. The optical flow estimation method according to claim 1, characterized in that, The estimation of the initial optical flow based on the first and second original feature maps includes: The current optical flow is initialized based on the first and second original feature maps at the first resolution. Based on the current optical flow, the first original feature map is spatially transformed to obtain the first aligned feature map; The initial optical flow is calculated based on the first aligned feature map and the second original feature map.

3. The optical flow estimation method according to claim 2, characterized in that, The calculation of the initial optical flow based on the first aligned feature map and the second original feature map includes: The first aligned feature map and the second original feature map are concatenated to obtain a fused feature map. The initial optical flow is obtained by performing a convolution operation on the fused feature map.

4. The optical flow estimation method according to any one of claims 1 to 3, characterized in that, The initial optical flow is a polar coordinate optical flow, including an initial amplitude component and an initial direction component.

5. The optical flow estimation method according to claim 1, characterized in that, The selection of multiple reference coordinate anchor points based on the initial optical flow includes: Based on the initial amplitude component and initial direction component in the initial optical flow, multiple reference amplitude components and multiple reference direction components are set respectively; The reference amplitude components and reference direction components are combined to generate multiple reference coordinate anchor points.

6. The optical flow estimation method according to claim 1, characterized in that, The generation of candidate optical flows corresponding to each of the aforementioned reference coordinate anchor points includes: Predicted optical flow corresponding to each of the aforementioned reference coordinate anchor points is generated through a first neural network; The predicted optical flow corresponding to each of the reference coordinate anchor points is converted into the candidate optical flow corresponding to each of the reference coordinate anchor points.

7. The optical flow estimation method according to claim 6, characterized in that, The step of generating the predicted optical flow corresponding to each of the reference coordinate anchor points through the first neural network includes: Each of the reference coordinate anchor points is input into the first neural network to obtain the predicted amplitude component and the predicted direction component corresponding to each of the reference coordinate anchor points. The predicted optical flow corresponding to each reference coordinate anchor point is determined based on the predicted amplitude component and the predicted direction component corresponding to each reference coordinate anchor point.

8. The optical flow estimation method according to claim 6, characterized in that, The step of converting the predicted optical flow corresponding to each of the reference coordinate anchor points into the candidate optical flow corresponding to each of the reference coordinate anchor points includes: Based on each of the reference coordinate anchor points and the predicted optical flow corresponding to each of the reference coordinate anchor points, calculate the first intermediate optical flow corresponding to each of the reference coordinate anchor points; The first intermediate optical flow is transformed to obtain the second intermediate optical flow, and the initialized current optical flow is superimposed to calculate the candidate optical flow corresponding to each of the reference coordinate anchor points.

9. The optical flow estimation method according to claim 8, characterized in that, The first intermediate optical flow is an optical flow in polar coordinates, while the second intermediate optical flow, the current optical flow, and the candidate optical flow are optical flows in spatial coordinates.

10. The optical flow estimation method according to claim 8, characterized in that, The step of calculating the first intermediate optical flow corresponding to each of the reference coordinate anchor points and the corresponding predicted optical flow includes: The intermediate amplitude component is obtained by multiplying the reference amplitude component in each of the reference coordinate anchor points with the predicted amplitude component in the corresponding predicted optical flow. The reference direction component in each of the reference coordinate anchor points is added to the prediction direction component in the corresponding prediction optical flow to obtain the intermediate direction component; The first intermediate optical flow is determined based on the intermediate amplitude component and the intermediate direction component.

11. The optical flow estimation method according to any one of claims 1 to 10, characterized in that, The step of fusing the candidate optical flows to obtain the target optical flow includes: The first original feature map and the second original feature map are input into the second neural network to obtain the first deep feature map and the second deep feature map; Based on each of the candidate optical flows, the first deep feature map is spatially transformed to obtain the second aligned feature map corresponding to each of the candidate optical flows; Each of the second aligned feature maps is matched with the second deep feature map to obtain the matching result; The candidate optical flows are fused according to the matching results to obtain the target optical flow.

12. The optical flow estimation method according to claim 11, characterized in that, The step of performing spatial transformation on the first deep feature map based on each of the candidate optical flows to obtain a second aligned feature map corresponding to each of the candidate optical flows includes: Based on the displacement vector in each candidate optical flow, the pixel positions in the first deep feature map are transformed to obtain the second aligned feature map corresponding to each candidate optical flow.

13. The optical flow estimation method according to claim 11, characterized in that, The matching results include a similarity score; The step of matching each of the second aligned feature maps with the second deep feature map to obtain the matching result includes: Calculate the similarity between each of the second aligned feature maps and the second deep feature map to obtain multiple similarity scores.

14. The optical flow estimation method according to claim 13, characterized in that, The step of fusing the candidate optical flows according to the matching results to obtain the target optical flow includes: The similarity scores are normalized to obtain the similarity coefficients corresponding to each similarity score; The candidate optical flows are weighted and summed using the similarity coefficients as weights to obtain the target optical flow.

15. The optical flow estimation method according to claim 11, characterized in that, After obtaining the target optical flow, the method further includes: Based on the target optical flow, a spatial transformation is performed on the first deep feature map to obtain a third aligned feature map; The third aligned feature map and the second deep feature map are input into the third neural network to obtain the optical flow increment; The target optical flow is adjusted according to the optical flow increment to obtain the adjusted target optical flow.

16. The optical flow estimation method according to claim 15, characterized in that, The method further includes: Based on the first deep feature map or the second deep feature map, the adjusted target optical flow is regularized to obtain the final target optical flow.

17. The optical flow estimation method according to claim 1, characterized in that, The method further includes: Each of the original images is input into the fourth neural network to obtain multiple first original feature maps and multiple second original feature maps with different resolutions.

18. The optical flow estimation method according to claim 1, characterized in that, The target optical flow is the output optical flow estimated based on a first original feature map and a second original feature map at a first resolution. The method further includes: Based on the first and second original feature maps at the current resolution, and the target optical flow estimated at the previous resolution, the target optical flow at the current resolution is estimated; wherein the current resolution is higher than the previous resolution. The target optical flow is estimated sequentially in order of increasing resolution until the target optical flow at the second resolution is estimated.

19. An optical flow estimation device, characterized in that, include: The image feature extraction module is used to obtain a first original feature map and a second original feature map, which are obtained by extracting features from two adjacent original images respectively. An initial optical flow generation module is used to estimate the initial optical flow based on the first and second original feature maps; A candidate optical flow generation module is used to select multiple reference coordinate anchor points based on the initial optical flow and generate candidate optical flows corresponding to each reference coordinate anchor point; The target optical flow generation module is used to fuse the candidate optical flows to obtain the target optical flow.

20. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processing unit, it implements the optical flow estimation method according to any one of claims 1 to 18.

21. An electronic device, characterized in that, include: Processing unit; as well as A storage unit for storing the executable instructions of the processing unit; The processing unit is configured to execute the optical flow estimation method of any one of claims 1 to 18 by executing the executable instructions.

Images