Optical flow calculation method based on global matching

Abstract

The invention relates to an optical flow calculation method based on global matching, and belongs to the field of optical flow estimation. The method comprises the following steps: acquiring two continuous video frames in a video stream, respectively extracting corresponding dense features from the two continuous video frames, and performing feature enhancement on the dense features; calculating the similarity between the pixel features of the two continuous video frames, and obtaining the corresponding relation of the corresponding pixel coordinates according to the differentiable matching layer, thereby obtaining the corresponding optical flow; performing optical flow prediction according to an optical flow propagation mode based on self-attention, and taking the predicted optical flow as a calculation result; and designing an L1 loss function, and supervising all optical flow calculations. According to the method, the optical flow is re-expressed as a global matching problem, the long-standing problem of large displacement is effectively solved, and meanwhile, the calculation efficiency is improved through parameter sharing and an attention mechanism.

Description

Technical Field

[0001] The invention belongs to the field of optical flow estimation and relates to an optical flow calculation method based on global matching. Background Art

[0002] Computer vision is an important branch of artificial intelligence that enables computers to understand images and videos like humans and extract useful information from them. Optical flow calculation is a core problem in computer vision, which aims to estimate the precise displacement of pixels between two adjacent frames in a video. Optical flow is closely related to the motion in the video. By calculating the optical flow, the motion information of the foreground objects and the background environment in the video can be obtained. Optical flow belongs to the low-level vision and is the basis of many advanced computer vision tasks and is widely used in daily life.

[0003] Motion detection technology is used to identify the movement of objects in the surrounding environment. It is widely used in smart transportation, smart home, surveillance and security, and is also important in autonomous driving. In the process of vehicle driving, in order to make reasonable decisions, it is necessary to analyze the road conditions around the vehicle: determine whether the surrounding pedestrians and vehicles are moving or stationary, and their speed. Optical flow algorithm is a common technology in motion detection tasks, which can provide motion information, especially in pure visual autonomous driving solutions, because this solution only uses cameras and does not rely on radar sensors. In this case, obtaining motion information can only rely on computer vision algorithms such as optical flow. Research on optical flow algorithms will help promote the development of autonomous driving technology.

[0004] In addition, optical flow is also widely used in the medical field, such as medical image registration and blood flow detection; optical flow is also used in multiple scenarios such as augmented reality, robot navigation, and film and television special effects production.

[0005] In summary, most visual problems related to motion can be solved by optical flow algorithms. Therefore, it is very important and necessary to study optical flow algorithms with higher accuracy. Early optical flow algorithms were mainly based on variational methods. In recent years, with the development of deep learning technology, optical flow algorithms based on deep learning have begun to dominate, and many optical flow network models have emerged. Yin et al. proposed the HD3 network, which uses a hierarchical discrete distribution decomposition method to estimate optical flow. This method predicts the probability distribution of optical flow and decodes the distribution through a regression method to obtain the final optical flow estimate. Yang et al. proposed the VCN network, which estimates optical flow by constructing a volume correlation map and refines the optical flow using a regression method, thereby improving both accuracy and efficiency. The SelFlow (Self-Supervised Flow) network proposed by Liu et al. uses self-supervised learning and regression methods for optical flow estimation. It self-supervises the learning of optical flow by utilizing the dense correlation of images, thus not relying on a large amount of labeled data. However, these regression-based methods have a major inherent limitation. That is, the cost volume needs to have a predefined size because the search space is regarded as the channel dimension of the subsequent regression, and the regression result is related to the cost volume. This requirement restricts the search space to a local range and makes it difficult to handle large displacements.

[0006] To alleviate the large displacement problem, RAFT proposes an iterative framework with a large number of iterative refinements, convolving different local cost quantities at different iteration stages to gradually reach a search space close to the global one, achieving excellent performance on standard benchmarks. Current state-of-the-art methods are based on this iterative architecture. Despite the outstanding performance, a large number of continuous refinements leads to a linear increase in inference time, making it difficult to achieve speed optimization, hindering its deployment in practical applications.

[0007] Traditional optical flow calculation methods are based on convolutional regression, but this regression-based method has an inherent limitation. The cost volume needs to have a predefined size because the search space is regarded as the channel dimension of the subsequent regression, and the regression result is related to the cost volume. This requirement limits the search space to a local range and makes it difficult to handle large displacements. Even if existing methods use different local cost volumes for convolution at different iteration stages, this large number of convolutions will lead to a linear increase in inference time and low efficiency. Summary of the invention

[0008] In view of this, the purpose of the present invention is to provide an optical flow calculation method based on global matching, which provides more flexible and accurate pixel motion modeling through global context modeling and adaptive feature learning. It is more effective in processing long-range dependencies and improves computational efficiency through parameter sharing and attention mechanisms.

[0009] In order to achieve the above object, the present invention provides the following technical solutions:

[0010] A method for calculating optical flow based on global matching, comprising the following steps:

[0011] S1, obtain two consecutive video frames in the video stream, extract corresponding dense features from the two consecutive video frames respectively, and perform feature enhancement on them;

[0012] S2, calculating the similarity between the pixel features of two consecutive video frames, and obtaining the corresponding relationship between the corresponding pixel coordinates according to the differentiable matching layer, so as to obtain the corresponding optical flow;

[0013] S3, predicting the optical flow according to the optical flow propagation method based on self-attention, and taking the predicted optical flow as the calculation result;

[0014] S4. Design the L1 loss function to supervise all optical flow calculations.

[0015] Further, in step S1, for two consecutive video frames I i and I j , proceed as follows:

[0016] S11, using weight-sharing convolutional network to extract video frame I i and I j The downsampled dense features F i ,F j ∈R H×W×D , where H, W, and D represent height, width, and feature dimensions, respectively;

[0017] S12, using the transformer structure based on the attention mechanism to simulate the downsampling of dense features F i and F j The relationship between them leads to higher quality identification features and

[0018] Further, in step S12, the following steps are included:

[0019] S121, respectively in feature F i and F j Add fixed two-dimensional sine and cosine position encoding;

[0020] S122, the features after adding the position encoding information are processed by multiple superposition of self-attention, cross-attention and feedforward network, and the feature F i and F j The symmetric processing is adopted, which can be expressed as:

[0021]

[0022] Where T represents the transformer, whose first input is the query and the second input is the key and value; P represents the added two-dimensional sine and cosine position encoding.

[0023] Further, in step S122, a moving local window attention strategy is used to reduce the quadratic computational complexity caused by the paired attention operation, which divides the features into a fixed number of local windows. The process is: the input features of size H×W are divided into K×K windows, and the size of each window is And perform self-attention and cross-attention independently in each local window; for every two consecutive local windows, move the window split To introduce cross-window connections.

[0024] Further, in step S2, the following steps are included:

[0025] S21, by calculation Each pixel in The correlation of all pixels in to compare their feature similarity:

[0026]

[0027] In the formula, each element in the correlation matrix C represents Median coordinate p i =(m,n) and Median coordinate p j = the correlation value between (k, l), is the normalization coefficient;

[0028] S22, using a differentiable matching layer, normalizes the correlation matrix C through a softmax operation to obtain a matching distribution:

[0029] M=softmax(C)∈R H×W×H×W

[0030] In the formula, M represents Each position relative to The correspondence between all positions in ;

[0031] S23, pixel grid G∈R H×W×2 The two-dimensional coordinates of are weighted averaged with the matching distribution M to obtain the corresponding relationship

[0032]

[0033] S24. Obtain the optical flow V by calculating the difference between the corresponding pixel coordinates:

[0034]

[0035] The optical flow V represents the intermediate prediction result for two consecutive video frames.

[0036] Further, in step S3, the final prediction of the optical flow is performed according to the optical flow propagation method based on self-attention, and the process includes:

[0037] The higher quality flow predictions in matched pixels are compared with the higher quality flow predictions in unmatched pixels by measuring the self-similarity of the features to obtain the final optical flow predictions:

[0038]

[0039] in:

[0040]

[0041] In the formula, represents the optical flow prediction from the softmax layer.

[0042] Further, in step S4, an L1 loss function is designed between the predicted value and the true value to supervise all optical flow calculations, which is expressed as:

[0043]

[0044] Where N is the number of traffic predictions, including intermediate predictions and final predictions, n∈N; γ is the weight, which increases exponentially; V n Represents the predicted value of the nth optical flow; V gt Represents the true value of optical flow.

[0045] The beneficial effects of the present invention are:

[0046] The optical flow calculation method based on global matching proposed in the present invention completely transforms the mainstream optical flow regression network by reformulating the optical flow as a global matching problem, thereby effectively solving the long-standing large displacement problem.

[0047] The present invention uses an optical flow computing network based on Transformer for feature enhancement to achieve global feature matching, and provides more flexible and accurate pixel motion modeling through global context modeling and adaptive feature learning. The present invention is more effective in processing long-range dependencies, and at the same time improves computational efficiency through parameter sharing and attention mechanisms.

[0048] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0050] Figure 1 It is the overall framework diagram of the optical flow calculation method based on global matching of the present invention;

[0051] Figure 2 Schematic diagram of the Transformer-based feature enhancement process. DETAILED DESCRIPTION

[0052] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0053] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0054] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0055] See also Figure 1-2 , which is an optical flow calculation method based on global matching.

[0056] Example

[0057] This embodiment proposes an optical flow calculation method based on global matching, such as Figure 1 As shown, it includes the following steps:

[0058] S1, obtain two consecutive video frames in the video stream, extract corresponding dense features from the two consecutive video frames respectively, and perform feature enhancement on them;

[0059] S2, calculating the similarity between the pixel features of two consecutive video frames, and obtaining the corresponding relationship between the corresponding pixel coordinates according to the differentiable matching layer, so as to obtain the corresponding optical flow;

[0060] S3, predicting the optical flow according to the optical flow propagation method based on self-attention, and taking the predicted optical flow as the calculation result;

[0061] S4. Design the L1 loss function to supervise all optical flow calculations.

[0062] In step S1, two consecutive video frames are obtained by extracting frames: the first video frame I i and the second video frame I j In this embodiment, the first video frame is recorded as I1 and the second video frame is recorded as I2. First, the weight-sharing convolutional network is used to extract the first video frame I i and the second video frame I j The downsampled dense features F i ,F j ∈R H×W×D , where H, W and D represent height, width and feature dimension respectively. In order to further consider the interdependence between them, the present invention uses the attention mechanism in the transformer architecture to simulate the mutual relationship between the two sets to obtain high-quality discriminative features for matching.

[0063] Since F1 and F2 are just two sets of features, they have no concept of spatial position, so first add fixed two-dimensional sine and cosine position encoding to the features. Adding position information also enables the matching process to consider not only the similarity of features but also their spatial distance, which helps resolve ambiguity and improves performance.

[0064] After adding location information, such as Figure 2As shown in Figure 1, the features are processed by the Transformer network six times with self-attention, cross-attention, and feed-forward network (FFN) to improve the quality of the initial features. Specifically, for the query, key, and value in self-attention, they are the same features. For cross-attention, the key and value are the same, but different from the query to introduce the mutual dependence between them, where the query, key, and value are used to perform matrix product operations with the corresponding weight matrix and feature map to generate the corresponding query, key, and value vectors:

[0065] Q=W Q F i

[0066] K=W K F i

[0067] V=W V F i

[0068] Among them, W Q , W K , W V are the weight matrices corresponding to query, key, and value respectively.

[0069] This process F1 and F2 are carried out symmetrically, that is:

[0070]

[0071] Here, T is the transformer, P is the positional encoding, the first input of T is the query, and the second input is the key and value.

[0072] One problem in the standard Transformer architecture is the quadratic computational complexity of the paired attention operation. To improve efficiency, this paper adopts the moving local window attention strategy in the Swin Transformer. However, unlike Swin, which uses a fixed window size, this paper splits the features into a fixed number of local windows so that the window size is adapted to the feature size. Specifically, the input features of size H×W are split into K×K windows (each window is of size ), and perform self-attention and cross-attention independently in each local window. For every two consecutive local windows, the window split is moved In order to introduce cross-window connections. In the framework of this embodiment, the network is divided into 2×2 windows (the size of each window is ), which represents a good trade-off between speed and accuracy.

[0073] In step S2, this embodiment provides the following steps:

[0074] Considering that the correspondence between the two frames should have a high similarity, we first calculate Each pixel in The correlation of all pixels in t is used to compare their feature similarity. This can be efficiently implemented by a simple matrix multiplication:

[0075]

[0076] Among them, each element in the correlation matrix C represents The median coordinate p1 = (m, n) and The correlation value between the coordinates p2 = (k, l), is a normalization coefficient to avoid the maximum value after the dot product operation. To identify the corresponding relationship, a simple method is to directly select the position with the highest correlation. However, this operation is indifferent, so end-to-end training is not possible. To solve this problem, the present invention uses a differentiable matching layer. Specifically, the last two dimensions of C are normalized using a softmax operation to obtain a matching distribution:

[0077] M=softmax(C)∈R H×W×H×W

[0078] In the formula, M represents Each position relative to Then, the pixel grid G∈R h×W×2 The corresponding relationship can be obtained by weighted average of the two-dimensional coordinates and the matching distribution M.

[0079]

[0080] Finally, by calculating the difference between the corresponding pixel coordinates, an intermediate prediction V of the optical flow can be obtained.

[0081]

[0082] This softmax-based approach not only enables end-to-end training but also provides sub-pixel accuracy.

[0083] In step S3, further optical flow prediction is performed. By observing the high structural similarity between the optical flow field and the image itself, the present invention proposes a method for propagating the optical flow field into the image.

[0084] The high-quality flow predictions in matched pixels are compared with those in unmatched pixels by measuring the self-similarity of the features. This can be efficiently implemented with a simple self-attention layer:

[0085]

[0086] in

[0087]

[0088] is the intermediate prediction of the optical flow from the softmax layer, which is obtained by substituting higher quality features into the sotfmax matching layer.

[0089] In step S4, an L1 loss function is designed between the predicted value and the true value to supervise all optical flow calculations, which is expressed as:

[0090]

[0091] Where N is the number of traffic forecasts, including intermediate and final forecasts, and γ (set to 0.9) is the weight. According to the RAFT method, the weight increases exponentially, giving higher weight to later forecasts.

[0092] This embodiment evaluates the optical flow calculation method based on global matching. This embodiment uses the endpoint error (EPE) as a performance evaluation indicator. The endpoint error is used to quantify the error between the estimated optical flow vector and the true optical flow vector. Specifically, the endpoint error is the Euclidean distance between the endpoint of the estimated optical flow vector and the endpoint of the true optical flow vector.

[0093] Assume that the actual optical flow vector of a pixel in frame i and frame i+1 is (u, v), and the optical flow vector estimated by the algorithm is Then the error formula of the endpoint is:

[0094]

[0095] This value represents the distance between the estimated optical flow vector and the true optical flow vector, which directly reflects the accuracy of the algorithm.

[0096] The datasets used in this example are MPI Sintel and KITTI. MPI Sintel is a dataset for optical flow evaluation with 1064 synthetic stereo images and ground truth data of disparity. Sintel is derived from the open source 3D animated short film Sintel. The dataset has 23 different scenes. The stereo images are RGB, while the disparity is grayscale. The resolution of both is 1024×436 pixels, 8 bits per channel. The KITTI dataset, which consists of several hours of traffic scenes recorded using various sensor modes, including high-resolution RGB, grayscale stereo cameras, and 3D laser scanners.

[0097] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. An optical flow calculation method based on global matching, characterized in that: It includes the following steps: S1, obtain two consecutive video frames in the video stream, extract corresponding dense features from the two consecutive video frames respectively, and perform feature enhancement on them; S2, calculating the similarity between the pixel features of two consecutive video frames, and obtaining the corresponding relationship between the corresponding pixel coordinates according to the differentiable matching layer, so as to obtain the corresponding optical flow; S3, predicting the optical flow according to the optical flow propagation method based on self-attention, and taking the predicted optical flow as the calculation result; S4. Design the L1 loss function to supervise all optical flow calculations.

2. The optical flow calculation method based on global matching according to claim 1, characterized in that: In step S1, for two consecutive video frames I i and I j , proceed as follows: S11, using weight-sharing convolutional network to extract video frame I i and I j The downsampled dense features F i ,F j ∈R H×W×D , where H, W, and D represent height, width, and feature dimensions, respectively; S12, using the transformer structure based on the attention mechanism to simulate the downsampling of dense features F i and F j The relationship between them leads to higher quality identification features and 3. The optical flow calculation method based on global matching according to claim 2, characterized in that: In step S12, the following steps are included: S121, respectively in feature F i and F j Add fixed two-dimensional sine and cosine position encoding; S122, the features after adding the position encoding information are processed by multiple superposition of self-attention, cross-attention and feedforward network, and the feature F i and F j The symmetric processing is adopted, which can be expressed as: Where T represents the transformer, whose first input is the query and the second input is the key and value; P represents the added two-dimensional sine and cosine position encoding.

4. The optical flow calculation method based on global matching according to claim 3 is characterized in that: In step S122, a moving local window attention strategy is used to reduce the quadratic computational complexity caused by the paired attention operation. The feature is divided into a fixed number of local windows. The process is as follows: the input feature of size H×W is divided into K×K windows, and the size of each window is And perform self-attention and cross-attention independently in each local window; for every two consecutive local windows, move the window split To introduce cross-window connections.

5. The optical flow calculation method based on global matching according to claim 2, characterized in that: In step S2, the following steps are included: S21, by calculation Each pixel in The correlation of all pixels in to compare their feature similarity: In the formula, each element in the correlation matrix C represents Median coordinate p i =(m,n) and Median coordinate p j = the correlation value between (k, l), is the normalization coefficient; S22, using a differentiable matching layer, normalizes the correlation matrix C through a softmax operation to obtain a matching distribution: M=softmax(C)∈R H×W×H×W In the formula, M represents Each position relative to The correspondence between all positions in ; S23, pixel grid G∈R H×W×2 The two-dimensional coordinates of are weighted averaged with the matching distribution M to obtain the corresponding relationship S24. Obtain the optical flow V by calculating the difference between the corresponding pixel coordinates: The optical flow V represents the intermediate prediction result for two consecutive video frames.

6. The optical flow calculation method based on global matching according to claim 5, characterized in that: In step S3, the final prediction of the optical flow is performed according to the optical flow propagation method based on self-attention, and the process includes: The higher quality flow predictions in matched pixels are compared with the higher quality flow predictions in unmatched pixels by measuring the self-similarity of the features to obtain the final prediction of optical flow: in: In the formula, represents the optical flow prediction from the softmax layer.

7. The optical flow calculation method based on global matching according to claim 6, characterized in that: In step S4, an L1 loss function is designed between the predicted value and the true value to supervise all optical flow calculations, which is expressed as: Where N is the number of traffic predictions, including intermediate predictions and final predictions, n∈N; γ is the weight, which increases exponentially; V n Represents the predicted value of the nth optical flow; V gt Represents the true value of optical flow.

Images