Optical flow calculation method, system, device and medium fusing shifting window attention

Abstract

The scheme relates to a kind of fusion shift window attention optical flow calculation method, system, computer equipment and storage medium.The method comprises: source image and target image are input into SwinTransformer network model to carry out feature extraction, and source image feature map, target image feature map are generated in combination with relative position coding;The correlation between source image feature map and target image feature map is calculated by shift window attention;Using shift window attention to analyze the correlation and perform iterative optical flow calculation.Because the similarity between pixels can be analyzed by attention mechanism, the movement of the occluded pixel can be deduced according to the known pixel, and the problem of occlusion is solved.Based on this, the attention mechanism is introduced into the optical flow calculation network to solve the local limitation of convolutional network;At the same time, in order to reduce the burden of image attention calculation, shift window attention is introduced, so as to improve the calculation accuracy of optical flow calculation network.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a light flow calculation method and system fusing shift window attention, computer equipment and storage medium. BACKGROUND

[0002] Light flow calculation aims to find the corresponding motion relationship between pixels in two images, and plays a role of a bottom module in the field of computer vision. It is widely used in visual odometry, automatic driving, scene reconstruction and action recognition applications. Light flow calculation has always been regarded as an optimization problem. In recent years, with the development of deep learning, many end-to-end deep networks have emerged to directly calculate light flow values. End-to-end light flow calculation networks are limited by motion blur, occlusion and large displacement. The main reason is that when a convolutional network is used to extract image features, only local features of the image are focused on, and the global self-similarity of the image is ignored, resulting in inaccurate prediction when pixels are occluded. The cause of occlusion is complex, and it is also the main factor affecting the calculation result. Occlusion refers to a pixel in the source image that has no corresponding pixel in the target image.

[0003] Many methods have been proposed to try to solve the problem of light flow calculation. Feature extraction and correlation body are considered to be key technologies to improve the accuracy of light flow calculation. However, when occlusion occurs, the source pixel at the occlusion will not have a corresponding pixel in the target image, which will cause the light flow calculation method based on feature extraction and correlation body to fail. Therefore, the traditional light flow calculation method has the problem of low calculation accuracy. SUMMARY

[0004] Therefore, in order to solve the above technical problems, a light flow calculation method and system fusing shift window attention, computer equipment and storage medium are provided, which can improve the calculation accuracy of light flow calculation.

[0005] A light flow calculation method fusing shift window attention, the method comprising:

[0006] Obtaining a source image and a target image, and inputting them into a Swin Transformer network model for feature extraction, and generating a source image feature map and a target image feature map in combination with relative position encoding;

[0007] Calculating the correlation between the source image feature map and all pixel points in the target image feature map by shift window attention for the pixel points in the source image feature map, to obtain the correlation body between the source image feature map and the target image feature map;

[0008] Resolving the correlation body using the shift window attention and performing iterative light flow calculation.

[0009] In one of the embodiments, the source image and the target image are obtained and input into a SwinTransformer network model for feature extraction, and source image feature maps and target image feature maps are generated in combination with relative position encoding, including:

[0010] The source image and the target image are separately subjected to feature extraction by the Swin Transformer network model, and source image feature maps and target image feature maps are generated in combination with relative position encoding; or

[0011] The source image and the target image are stacked in channels to obtain a to-be-processed image, and the to-be-processed image is subjected to feature extraction by the SwinTransformer network model, and source image feature maps and target image feature maps are generated in combination with relative position encoding.

[0012] In one of the embodiments, before the source image feature maps and the target image feature maps are generated in combination with the relative position encoding, the method further includes:

[0013] The occluded part in the source image is taken as an occlusion image, and the occlusion image is taken as position encoding;

[0014] A target pixel point in the occlusion image is determined, and attention weights between the target point and all other pixel points in the occlusion image are calculated;

[0015] The relative position encoding is determined according to the position encoding and the attention weights.

[0016] In one of the embodiments, A(i,j) is a matrix that specifies the attention weights between the target pixel point (i,j) and all other pixel points, and the relative position bias B is a rectangular region with a distance of r around the point (i,j); wherein,

[0017] In one of the embodiments, the correlation between the source image feature maps and the target image feature maps is calculated by the shift window attention for the pixel points in the source image feature maps and all pixel points in the target image feature maps, including:

[0018] The pixel points in the source image feature maps are obtained respectively all pixel points in the target image feature maps

[0019] The initial correlation is calculated by the pixel point and the pixel point point multiplication,

[0020] The attention weight of each pixel point and other pixel points around is obtained by softmax, and the stored value is updated to obtain a correlation body,

[0021] In one of the embodiments, the correlation body is parsed using the shift window attention, including:

[0022] The correlation body is parsed using the shift window attention to obtain a two-dimensional motion vector;

[0023] The two-dimensional motion vector and the source image feature map are input into a shift window, and an optical flow increment is output after a GRU neural network.

[0024] In one of the embodiments, the correlation body is parsed using the shift window attention and iterative optical flow calculation is performed, including:

[0025] The correlation body is divided into different windows, and attention calculation is performed in each window to obtain the optical flow increment;

[0026] The shift window is used to perform iterative optical flow increment calculation to obtain an optical flow calculation result.

[0027] An optical flow calculation system fusing a shift window attention, the system comprising:

[0028] A feature extraction module is configured to obtain a source image and a target image, and input the source image and the target image into a Swin Transformer network model for feature extraction, and generate a source image feature map and a target image feature map in combination with relative position encoding;

[0029] A correlation body calculation module is configured to calculate, by using a shift window attention, a correlation between a pixel point in the source image feature map and all pixel points in the target image feature map, to obtain a correlation body between the source image feature map and the target image feature map;

[0030] An optical flow calculation module is configured to parse the correlation body using the shift window attention and perform iterative optical flow calculation.

[0031] A computer device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the following steps when executing the computer program:

[0032] A source image and a target image are obtained and input into a Swin Transformer network model for feature extraction, and a source image feature map and a target image feature map are generated in combination with relative position encoding;

[0033] correlation body between the source image feature map and the target image feature map is obtained by calculating the correlation of the pixel points in the source image feature map with all pixel points in the target image feature map through the shifted window attention;

[0034] The correlation body is resolved using the shifted window attention and iterative optical flow calculation is performed.

[0035] A computer readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the following steps:

[0036] A source image and a target image are obtained and input into a Swin Transformer network model for feature extraction, and a source image feature map and a target image feature map are generated in combination with relative position encoding;

[0037] A correlation body between the source image feature map and the target image feature map is obtained by calculating the correlation of the pixel points in the source image feature map with all pixel points in the target image feature map through the shifted window attention;

[0038] The correlation body is resolved using the shifted window attention and iterative optical flow calculation is performed.

[0039] The optical flow calculation method, system, computer device and storage medium incorporating shifted window attention described above, by obtaining a source image and a target image and inputting them into a Swin Transformer network model for feature extraction, a source image feature map and a target image feature map are generated in combination with relative position encoding; a correlation body between the source image feature map and the target image feature map is obtained by calculating the correlation of the pixel points in the source image feature map with all pixel points in the target image feature map through the shifted window attention; the correlation body is resolved using the shifted window attention and iterative optical flow calculation is performed. Since the similarity between pixels can be analyzed through the attention mechanism, the movement of the occluded pixels can be deduced according to the known pixels, and the occlusion problem is solved. Based on this, the attention mechanism is introduced into the optical flow calculation network to solve the local limitation of the convolutional network; at the same time, in order to reduce the burden of image attention calculation, the shifted window attention is introduced, thereby improving the calculation accuracy of the optical flow calculation network. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 It is an application environment diagram of the optical flow calculation method incorporating shifted window attention in one embodiment;

[0041] Figure 2 It is a flowchart of the optical flow calculation method incorporating shifted window attention in one embodiment;

[0042] Figure 3 A network architecture diagram of SWin in an embodiment;

[0043] Figure 4 A network architecture diagram of SWin in an embodiment; Figure 3 A detailed structure diagram of an attention module in the network architecture of SWin;

[0044] Figure 5 A network structure diagram of optical flow calculation in an embodiment of the fusion of shift window attention;

[0045] Figure 6 A part of scene diagram under Sintel in the experimental process;

[0046] Figure 7 A feature map and an attention feature map in the experimental process;

[0047] Figure 8 A graphical display diagram of optical flow calculation result on the KITTI dataset in the experimental process;

[0048] Figure 9 A graphical display diagram of optical flow calculation result on the FlyingChairs dataset in the experimental process;

[0049] Figure 10 A graphical display diagram of optical flow calculation result on the Sintel dataset in the experimental process;

[0050] Figure 11 A structure block diagram of the optical flow calculation system in an embodiment of the fusion of shift window attention;

[0051] Figure 12 An internal structure diagram of a computer device in an embodiment. DETAILED DESCRIPTION

[0052] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0053] The optical flow calculation method of the fusion of shift window attention provided by the embodiments of the present application can be applied in the application environment as shown in the figure. Figure 1 Figure 1 ​As shown, the application environment includes a computer device 110. The computer device 110 can obtain a source image and a target image, and input them into a Swin Transformer network model for feature extraction, and generate a source image feature map and a target image feature map in combination with relative position encoding. The computer device 110 can calculate the correlation between all pixel points in the target image feature map and the pixel points in the source image feature map by using the shift window attention, to obtain a correlation body between the source image feature map and the target image feature map. The computer device 110 can use the shift window attention to analyze the correlation body and perform iterative optical flow calculation. The computer device 110 can be, but is not limited to, various personal computers, notebook computers, robots, unmanned aerial vehicles, tablet computers and the like.

[0054] In one embodiment, as shown in FIG. 1, a method for optical flow calculation with fusion of shift window attention is provided, including the following steps: Figure 2 As shown in FIG. 1, a method for optical flow calculation with fusion of shift window attention is provided, including the following steps:

[0055] Step 202, obtaining a source image and a target image, and inputting them into a Swin Transformer network model for feature extraction, and generating a source image feature map and a target image feature map in combination with relative position encoding.

[0056] The computer device can obtain a source image and a target image, wherein I1 is defined as the source image, I2 is defined as the target image, the height and the width are defined as H and W respectively, and the image channel of each pixel is 3. Therefore, a tensor with a dimension of [3, H, W] can be used to store a picture in the computer device.

[0057] After the computer device obtains the source image and the target image, the source image and the target image can be input into the Swin Transformer network model for feature extraction. A common feature extraction method is to input an image into a convolutional network for processing to obtain a feature map. This method can effectively avoid damage to the calculation result caused by color changes, noise and the like of the image; however, when occlusion occurs, the source pixels at the occlusion do not have corresponding pixels in the target image, which will cause the feature extraction to fail, because the convolutional network has a local feature during feature extraction, that is, the convolution kernel only concerns a part of the area around the pixel, and cannot solve the global correlation problem of the image pixels. If occlusion occurs, it is difficult to infer the optical flow according to other pixels. Therefore, the scheme performs feature extraction by fusing a shift window attention mechanism, that is, a Swin Transformer network model. The attention mechanism can obtain more global information and can obtain the global autocorrelation of the image pixels. The Transformer network encapsulates self-attention and has dominated the field of natural language processing. At present, research in the field of vision is also increasingly widespread. The Transformer network can obtain basic image semantic level information by calculating the pixel autocorrelation of the source image. In this way, when occlusion or blur occurs, the optical flow can be extracted by using such information.

[0058] In the embodiment, the shift window attention is mainly used for feature enhancement. Previously, feature extraction was performed by a convolutional network to obtain a so-called feature map from the pixels. However, the local feature of the convolutional network limits the extracted features to the vicinity of the pixels. In particular, when occlusion occurs, the effectiveness of the features obtained by convolution is greatly reduced. In order to extract features with better recognition and effectiveness, it is necessary to introduce an attention mechanism to express the correlation between the pixels and the global correlation. Therefore, the source image and the target image are input into the Swin Transformer network model to obtain features with better recognition and global correlation, so as to generate a source image feature map F1 and a target image feature map F2 in combination with relative position encoding. The formula for generating the feature map is: wherein T is a Transformer, P is relative position encoding, the first input parameter of T is Q, and the second input parameter is K and V, wherein V is mapped from the relevant body, and K and Q are mapped from the background features, that is, the introduced attention mechanism is: After improvement, a multi-head self-attention mechanism can be obtained: MSA=Concat(Attention1,...,Attention n )W.

[0059] The generated source image feature map F1 and target image feature map F2 have a length and a width of 1 / 8 of the original image, and each feature point has D channels. The computer device can store a feature map using a tensor with dimensions of [D, H / 8, W / 8]. Taking the source image feature map F1 as a plane, each feature point has its own plane coordinate x, and the feature point located at x is denoted as F1(x), and the optical flow located at x is denoted as v1(x).

[0060] In step 204, the correlation between the source image feature map and the target image feature map is calculated by shifting the window attention for the pixel points in the source image feature map, and the correlation volume between the source image feature map and the target image feature map is obtained.

[0061] The correlation volume is denoted as Correlation Volume or Cost Volume, and records the pixel similarity of the two feature maps. For each F1(x), the correlation with all points in F2 is calculated, with dimensions of [H', W'], and therefore the entire correlation volume has dimensions of [H', W', H', W']. The correlation volume records the correlation of the two feature maps, which is usually obtained by point-by-point multiplication, and the dimensions are usually H*W*H*W. When occlusion occurs, the recorded correlation is ambiguous. At this time, the attention mechanism is still applied to expand the correlation recorded by each pixel.

[0062] The difficulty of applying attention to image processing is that the computational complexity increases quadratically with the growth of image pixels. Therefore, the image is first divided into several windows, each window containing M*M blocks, and then the attention calculation is performed in each window, and then the results are summarized. However, this will cause the relationship between windows to be ignored, so SWin further proposes a window shifting method, that is, moving the position of all windows and then performing attention calculation again, that is, using the shifted window attention calculation method. Figure 3 As shown in Figure 3 is the network architecture of SWin, which represents the iterative processing procedure starting from the input image. This process repeatedly performs block merging and shifted window attention. Since each window contains a fixed number of blocks (M = 4), block merging can generate image windows of different resolutions, thereby generating hierarchical input data.

[0063] As shown in Figure 4 , and Figure 4 is Figure 3The detailed structure of an attention module in the network architecture of SWin, which contains two sub-networks. The difference between the two is that the first one directly performs attention calculation within the window, and the second one first modifies the position of the window, i.e. each window moves M2 pixel positions to the right and down, and then performs calculation. LN in the figure represents layer normalization, which can use the nn.LayerNorm function provided by Pytorch; MLP is a multi-layer perceptron composed of two fully connected layers nn.Linear and GELU activation function; both window attention (W-MSA) and shifted window attention (SW-MSA) are special cases based on the multi-head self-attention mechanism, i.e. the input data is a data block within the window.

[0064] The overall calculation process of the shifted window attention can be represented as:

[0065]

[0066]

[0067]

[0068]

[0069] wherein z l-1 is the input image block feature, after layer normalization LN and window attention W-MSA, it is added to z l-1 , and then z l is obtained by LN and MLP, and the subsequent processing process is similar, except that the shifted window attention SW-MSA is used for attention calculation.

[0070] Step 206, using shifted window attention to analyze the correlation body and performing iterative optical flow calculation.

[0071] The shifted window attention is used to analyze the correlation body, and the window attention in feature enhancement is shared with the weight parameters.

[0072] In this embodiment, since the similarity between pixels can be analyzed by the attention mechanism, the motion of the occluded pixels can be inferred according to the known pixels, and the problem of occlusion can be solved. Based on this, the attention mechanism is introduced into the optical flow calculation network to solve the local limitation of the convolutional network; at the same time, in order to reduce the burden of image attention calculation, the shifted window attention is introduced, thereby improving the calculation accuracy of the optical flow calculation network.

[0073] In one embodiment, the optical flow calculation method fusing the shift window attention further includes a feature enhancement process of extracting image features through a moving window attention mechanism, and the specific process includes: separately performing feature extraction on the source image and the target image through a SwinTransformer network model, and combining relative position encoding to generate a source image feature map and a target image feature map; or stacking the source image and the target image according to a channel to obtain a to-be-processed image, performing feature extraction on the to-be-processed image through a Swin Transformer network model, and combining relative position encoding to generate a source image feature map and a target image feature map.

[0074] The source image and the target image are input into the Swin Transformer network model, and more recognizable and globally relevant features can be obtained, wherein the features can be extracted separately or in combination with CNN; one image can be processed separately, or the two images can be stacked according to a channel and then processed.

[0075] In one embodiment, the optical flow calculation method fusing the shift window attention further includes a process of determining relative position encoding, and the specific process includes: regarding the occluded part in the source image as an occlusion image, and regarding the occlusion image as position encoding; determining a target pixel point in the occlusion image, and calculating attention weights between the target point and all other pixel points in the occlusion image; determining the relative position encoding according to the position encoding and the attention weights.

[0076] When calculating self-attention, the spatial position information of the input sequence is ignored, and therefore, position encoding P needs to be added additionally, and relative position encoding is adopted, that is, the offset of a pixel from other positions. In order to emphasize the learning of the occlusion in the attention learning process, the occlusion map of the image is used as position encoding and participates in the learning process.

[0077] In one embodiment, A(i,j) is a matrix that specifies the attention weights between the target pixel point (i,j) and all other pixel points, and the relative position bias B is a rectangular region with a distance of r around the point (i,j); wherein, During the training process, the value of r can be set to 5, and after training, the relative position distribution is relatively obvious, that is, the weight of the pixel position 2 pixels away from the position (i,j) is the largest, indicating that the pixel is affected by the surrounding elements.

[0078] In one embodiment, the optical flow calculation method fusing the shift window attention further includes a process of calculating a correlation body, and the specific process includes: obtaining pixel points all pixel points in the target image feature map by combining the pixel points with the pixel points The initial correlation body is calculated in a dot multiplication manner, The attention weight of each pixel point and other pixel points around is obtained by softmax, and the stored value is updated to obtain the correlation body,

[0079] In one embodiment, the optical flow calculation method fusing the shift window attention further includes a process of analyzing the correlation body, and the specific process includes: using the shift window attention to analyze the correlation body to obtain a two-dimensional motion vector; and inputting the two-dimensional motion vector and the source image feature map into the shift window and outputting an optical flow increment through a GRU neural network.

[0080] After the correlation body is analyzed using the shift window attention, a 2D motion vector can be obtained, the motion vector is divided into x-direction and y-direction vectors, so that two types of 1D motion vectors are obtained, and the dimensions of the two types of 1D motion vectors are [H, W, X] respectively, and then the motion vectors and the background feature Figure 1 are input into the window attention module, and the output vector maintains the same dimension, and the iterative incremental optical flow calculation is performed through the GRU module.

[0081] In one embodiment, the optical flow calculation method fusing the shift window attention further includes a process of performing iterative optical flow calculation, and the specific process includes: dividing the correlation body into different windows, and performing attention calculation in each window to obtain an optical flow increment; shifting the window to perform iterative optical flow increment calculation to obtain an optical flow calculation result. Through the iterative optical flow increment calculation, the optical flow prediction accuracy can be continuously improved, so that the final optical flow calculation result is obtained.

[0082] In one embodiment, the loss function formula is represented in a supervised training manner as follows: Wherein N represents the number of iterations, and γ n-i represents that more weight is given to the latest prediction.

[0083] In one embodiment, the network structure of the optical flow calculation method fusing the shift window attention is as shown in Figure 5 The network structure follows the overall framework of the RAFT optical flow calculation network. The main features are: 1. calculating the correlation body of all pixel pairs; and 2. performing iterative optical flow increment calculation through a gate recurrent unit (GRU). The attention mechanism is introduced in the feature extraction, correlation body calculation and iterative optical flow calculation links, the SWin is applied to the feature attention extraction, which can also be called feature enhancement, so as to encode the relationship between the pixel points into each point through the attention mechanism.

[0084] Similar to GMA, CRAFT and other methods, attention is introduced to process the correlation volume and resolve the calculation of optical flow, because the dimension of the correlation volume itself is too large if directly calculated using self-attention, which will occupy too many computing resources and affect the optical flow calculation time. Therefore, a shift window attention is introduced, which divides the correlation volume into different windows, respectively calculates the attention in each window, and shifts the window to perform the next cycle calculation. After the shift window attention, the optical flow increment is obtained, and after the iterative optical flow increment calculation, the optical flow prediction accuracy is continuously improved to obtain the final optical flow calculation result.

[0085] It should be understood that although each step in the above flowchart is displayed in sequence according to the direction of the arrow, these steps are not necessarily executed in the order indicated by the arrow. Unless otherwise specified herein, there is no strict order limitation for the execution of these steps, and these steps can be executed in other orders. Moreover, at least part of the steps in the above flowchart can include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily sequential, but can be alternately executed with other steps or at least part of the sub-steps or stages of other steps.

[0086] In one embodiment, the optical flow calculation method provided by the fusion shift window attention is experimented, and the experimental process and experimental data are as follows:

[0087] Firstly, the model training is divided into three stages, the first stage is trained on the FlyingChairs dataset; the second stage is trained on the FlyingThings dataset; the third stage is fine-tuned based on the data of Sintel, KITTI and HD1K datasets. The batchsize of the first and second stages is 8, and each performs 120000 iterations of training; the batchsize of the third stage is 6, and performs 120000 iterations of training (if the evaluation experiment is performed on the KITTI dataset, 50000 iterations of training are performed).

[0088] The evaluation experiments are conducted on Sintel and KITTI datasets. Sintel dataset is extracted from a movie; KITTI dataset is from real scenes, which can be subdivided into KITTI 2012 and KITTI 2015 according to the release year. Unless otherwise specified, KITTI referred to in this paper refers to KITTI 2015. The Sintel dataset sequences include Clean, Final and Albedo. The Clean sequence uses brightness rendering, including specular reflection, etc. The scene of the Final sequence is more complex, including motion blur, etc. The scene of the Albedo sequence has no brightness change. Figure 6 are part of the scene pictures under Sintel.

[0089] A RTX2080Ti GPU with 64G memory is used in all experiments, based on Cuda 11.1, Pytorch1.8, torchvision 0.9, Ubuntu18.04.

[0090] The evaluation indicators used in the evaluation experiment are mainly AEPE (Average Endpoint Error) and ER (Error Rates). AEPE is also the ranking evaluation scheme recommended by the Sintel dataset. KITTI dataset uses two evaluation standards, F1-fg and F1-All, which represent the percentage of error instances in the foreground area or the whole image area (error instances refer to EPE>3 pixels or >5% label optical flow).

[0091] In order to compare with the scheme, representative optical flow calculation networks GMFlow, CRAFT, GMA, FlowFormer and RAFT, SCV are selected, among which RAFT and SCV networks do not use attention mechanism.

[0092] Table 1 shows the quantitative comparison with existing networks on Sintel dataset, reaching or approaching the latest calculation effect on Clean and Final sequences, where the data is from the test result data in the respective papers.

[0093] Table 1 Comparison of results on Sintel dataset

[0094]

[0095] Table 2 is the calculation result comparison under different occlusion conditions, where Noc represents the non-occlusion area, Occ represents the occlusion area, Occ-In and Occ-out respectively represent whether the corresponding pixels in the target image move to the image area outside, and All is the calculation result in all areas.

[0096] Table 2 compares the results for different regions on the Sintel dataset.

[0097]

[0098] Table 3 shows the quantitative comparison on the KITTI dataset. As can be seen from the table, there is still a gap compared with the latest supervised training, but it has reached or exceeded some of the supervised training results.

[0099] Table 3 shows the comparison of results on the KITTI dataset.

[0100]

[0101] Table 4 shows a comparison of the data before and after the introduction of attention to compute the relevant volume. It can be seen that EPE is significantly improved after the introduction of attention.

[0102] Table 4 Comparison of results before and after introducing attention-based computational entities.

[0103]

[0104] like Figure 7 As shown, Figure 7 The graph shows the features before and after passing through the attention node. The second column shows the feature map obtained after the convolutional network; the third column shows the feature map after adding relative position encoding; and the fourth column shows the enhanced feature map after passing through the attention module. In the attention feature map, it is evident that more image details have been resolved, including the outline of the person. The image features are more concentrated and clearer in both spatial and channel dimensions, facilitating subsequent optical flow calculations.

[0105] like Figure 8 As shown, Figure 8 The results of optical flow calculations on the KITTI dataset are graphically displayed. The images in the KITTI dataset come from real-world scenes. Many optical flow calculation methods perform poorly on the KITTI dataset, but after attention calculations, it can be seen from the image that it can correctly capture the movement of vehicles and bicycles outdoors.

[0106] Figure 9 This is a graphical representation of the optical flow calculation results on the FlyingChairs dataset, an artificially synthesized dataset in which 3D chair models are randomly placed on images. The second column shows the label data, the third column shows the calculated optical flow values, and the last column is the occlusion map, with white pixels representing occluded areas. As can be seen, even in occluded areas, the predicted optical flow still closely matches the label values.

[0107] Figure 10are the optical flow results on Sintel dataset. Four pairs of images are selected from different sequences, including fast moving, large displacement and occlusion. The second column is the result of the proposed model; the third column is the inverse optical flow obtained by swapping the image order; the fourth column shows the occluded area by white color. It can be seen from Figure 10 that the proposed model can capture the motion details in the case of fast moving and occlusion.

[0108] In one embodiment, as shown in Figure 11 , a system for optical flow calculation based on fusion of shift window attention is provided, comprising a feature extraction module 1110, a correlation body calculation module 1120 and an optical flow calculation module 1130, wherein:

[0109] The feature extraction module 1110 is configured to obtain a source image and a target image, and input them into a Swin Transformer network model for feature extraction, to generate a source image feature map and a target image feature map in combination with relative position encoding.

[0110] The correlation body calculation module 1120 is configured to calculate the correlation between the pixel points in the source image feature map and all the pixel points in the target image feature map by shift window attention, to obtain the correlation body between the source image feature map and the target image feature map.

[0111] The optical flow calculation module 1130 is configured to parse the correlation body using shift window attention and perform iterative optical flow calculation.

[0112] In one embodiment, the feature extraction module 1110 is further configured to separately extract features of the source image and the target image by the Swin Transformer network model, to generate the source image feature map and the target image feature map in combination with the relative position encoding; or to stack the source image and the target image according to the channel to obtain a to-be-processed image, to extract features of the to-be-processed image by the Swin Transformer network model, to generate the source image feature map and the target image feature map in combination with the relative position encoding.

[0113] In one embodiment, the feature extraction module 1110 is further configured to take the occluded part of the source image as an occlusion image, and take the occlusion image as the position encoding; determine a target pixel point in the occlusion image, and calculate the attention weight between the target point and all other pixel points in the occlusion image; determine the relative position encoding according to the position encoding and the attention weight.

[0114] In one embodiment, the correlation body calculation module 1120 is further configured to obtain the pixel points in the source image feature map and all the pixel points in the target image feature map by shifting the window attention to the pixel points in the source image feature map with the pixel points The initial correlation body is calculated in a dot multiplication manner, The attention weight of each pixel point with other pixel points around is obtained by softmax, and the stored value is updated to obtain the correlation body,

[0115] In one embodiment, the optical flow calculation module 1130 is further configured to parse the correlation body using the shift window attention to obtain a two-dimensional motion vector; input the two-dimensional motion vector and the source image feature map into the shift window, and output the optical flow increment through the GRU neural network.

[0116] In one embodiment, the optical flow calculation module 1130 is further configured to split the correlation body into different windows, and perform attention calculation in each window to obtain an optical flow increment; shift the window, and perform iterative optical flow increment calculation to obtain an optical flow calculation result.

[0117] In one embodiment, a computer device is provided, which can be a terminal, and an internal structure diagram of the computer device can be as shown in Figure 12 The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The network interface of the computer device is configured to communicate with external terminals through network connection. The computer program is executed by the processor to implement an optical flow calculation method fusing shift window attention. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or a key, trackball or touchpad arranged on the shell of the computer device, or an external keyboard, touchpad or mouse, etc.

[0118] Those skilled in the art can understand that Figure 12 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0119] In one embodiment, a computer device is provided, which includes a memory and a processor, and the memory stores a computer program. When the processor executes the computer program, the following steps are implemented:

[0120] The source image and the target image are obtained and input into a Swin Transformer network model for feature extraction, and relative position encoding is combined to generate a source image feature map and a target image feature map;

[0121] The correlation between the source image feature map and the target image feature map is obtained by calculating the correlation between the pixel points in the source image feature map and all pixel points in the target image feature map through the shift window attention.

[0122] The correlation is analyzed using the shift window attention and iterative optical flow calculation is performed.

[0123] In one embodiment, when the processor executes the computer program, the following steps are also implemented: the source image and the target image are separately extracted by the Swin Transformer network model, and the relative position encoding is combined to generate the source image feature map and the target image feature map; or the source image and the target image are stacked according to the channel to obtain a to-be-processed image, and the to-be-processed image is extracted by the Swin Transformer network model, and the relative position encoding is combined to generate the source image feature map and the target image feature map.

[0124] In one embodiment, when the processor executes the computer program, the following steps are also implemented: the occluded part in the source image is taken as an occlusion image, and the occlusion image is taken as the position encoding; the target pixel point in the occlusion image is determined, and the attention weight between the target point and all other pixel points in the occlusion image is calculated; and the relative position encoding is determined according to the position encoding and the attention weight.

[0125] In one embodiment, when the processor executes the computer program, the following steps are also implemented: the pixel points in the source image feature map all pixel points in the target image feature map The initial correlation is calculated by multiplying the pixel points and the pixel points The attention weight of each pixel point and the surrounding other pixel points is obtained by softmax, and the stored value is updated to obtain the correlation,

[0126] In one embodiment, when the processor executes the computer program, the following steps are also implemented: the correlation is analyzed using the shift window attention to obtain a two-dimensional motion vector; and the two-dimensional motion vector and the source image feature map are input into the shift window, and the optical flow increment is output through the GRU neural network.

[0127] ​In one embodiment, the processor further implements the following steps when executing the computer program: dividing the correlation volume into different windows, and performing attention calculation in each window to obtain an optical flow increment; and shifting the window to perform iterative optical flow increment calculation to obtain an optical flow calculation result.

[0128] In one embodiment, a computer readable storage medium is provided, and the computer readable storage medium stores a computer program. The computer program is executed by a processor to implement the following steps:

[0129] The source image and the target image are obtained and input into a Swin Transformer network model for feature extraction, and relative position encoding is combined to generate a source image feature map and a target image feature map.

[0130] The correlation between the source image feature map and the target image feature map is calculated by shifting the window attention to the pixel points in the source image feature map and all pixel points in the target image feature map to obtain a correlation volume.

[0131] The correlation volume is analyzed by the shifting window attention, and iterative optical flow calculation is performed.

[0132] In one embodiment, the computer program is executed by the processor to further implement the following steps: the source image and the target image are separately extracted by the Swin Transformer network model, and the source image feature map and the target image feature map are generated by combining the relative position encoding; or the source image and the target image are stacked according to the channel to obtain a to-be-processed image, and the to-be-processed image is extracted by the Swin Transformer network model, and the source image feature map and the target image feature map are generated by combining the relative position encoding.

[0133] In one embodiment, the computer program is executed by the processor to further implement the following steps: the occluded part in the source image is taken as an occlusion image, and the occlusion image is taken as a position encoding; the target pixel point in the occlusion image is determined, and the attention weight between the target point and all other pixel points in the occlusion image is calculated; and the relative position encoding is determined according to the position encoding and the attention weight.

[0134] In one embodiment, the computer program is executed by the processor to further implement the following steps: the pixel points in the source image feature map all pixel points in the target image feature map are obtained respectively, and the pixel points are multiplied to calculate an initial correlation volume, the attention weight of each pixel point and the surrounding other pixel points is obtained by softmax, and the stored value is updated to obtain the correlation volume,

[0135] In one embodiment, the computer program, when executed by the processor, further implements the following steps: using the shift window attention to parse the correlation body to obtain a two-dimensional motion vector; inputting the two-dimensional motion vector and the source image feature map into the shift window, and outputting the optical flow increment after passing through the GRU neural network.

[0136] In one embodiment, the computer program, when executed by the processor, further implements the following steps: dividing the correlation body into different windows, and performing attention calculation in each window to obtain an optical flow increment; shifting the window, and performing iterative optical flow increment calculation to obtain an optical flow calculation result.

[0137] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. Any reference to memory, storage, database or other medium used in the embodiments provided in the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0138] The technical features of the above embodiments can be combined in any way. To make the description concise, not all possible combinations of the technical features in the above embodiments are described, but as long as the combinations of the technical features do not contradict, they should be considered within the scope of the present disclosure.

[0139] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent. It should be pointed out that for ordinary skilled persons in the art, without departing from the concept of the present application, some modifications and improvements can be made, which are within the scope of the present application. Therefore, the scope of the patent of the present application should be subject to the appended claims.

Claims

1. A method for calculating optical flow by fusing shifted window attention, characterized in that, The method includes: The source image and target image are acquired and input into the Swing Transformer network model for feature extraction. The source image feature map and target image feature map are generated by combining relative position encoding. Before generating the source image feature map and the target image feature map by combining the relative position encoding, the method further includes: using the occluded portion of the source image as the occluded image, and using the occluded image as the position encoding; determining the target pixel in the occluded image, and calculating the attention weights between the target pixel and all other pixels in the occluded image; determining the relative position encoding based on the position encoding and the attention weights; A(i,j) is a matrix specifying the attention weights between the target pixel (i,j) and all other pixels, and the relative position bias B is a rectangular region with a distance of r added around the point (i,j); where... ; By using a shift window to focus on the pixels in the source image feature map, the correlation between the source image feature map and all pixels in the target image feature map is calculated, thus obtaining the correlation volume between the source image feature map and the target image feature map; The method of parsing the correlated volume using the shifted window attention and performing iterative optical flow calculation includes: parsing the correlated volume using the shifted window attention to obtain a two-dimensional motion vector; inputting the two-dimensional motion vector and the source image feature map into the shifted window, and outputting the optical flow increment after passing through a GRU neural network; dividing the correlated volume into different windows, and performing attention calculation in each window to obtain the optical flow increment; shifting the window, performing iterative optical flow increment calculation, and obtaining the optical flow calculation result.

2. The optical flow calculation method for fusing shifted window attention according to claim 1, characterized in that, The process of acquiring the source image and target image, inputting them into the Swin Transformer network model for feature extraction, and combining relative position encoding to generate source image feature maps and target image feature maps includes: The source image and the target image are extracted separately using the Swin Transformer network model, and feature maps of the source image and the target image are generated by combining relative position encoding; or The source image and the target image are superimposed according to channels to obtain the image to be processed. The SwinTransformer network model is used to extract features from the image to be processed, and the source image feature map and the target image feature map are generated by combining relative position encoding.

3. The optical flow calculation method for fusing shifted window attention according to claim 1, characterized in that, The step of using shifted window attention to focus on pixels in the source image feature map and calculating the correlation with all pixels in the target image feature map to obtain the correlation body between the source image feature map and the target image feature map includes: Pixels in the feature map of the source image are obtained respectively. All pixels in the feature map of the target image ; By pixel With pixels The initial related volume is calculated using the dot product method. ; The attention weights between each pixel and its surrounding pixels are obtained using softmax, and the stored values ​​are updated to obtain the relevant volume. .

4. An optical flow calculation system incorporating shifted window attention, characterized in that, The system includes: The feature extraction module is used to acquire source and target images and input them into the Swing Transformer network model for feature extraction. It combines relative position encoding to generate source image feature maps and target image feature maps. Before generating the source image feature map and the target image feature map by combining the relative position encoding, the method further includes: using the occluded portion of the source image as the occluded image, and using the occluded image as the position encoding; determining the target pixel in the occluded image, and calculating the attention weights between the target pixel and all other pixels in the occluded image; determining the relative position encoding based on the position encoding and the attention weights; A(i,j) is a matrix specifying the attention weights between the target pixel (i,j) and all other pixels, and the relative position bias B is a rectangular region with a distance of r added around the point (i,j); where... ; The correlation volume calculation module is used to calculate the correlation between the source image feature map and all pixels in the target image feature map by using shift window attention on the pixels in the source image feature map, so as to obtain the correlation volume between the source image feature map and the target image feature map; An optical flow calculation module is used to parse the relevant volume using the shifted window attention and perform iterative optical flow calculation, including: parsing the relevant volume using the shifted window attention to obtain a two-dimensional motion vector; inputting the two-dimensional motion vector and the source image feature map into the shifted window, and outputting the optical flow increment after passing through a GRU neural network; dividing the relevant volume into different windows, and performing attention calculation in each window to obtain the optical flow increment; shifting the window, performing iterative optical flow increment calculation, and obtaining the optical flow calculation result.

5. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 3.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 3.

Images