2.5d monocular scene flow estimation method based on attention decoding structure

Abstract

The application provides a 2.5D monocular scene flow estimation method based on an attention decoding structure, comprising the following steps: S1, constructing a camera pose estimation subnetwork and a shared pyramid feature encoder; S2, constructing a parallel deep decoder based on cavity self-attention and an optical flow decoder based on non-local cross-frame attention; S3, constructing an overall network loss function; S4, inputting a monocular image sequence to perform end-to-end training on the network in a self-supervised manner; and S5, when inputting two continuous image frames to test the model, two components of the 2.5D scene flow, i.e., optical flow and depth, can be obtained; when inputting a single image to test the model, a depth estimation result can be obtained. The application estimates the 2.5D monocular scene flow by using the decoding structure based on attention, wherein attention is introduced in the decoding structure to strengthen the feature expression capability and pixel-related calculation, and the above design is used to improve the scene flow estimation precision.

Description

TECHNICAL FIELD

[0001] The application provides a 2.5D monocular scene flow estimation method, which utilizes a parallel deep decoding structure based on cavity self-attention and an optical flow decoding structure based on non-local cross-frame attention, and belongs to the field of computer vision. BACKGROUND

[0002] As an important research direction of computer vision, scene flow estimation has a very broad application prospect in the fields of autonomous driving, intelligent robots and target tracking. In recent years, deep learning has developed rapidly, and a large number of researchers have also begun to use deep learning technology to process scene flow estimation problems. The method of deep learning has the advantages of fast running speed and high precision, and has achieved leading results on multiple public data sets.

[0003] Compared with other forms of scene flow estimation, 2.5D monocular scene flow is more convenient in application because its input is a monocular image sequence.

[0004] The application estimates two components of 2.5D monocular scene flow, depth and optical flow, through a deep decoding structure based on cavity self-attention and an optical flow decoding structure based on non-local cross-frame attention. In this method, the camera pose, depth and optical flow are estimated simultaneously by using a parallel structure. The encoder for estimating depth and optical flow adopts a shared pyramid feature encoder, and the decoder is a depth decoder based on cavity self-attention and an optical flow decoder based on non-local cross-frame attention. In the decoding structure, appropriate self-attention and cross-attention are added, which is beneficial to improve the expression ability of features and the correlation between pixels, and further improves the estimation accuracy. SUMMARY

[0005] The application provides a 2.5D monocular scene flow estimation method based on an attention decoding structure, aiming to enhance the generalization expression ability of the network and the connection between pixels.

[0006] The application is achieved by the following steps:

[0007] S1. Constructing a camera pose estimation subnetwork and a shared pyramid feature encoder;

[0008] S2. Constructing a parallel depth decoder based on cavity self-attention and an optical flow decoder based on non-local cross-frame attention;

[0009] S3. Constructing an overall network loss function;

[0010] S4. Inputting a monocular image sequence to train the network in an end-to-end manner in a self-supervised manner;

[0011] S5. When inputting two continuous frame image pairs to test the model, two components of the 2.5D scene flow can be obtained: optical flow and depth. When inputting a single image to test the model, a depth estimation result can be obtained.

[0012] The application also includes the following structural features:

[0013] 1. The step S2 of constructing the parallel depth decoder is a depth decoder based on cavity self-attention. The attention mechanism is connected after the first layer convolution operation of the decoder to improve the expression ability of the extracted features. The cavity self-attention uses a 1x1 convolution operation on the feature map after convolution, and then uses convolution operations with different dilation rates to capture multi-scale context information. Different feature maps are added, and a SiLU activation function is applied to obtain a Query query matrix Q. The dot product operation is performed on the Query query matrix and the Key matrix, and the obtained result is dot multiplied with the Value matrix to obtain the output of the cavity self-attention.

[0014] 2. The step S2 of constructing the parallel optical flow decoder is an optical flow decoder based on non-local cross-frame attention. The non-local attention can capture long-distance correspondence in time and space. First, the features input from the encoder are normalized to obtain normalized adjacent frame features. According to the convolution operation, the Value matrix, the Key matrix and the Query matrix of another frame are obtained. In the application, the correlation matrix Γ is obtained by dot multiplying the one-dimensional local vector Q p in the Query matrix with the Key matrix and then performing softmax. In order to reduce the influence of the wrong matching pixels, each slice in the second dimension of the correlation matrix Γ is multiplied by a Gaussian weight. Finally, the final output of the cross-frame non-local cross-attention module can be written as formula (1):

[0015]

[0016] wherein, represents the product operation, i.e. Hadamard product, the matrix G represents the dot product result of the Key matrix and the vector Q p , V represents the Value matrix obtained after the convolution operation, Γ represents the correlation matrix, and X t represents the feature map output by the non-local cross-attention.

[0017] 3. The step S3 of constructing the overall network loss function, the learning method of the application is self-supervised learning, which can learn the depth, camera motion pose and optical flow information in the scene from unlabeled monocular image sequences. By defining a multi-task learning loss function, the sub-networks can work cooperatively. The loss function includes image reconstruction loss, spatial smoothing loss and geometric consistency loss.

[0018] The image reconstruction loss function calculates the brightness error between the target image and the transformed image, and calculates the image similarity between the two images. Its specific definition formula (2) is shown below:

[0019]

[0020] Where L1 represents the depth image reconstruction loss, x t Image I t In the image I, the pixels are α, which is a balance factor. t Image from the target's perspective. For an image after depth and camera pose motion changes, SSIM represents the structural similarity function, and |·| represents the absolute value operation.

[0021] For optical flow motion fields, it can also be used to modify the original image I. s The image is obtained after transformation. The image reconstruction loss can be calculated using this transformation, and its specific definition is shown in formula (3):

[0022]

[0023] Where L2 represents the optical flow image reconstruction loss, x t To represent image I t In the image I, the pixels are α, which is a balance factor. t Image from the target's perspective. For the image after optical flow transformation, SSIM represents the structural similarity function, and |·| represents the absolute value operation.

[0024] For the depth map estimated by the network, the definition of the depth space smoothing loss is as shown in Equation (4):

[0025]

[0026] Where L3 represents the depth-space smoothing loss, x t Image I t The pixels in the image I t For the target view image, D(x) t ) is in pixel x t The estimated depth is obtained at that point. represents the gradient operator, and |·| represents the modulo operation.

[0027] For the optical flow map obtained by network estimation, the definition of the optical flow spatial smoothing loss is as shown in Equation (5):

[0028]

[0029] Where L4 represents the optical flow smoothing loss, xt represents a pixel point in image I t , image I t is a target view image, f t→s (x t ) is an optical flow field estimated at point x t , represents a gradient operator, and |·| represents a modulus operation.

[0030] The geometric consistency loss is used to reduce the interference from the occluded area, which is defined as formula (6):

[0031]

[0032] where L5 represents the geometric consistency loss, Δf t→s (x t ) is the difference between the forward flow field and the reverse flow field at pixel point x t in image I t , |·| represents a modulus operation, and δ(x t ) is defined as formula (7):

[0033] |Δf t→s (x t )|<max{γ,η|Δf t→s (x t )|} (7)

[0034] where γ and η are threshold parameters, max(·) is a maximum function, and |·| represents a modulus operation.

[0035] Through the definition of the above loss, the overall loss of the network can be obtained, and the definition L is the loss of the overall network, as shown in formula (8):

[0036] L=w1L1+w2L2+w3L3+w4L4+w5L5 (8)

[0037] where L represents the overall loss of the network, w1, w2, w3, w4, and w5 represent the weights of each loss, respectively, L1 represents the image reconstruction loss through depth change, L2 represents the image reconstruction loss through optical flow change, L3 represents the spatial smoothing loss in the depth map, L4 represents the spatial smoothing loss in the optical flow map, and L5 represents the geometric consistency loss.

[0038] Compared with existing technologies, the advantages of this invention are as follows: This invention estimates two components of a 2.5D monocular scene stream—depth and optical flow—by utilizing a depth decoding structure based on dilated self-attention and an optical flow decoding structure based on nonlocal cross-frame attention. In this method, camera pose, depth, and optical flow are estimated simultaneously using a parallel structure. The encoder used for depth and optical flow estimation employs a shared pyramid feature encoder, and the decoders are a depth decoder based on dilated self-attention and an optical flow decoder based on nonlocal cross-frame attention, respectively. Adding appropriate self-attention and cross-attention to the decoding structure improves the expressive power of features and the correlation representation between pixels, further enhancing the accuracy of the estimation. Attached Figure Description

[0039] Figure 1 This is a flowchart of the present invention;

[0040] Figure 2 This is the overall architecture diagram of this design;

[0041] Figure 3 For the shared pyramid feature encoder structure diagram;

[0042] Figure 4 Diagram of a hollow self-attention structure;

[0043] Figure 5 This is a diagram of the nonlocal cross-frame attention structure. Detailed Implementation

[0044] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0045] This invention is achieved through the following steps:

[0046] S1. As attached Figure 2 As shown, a camera pose estimation subnetwork is constructed, consisting of 7 ordinary convolutional layers. The stride of the convolutional layers from the prediction layer is 2, 2, 2, 2, 2, 1, and 1, respectively, resulting in 16, 32, 64, 128, 256, 256, 256, and 6 channels in the output feature maps. Finally, the camera pose is estimated using a global average pooling layer.

[0047] Construct a shared pyramid feature encoder. (See attached image) Figure 3 As shown, the pyramid feature extraction module consists of 6 convolutional layers, with output feature maps having 16, 32, 64, 96, 128, and 256 channels, corresponding to the 6 levels of the pyramid structure in the image. The stride is 2 for each layer, and the feature map sizes are 1 / 2, 1 / 4, 1 / 8, 1 / 16, 1 / 32, and 1 / 64 of the original image size, respectively. All convolutional layers are followed by a ReLU activation function.

[0048] S2. As attached Figure 2 As shown, a parallel deep decoder is constructed. The decoding part mainly consists of a series of deconvolutional layers, with the number of output feature map channels being 512, 256, 128, 64, 32, and 16, respectively. A dilated self-attention layer is added after the first convolutional layer; the dilated self-attention structure is shown in the attached figure. Figure 4 As shown, a 1×1 convolution operation is first applied to the input feature map, followed by convolution operations with different dilation rates to capture contextual information. The different feature maps are summed and then activated by the SiLU function to obtain the query matrix Q. The specific operation is shown in formula (1):

[0049]

[0050] Among them, W q The weights are represented by the query matrix, d represents the number of feature channels, r and g represent the dilation coefficient and the number of groups in the convolution, respectively, and Conv represents the convolution operation. This represents the feature map after the convolution operation. The number of groups is set to the number of feature channels to further reduce the number of parameters. As calculated by formula (2):

[0051]

[0052] Where X represents the input feature map, W q The values ​​represent the weights of the query matrix, Conv represents the convolution operation, and r and g represent the dilation coefficient and the number of groups in the convolution, respectively. After the above operations, the query matrix and the key matrix are multiplied by a dot product, and the result is then multiplied by a dot product with the value matrix to obtain the output of the dilated self-attention.

[0053] A parallel optical flow decoder based on nonlocal cross-frame attention is constructed. Nonlocal attention can capture long-distance correspondences in time and space. (See attached diagram) Figure 5 As shown, group normalization is first used to normalize the input features, resulting in normalized features of adjacent frames. Then, a convolution operation is performed to obtain the value matrix and key matrix of one frame, and the query matrix of another frame. In this invention, the relevance matrix Γ is derived from the one-dimensional local vector Q in the query matrix. p The result is obtained by dot product of the key matrix and then softmax. To mitigate the impact of mismatched pixels, each slice in the second dimension of the correlation matrix Γ is multiplied by a Gaussian weight. Finally, the final output of the cross-frame nonlocal cross-attention module can be written as Equation (3):

[0054]

[0055] in, represents the product operation, i.e., Hadamard product, matrix G represents the point multiplication result of Key matrix and vector Q p , V represents the Value matrix obtained after convolution operation, and represents the correlation matrix t represents the feature map output by the non-local cross attention. In the optical flow decoder, each layer of feature maps is divided into 16 blocks, and then two feature maps of each layer are input into the non-local cross-frame attention module to perform cross attention to obtain a correlation matrix of the two image features, and the matrix is input into a convolution layer and a prediction layer to obtain the optical flow estimation result of each layer. Finally, the optical flow results of each layer are fused to obtain the final optical flow output.

[0056] S3. Constructing the overall network loss function, the learning method of the application is self-supervised learning, which can learn the depth, camera motion pose and optical flow information in the scene from unlabeled monocular image sequences. By defining a multi-task learning loss function, each sub-network can work cooperatively. The loss function includes image reconstruction loss, spatial smoothing loss and geometric consistency loss.

[0057] The image reconstruction loss function calculates the brightness error between the target image and the transformed image, and calculates the image similarity between the two images, which is specifically defined as shown in formula (4):

[0058]

[0059] Wherein, L1 represents the depth image reconstruction loss, x t represents a pixel point in image I t , and a is a balance factor. Image I t is a target view image, is an image after depth and camera pose motion change, and SSIM represents a structural similarity function. |·| represents an absolute value operation.

[0060] For the optical flow motion field, the original image I s can also be transformed using the optical flow motion field, and the transformed image is obtained. The image reconstruction loss can be calculated through the transformation, and the specific definition is shown in formula (5):

[0061]

[0062] Wherein, L2 represents the optical flow image reconstruction loss, x t represents a pixel point in image I t , and a is a balance factor. Image I t is a target view image, is an image after optical flow change, and SSIM represents a structural similarity function. |·| represents an absolute value operation.

[0063] For the depth map estimated by the network, the definition of the depth space smoothing loss is as shown in Equation (6):

[0064]

[0065] Where L3 represents the depth-space smoothing loss, x t Image I t The pixels in the image I t For the target view image, D(x) t ) is in pixel x t The estimated depth is obtained at that point. represents the gradient operator, and |·| represents the modulo operation.

[0066] For the optical flow map obtained by network estimation, the definition of the optical flow spatial smoothing loss is as shown in Equation (7):

[0067]

[0068] Where L4 represents the optical flow smoothing loss, x t Image I t The pixels in the image I t For the target view image, f t→s (x t ) is at point x t The estimated optical flow field is obtained at that location. represents the gradient operator, and |·| represents the modulo operation.

[0069] The geometric consistency loss is used to reduce interference from occluded areas, and its definition is shown in Equation (8):

[0070]

[0071] Where L5 represents the geometric consistency loss, Δf t→s (x t Image I t At pixel x t The difference between the forward and reverse flow fields, |·| represents the modulo operation, δ(x) t The definition of formula (9) is shown below:

[0072] |Δf t→s (x t )|<max{γ,η|Δf t→s (x t (9)

[0073] Where γ and η are threshold parameters, max(·) is the maximum value function, and |·| represents the modulo operation.

[0074] By the definition of the above loss, the overall loss of the network can be obtained, and the definition L is the loss of the overall network, as shown in formula (10):

[0075] L = w1L1 + w2L2 + w3L3 + w4L4 + w5L5 (10)

[0076] Wherein L represents the overall loss of the network, w1, w2, w3, w4, w5 represent the weight of each loss respectively, L1 represents the image reconstruction loss through the depth change, L2 represents the image reconstruction loss through the optical flow change, L3 represents the spatial smoothing loss in the depth map, L4 represents the spatial smoothing loss in the optical flow map, and L5 represents the geometric consistency loss.

[0077] S4. Input monocular image sequence to train the network in a self-supervised manner.

[0078] S5. When inputting two continuous image frames to test the model, two components of the 2.5D scene flow, optical flow and depth, can be obtained. When inputting a single image to test the model, the depth estimation result can be obtained.

[0079] In summary, the purpose of the present application is to provide a 2.5D monocular scene flow estimation method based on an attention decoding structure. The method comprises the following steps: S1, constructing a camera pose estimation sub-network and a shared pyramid feature encoder; S2, constructing a parallel depth decoder based on the attention of the cavity and an optical flow decoder based on the attention of the non-local cross-frame; S3, constructing an overall network loss function; S4, inputting monocular image sequence to train the network in a self-supervised manner; S5, when inputting two continuous image frames to test the model, two components of the 2.5D scene flow, optical flow and depth, can be obtained. When inputting a single image to test the model, the depth estimation result can be obtained. The present application estimates the 2.5D monocular scene flow by using the decoding structure based on attention, wherein attention is introduced in the decoding structure to strengthen the feature expression ability and pixel-related calculation, and the above design is used to achieve the purpose of improving the scene flow estimation accuracy.

Claims

1. A 2.5D monocular scene flow estimation method based on an attention decoding structure, characterized in that, The steps are as follows: S1. Construct a camera pose estimation subnetwork and a shared pyramid feature encoder; The camera pose estimation subnetwork includes multiple convolutional layers, and the output feature map is used to estimate the camera motion pose through a global average pooling layer. The shared pyramid feature encoder includes multiple convolutional layers, with the number of channels in the output feature map increasing sequentially, corresponding to the levels of the pyramid structure. All convolutional layers are followed by a ReLU activation function. S2. Construct a parallel deep decoder based on hole self-attention and an optical flow decoder based on nonlocal cross-frame attention; The dilated self-attention-based deep decoder adds dilated self-attention after the first convolutional layer, applies convolution operations to the input feature map, and then uses convolution operations with different dilation rates to capture contextual information. The sum of the different feature maps is then processed by the SiLU activation function to obtain the query matrix Q. Among them, W q The Query matrix represents the weights; d represents the number of feature channels; r and g represent the dilation coefficient and the number of groups in the convolution, respectively; Conv represents the convolution operation. This represents the feature map after the convolution operation. X represents the input feature map; Perform a dot product operation on the query matrix and the key matrix, and then perform a dot product operation on the result with the value matrix to obtain the output of the void self-attention; The optical flow decoder based on nonlocal cross-frame attention normalizes the input features to obtain normalized features of adjacent frames. It then uses convolution operations to obtain the Value matrix and Key matrix of one frame, and the Query matrix of another frame. The final output feature map X from the nonlocal cross-attention is... t : in, The product operation is represented by the Hadamard product; Γ represents the correlation matrix, which is a one-dimensional local vector Q in the query matrix. p The key matrix is ​​multiplied by the key matrix and then softmax is applied. Each slice of the second dimension of the correlation matrix Γ is multiplied by a Gaussian weight. Matrix G represents the key matrix and vector Q. p The dot product result; V represents the Value matrix; In the optical flow decoder based on nonlocal cross-frame attention, the feature map of each layer is divided into blocks, and the two feature maps of each layer are input into the nonlocal cross-frame attention module to obtain the correlation matrix of the two image features. This matrix is ​​then input into the convolutional layer and the prediction layer to obtain the optical flow estimation result of each layer. Finally, the optical flow results of each layer are fused to obtain the final optical flow output. S3. Construct the overall network loss function; S4. Input monocular image sequences and train the network end-to-end using a self-supervised approach; S5. When testing the network with two consecutive frames of images, two components of the 2.5D scene flow are obtained: optical flow and depth; when testing the network with a single image, the depth estimation result is obtained.

2. The 2.5D monocular scene flow estimation method based on attention decoding structure according to claim 1, characterized in that: Step S3, constructing the overall network loss function, specifically involves calculating the brightness error between the target image and the transformed image, and also calculating the image similarity between the two images. Its specific definition formula is as follows: Where L1 represents the depth image reconstruction loss, x t Image I t In the image I, the pixels are α, which is a balance factor. t Image from the target's perspective. For images after depth and camera pose motion changes, SSIM represents the structural similarity function, and |·| represents the modulo operation; For optical flow motion fields, their application to the original image I s Perform a transformation to obtain the image. This transformation allows us to calculate the image reconstruction loss, and its specific formula is as follows: Where L2 represents the optical flow image reconstruction loss, The image after optical flow transformation; For the depth map estimated by the network, the depth space smoothing loss is defined as: Where L3 represents the depth-space smoothing loss, D(x) t ) is in pixel x t The estimated depth is obtained at that point. Represents the gradient operator; For the optical flow map estimated by the network, the definition of the optical flow spatial smoothing loss is: Where L4 represents the optical flow smoothing loss, f t→s (x t ) is at point x t The optical flow field is estimated at this location; Geometric consistency loss is used to reduce interference from occluded regions, and it is defined as: Where L5 represents the geometric consistency loss, Δf t→s (x t Image I t At pixel x t The difference between the forward and reverse flow fields; Based on the above definition of loss, we obtain the overall loss of the network, defined as L, as shown in the following formula: L = w1L1 + w2L2 + w3L3 + w4L4 + w5L5 Where L represents the overall loss of the network, and w1, w2, w3, w4, and w5 represent the weights of each loss term.

Images