A transformer-based end-to-end multi-frame joint pose estimation method and device

Abstract

The present invention proposes a transformer-based end-to-end multi-frame joint pose estimation method and device. The method uses an optical flow estimation network to extract the original optical flow between multiple frames of images; and generates an internal reference layer according to the camera's internal reference matrix; each original optical flow is connected to the internal reference layer, and after centralized random shearing, it is spliced ​​into a multi-frame fusion feature map; then the transformer network is used to calculate the attention weight of the fusion feature map and output the feature vector; finally, the camera's pose is restored through a fully connected network, so that the accuracy of the end-to-end visual odometer is improved.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision perception, and in particular relates to a transformer-based end-to-end multi-frame joint pose estimation method and device. Background Art

[0002] Visual Odometry (VO), as a core problem in computer vision, is widely used in autonomous driving, 3D reconstruction, Simultaneous Localization and Mapping (SLAM) and other fields.

[0003] Traditional VO implementation methods can be mainly divided into feature point method, optical flow method, direct method, etc. In recent years, with the continuous development of deep learning technology, deep learning technology has also received widespread attention and application in VO. Using neural networks to extract feature points, it has shown performance that exceeds traditional manual features in terms of robustness and matching accuracy. Some end-to-end VO methods avoid the cumbersome traditional framework and rely on the powerful understanding ability of neural networks to directly estimate the corresponding pose transformation by only inputting image sequences. This end-to-end VO method is gradually becoming a new research hotspot due to its great potential in complex environments, high-level visual image understanding (such as semantics), and data-driven (such as unsupervised, large models, multimodal) and other aspects.

[0004] However, existing end-to-end VO methods usually use a combination of multi-layer convolutional neural networks as encoders to extract image features. Single-layer convolutional neural networks are limited by the size of the receptive field and have good feature extraction capabilities locally, but cannot extract features with good correlation for the entire image frame. Therefore, layer-by-layer encoding and extraction is often performed through a combination of multi-layer convolutional neural networks to achieve the purpose of expanding the receptive field at different scales of the image. However, this also usually leads to problems such as network structure redundancy and difficulty in network gradient transmission. As a new type of neural network, Transformer, unlike convolutional neural networks, does not require multiple convolution calculations. It only needs to calculate the attention relationship of the input feature vector itself to achieve excellent deep-level feature extraction capabilities. In addition, existing end-to-end methods mostly perform frame-to-frame pose estimation for two consecutive frames, estimate the relative pose of each two frames of the camera, and thereby restore the motion trajectory of the entire camera. This inevitably leads to the problem of cumulative drift.

[0005] Therefore, there is an urgent need for a method that can introduce advanced Transformer networks, efficiently extract image features, and effectively improve the cumulative drift problem in frame-to-frame pose estimation. Summary of the invention

[0006] In view of the shortcomings of the prior art, the present invention proposes a transformer-based end-to-end multi-frame joint pose estimation method and device. The method integrates the original optical flow information of multi-frame images and the intrinsic parameters of the camera, introduces an advanced Transformer network model, and realizes efficient image feature attention calculation and extraction. The multi-frame joint pose estimation effectively solves the problem of cumulative drift in frame-to-frame pose estimation, thereby improving the accuracy of the end-to-end visual odometer.

[0007] The present invention is achieved through the following scheme:

[0008] A transformer-based end-to-end multi-frame joint pose estimation method, the method comprising the following steps:

[0009] (1) Extracting original optical flows from multiple frames of images through the optical flow estimation network;

[0010] (2) Generate an intrinsic parameter layer based on the camera’s intrinsic parameter matrix, and connect each original optical flow with the intrinsic parameter layer to obtain an optical flow-intrinsic parameter layer composite representation;

[0011] (3) Perform random shearing to keep the center of the composite representation of optical flow-intrinsic reference layer, and concatenate every four processed composite representations into a fused feature map;

[0012] (4) Use the transformer network to calculate the attention weight of the fused feature map and output the feature vector;

[0013] (5) Based on the feature vector output in step (4), the camera pose is restored through a three-layer fully connected network.

[0014] Furthermore, the step (1) is specifically as follows: the optical flow estimation network performs an input image sequence ( , , …) to generate the original optical flow, and each adjacent two frames of images generate an optical flow map; when two frames of images , In the optical flow estimation network, an optical flow map is generated through multi-layer encoding and decoding , and so on, we can get the original optical flow sequence ( ).

[0015] Furthermore, the step (2) is specifically as follows: first, x Axis and y Axis generates two index matrices and ; According to the camera's internal parameters , for two index matrices and Perform projection calculations to generate internal reference layers Two channels and ; and compared with the original optical flow sequence ( ) are connected to form an optical flow-internal reference layer composite representation ( ).

[0016] Furthermore, the generation of the internal reference layer Two channels and , the specific calculation formula is as follows:

[0017] ;

[0018] in and represents the focal length, and Represents the principal point coordinates.

[0019] Furthermore, the step (3) is specifically: the optical flow-intrinsic reference layer composite representation ( ) While maintaining the centralized random shearing process, its length and width are reduced to 1 / 2 of the original, and the channel remains unchanged; in this way, the composite representation of the optical flow-intrinsic reference layer after every four random shearing processes is spliced ​​into a fused feature map.

[0020] Furthermore, the step (4) is specifically as follows: the fused feature map is input into the transformer, firstly using a convolution kernel size of The convolution block divides the fused feature map into non-overlapping image blocks; and Represents the length and width of the image; each image block is flattened into a feature vector The image is input to Swin TransformerBlocks for attention calculation, extraction and processing. Image blocks are merged between each layer for downsampling. The processed feature vector will be used for subsequent posture recovery.

[0021] Furthermore, the structure of the Swin Transformer Blocks is represented by the following formula to express its calculation process:

[0022] ;

[0023] ;

[0024] ;

[0025] ;

[0026] in, and Respectively represent The results of the output features of the SW-MSA module and the MLP module in the Transformer Blocks; W-MSA and SW-MSA represent the multi-head self-attention mechanism of conventional and shifted window partitions, respectively.

[0027] The present invention also provides a transformer-based end-to-end multi-frame joint pose estimation device, which includes the following modules:

[0028] Optical flow extraction module: extracts original optical flow from multiple frames of images through the optical flow estimation network;

[0029] Optical flow-intrinsic reference layer connection module: Generates the intrinsic reference layer according to the camera's intrinsic reference matrix, and connects each original optical flow with the intrinsic reference layer to obtain an optical flow-intrinsic reference layer composite representation;

[0030] Cutting and splicing module: Perform random cutting processing on the composite representation of optical flow-intrinsic reference layer to keep the center, and splice every four processed composite representations into a fused feature map;

[0031] Calculation output module: Use the transformer network to calculate the attention weight of the fused feature map and output the feature vector;

[0032] Posture recovery module: Based on the feature vector output by the calculation output module, the posture of the camera is restored through a three-layer fully connected network.

[0033] An electronic device is also provided, comprising:

[0034] one or more processors;

[0035] A memory for storing one or more programs;

[0036] When the one or more programs are executed by the one or more processors, the one or more processors implement the transformer-based end-to-end multi-frame joint pose estimation method.

[0037] A computer-readable storage medium is provided, on which computer instructions are stored, and when the instructions are executed by a processor, the steps of the transformer-based end-to-end multi-frame joint pose estimation method are implemented.

[0038] The beneficial effects of the present invention are as follows: (1) The original optical flow is extracted as an intermediate expression, which clarifies the 2D matching relationship between images and strengthens the motion clues; (2) Connecting the intrinsic reference layer and the original optical flow can introduce the intrinsic parameters of the camera into the model calculation, which on the one hand increases the spatial scale relationship of the image features and on the other hand is conducive to the generalization of the model across camera images; (3) The introduction of the advanced Transformer model realizes efficient attention calculation and extraction of image features; (4) Multi-frame joint pose estimation effectively solves the problem of cumulative drift in frame-to-frame pose estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a schematic diagram of the overall network structure of the present invention;

[0040] Figure 2 It is a schematic diagram of generating an internal reference layer of the present invention;

[0041] Figure 3 It is a schematic diagram of the centralized random shearing and splicing fusion features of the present invention;

[0042] Figure 4 Schematic diagram of the Transformer block of the present invention. DETAILED DESCRIPTION

[0043] Here, exemplary embodiments are described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application.

[0044] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0045] Example 1

[0046] As attached Figure 1 As shown in Figure 1, a transformer-based end-to-end multi-frame joint pose estimation method is shown in Figure 2. The specific steps are as follows:

[0047] S1: Extract the original optical flow from multiple frames of images through the optical flow estimation network;

[0048] S2: Generate an intrinsic reference layer according to the intrinsic reference matrix of the camera, and connect each original optical flow with the intrinsic reference layer to obtain an optical flow-intrinsic reference layer composite representation;

[0049] S3: Perform random shearing to keep the center of the composite representation of optical flow-intrinsic reference layer, and concatenate every four processed composite representations into a fused feature map;

[0050] S4: Use the transformer network to calculate the attention weight of the fused feature map and output the feature vector;

[0051] S5: Based on the feature vector output by S4, the camera pose is restored through a three-layer fully connected network;

[0052] S1: Input image sequence ( ), the optical flow estimation network is shown in the attached Figure 2 As shown. After each adjacent two frames of images are processed by the optical flow estimation network with multiple layers of encoding and decoding, an original optical flow is generated. The optical flow estimation network can adopt the advanced open source work in this field. In this embodiment, the RAFT network is adopted. , For example, , Generate an optical flow map under the action of the optical flow estimation network By analogy, the original optical flow sequence obtained for the entire image sequence is ( ).

[0053] In S2: Camera internal parameters ,in and represents the focal length, and Indicates the coordinates of the principal point. To generate an internal reference layer of the same scale as the original optical flow , first of all, the two-dimensional image frame x Axis and y Axis generates two index matrices and ; Then for the two index matrices and Perform projection calculations to generate internal reference layers Two channels and This process is shown in the attached Figure 3 As shown, the specific calculation formula is as follows: ;

[0054] Subsequently, the original optical flow sequence described in S1 ( ) ( H and W Represents the height and width of the image) and the internal reference layer are connected to form an optical flow-internal reference layer composite representation ( , ) The process is as shown in the attached Figure 4 shown.

[0055] S3: The composite representation of the optical flow-intrinsic layer described in S2 Under the random clipping process of keeping the center, each optical flow-intrinsic layer composite representation ( ) The length and width are reduced to 1 / 2 of the original, and the channel remains unchanged, that is, ; In this way, the composite representation of the optical flow-intrinsic reference layer after four random shearing processes is spliced ​​into a fused feature map The process is as follows Figure 1 shown.

[0056] S4: The fused feature map described in S3 Input into the transformer network, where and Represents the length and width of the image; the specific structure of the transformer network is shown in the attached Figure 1 As shown. First, use the convolution kernel size of The convolution block divides the fused feature map into non-overlapping image patches; each patch is flattened into a feature vector , and input to the multi-layer SwinTransformer Blocks for attention calculation extraction and processing, where image blocks are merged between each layer for downsampling to reduce the network calculation burden.

[0057] The structure of Swin Transformer Blocks is as follows: Figure 4 As shown, the calculation process can be expressed by the following formula:

[0058] ;

[0059] ;

[0060] ;

[0061] ;

[0062] in, and Respectively represent The result of outputting features of (S)W-MSA modules and MLP modules in Transformer Blocks; W-MSA and SW-MSA represent conventional and shifted window partitioned multi-head self-attention mechanisms, respectively.

[0063] In S5:

[0064] The transformer network described in S4 processes the multi-frame fusion feature maps and finally outputs a feature vector containing rich geometric clues. This feature vector is passed through a 3-layer fully connected network to recover the translation vector and rotation vector representing the relative position of the camera.

[0065] The present invention also provides a transformer-based end-to-end multi-frame joint pose estimation device, which includes the following modules:

[0066] Optical flow extraction module: extracts original optical flow from multiple frames of images through the optical flow estimation network;

[0067] Optical flow-intrinsic reference layer connection module: Generates the intrinsic reference layer according to the camera's intrinsic reference matrix, and connects each original optical flow with the intrinsic reference layer to obtain an optical flow-intrinsic reference layer composite representation;

[0068] Cutting and splicing module: Perform random cutting processing on the composite representation of optical flow-intrinsic reference layer to keep the center, and splice every four processed composite representations into a fused feature map;

[0069] Calculation output module: Use the transformer network to calculate the attention weight of the fused feature map and output the feature vector;

[0070] Posture recovery module: Based on the feature vector output by the calculation output module, the posture of the camera is restored through a three-layer fully connected network.

[0071] Also provided is an electronic device, comprising:

[0072] one or more processors;

[0073] A memory for storing one or more programs;

[0074] When the one or more programs are executed by the one or more processors, the one or more processors implement the transformer-based end-to-end multi-frame joint pose estimation method.

[0075] And a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the steps of the transformer-based end-to-end multi-frame joint pose estimation method.

[0076] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the contents disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not disclosed in the present application.

[0077] It should be understood that the present application is not limited to the exact construction that has been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof.

Claims

1. A transformer-based end-to-end multi-frame joint pose estimation method, characterized in that: The method comprises the following steps: (1) Extracting original optical flows from multiple frames of images through the optical flow estimation network; (2) Generate an intrinsic reference layer based on the camera’s intrinsic parameter matrix, and connect each original optical flow with the intrinsic reference layer to obtain an optical flow-intrinsic reference layer composite representation; (3) Perform random shearing to keep the center of the composite representation of optical flow-intrinsic reference layer, and concatenate every four processed composite representations into a fused feature map; (4) Use the transformer network to calculate the attention weight of the fused feature map and output the feature vector; specifically, the fused feature map is input into the transformer, and the fused feature map is first divided into non-overlapping image blocks; where H and W represent the length and width of the image; each image block is flattened into a feature vector The input is sent to Swin Transformer Blocks for attention calculation extraction and processing. Image blocks are merged between each layer for downsampling. The processed feature vector will be used for subsequent posture recovery. (5) Based on the feature vector output in step (4), the camera pose is restored through a three-layer fully connected network.

2. According to claim 1, a transformer-based end-to-end multi-frame joint pose estimation method is characterized in that: The step (1) is specifically as follows: the optical flow estimation network generates an original optical flow for the input image sequence (I1, I2, ...), and generates an optical flow map for each two adjacent frames of images; when the two frames of images I1 and I2 are encoded and decoded in the optical flow estimation network through multiple layers, an optical flow map F is generated. 1，2 , and so on, we can get the original optical flow sequence (F 1,2 、F 2,3 ,…F i，i+1 ).

3. The transformer-based end-to-end multi-frame joint pose estimation method according to claim 1, characterized in that: The step (2) is specifically as follows: firstly, two index matrices X are generated for the x-axis and y-axis in the two-dimensional image frame. ind and V ind ; According to the camera's internal parameter N = {f x , f y , o x , o y }, for two index matrices X ind and Y ind Perform projection calculations to generate the internal reference layer K c Two channels and And compared with the original optical flow sequence (F 1,2 、F 2,3 ,…F i,i+1 ) in each original optical flow F i，j+1 are connected to form an optical flow-internal reference layer composite representation (F i，i+1 , K c ).

4. The transformer-based end-to-end multi-frame joint pose estimation method according to claim 3, characterized in that: The generated internal reference layer K c Two channels and The specific calculation formula is as follows: where f x and f y represents the focal length, o x and y Represents the principal point coordinates.

5. The transformer-based end-to-end multi-frame joint pose estimation method according to claim 1, characterized in that: The step (3) is specifically: the optical flow-intrinsic reference layer composite representation (F i，i+1 , K c ) While maintaining the centralized random shearing process, its length and width are reduced to 1 / 2 of the original, and the channel remains unchanged; in this way, the composite representation of the optical flow-intrinsic reference layer after every four random shearing processes is spliced ​​into a fused feature map.

6. The transformer-based end-to-end multi-frame joint pose estimation method according to claim 1, characterized in that: The structure of the Swin Transformer Blocks is expressed in the following formula to express its calculation process: in, and z l They respectively represent the results of the output features of the SW-MSA module and the MLP module in the lth Transformer Blocks; W-MSA and SW-MSA represent the conventional and shifted window partitioned multi-head self-attention mechanisms, respectively.

7. A transformer-based end-to-end multi-frame joint pose estimation device, characterized in that: The device includes the following modules: an optical flow extraction module: extracting original optical flows from multiple frames of images respectively through an optical flow estimation network; Optical flow-intrinsic reference layer connection module: Generates the intrinsic reference layer according to the camera's intrinsic reference matrix, and connects each original optical flow with the intrinsic reference layer to obtain an optical flow-intrinsic reference layer composite representation; Cutting and splicing module: Perform random cutting processing on the composite representation of optical flow-intrinsic reference layer to keep the center, and splice every four processed composite representations into a fused feature map; Calculation output module: Use the transformer network to calculate the attention weight of the fused feature map and output the feature vector; Specifically, the fused feature map is input into the transformer, and the fused feature map is first divided into non-overlapping image blocks; where H and W represent the length and width of the image; each image block is flattened into a feature vector The input is sent to Swin Transformer Blocks for attention calculation extraction and processing. Image blocks are merged between each layer for downsampling. The processed feature vector will be used for subsequent posture recovery. Posture recovery module: Based on the feature vector output by the calculation output module, the posture of the camera is restored through a three-layer fully connected network.

8. An electronic device, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the transformer-based end-to-end multi-frame joint pose estimation method as described in any one of claims 1-6.

9. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the instruction is executed by the processor, the steps of the transformer-based end-to-end multi-frame joint pose estimation method are implemented as described in any one of claims 1-6.

Images