Panoramic image stitching method based on deformable attention mechanism

Abstract

The application discloses a panoramic image splicing method based on a deformable attention mechanism, which comprises the following steps: sending multiple-view images into a backbone network to obtain multi-scale feature maps; generating a learnable Bev Queries model, randomly generating a grid-shaped Bev Queries, each grid of the Bev Queries representing an actual physical size and corresponding to a three-dimensional coordinate of a self-vehicle coordinate system, querying and aggregating relevant feature information in the multi-view multi-scale pictures through an attention mechanism, and generating a panoramic bird's eye view through transposed convolution; finally, training the model according to the obtained Bev Queries model, learning the structure of the graph using MS-SSIM Loss, learning edge features and correcting the color and removing the image noise of the image when the MS-SSIM Loss is reduced to a set threshold, and obtaining a panoramic image. The application can not only construct a more smooth and less distorted and distorted panoramic bird's eye view, but also has strong robustness to camera internal and external parameter drift in a complex environment and has a fast calculation speed.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of panoramic stitching of multi-view images, relates to the auxiliary driving technology of a vehicle, and mainly relates to a panoramic image stitching method based on a deformable attention mechanism. BACKGROUND

[0002] In many traffic accidents, accidents caused by the blind area of a vehicle account for a relatively large proportion, and therefore, the use of panoramic bird's-eye view to solve the visual blind area problem of a driver is increasingly valued.

[0003] The current mainstream method is to obtain a panoramic image through inverse perspective mapping (IPM) after multi-view images are collected by an image collection device. Specifically, features are extracted from the multi-view images through a SIFT method or the like, a homography matrix is obtained, and then, pixel-by-pixel perspective projection is performed according to the homography matrix and converted into a panoramic bird's-eye view perspective. Then, the panoramic bird's-eye view formed by the multi-view images is stitched and fused at the interface. However, the IPM algorithm has two assumptions: 1. the road surface is parallel to the world coordinate system, and 2. the coordinate system of the vehicle itself is parallel to the world coordinate system. The former does not meet the condition in the case of a non-flat road surface, and the latter relies on real-time correction of the vehicle attitude parameters (Pitch and Roll) and has a high accuracy requirement, which is very difficult to achieve. Moreover, during the driving of a commercial vehicle, the external environment is relatively complex, the camera of the commercial vehicle may deviate from the calibrated position, and the internal and external parameters of the camera may also drift, which has a great impact on the pixel-by-pixel projection method. The image stitching result is prone to have a large graphical distortion or even projection errors, and therefore, the camera of certain commercial vehicles needs to be recalibrated regularly, which is not practical and economical. In addition, the artifacts caused by image parallax cannot be ignored. Therefore, the image stitching method based on IPM is prone to have a serious distortion, misaligned images, pixel voids and other problems, and is not ideal in actual use.

[0004] In view of the problems existing in the current mainstream image stitching method, in order to better solve the vehicle blind area problem of a commercial vehicle driver and provide a smooth panoramic bird's-eye view without distortion for the driver, a better method is urgently needed to construct a more smooth and less distorted panoramic bird's-eye view, which can overcome the problems of complex environment and camera internal and external parameter drift. SUMMARY

[0005] The present application is just for the problems of serious distortion, image misalignment and artifacts in the prior art traditional method, and provides a panoramic image stitching method based on a deformable attention mechanism, multi-view images are sent into a backbone network to obtain multi-scale feature maps; a learnable Bev Queries model is generated, Bev Queries in a grid shape are randomly generated, each cell of the Bev Queries represents an actual physical size and corresponds to a three-dimensional coordinate of a self-vehicle coordinate system, relevant feature information is queried and aggregated in multi-view multi-scale pictures through an attention mechanism, and a panoramic bird's eye view is generated through transposed convolution; finally, the Bev Queries model obtained is used for model training, MS-SSIM Loss is used to learn the structure of the graph, when the MS-SSIM Loss decreases to a set threshold, an L2 Loss loss function is used to learn edge features and correct the color of the image, finally, an L1 Loss loss function is used to remove image noise, and a panoramic image is obtained. The present application can not only construct a more smooth and less distorted panoramic bird's eye view, but also has strong robustness to camera internal and external parameter drift in a complex environment, can better face a complex environment, and has a faster calculation speed because only local attention calculation is performed.

[0006] In order to achieve the above-mentioned purpose, the technical scheme adopted by the present application is as follows: a panoramic image stitching method based on a deformable attention mechanism, comprising the following steps:

[0007] S1, multi-scale feature extraction of multi-view images: multi-view images are sent into a backbone network to obtain multi-scale feature maps, and the position information of feature points in the multi-scale feature maps is obtained by adding feature layer position encoding and position encoding calculated based on a trigonometric function formula;

[0008] S2, generating a learnable Bev Queries model: randomly generating Bev Queries in a grid shape, each cell of the Bev Queries represents an actual physical size and corresponds to a three-dimensional coordinate of a self-vehicle coordinate system, relevant feature information is queried and aggregated in multi-view multi-scale pictures through an attention mechanism, and a panoramic bird's eye view is generated through transposed convolution;

[0009] S3, model training: using the Bev Queries model obtained in step S2, using MS-SSIM Loss to learn the structure of the graph, when the MS-SSIM Loss decreases to a set threshold, using an L2 Loss loss function to learn edge features and correct the color of the image, finally, using an L1 Loss loss function to remove image noise, and obtaining a panoramic image.

[0010] As an improvement of the present application, EfficientNet-B4 is used as the backbone network in step S1.

[0011] As an improvement of the present application, step S2 is specifically: randomly initializing Bev Queries∈R H×W×C , H, W, C are the height, width and channel number of Bev Queries respectively, Bev Queries is divided into HxW grids, each grid represents the actual physical size and corresponds to a three-dimensional coordinate (x, y, z) of the ego vehicle coordinate system, and all grids share a Z coordinate around 0;

[0012] Each grid of Bev Queries queries the relevant feature information generated based on geometric prior and attention mechanism, continuously updates Bev Queries according to the query result, and finally generates a panoramic bird's eye view through transposed convolution.

[0013] As an improvement of the present application, in step S2, the feature range that each grid of Bev Queries queries includes K h x K w fixed region and deformed sampling region:

[0014] K h x K w The fixed region refers to a K h x K w region centered on , and the relevant features of the fixed region that each grid will query are where N view , N scale , c, K h , K w are the number of viewing angles, the number of feature map scales, the number of channels, the height of the sampling region, and the width of the sampling region, respectively. Refers to the two-dimensional coordinates of each grid in different viewing angles and different size feature maps;

[0015] The deformed sampling region refers to a group of position offset regions generated by Bev Queries through linear mapping.

[0016] As another improvement of the present application, the generation method of the two-dimensional coordinates is specifically:

[0017] S21: determining the three-dimensional coordinates p i (x, y, z) of each grid of Bev Queries, i represents the grid number of Bev Queries;

[0018] S22: converting the three-dimensional coordinates pi (x,y,z) is projected onto the multi-view image according to the internal and external parameters of the camera to obtain its corresponding two-dimensional coordinates The projection is calculated as follows:

[0019]

[0020] Wherein, v represents different views; R v is the camera external parameter; K v is the camera internal parameter;

[0021] S23: The two-dimensional coordinates obtained in step S22 are normalized to obtain the corresponding two-dimensional coordinates on the multi-scale feature map s represents a feature map of different scales;

[0022] S24: The Round function is used to obtain integer coordinates, and the Round function rounds the numerical value and returns the integer value:

[0023]

[0024] As another improvement of the application, the updating method of Bev Queries in step S2 is specifically: generating all weights corresponding to the sampling area by full connection layer and Softmax layer for Bev Queries, updating Bev Queries according to the feature value of the sampling area and the corresponding weight, and the calculation formula is as follows

[0025]

[0026] Wherein, Q represents Bev Queries, F represents multi-scale feature map, m is the number of attention heads, Delta mq respectively are fixed sampling area and deformed sampling area, A mqk is the attention weight, W m and W′ m both represent linear mapping;

[0027] After calculation, the generated result and the input Bev Queries are added and then normalized by layer, and then the output of the feedforward network layer and the input are added and normalized, which is a complete encoding layer; the Bev Queries model comprises 6 encoding layers.

[0028] As another improvement of the application, the method for generating a panoramic bird's eye view through transposed convolution in step S2 is specifically: after 6 encoding layers, multi-layer transposed convolution is used to convert high-dimensional small-size Bev Queries into a 3-channel 200x200 BEV image.

[0029] As a further improvement of the present application, the calculation formula of MS-SSIM Loss learning the graph structure in step S3 is:

[0030]

[0031] Where m represents different scales, μ p , μ g represents the mean of the predicted image and the true value respectively, σ p , σ g represents the standard deviation between the predicted image and the true value, σ pg represents the covariance of the predicted image and the true value, β m , γ m represent the relative importance between the two terms, and c1 and c2 are constants.

[0032] Compared with the prior art, the present application has the beneficial effects: a panoramic image stitching method based on deformable attention mechanism is proposed, in order to better solve the problem of blind area of commercial vehicle driver, a smooth and distortion-free panoramic bird's eye view is provided for the driver, the method is proposed by using the geometric prior knowledge (IPM) of camera projection and the characteristics of attention mechanism focusing on part of the main information, not only can construct a more smooth and less distortion and distortion panoramic bird's eye view, but also has strong robustness to camera internal and external parameter drift in complex environment, can better face complex environment, at the same time because only local attention calculation is carried out, the method has fast calculation speed, the image stitching framework constructed by the present application has good expansibility, not only can be applied to panoramic image stitching of commercial vehicle, but also can be extended to medical image, three-dimensional real scene modeling and virtual reality and other task scenes in the future. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 is the overall structural framework diagram of the method of the present application;

[0034] Figure 2 is the data flow framework diagram of the sparse spatial deformable cross-attention in embodiment 1 of the present application;

[0035] Figure 3 is a comparison diagram of the projection process of the traditional IPM method and the method of the present application;

[0036] Figure 4 is a sampling area illustration diagram of each grid of Bev Queries in embodiment 1 of the present application;

[0037] Figure 5 is a step flow chart of the method of the present application. DETAILED DESCRIPTION

[0038] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are for illustrative purposes only and are not intended to limit the scope of the invention.

[0039] Example 1

[0040] Traditional bird's-eye view generation methods suffer from significant distortion and poor robustness. To effectively address these drawbacks and provide drivers with a smoother, less distorted panoramic bird's-eye view of the vehicle's surroundings, this invention discloses a panoramic image stitching method based on a deformable attention mechanism, such as... Figure 5 As shown, the specific steps are as follows:

[0041] Step S1: Extract multi-scale features from multi-view images

[0042] like Figure 1 As shown in the overall framework diagram, this invention uses EfficientNet-B4 (Tan, M., & Le, QV (2019). EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks.) as the backbone network for multi-view image sharing. After feeding multi-view images into the backbone network, multi-scale feature maps are obtained. This provides multi-scale feature information for the next step of the spatial cross-attention mechanism. Multi-scale feature information focuses on richer feature information, making the image stitching result smoother. Based on the multi-scale fusion characteristic of the deformable attention mechanism, although multi-scale features are used, this invention does not use traditional Feature Pyramid Networks (FPNs), thus accelerating the model's inference speed.

[0043] The input to the attention mechanism is sequential. When stitching together multi-view, multi-scale images, this invention first uses trigonometric function positional encoding to represent the positional relationships between different feature points. Simultaneously, because feature points in feature maps of different scales may have the same two-dimensional positional coordinates, this invention uses learnable positional encoding based on feature layers; that is, all feature points in the same feature layer correspond to the same feature layer positional encoding. In practical use, the feature layer positional encoding and the positional encoding calculated based on trigonometric function formulas are added together as the embedded positional information.

[0044] Step S2: Generate a learnable Bev Queries model

[0045] Bev Queries (Li, Z., BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers) is a grid-shaped learnable parameter that describes the features of the panoramic bird's-eye view.

[0046] Randomly initialize Bev Queries∈R H×W×C H, W, C are the height, width and channel number of Bev Queries respectively, and its function is to query and aggregate related feature information in multi-view multi-scale pictures through attention mechanism, and finally generate a panoramic bird's-eye view. Bev Queries is divided into HxW grids, and each grid represents the actual physical size and corresponds to a three-dimensional coordinate (x, y, z) of the ego vehicle coordinate system. Since the panoramic bird's-eye view of the top view is to be spliced, all grids will share a Z coordinate around 0.

[0047] Each grid of Bev Queries will query some related feature information generated based on geometric prior and attention mechanism, and then update Bev Queries based on the query results. Finally, the panoramic bird's-eye view is generated by transposed convolution of Bev Queries.

[0048] The present application constructs a sparse spatial deformable attention mechanism with good robustness based on a deformable attention mechanism, and has a faster inference speed. The data flow framework diagram is as shown in Figure 2 Each Query obtains the required weight and offset through linear mapping, and the deformation sampling area can be obtained according to the offset. At the same time, the multi-scale feature map is linearly mapped to obtain the corresponding value. Finally, the attention mechanism calculation is performed according to the weight, sampling area and feature corresponding value. The sparse spatial deformable attention mechanism makes the present application still have good splicing effect and good robustness when the internal and external parameters of the car drift, and also expands the practical application scenarios of the present application.

[0049] In the traditional IPM method, the projection process is as shown in the left part of Figure 3 The camera internal and external parameters are projected from the multi-view image to the panoramic image, which will cause some pixels to overlap and have holes due to parallax, which exacerbates the non-smoothness of the panoramic image. In order to overcome this shortcoming, the present application selects to project from the panoramic bird's-eye view to the multi-view image according to the camera internal and external parameters. The general process of projection is as shown in the right part of Figure 3 This solves the problem of parallax. Because each grid of Bev Queries has a three-dimensional coordinate p i(x, y, z), i represents each grid of Bev Queries, so it can be projected onto the multi-view image according to the internal and external parameters of the camera, and the corresponding two-dimensional coordinates F are obtained i v (x, y), v represents different views. Then the two-dimensional coordinates are normalized to obtain the corresponding two-dimensional coordinates F on the multi-scale feature map i sv (x, y), s represents different scale feature maps, and then the Round function is used to obtain integer coordinates. The Round function rounds the numerical value and returns its integer value. The calculation process of the projection is as follows (where R v is the camera extrinsic parameter, K v is the camera intrinsic parameter):

[0050] F i v = K v × R v × P i

[0051]

[0052] Therefore, each grid of Bev Queries will obtain some two-dimensional coordinates in different views and different size feature maps Because a visual element only needs to establish a connection with part of the feature information, and in order to improve the robustness of the model, the present application will sample a K h × K w region centered on If the sampling range exceeds the boundary of the feature map, the corresponding feature value will be set to zero. Therefore, the relevant features of the fixed region that each Query will query are N view , N scale , C, K h , K w are the number of views, the number of feature map scales, the number of channels, the height of the sampling region, and the width of the sampling region. In this way, even if the internal and external parameters of the camera drift slightly, K h × K wThe fixed sampling region also covers the projection center after the camera internal and external parameter drifts, because the attention mechanism has permutation invariance and dynamic weight, and slight changes in the position of the projection center will not have a great impact on the stitching result, so the camera drift of smaller amplitude will not affect the result, which greatly improves the robustness of the model. At the same time, it also reduces the computational complexity and the required space resources, and speeds up the model convergence. The correspondence between each grid of Bev Queries and the sampling position of the fixed region is fixed, so the fixed sampling table can be calculated in advance and established accordingly. When needed, directly look up the sampling feature position according to the table. Using the fixed sampling table can make the model free from the camera internal and external parameters, saving the calculation time of using the camera internal and external parameters for projection, and improving the inference speed.

[0053] At the same time, in order to deal with some special situations, such as some special graphics such as lane lines, which need some direction larger range of feature information, relying only on K h ×K w fixed region feature information is not enough, so the application borrows the idea of deformable convolution, generates a set of position offsets through linear mapping of Bev Queries, to pay attention to those feature information that cannot be paid attention to by K h ×K w fixed region features, which will improve the robustness of the model and better cope with the drift of the camera internal and external parameters. Therefore, the feature range that each Query of Bev Queries will pay attention to includes K h ×K w fixed region and deformed sampling region, as shown in Figure 4 , the middle rectangular region is K h ×K w fixed region, and the discrete region around it is deformed sampling region, and the overall is the feature region that each Query will query. Based on the idea of deformable convolution, Bev Queries generates all the weights corresponding to the sampling region through the fully connected layer and the Softmax layer, and then updates Bev Queries according to the feature value of the sampling region and the corresponding weight, which is a sparse spatial attention mechanism calculation, and the calculation formula is

[0054]

[0055] Where Q represents Bev Queries, F represents multi-scale feature maps, m is the number of attention heads, Δ mq are the fixed sampling region and the deformed sampling region respectively, A mqk is the attention weight, W m and W′ mBoth represent linear mappings.

[0056] After the calculation of the cross-attention mechanism, the generated results and the input Bev Queries are added and then normalized by layer, and then pass through a feedforward network layer, and finally the output of the feedforward network layer and the input are added and normalized, which is a complete encoding layer, and the model of the application uses 6 encoding layers to generate more expressive Bev Queries.

[0057] As shown in Figure 1 After 6 encoding layers, Bev Queries have aggregated enough information, in order to generate a 200x200 size panoramic bird's eye view, the application uses multi-layer transposed convolution to convert high-dimensional small-size Bev Queries into 3-channel 200x200 BEV images. This step is all the forward propagation process of the model of the application.

[0058] Step S3: model training

[0059] There is no large public data set for image stitching, so the application selects UDIS-D, CROSS and HPatches data sets, and also constructs the data required for training the model according to some public data sets such as COCO.

[0060] The method of the application generates a panoramic bird's eye view according to the multi-scale feature maps of multi-view images, so the loss function is fully referenced from the loss functions of image inpainting and image generation models. The training strategy of the application is to first use MS-SSIM Loss to learn the structure of the graph, and the formula of MS-SSIM Loss is as follows:

[0061]

[0062] Where m represents different scales, mu p , mu g respectively represent the mean of the predicted image and the true value, sigma p , sigma g respectively represent the standard deviation between the predicted image and the true value, sigma pg represents the covariance of the predicted image and the true value, beta m , gamma mThe relative importance between two items is represented, c1 and c2 are constants, and the divisor is prevented from being 0. MS-SSIM is a structural similarity index, which can be used to measure the similarity between two pictures, and is a perceptual model, which is more in line with the intuitive feeling of the human eye. When the MS-SSIM loss is reduced to a certain threshold, the next stage will use the L2Loss loss function to learn some edge features and correct the color of the image, and the last stage will use the L1Loss loss function to remove some image noise to achieve panoramic image stitching.

[0063] The application constructs a relatively general image stitching framework, which has good expansibility and can be used not only in panoramic image stitching of a vehicle, but also in more task scenarios, and can be used in medical images, three-dimensional real scene modeling and virtual reality scenarios in the future. The application alleviates the problems of serious distortion, misalignment of images and artifacts existing in the traditional IPM method, and realizes a more smooth panoramic bird's eye view stitching method. Meanwhile, the application can better cope with various complex situations in commercial vehicles, and has good robustness to drift of camera internal and external parameters.

[0064] It should be noted that the above content only illustrates the technical idea of the application, and cannot limit the protection scope of the application. For ordinary skilled persons in the art, some improvements and refinements can be made without departing from the principles of the application, and these improvements and refinements fall within the protection scope of the claims of the application.

Claims

1. A panoramic image stitching method based on a deformable attention mechanism, characterized in that, The method comprises the following steps: S1, multi-scale feature extraction of multi-view images: after the multi-view images are input into a backbone network, multi-scale feature maps are obtained, and the position information of feature points in the multi-scale feature maps is obtained by adding feature layer position encoding and position encoding calculated based on a trigonometric function formula; S2, generating a learnable Bev Queries model: randomly generating a grid-shaped Bev Queries, each grid of the Bev Queries represents an actual physical size and corresponds to a three-dimensional coordinate of a vehicle coordinate system, and relevant feature information is queried and aggregated in multi-view multi-scale pictures through an attention mechanism, and a panoramic bird's eye view is generated through transposed convolution; specifically: Randomly initialize Bev Queries , are the height, width and channel number of Bev Queries, Bev Queries are divided into grids, each grid represents the actual physical size and corresponds to a three-dimensional coordinate (x, y, z) of the ego vehicle coordinate system, and all grids share a Z coordinate around 0; Each grid of the Bev Queries queries relevant feature information generated based on geometric priors and an attention mechanism, and the Bev Queries are constantly updated according to the query results, and finally the Bev Queries are converted into a panoramic bird's eye view through transposed convolution; The range of features that each Bev Query will attend to includes fixed regions and warped regions: The fixed regions refer to the regions centered at The relevant features of the fixed regions that each grid will query are where and are the number of views, the number of feature map scales, the number of channels, the height of the sampling region, and the width of the sampling region, respectively; The 2D coordinates of each grid in different views and different size feature maps; the warped regions refer to the regions that are generated by Bev Queries through linear mapping to generate a set of position offset regions; S3, model training: according to the Bev Queries model obtained in step S2, MS-SSIM Loss is used to learn the structure of the graph, when the MS-SSIM Loss is reduced to a set threshold, L2 Loss loss function is used to learn edge features and correct the color of the image, and finally L1 Loss loss function is used to remove image noise, and a panoramic image is obtained. 2.The panorama image stitching method based on deformable attention mechanism according to claim 1, wherein: In the step S1, EfficientNet-B4 is used as the backbone network. 3.The panorama image stitching method based on deformable attention mechanism according to claim 2, wherein: The two-dimensional coordinates The generation mode is specifically: S21: Determine the three-dimensional coordinates of each cell of Bev Queries , represents the cell number of Bev Queries; S22: obtaining three-dimensional coordinates of the object in the three-dimensional space according to the three-dimensional coordinates of the object in the three-dimensional space and the three-dimensional coordinates of the object in the three-dimensional space According to the internal and external parameters of the camera, the three-dimensional coordinates of the object in the three-dimensional space are projected onto the multi-view image to obtain the corresponding two-dimensional coordinates The calculation method of the projection is as follows: ； wherein, represent different view angles; is a camera extrinsic parameter; is a camera intrinsic parameter; S23: normalizing the two-dimensional coordinates obtained in step S22 to obtain the corresponding two-dimensional coordinates on the multi-scale feature map , represent feature maps of different scales S24: obtaining integer coordinates by using a Round function, which rounds the numerical value and returns the integer value thereof: 。 4.The panorama image stitching method based on deformable attention mechanism according to claim 3, wherein: In the step S2, the updating method of the Bev Queries is specifically as follows: the Bev Queries are generated through a fully connected layer and a Softmax layer to generate all weights corresponding to the sampling area, the Bev Queries are updated according to the feature values of the sampling area and the corresponding weights, and the calculation formula is as follows: ； where Q represents Bev Queries, F represents multi-scale feature maps, m is the number of attention heads, are fixed sampling regions and deformed sampling regions, respectively, is an attention weight, and represent linear mappings; After calculation, the generated result and the input Bev Queries are added and then normalized, and then a feedforward network layer is passed, and then the output of the feedforward network layer and the input are added and normalized, which is a complete encoding layer; the Bev Queries model comprises six encoding layers. 5.The panorama image stitching method based on deformable attention mechanism according to claim 4, wherein: The method of generating the panoramic bird's eye view in the step S2 is specifically: after passing through 6 encoding layers, using multi-layer transposed convolution, converting the high-dimensional small-size Bev Queries into 3-channel BEV images. 6.The panorama image stitching method based on deformable attention mechanism according to claim 5, wherein: In the step S3, the calculation formula of the MS-SSIM Loss for learning the graph structure is as follows: ； where m represents different scales, , denote the mean of the predicted image and ground truth, respectively, , denote the standard deviation between the predicted image and ground truth, respectively, denote the covariance of the predicted image and ground truth, , denote the relative importance between the two terms, and are constants.

Images