Infrared image real-time splicing method

Abstract

The application discloses an infrared image real-time splicing method, which comprises the following steps: S100, adopting a lightweight feature matching method based on deep learning to register infrared images; and S200, adopting an adaptive weighted fusion method to fuse the registered images. The method has the characteristics of short delay, good splicing quality and real-time splicing support, and can be widely applied in the fields of surround view detection, panoramic exploration, intelligent driving and the like.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision, panorama fusion and artificial intelligence, and particularly relates to an infrared image real-time splicing method. BACKGROUND

[0002] With the breakthrough of computer vision theory and the rapid development of graphical hardware, visual technology based on deep learning is increasingly applied to daily life. Among them, the target detection algorithm can identify the position and category of a specific target in an image or video, and plays a decisive role in fields such as intelligent monitoring, traffic management, virtual reality, etc. However, due to the diversity of application scenarios, people's requirements for visual systems have become increasingly high, and traditional target detection algorithms have been unable to meet the effective detection of complex scenes.

[0003] Since the view angle of a single fixed camera is fixed, it cannot monitor the scene in all directions, so a wide-view panoramic image needs to be generated by looking around splicing to effectively detect the scene. In military applications, looking around panorama can effectively analyze the battlefield situation, for example, in the field of field reconnaissance, reconnaissance aircraft can use looking around panorama to construct a wide-view monitoring image, quickly and effectively detect and identify potential dangers, and take appropriate measures in a timely manner. Looking around splicing technology can also be applied to unmanned aerial vehicle aerial photography, making it easier for observers to detect the environment in all directions from the looking around image sequence and make corresponding decisions. Although existing visible light looking around splicing methods are relatively mature, they have limited effect on infrared images, and the splicing takes a long time, making it difficult to perform real-time splicing. SUMMARY

[0004] To solve the above technical problems, the present application discloses an infrared image real-time splicing method, which comprises the following steps:

[0005] S100: using a lightweight feature matching method based on deep learning to register the infrared images;

[0006] S200: using a self-adaptive weighted fusion method to fuse the registered images.

[0007] Through the above technical solution, the infrared image real-time splicing method has the characteristics of short delay, good splicing quality and support for real-time generation. This method is not only suitable for infrared image real-time splicing, but also suitable for visible light image real-time splicing, and can be widely applied to fields such as military reconnaissance, intelligent security, intelligent transportation, etc. This method can realize real-time and stable infrared looking around splicing on a mobile terminal. BRIEF DESCRIPTION OF DRAWINGS

[0008] Figure 1 is a flowchart of an infrared image real-time splicing method provided in an embodiment of the present application;

[0009] Figure 2 is a structure diagram of an infrared image real-time splicing method provided in an embodiment of the present disclosure;

[0010] Figure 3 is a deep separable convolution diagram provided in an embodiment of the present disclosure;

[0011] Figure 4 is a self-attention module diagram provided in an embodiment of the present disclosure;

[0012] Figure 5 is a weighted average fusion algorithm diagram provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0013] In order for those skilled in the art to understand the technical solutions disclosed in the present application, the following will combine embodiments and related drawings to describe the technical solutions of the embodiments. Figures 1 to 5 The technical solutions of the embodiments are described, and the described embodiments are part of the embodiments of the present application, not all embodiments.

[0014] In this paper, the reference to "embodiments" means that the specific features, structures or characteristics described in conjunction with the embodiments can be included in at least one embodiment of the present application. The phrase appears at various places in the specification does not necessarily refer to the same embodiment, nor is it independent or alternative to other embodiments. Those skilled in the art can understand that the embodiments described herein can be combined with other embodiments.

[0015] Referring to Figure 1 In one embodiment, the present application discloses an infrared image real-time splicing method, which comprises the following steps:

[0016] S100: using a lightweight feature matching method based on deep learning to register infrared images;

[0017] S200: using an adaptive weighted fusion method to fuse the registered images.

[0018] In this embodiment, the present method obtains accurate matching results of infrared images by using a lightweight feature matching algorithm based on deep learning, and then fuses the matched images by using an adaptive weighted fusion method, so as to finally obtain a spliced infrared all-around panoramic image.

[0019] The structure diagram of the infrared image real-time splicing method is as shown in Figure 2The method is shown. The lightweight infrared feature matching algorithm based on deep learning proposed by the method is composed of two stages of feature extraction and feature matching. The input data of the algorithm is an infrared image pair to be registered. First, the image is extracted by the feature extraction network, then the self-attention mechanism module is designed to enhance the ability of capturing long-distance information of the features, and global information is introduced into the local features. Then, the level-3 features are searched by element nearest neighbor search to obtain the feature matching point pair and the coarse-grained transformation matrix H0. According to the feature point pair in level-3, the level-2 to level-0 feature point receptive field is constructed step by step, and the coarse-grained transformation matrix is fine-tuned in the receptive field to obtain the feature point matching pair and the fine-grained transformation matrix H1 of the input image pair. Finally, according to the matching result, the omnidirectional splicing image is obtained through adaptive weighted fusion.

[0020] Specifically, for input images A and B, first, the down-sampled 1 / 2, 1 / 4, 1 / 8 and 1 / 16 features are extracted by the feature extraction network, respectively, and are denoted as level-0, level-1, level-2 and level-3; then, element-wise nearest neighbor search is performed in the level-3 features to obtain a coarse matching point set M3; then, according to the coarse matching point set M3, a homography matrix H0 is calculated by the random sample consensus algorithm, and the matrix is denoted as a coarse matching matrix; in the coarse matching point set, each matching point is up-sampled to obtain a matching point set M2 in the level-2 feature layer, and the same method is used to obtain a matching point set M0 in the final level-0 feature layer. The point set is regarded as a fine matching result, and a homography matrix is calculated according to the matching result to obtain a fine-grained transformation matrix H1. Through homography mapping, images A and B are mapped to the same plane, and finally an omnidirectional splicing image is obtained through adaptive weighted fusion.

[0021] Compared with traditional feature matching algorithms, considering the characteristics of blurred edges and weak texture of infrared images, the method does not select hand-crafted features such as SIFT and ORB to match images, but directly performs dense nearest neighbor search in the down-sampled feature map to obtain more matching feature pairs. Compared with deep learning-based feature matching algorithms, considering the deployment and real-time requirements of the algorithm, the method uses a single-stage matching, and uses a traditional nearest neighbor search algorithm in the matching stage to obtain a feature point receptive field by directly matching features element by element, and optimizes the matching result from coarse to fine to obtain a geometric transformation relationship matrix. Since the algorithm only needs to extract features through a neural network, it reduces the deployment difficulty compared with other network-based matching algorithms and ensures that the overall speed of the algorithm is fast.

[0022] In another embodiment, the step S100 further comprises:

[0023] S101: feature extraction is performed on the infrared image to obtain a feature map;

[0024] S102: coarse matching results between infrared images are obtained through dense nearest neighbor search of the obtained feature map;

[0025] S103: the coarse matching results are optimized step by step upwards through the obtained feature map and the coarse matching results to obtain final accurate matching results.

[0026] For this embodiment, the overall process of feature extraction is as follows: the input infrared image is 640x480, after equal scaling, the long side of the input will be scaled to 224, and the short side will be scaled to 168, and then the short side will be completed to 224 using pure black pixels. Therefore, the image input to the network is 224x224, after four-stage feature extraction, feature maps down-sampled by 2, 4, 8 and 16 times are obtained, respectively, and the sizes are 112x112, 56x56, 28x28 and 14x14. Each pixel point on each layer of feature map corresponds to a block region of 2x2, 4x4, 8x8 and 16x16 on the original image, respectively, and finally the 16x16 features are enhanced through the self-attention mechanism.

[0027] After obtaining features of different sizes, the next step is to match the features.

[0028] The core of the feature matching module can be divided into two stages. The first stage is the dense nearest neighbor search stage, which uses the feature map down-sampled by 1 / 16 for coarse-grained matching to obtain coarse matching feature point pairs and a feature matching matrix H0; the second stage is the hierarchical refinement stage, which uses the feature maps of 1 / 8, 1 / 4 and 1 / 2 in the feature pyramid to gradually adjust the feature matching points upwards, and finally obtains fine feature point pairs and a feature matching matrix H1.

[0029] Although deep features have strong robustness to geometric changes and illumination changes, too much pixel-level information is lost, and direct up-sampling to the original resolution will result in poor matching performance. On the other hand, since dense nearest neighbor search is only performed in the feature layer down-sampled by 1 / 16, the time consumption of the dense nearest neighbor search algorithm is small at this time, but as the feature layer is gradually up-sampled, the feature layer becomes larger and larger, and the time required for using the dense nearest neighbor search increases by several times, so the coarse matching results need to be further optimized.

[0030] To solve the above optimization problem, the method adopts a refinement method from coarse to fine to optimize the coarse-grained feature matching matrix. Specifically, first, the feature acceptance domain is determined according to the position of the feature matching point of the last layer, the search area is limited by the acceptance domain, and then the nearest neighbor search is performed in the acceptance domain, and then the feature matching matrix obtained by the coarse matching is optimized by the hierarchical refinement method, and finally the refined feature matching point pair and the feature matching matrix H1 are obtained.

[0031] In another embodiment, the step S101 further comprises:

[0032] S1011: The input infrared image is subjected to a backbone network composed of a stack of lightweight convolutional layers for feature extraction;

[0033] S1012: The extracted features are enhanced by a self-attention module and output corresponding feature maps.

[0034] In another embodiment, the lightweight convolutional layer is a depth separable convolution, which divides the one-time convolution into two parts: depth convolution and pointwise convolution.

[0035] In this embodiment, the feature extraction module is responsible for feature extraction of the input infrared image, so that the feature matching module can process the features at different levels. The module is composed of convolution stacks of different scales, combined with a self-attention structure. In order to reduce the overall computational complexity of the model, the standard convolutional layer of the module is replaced by a depth separable convolution. The depth separable convolution layer is as shown in Figure 3 Specifically, the depth separable convolution divides the one-time convolution into two parts: depth convolution and pointwise convolution. For an input feature with M channels, first, the depth convolution is used to process each channel of the feature with an n x n convolution kernel, and finally, M convolution kernels are used to obtain a feature with M channels. Since the depth convolution calculates each channel independently, it cuts off the implicit information interaction between different channels, reducing the non-linear transformation ability of the network, so a 1 x 1 pointwise convolution is added after the depth convolution to integrate the channel information and obtain the final N-channel output feature.

[0036] Assuming that the size of the input feature is H x W x M, and the size of the convolution kernel is K x K, for N convolution kernels, the calculation amount of the standard convolution is H x W x M x K x K x N; and for the depth separable convolution, under the same scale, the calculation amount of the depth convolution is H x W x M x K x K, and the calculation amount of the pointwise convolution is H x W x M x N. The calculation amount ratio of the standard convolution and the depth separable convolution is as shown in the following formula,

[0037]

[0038] Compared with the standard convolution, the depth separable convolution can greatly reduce the calculation amount, and the network is significantly improved in lightweight.

[0039] In another embodiment, the self-attention module is a Transformer-based unfolding-warping-aggregation structure.

[0040] For this embodiment, since the convolutional network usually focuses on extracting semantic information or pixel information from a local area, it is difficult to model the representation learning from a global perspective. Inspired by the MobileVit network, the method introduces a self-attention mechanism layer in the feature map down-sampled by 1 / 16 to learn global information. Through the learning of global features, more global information can be used in the feature point coarse matching, so that the overall matching effect is better. The self-attention module is as shown in Figure 4 . Specifically, assuming that the feature size after 1 / 16 down-sampling is (H, W, C), first, the local feature is modeled through the depth separable convolution (DW Conv), and then the global feature is modeled through the Unfold-Transformer-Fold structure. Specifically, in the Unfold stage, the feature map is divided according to the Patch, and the Patch is set to 2 in this method, i.e., each Patch is composed of 4 pixels, to obtain the changed feature (H x W / patch 2 , patch 2 , C), then the transformed feature map is input into the multi-head attention module (MultiHeadAttention) for calculation. When attention is performed, the pixels in each Patch are calculated with the pixels at the corresponding positions of other Patches. Then, the multi-layer perceptron (MLP) is used to fuse the global semantic information for each pixel in the feature map, and the Fold operation is used to transform the feature size to (H, W, C) again. Then, a depth separable convolution (DW Conv) and a residual branch are used to obtain the final output feature.

[0041] In another embodiment, the step S102 further includes: performing coarse-grained matching with the feature map down-sampled by 1 / 16 to obtain coarse matching feature point pairs and a coarse-grained feature matching matrix H0.

[0042] In another embodiment, the specific way of the dense nearest neighbor search is: given the feature maps F A and F B , the dense nearest neighbor search algorithm finds the nearest neighbor match of each element in the feature layer through the absolute distance between two points. For the point p A in the feature map F 4 , if the best matching point p Bthe distance between the sub-optimal matching point and point p A is lower than a given Lowe matching threshold, then point p B is matched; likewise, when the corresponding point p B is also matched with p 4 , then point p A is matched with point p B As a result, the matching point pairs are determined and returned, and finally a coarse-grained feature matching matrix H0 is constructed according to the re-projection error of the determined matching point pairs.

[0043] In another embodiment, the step S103 further comprises the following steps:

[0044] S1031: determining a feature receptive field according to the positions of the matching point pairs of the previous layer, wherein the feature receptive field is composed of four points, i.e., the upper, lower, left and right points of the matching point pairs;

[0045] S1032: performing a dense nearest neighbor search in the feature receptive field point set;

[0046] S1033: optimizing the coarse-grained feature matching matrix H0 according to the result of the dense nearest neighbor search in the feature receptive field point set.

[0047] For this embodiment, the matching point pairs of the nth layer and the feature map of the (n-1)th layer The hierarchical refinement algorithm first constructs a receptive field point set for each point in the matching point pairs of the nth layer Specifically, the points in are up-sampled by 2 times, and the right, lower and lower-right points are taken as the receptive field, and a nearest neighbor search is performed in the receptive field range on the feature map of the (n-1)th layer to finally obtain the matching point set of the (n-1)th layer.

[0048] After the registration between different infrared images is completed, the infrared images can be mapped to the same plane through geometric relationship matrix mapping. However, due to the differences in shooting angles, exposure times and other aspects, there will be obvious splicing gaps in the overlapping area of the images converted to the same plane, so the images need to be fused.

[0049] In another embodiment, the step S200 further comprises: calculating the pixel value corresponding to the fusion area by weighted average fusion for the registered images, and obtaining the fusion result according to the pixel value.

[0050] In another embodiment, the specific calculation method of the pixel value is as follows:

[0051] f(x, y) = wf1(x, y) + (1-w)f2(x, y)

[0052] Where f1(x,y) and f2(x,y) are the registered images, f(x,y) is the weighted average fused pixel value of the region to be fused between x1 and x2, x1 and x2 represent the start and end positions of the image fusion region, respectively; w represents the weighting coefficient.

[0053] In this embodiment, a specific algorithm diagram is shown below. Figure 5 As shown in the formula. x1 < x < x2. This method has a good fusion effect and fast fusion speed in areas with inconsistent exposure, which can meet the real-time requirements of the infrared surround view stitching algorithm designed in this chapter.

[0054] Finally, it should be noted that those skilled in the art can make many variations based on the guidance of this specification and without departing from the scope of protection of the claims of this invention, and these variations are all within the scope of protection of this invention.

Claims

1. A method for real-time infrared image stitching, the method comprising the following steps: S100: A lightweight feature matching method based on deep learning is used to register infrared images; S200: An adaptive weighted fusion method is used to fuse the registered images; in, Step S100 further includes: S101: Extract features from the infrared image to obtain a feature map; S102: Obtain coarse matching results between infrared images by performing dense nearest neighbor search on the obtained feature map; S103: Using the obtained feature map and the coarse matching result, the coarse matching result is optimized step by step upwards to obtain the final accurate matching result; Step S101 further includes: S1011: The input infrared image is processed through a backbone network consisting of stacked lightweight convolutional layers for feature extraction. S1012: The extracted features are enhanced by the self-attention module and the corresponding feature map is output. Step S102 further includes: performing coarse-grained matching using a feature map downsampled by 1 / 16, to obtain coarse-matched feature point pairs and a coarse-grained feature matching matrix. ; The input data for the lightweight infrared feature matching method based on deep learning consists of infrared image pairs to be registered. For input images A and B, a feature extraction network first extracts features downsampled to 1 / 2, 1 / 4, 1 / 8, and 1 / 16, respectively, denoted as level-0, level-1, level-2, and level-3. Based on the feature point pairs in level-3, feature point receptive fields from level-2 to level-0 are constructed progressively upwards. Within these receptive fields, the coarse-grained transformation matrix is ​​fine-tuned to obtain the feature point matching pairs and the fine-grained transformation matrix for the input image pairs. ; The self-attention module is based on the Transformer's unfold-deform-aggregate structure. In the Unfold stage, the feature map is divided according to the Patch. Then, the transformed feature map is fed into the multi-head attention module for calculation. Then, the global semantic information is fused for each pixel in the feature map through a multilayer perceptron. The feature size is transformed back to the feature size after 1 / 16 downsampling through the Fold operation. Finally, the output feature is obtained by a depthwise separable convolution and a residual branch.

2. The method as described in claim 1, wherein the lightweight convolutional layer is a depthwise separable convolution, and the depthwise separable convolution divides a single convolution into two parts: a depthwise convolution and a pointwise convolution.

3. The method as described in claim 1, wherein the dense nearest neighbor search is performed as follows: given a feature map and Dense nearest neighbor search (DNN) algorithms find the nearest neighbor match for each element in the feature layer by using the absolute distance between two points. For a feature map... Points in If the best matching point If the ratio of the distance to the suboptimal matching point is lower than a given Lowe matching threshold, then the point... With point Matching; similarly, when corresponding points also with When matching, the point will be... With point The determined matching point pairs are returned, and the reprojection error is calculated based on the determined matching point pairs to construct a coarse-grained feature matching matrix. .

4. The method of claim 1, wherein step S103 further comprises the following steps: S1031: The feature receptive field is determined based on the position of the feature matching point pair in the previous layer. The feature receptive field is the four points above, below, left, and right of the feature matching point pair. S1032: Perform dense nearest neighbor search within the feature receptive domain point set; S1033: Based on the dense nearest neighbor search results in the feature receptive domain point set, perform coarse-grained feature matching matrix... Optimize.

5. The method of claim 1, wherein step S200 further comprises: For the registered image, the pixel values ​​corresponding to the fusion region are calculated by weighted average fusion, and the fusion result is obtained based on the pixel values.

6. The method as described in claim 5, wherein the pixel value is specifically calculated as follows: ， in, and For the registered image, for and The weighted average fused pixel values ​​of the regions to be fused between them. and These represent the start and end positions of the image fusion region, respectively. This represents the weighting coefficient.

Images