Aerial image splicing method based on human vision system

Abstract

The invention discloses an aerial image splicing method based on a human vision system, and the method comprises the steps: designing a new registration network in a registration stage, and fusing a replacement attention module into an accurate and lightweight key point detection and descriptor extraction (ALIKE) network, so that the network can better capture the space and channel information, and improves the registration performance; in the fusion stage, an energy function containing color, structure and saliency difference is designed based on a human vision system. The method has the advantages that the color difference between aerial images can be effectively eliminated, artifacts can be reduced, and the method is more in line with human impressions.

Description

Technical Field

[0001] The present invention relates to the technical field of image stitching, and in particular to an aerial image stitching method based on the human visual system. Background Art

[0002] Image stitching refers to the technology of obtaining a high-definition image with a larger viewing angle by processing a group of partially overlapping images through image registration and image fusion. Among them, aerial image stitching is widely used in urban construction planning, disaster exploration and environmental monitoring due to its low cost and high flexibility.

[0003] Due to the special way of drone aerial photography and the complexity of the environment, there are still many challenges. For aerial images with weak textures, an improved VGG16 (Visual Geometry Group 16-layerNetwork) twin network drone remote sensing image stitching model is proposed in the prior art. It uses the first 13 layers of the VGG16 network and introduces an improved spatial attention module to effectively extract the features of the overlapping areas of the image;

[0004] In addition, a complete unsupervised deep learning aerial image stitching framework is proposed to address the problem of poor robustness in complex aerial scenes, which consists of an unsupervised deep homography estimation network and an unsupervised image fusion network;

[0005] Considering the stitching of images with complex textures and continuous depth changes, additional depth information is introduced into the energy function to guide the seams to pass through continuous areas with high similarity in the image, thereby obtaining high-quality stitching results;

[0006] As for the problems of ghosting and distortion that may occur in aerial image stitching algorithms when processing complex images with parallax, some people have proposed an aerial image stitching method based on straight line feature constrained grid deformation, which can reduce the problems of ghosting in overlapping areas and distortion in non-overlapping areas.

[0007] Although the above method can solve the problems of ghosting and distortion in aerial image stitching, it ignores the perception of the human eye on images. Summary of the invention

[0008] The technical problem to be solved by the present invention is to overcome the above technical defects and propose a UAV aerial image stitching strategy based on deep learning and optimal seams.

[0009] To solve the above technical problems, the technical solution provided by the present invention is: a method for stitching aerial images based on the human visual system, comprising the following steps: Step 1: improving the ALIKE network to build a deep learning model to extract key points to complete image registration; Step 2: designing an energy function based on the human visual system; Step 3: selecting the optimal seam to complete image fusion and obtain a stitched image that conforms to human perception.

[0010] Preferably, the improved ALIKE network includes a feature encoder, a feature aggregation module, a feature extraction head, and a differentiable key point detection and description module;

[0011] The feature encoder extracts input image features to generate a feature map, the feature aggregation module includes aggregating multi-level features from the encoder, the feature extraction head includes performing score map processing, and the differentiable key point detection and description module includes using a DKD module to extract key points from the score map and then extracting descriptors corresponding to the key points from a dense descriptor map.

[0012] Preferably, the image registration further comprises the following steps:

[0013] S1.1: After constructing key points and descriptors, use the K nearest neighbor algorithm to perform rough matching between images;

[0014] S1.2: Use the random sampling consistency algorithm to screen the feature points of the rough match, eliminate the wrong matches, estimate the homography matrix based on the final feature points, and complete the image registration.

[0015] Preferably, the energy function includes constraining the seam in terms of color, structure, and visual significance, and the energy function is:

[0016] E=E color +E structrue +E VS .

[0017] Preferably, the color constraint includes detecting the overlapping area of ​​the two registered images, constructing a difference matrix to represent the similarity of the overlapping area, and calculating the color difference using the LAB color space:

[0018]

[0019]

[0020] ΔA=4×(O G -O′ G );

[0021]

[0022]

[0023] Where O and O′ are the overlapping areas of the two registered images, E color It's the color difference, R ,O G ,O B are the values ​​of the red, green, and blue channels of region O respectively. R ,O′ G ,O′ B They are the values ​​of the red, green and blue channels of area O′ respectively.

[0024] Preferably, the structural constraint includes constructing a structural difference metric using a high frequency portion of the overlapping region, applying Gaussian filtering to the overlapping region to obtain O1 and O2, calculating the structural difference between O1 and O2 by Gaussian difference edge detection, and detecting line segment information of the overlapping region using a line segment detector:

[0025]

[0026]

[0027] Where E structrue is the structural difference, σ2 and σ3 are the parameters of Gaussian difference edge detection.

[0028] Preferably, the visual saliency constraint comprises using the SDSP saliency detection method:

[0029] VS(X)=VS F (X)×VS D (X)×VS C (X);

[0030] E VS =VS(O)-VS(O′);

[0031] Where X is the different pixels in the input image pair, and the final saliency map VS combines the three saliency maps: VS F (X)VS D (X) and VS C (X);

[0032] VS F (X) is the frequency domain saliency map obtained by bandpass filtering modeling;

[0033] VS D (X) is the position-based saliency map;

[0034] VS C (X) is the saliency map based on color temperature.

[0035] Preferably, the selection of the optimal seam comprises using a dynamic seam line generation algorithm based on a threshold constraint, comprising the following steps:

[0036] S3.1: Set a threshold e to limit the difference of pixels on the stitching line to a range less than e;

[0037] S3.2: Under the minimum threshold e condition, the eight-connected region where the start point and the end point are located is recorded as R, and the search area of ​​the seam is limited to R;

[0038] S3.3: Each pixel in the search area R has a specific difference value;

[0039] S3.4: Starting from the starting point, expand in eight adjacent directions, update the pixel difference value in R to the minimum difference sum from the starting point to the point, and use the updated pixel as the new expansion point until it expands to the end point;

[0040] S3.5: Starting from the end point, follow the pixel path with the smallest difference and value and return to the starting point to obtain the optimal seam.

[0041] The advantages of the present invention compared with the prior art are: in the registration stage, a new registration network is designed, and the permutation attention module is integrated into the accurate and lightweight key point detection and descriptor extraction (ALIKE) network, so that the network can better capture spatial and channel information and improve the registration performance; in the fusion stage, an energy function containing color, structure and significance differences is designed based on the human visual system to avoid seams passing through images with straight structure and high significance areas, solve the artifact problem, and make the spliced ​​image more in line with human perception. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a schematic diagram of the network structure.

[0043] Figure 2 It is a permutation attention model.

[0044] Figure 3 This is a simple diagram for finding seams. DETAILED DESCRIPTION

[0045] The present invention is further described in detail below in conjunction with the accompanying drawings.

[0046] References to embodiments of the present invention will now be provided in detail, one or more examples of which are described below. Each example is provided as an explanation rather than a limitation of the present invention. In fact, it will be apparent to those skilled in the art that various modifications and variations may be made to the present invention without departing from the scope or spirit of the present invention. For example, a feature illustrated or described as part of one embodiment may be used in another embodiment to produce a further embodiment.

[0047] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those generally understood by those skilled in the art of the present invention. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more related listed items.

[0048] Combined with Figure 1-3 As shown:

[0049] Image Registration:

[0050] Image registration refers to establishing a data relationship between multiple pairs of images describing the same scene. Assume that the original image is I, the target image is I′, and p = [x, y] T and p′=[x′,y′] T are the matching points on the original image and the target image respectively. The matching point pairs between the original image and the target image can be mapped to the same coordinate system through the homography matrix H:

[0051]

[0052] where H∈R 3×3 .

[0053] The improved ALIKE network consists of four parts. The network structure is as follows: Figure 1 shown.

[0054] Feature encoder: used to extract input image features to generate feature maps. The encoder contains four modules. The first module contains two 3*3 convolutional layers with ReLu (Rectified Linear Unit) as the activation function. The feature map is compressed by the maximum pooling operation, and then the SA_ResNet (Residual Network with Shuffle Attention) module is used to further extract features. The original ResNet (Residual Network) module consists of 3 groups of convolutional layers + BN (BatchNormalization) + ReLu activation function, while the SA_ResNet module adds a permutation attention module before the last group of ReLu activation functions to aggregate the extracted features and pay more attention to features that are beneficial to registration. The permutation attention model is as follows: Figure 2 shown.

[0055] The permutation attention model groups feature maps and processes channel and spatial information separately. For channel information, a simple global average pooling is used to embed global information to generate channel statistics s∈R C2G×1×1 , can be achieved by compressing X in the spatial dimension k1 To calculate:

[0056]

[0057] Then it is linearly fused with the original feature map.

[0058] Feature aggregation module: Aggregate multi-level features from the encoder, first use a 1*1 convolution and bilinear upsampling to adjust the number of channels, and then simply splice them together.

[0059] Feature extraction head: Outputs a H*W*(dim+1) feature map, where the first dim channels are normalized by the L2 norm to become the descriptor map D, and the last channel is normalized by the Sigmoid activation function to become the score map S.

[0060] Differentiable keypoint detection and description: Use the DKD (Differentiable Keypoint Detection) module to extract keypoints from the score map, and then extract the descriptors corresponding to the keypoints from the dense descriptor map. In order to detect keypoints in the score map S, a widely used method is non-maximum suppression (NMS). Find the pixel with the highest score in the local window. This operation is equivalent to performing a maximum value (argmax) operation in the local N×N window.

[0061]

[0062] Where s(i,j) represents the score of the local position (i,j). In this formula, the output position is decoupled from the score map. This is a non-differentiable operation and cannot be back-propagated.

[0063] In order to couple the key points with the score map, a soft maximum (softargmax) is used to extract differentiable key points from the local window. First, the score map processed by NMS is obtained by suppressing the non-maximum score s = S (u, v) in the local N × N window:

[0064]

[0065] Use threshold th to filter low response scores, and then use s max Centered on the soft maximum, local soft coordinates are extracted Normalize the scores using the softmax function in an N×N window:

[0066]

[0067] Among them, t det is the weight coefficient for controlling sharpness. s′(i,j) is the position [i,j]T is the probability of the key point, and the expected position of the key point in the local window can be given by integral regression:

[0068]

[0069] Sub-pixel key points are constructed:

[0070]

[0071] in, represents the pixel-level key point position and is not differentiable. express It is associated with the score in the local N×N window, making it differentiable within this window, which allows the entire network to backpropagate to construct key points and descriptors.

[0072] After extracting key points through the improved ALIKE network, the K nearest neighbor algorithm is first used to perform coarse matching between images, and then the random sample consensus algorithm (RANSCA) is used to screen the feature points of the coarse matching and eliminate false matches. The homography matrix is ​​estimated based on the final feature points to complete the image registration.

[0073] Image Fusion:

[0074] After completing the image registration, the registered images are fused. Common fusion methods include weighted average fusion and optimal seam fusion. However, since aerial images are greatly affected by lighting, weighted average fusion will cause color differences in overlapping areas, and ghost images will appear when facing moving objects.

[0075] The best seam algorithm is used to search for a seam line with the minimum energy in the overlapping area of ​​the images to be stitched, and the images are merged according to the seam line, which can effectively avoid the ghosting phenomenon.

[0076] Design energy function based on human visual system:

[0077] First, detect the overlapping area of ​​the two registered images, denoted as O and O'. Then construct a difference matrix to represent the similarity of the overlapping area of ​​the two images. In order to better cater to the human eye's perception of color, use the LAB color space to calculate the color difference:

[0078]

[0079] ΔA=4×(O G -O′ G ) (11)

[0080]

[0081] Here, E color It's the color difference, R ,O G ,O B are the values ​​of the red, green, and blue channels of region O respectively. R ,O′ G ,O′ B are the values ​​of the red, green and blue channels of region O′ respectively;

[0082] In the process of stitching aerial images, edge discontinuities caused by registration errors can significantly reduce the quality of the stitched image. However, in low-frequency areas of the image, such as smooth texture areas such as roads and grass, slight registration deviations have relatively little impact on the stitching quality, so this application uses the high-frequency part of the overlapping area to construct a structural difference metric, while reducing the structural difference weight of the low-frequency part;

[0083] The aim is to guide the seam position to the low-frequency area of ​​the image. Gaussian filtering is used on the overlapping area to obtain O1 and O2, and then the structural difference between O1 and O2 is calculated by Gaussian difference edge detection. In addition, in order to make the difference of structural objects more obvious, line segment difference is introduced. The line segment detector (LSD) is used to detect the line segment information of the overlapping area to avoid the seam passing through the structure with straight lines. The structural difference function is shown as follows:

[0084]

[0085] Among them, E structrue is the structural difference, σ2 and σ3 are the parameters of Gaussian difference edge detection;

[0086] Pixels with high saliency usually have a more significant impact on the overall quality and visual perception of the image. Therefore, when determining the seam location, avoiding high saliency areas becomes an important consideration. Saliency analysis is integrated into the energy function, and a simple and efficient saliency detection method, SDSP, is used. The calculation formula of SDSP is:

[0087] VS(X)=VS F (X)×VS D (X)×VS C (X) (16)

[0088] E VS =VS(O)-VS(O′) (17)

[0089] Where X is the different pixels in the input image pair. The final saliency map VS combines the three saliency maps: VS F (X)VSD (X) and VS C (X). VS F (X) is the frequency domain saliency map obtained by bandpass filtering modeling, VS D (X) is the position-based saliency map, which reflects the phenomenon that the central area of ​​the image usually attracts more attention than the edge area. C (X) is a color temperature-based saliency map that exploits the visual property that warm colors are usually more eye-catching than cool colors.

[0090] By introducing the significant difference metric, we can more effectively identify visually significant areas and guide the seams to avoid these areas, making the final stitching result more natural and visually coherent.

[0091] Taking the three energy functions into consideration, the total energy function of the optimal seam is:

[0092] E=E color +E structrue +E VS (18)

[0093] In order to optimize the seam quality of image stitching, a dynamic seam line generation algorithm based on threshold constraints is used. A threshold e is set, and the difference of pixels on the seam line is limited to a range less than e. Under the minimum threshold e condition, the eight-connected region where the starting point and the end point are located is recorded as R, and the search area of ​​the seam is limited to R. Each pixel in the search area R has a specific difference value. Starting from the starting point, it is expanded in eight adjacent directions, and the pixel difference value in R is updated to the minimum difference sum from the starting point to this point, and the updated pixel is used as the new extension point until it is extended to the end point. Then, starting from the end point, follow the pixel path with the smallest difference sum value and return to the starting point to obtain the optimal seam. The core idea of ​​the algorithm is to improve the naturalness and continuity of the stitching by minimizing the pixel difference on the seam line. The schematic diagram of the algorithm is shown in the figure. Figure 3 shown.

[0094] exist Figure 3 In (a), red represents the starting point, green represents the end point, and the yellow area represents the search area R. Figure 3 In (b), the red color represents the determined path. After finding the optimal seam, the images are fused according to the seam to obtain the stitched image.

[0095] The contents not described in detail in this specification belong to the prior art known to professional and technical personnel in this field.

[0096] The present invention and its embodiments are described above, and such description is not restrictive. The drawings show only one embodiment of the present invention, and the actual structure is not limited thereto. In short, if ordinary technicians in the field are inspired by it, without departing from the purpose of the invention, they can design a structure and embodiment similar to the technical solution without creativity, which should belong to the protection scope of the present invention.

Claims

1. A method for stitching aerial images based on the human visual system, characterized in that: The following steps are involved: Step 1: Improve the ALIKE network to build a deep learning model to extract key points and complete image registration; Step 2: Design energy function based on human visual system; Step 3: Select the optimal seam to complete image fusion and obtain a stitched image that conforms to human perception.

2. The method for stitching aerial images based on the human visual system according to claim 1, characterized in that: The improved ALIKE network includes a feature encoder, a feature aggregation module, a feature extraction head, and a differentiable key point detection and description module; The feature encoder extracts input image features to generate a feature map, the feature aggregation module includes aggregating multi-level features from the encoder, the feature extraction head includes performing score map processing, and the differentiable key point detection and description module includes using a DKD module to extract key points from the score map and then extracting descriptors corresponding to the key points from a dense descriptor map.

3. The method for stitching aerial images based on the human visual system according to claim 2, characterized in that: The image registration further comprises the following steps: S1.1: After constructing key points and descriptors, use the K nearest neighbor algorithm to perform rough matching between images; S1.2: Use the random sampling consistency algorithm to screen the feature points of the rough match, eliminate the wrong matches, estimate the homography matrix based on the final feature points, and complete the image registration.

4. The method for stitching aerial images based on the human visual system according to claim 1, characterized in that: The energy function includes constraints on the seams in terms of color, structure, and visual saliency. The energy function is: E=E color +E structrue +E VS 。 5. The method for stitching aerial images based on the human visual system according to claim 4, characterized in that: The color constraint includes detecting the overlapping area of ​​the two registered images, constructing a difference matrix to represent the similarity of the overlapping area, and calculating the color difference using the LAB color space: ΔA=4×(O G -O′ G ); Where O and O′ are the overlapping areas of the two registered images, E color It's the color difference, R ,O G ,O B are the values ​​of the red, green, and blue channels of region O respectively. R ,O′ G ,O′ B They are the values ​​of the red, green and blue channels of area O′ respectively.

6. The method for stitching aerial images based on the human visual system according to claim 4, characterized in that: The structural constraint includes constructing a structural difference metric using the high-frequency part of the overlapping area, applying Gaussian filtering to the overlapping area to obtain O1 and O2, calculating the structural difference between O1 and O2 by Gaussian difference edge detection, and using a line segment detector to detect the line segment information of the overlapping area: Where E structrue is the structural difference, σ2 and σ3 are the parameters of Gaussian difference edge detection.

7. The method for stitching aerial images based on the human visual system according to claim 4, characterized in that: The visual saliency constraints include using the SDSP saliency detection method: VS(X)=VS F (X)×VS D (X)×VS C (X); E VS =VS(O)-VS(O′); Where X is the different pixels in the input image pair, and the final saliency map VS combines the three saliency maps: VS F (X)VS D (X) and VS C (X); VS F (X) is the frequency domain saliency map obtained by bandpass filtering modeling; VS D (X) is the position-based saliency map; VS C (X) is the saliency map based on color temperature.

8. The method for stitching aerial images based on the human visual system according to any one of claims 1 to 7, characterized in that: The selection of the optimal seam includes using a dynamic seam line generation algorithm based on a threshold constraint, including the following steps: S3.1: Set a threshold e to limit the difference of pixels on the stitching line to a range less than e; S3.2: Under the minimum threshold e condition, the eight-connected region where the start point and the end point are located is recorded as R, and the search area of ​​the seam is limited to R; S3.3: Each pixel in the search area R has a specific difference value; S3.4: Starting from the starting point, expand in eight adjacent directions, update the pixel difference value in R to the minimum difference sum from the starting point to the point, and use the updated pixel as the new expansion point until it expands to the end point; S3.5: Starting from the end point, follow the pixel path with the smallest difference and value and return to the starting point to obtain the optimal seam.

Images