A high-resolution stitching method for multiple lane images based on unsupervised deep learning

Abstract

The present invention relates to the field of intelligent mining technology, and in particular to a method for high-resolution stitching of multiple images in a tunnel based on unsupervised deep learning, comprising: collecting an image sequence in a tunnel, wherein the images in the image sequence have different acquisition viewing angles and adjacent images have overlapping areas; stitching adjacent image sequences using an unsupervised image stitching method to obtain a high-resolution seamless stitching image; stitching the seamless stitching image using a key point matching image stitching method to obtain a panoramic image; and performing projection correction and rectangularization processing on the panoramic image to complete high-resolution stitching of multiple images in the tunnel. Aiming at the limitations of deep learning methods in processing high-resolution stitching of multiple images, the present invention adopts a traditional key point matching stitching algorithm for panoramic stitching; and by introducing multi-image sequence matching, projection correction technology, and image rectangularization processing, a better image foundation is provided for subsequent real-time video intelligent analysis and processing.

Description

Technical Field

[0001] The present invention relates to the field of intelligent mining technology, and in particular to a tunnel multi-image high-resolution stitching method based on unsupervised deep learning. Background Art

[0002] The illumination of mine tunnels is uneven, filled with dust and mist, and the imaging conditions are poor. Due to the limitation of camera focal length, a single fixed-angle camera is difficult to obtain panoramic images of important scenes in the tunnel. To solve this problem, multiple cameras are needed to obtain images with overlapping areas from different angles, and these images are seamlessly stitched using image stitching technology to form a wide panoramic image.

[0003] Traditional stitching methods rely on the quality of feature detection and do not perform well in lane environments. Stitching methods based on deep learning have significant advantages in feature extraction, but have limitations in processing multi-image high-resolution stitching. Currently, the research and application of multi-image high-resolution stitching based on deep learning is relatively limited, especially in complex lane conditions. Therefore, there is an urgent need for a lane multi-image high-resolution stitching method based on unsupervised deep learning. Summary of the invention

[0004] The purpose of the present invention is to provide a method for high-resolution stitching of multiple images in a lane based on unsupervised deep learning. Aiming at the limitations of deep learning methods in processing high-resolution stitching of multiple images, a panoramic stitching algorithm based on traditional key point matching is adopted; and by introducing multi-image sequence matching, projection correction technology and image rectangularization processing, a better image foundation is provided for subsequent real-time video intelligent analysis and processing.

[0005] To achieve the above object, the present invention provides the following solutions:

[0006] A high-resolution stitching method for multiple lane images based on unsupervised deep learning, comprising:

[0007] Acquiring an image sequence in the lane, wherein the images in the image sequence are acquired from different viewing angles and adjacent images have overlapping areas;

[0008] Adopting an unsupervised image stitching method to stitch adjacent image sequences to obtain a high-resolution seamless stitching image;

[0009] Using a key point matching image stitching method to stitch the seamless stitching images to obtain a panoramic image;

[0010] Projection correction and rectangularization are performed on the panoramic image to complete high-resolution stitching of multiple images of the lane.

[0011] Optionally, collecting an image sequence in the lane includes:

[0012] Arranging a camera device in the lane, and collecting images through the camera device;

[0013] Based on the arrangement position of the camera device, the images are combined into an image sequence in combination with an image sequence matching algorithm, wherein the image sequence matching algorithm includes a brute force matching algorithm and a K-nearest neighbor matching algorithm.

[0014] Optionally, combining the images into an image sequence based on the arrangement positions of the camera devices and in combination with an image sequence matching algorithm comprises:

[0015] Extracting features from the image based on scale-invariant feature transformation to obtain feature points;

[0016] The feature points are processed using the brute force matching algorithm and the K-nearest neighbor matching algorithm to obtain the nearest neighbor feature points and the next nearest neighbor feature points of each feature point, and to construct a data set;

[0017] The data set is processed by eliminating outliers through a random sampling consensus algorithm to obtain the image sequence.

[0018] Optionally, the unsupervised image stitching method comprises: an unsupervised image coarse alignment algorithm and an unsupervised image reconstruction algorithm, wherein the unsupervised image coarse alignment algorithm performs image enhancement and feature extraction on adjacent image sequences and then inputs the features into a regression network to perform coarse alignment on adjacent image sequences to obtain a coarsely aligned image;

[0019] The unsupervised image reconstruction algorithm obtains a high-resolution seamless stitching image by inputting the coarsely aligned image into a low-resolution deformation branch and a high-resolution refinement branch in sequence, wherein the low-resolution deformation branch is used to receive the coarsely aligned image and ensure the integrity of the coarsely aligned image, and the high-resolution refinement branch is used to process the coarsely aligned image through content mask constraints and seam mask constraints to output a high-resolution seamless stitching image.

[0020] Optionally, stitching the seamless stitching images using a key point matching image stitching method to obtain a panoramic image includes:

[0021] Adjusting the seamless stitching image to a medium resolution and a low resolution respectively to obtain a medium resolution image and a low resolution image;

[0022] Performing feature extraction, feature matching, and feature point purification on the medium-resolution image to obtain estimated camera parameters, and generating transformation parameters based on the camera parameters;

[0023] Performing image transformation on the low-resolution image based on the transformation parameters to obtain estimated exposure parameters and a first transformed image;

[0024] The seamless stitching image is transformed based on the transformation parameters to obtain a second transformed image, the exposure error of the second transformed image is compensated in combination with the exposure parameters, and then the first transformed image and the second transformed image are mixed to obtain the panoramic image.

[0025] Optionally, performing projection correction on the panoramic image includes:

[0026] Based on the camera parameters, the camera intrinsic parameters and the rotation matrix are obtained, and the world coordinates corresponding to the pixel points in the panoramic image are calculated;

[0027] The two-dimensional coordinates after projection transformation are calculated by using the world coordinates to obtain a cylindrical projection image.

[0028] Optionally, a method for calculating the world coordinates corresponding to the pixel points in the panoramic image is:

[0029]

[0030] Among them, (X, Y, Z) is the world coordinate corresponding to the pixel point, R is the rotation matrix, K is the camera intrinsic parameter, and (x, y) is the two-dimensional coordinate of the pixel point;

[0031] The method for calculating the two-dimensional coordinates after projection transformation through the world coordinates is:

[0032]

[0033] Among them, (u, v) is the two-dimensional coordinate after cylindrical projection, and s is the transformation scale of the spliced ​​image.

[0034] Optionally, performing rectangular processing on the panoramic image includes:

[0035] Extending the frame of the cylindrical projection image to obtain a frame image;

[0036] The pixel points whose pixel values ​​in the border image are greater than a preset value are set to white, and the pixel points whose pixel values ​​are less than the preset value are set to black, to generate a first mask;

[0037] Calculate the minimum rectangular area that accommodates the first mask to obtain the second mask;

[0038] Cyclic erosion-subtraction operations are performed on the second mask and the first mask until the pixels are subtracted to zero, thereby obtaining a third mask;

[0039] The third mask is spliced ​​with the cylindrical projection image to obtain a rectangular panoramic image.

[0040] The beneficial effects of the present invention are:

[0041] The present invention provides a tunnel multi-image high-resolution stitching method based on unsupervised deep learning, which is more suitable for mine tunnel scenes than traditional stitching methods. In view of the limitations of deep learning methods in processing multi-image high-resolution stitching, panoramic stitching is performed based on traditional key point matching stitching algorithm. By introducing multi-image sequence matching, the method is applicable to fixed and moving image acquisition. At the same time, projection correction technology and image rectangularization processing are introduced to provide a better image foundation for subsequent real-time video intelligent analysis and processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0043] Figure 1 A flow chart of a method for high-resolution stitching of multiple lane images based on unsupervised deep learning according to an embodiment of the present invention;

[0044] Figure 2 A schematic diagram of the arrangement of laneway cameras according to an embodiment of the present invention;

[0045] Figure 3 A schematic diagram of scale space and extreme point precise positioning according to an embodiment of the present invention;

[0046] Figure 4 A schematic diagram of coordinates moving to the main direction of a feature point according to an embodiment of the present invention;

[0047] Figure 5 A schematic diagram of a specific process of brute force matching and K-nearest neighbor matching in an embodiment of the present invention;

[0048] Figure 6 Schematic diagram of the RANSAC fitting process of an embodiment of the present invention;

[0049] Figure 7 This is a structural diagram of the unsupervised coarse alignment algorithm framework of an embodiment of the present invention;

[0050] Figure 8 This is a structural diagram of the unsupervised image reconstruction algorithm framework of an embodiment of the present invention;

[0051] Fig. 9 This is a stitching image splicing flow chart of an embodiment of the present invention;

[0052] Fig.10 A schematic diagram of cylindrical projection correction according to an embodiment of the present invention;

[0053] Fig.11 The figure is a flowchart of image rectangularization according to an embodiment of the present invention. DETAILED DESCRIPTION

[0054] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0055] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0056] This embodiment provides a method for high-resolution stitching of multiple lane images based on unsupervised deep learning. Figure 1 As shown, including:

[0057] Step 1: Collect image sequences in the lane;

[0058] Arranging a camera device in the lane and collecting images through the camera device;

[0059] Based on the arrangement position of the camera device, the images are combined with an image sequence matching algorithm to form an image sequence, wherein the image sequence matching algorithm includes a brute force matching algorithm and a K-nearest neighbor matching algorithm, and the images in the image sequence are acquired from different perspectives and adjacent images have overlapping areas.

[0060] Specifically, based on the arrangement position of the camera device, combining the images into an image sequence in combination with an image sequence matching algorithm includes:

[0061] Extract features from the image based on scale-invariant feature transformation to obtain feature points;

[0062] The brute force matching algorithm and K-nearest neighbor matching algorithm are used to process the feature points, obtain the nearest neighbor feature points and the next nearest neighbor feature points of each feature point, and construct the data set;

[0063] The data set is processed to remove outliers through a random sampling consensus algorithm to obtain an image sequence.

[0064] Step 2: stitching adjacent image sequences to obtain a high-resolution seamless stitching image;

[0065] Adjacent image sequences are stitched together using an unsupervised image stitching method to obtain a high-resolution seamless stitching image, wherein the unsupervised image stitching method includes an unsupervised image coarse alignment algorithm and an unsupervised image reconstruction algorithm. The unsupervised image coarse alignment algorithm performs image enhancement and feature extraction on adjacent image sequences and then inputs the features into a regression network to perform coarse alignment on adjacent image sequences to obtain a coarsely aligned image.

[0066] The unsupervised image reconstruction algorithm obtains a high-resolution seamless stitching image by inputting the coarsely aligned image into a low-resolution deformation branch and a high-resolution refinement branch in sequence. The low-resolution deformation branch is used to receive the coarsely aligned image and ensure the integrity of the coarsely aligned image, and the high-resolution refinement branch is used to process the coarsely aligned image through content mask constraints and seam mask constraints to output a high-resolution seamless stitching image.

[0067] Step 3, stitching the seamless stitching images to obtain a panoramic image;

[0068] The seamless stitching images are adjusted to medium resolution and low resolution respectively to obtain medium resolution images and low resolution images;

[0069] Perform feature extraction, feature matching, and feature point purification on medium-resolution images to obtain estimated camera parameters and generate transformation parameters based on the camera parameters;

[0070] Performing image transformation on the low-resolution image based on the transformation parameters to obtain an estimated exposure parameter and a first transformed image;

[0071] The seamless stitching image is transformed based on the transformation parameters to obtain a second transformed image. After compensating for exposure errors in the second transformed image in combination with the exposure parameters, the first transformed image is mixed with the second transformed image to obtain a panoramic image.

[0072] Step 4: Perform projection correction and rectangularization on the panoramic image to complete high-resolution stitching of multiple images of the lane;

[0073] Specifically, the projection correction of the panoramic image includes:

[0074] Based on the camera parameters, the camera intrinsic parameters and rotation matrix are obtained to calculate the world coordinates corresponding to the pixel points in the panoramic image;

[0075] The two-dimensional coordinates after projection transformation are calculated through the world coordinates to obtain the cylindrical projection image.

[0076] Specifically, the rectangularization process of the panoramic image includes:

[0077] Extending the border of the cylindrical projection image to obtain a border image;

[0078] The pixel points whose pixel values ​​in the border image are greater than a preset value are set to white, and the pixel points whose pixel values ​​are less than the preset value are set to black, to generate a first mask;

[0079] Calculate the minimum rectangular area that accommodates the first mask and obtain the second mask;

[0080] The second mask and the first mask are cyclically eroded-subtracted until the pixels are subtracted to zero, thereby obtaining a third mask;

[0081] The third mask is stitched with the cylindrical projection image to obtain a rectangular panoramic image.

[0082] The following combination Figure 1-11 The technical solution proposed in this embodiment is specifically described, including the following steps:

[0083] Step 1: Collect image sequences for important scenes in the lane;

[0084] For important scenes in the lane, the camera collects image sequences with overlapping areas from different perspectives;

[0085] Through fixed cameras in the lanes, fixed acquisition is performed and image matching sequences are established based on the camera arrangement positions;

[0086] Laneways are usually Figure 2 The fixed camera arrangement is s1, s2, s3, s4, and the corresponding captured images are I1, I2, I3, I4. Then the image matching sequence can be established according to the camera position, that is, I1, I2, I3, I4.

[0087] Through the acquisition of images by mobile devices, an image sequence matching algorithm based on feature point matching is used to establish an image matching sequence.

[0088] The image sequence matching algorithm used in this embodiment is based on the extraction of scale-invariant feature transform (SIFT) features, and compares feature point descriptors through brute force matching (BFMatcher) to generate a matching list. Then, K-nearest neighbor matching (KnnMatch) is used to find the nearest neighbor and next nearest neighbor feature points for each feature point.

[0089] Step 1.1, SIFT feature extraction;

[0090] (1) Scale space construction;

[0091] The image is subjected to Gaussian filtering of different scales, that is, an image in each layer of the image pyramid is Gaussian blurred using different parameters, so that each layer of the pyramid contains multiple Gaussian blurred images. The multiple images in each layer of the pyramid are collectively called a group (Octave). There is only one group of images in each layer of the pyramid. The number of groups is equal to the number of pyramid layers, and each group contains multiple images. The first image of the last group of the Gaussian pyramid is 1 / 2 of the third to last image of the previous group, that is, the image size is 1 / 2 of the previous group. For example, if the input of this solution is a 512*512 image, and the top image of the pyramid is set to 2*2, then an 8-layer pyramid is required.

[0092] By performing differential processing on the images convolved with different convolution kernels in the same Octave of the Gaussian pyramid, a new set of Octaves will be obtained, thus obtaining a Gaussian difference pyramid, and finally obtaining the following Figure 3 The scale space shown.

[0093] (2) Accurate positioning of extreme points;

[0094] Taking the local extreme points in the Gaussian difference pyramid as feature points, finding the extreme points requires comparing each pixel with its adjacent points in the pixel domain and the scale domain. That is, in addition to considering the points in the x and y directions, we also need to consider the points in the direction of the smoothing factor σ, as shown below: Figure 3 shown.

[0095] (3) extreme point direction assignment;

[0096] The main direction of the feature point is obtained based on the gradient histogram statistics method, where the gradient amplitude and angle are calculated as:

[0097]

[0098]

[0099] Where I(x,y) is the grayscale image, (x,y) is the pixel coordinates, G(x,y) is the Gaussian kernel function, and L(x,y) is the image after Gaussian kernel convolution.

[0100] (4) Feature point description;

[0101] After the above processing, each feature point has three kinds of information (x, y, σ, θ): position (x, y), scale σ, and direction θ. Finally, a vector is constructed to describe the feature point information in the image. First, the image area required by the descriptor is calculated. The image area radius formula is as follows

[0102]

[0103] Among them, d = 4 represents the division into 4*4 sub-blocks. Then, the coordinates of the radius area where the feature point is located are moved to the main direction of the feature point, such as Figure 4 shown.

[0104] Finally, the feature point descriptor is generated. First, the 16*16 neighborhood is divided into 4 8*8 regions around the feature point. Then, the 8*8 region is divided into 4 regions, each of which is a 4*4 region. The gradient direction histogram is then counted for each 4*4 region, where the gradient direction histogram is 8 vectors, a total of 4*4*8=128 vectors. A 128-dimensional vector is generated, which is the feature descriptor of the point.

[0105] Step 1.2, brute force matching and K-nearest neighbor matching;

[0106] Brute Force Matcher (BFMatcher) compares feature point descriptors and generates a matching list. Then, K-Nearest Neighbor Match (KnnMatch) is used to find the nearest and next nearest neighbor feature points for each feature point. The specific process is as follows Figure 5 shown.

[0107] Step 1.3, RANSAC;

[0108] Finally, the random sample and consensus algorithm (RANSAC) is used to remove outliers and obtain the final matching sequence.

[0109] The basic idea of ​​the algorithm is to randomly find as few points as possible from all the observation points to fit the model, and then calculate the residuals between the model prediction value and the true value after fitting. When the difference is less than the given threshold, it is judged as an inlier; when it is above the threshold, it is judged as an outlier. Count the number of inliers, and then randomly select points to fit the model. If the number of inliers counted in the result of this fitting model is greater than that of the previous model, the old model will be iterated into a new model. Repeatedly extract data for estimation until the most suitable model is obtained. The fitting process is as follows: Figure 6 As shown:

[0110] a) The original data set for which the straight line model needs to be found, and b) a straight line obtained by estimating the model of the experimental data set through the RANSAC algorithm. From b), we can see that the result of the model estimation is to find a straight line model that best satisfies the data point set. That is, when the inner points are basically on the straight line and the outer points deviate seriously from the straight line, a more suitable model is found. The steps for deriving the number of iterations are as follows:

[0111] Assume that the proportion of the internal points in the value is t, that is, the calculation formula is:

[0112]

[0113] Each time the calculation model selects N points, the probability that the selected points have at least one external point is 1-t N Then, in the case of k iterations, the probability that the selected point has at least one external point is (1-t N ) k It can be concluded that the probability of sampling the correct N points and calculating the correct model is:

[0114] P=1-(1-t N ) k (5)

[0115] From the above formula, we can deduce:

[0116]

[0117] In the case of unknown value t, this result can be obtained by using the adaptive iteration method, that is, initially setting t to an infinite number of iterations, then calculating the current internal point ratio as t to estimate the number of iterations, and then iteratively updating the model parameters.

[0118] Step 2: Image stitching based on unsupervised deep learning;

[0119] The unsupervised image stitching method adopted in this embodiment includes two stages: unsupervised image coarse alignment and unsupervised image reconstruction.

[0120] Step 2.1, unsupervised image coarse alignment stage;

[0121] The algorithm framework of the unsupervised rough alignment stage is shown in Figure 7 , taking the complete image as input, the homography is estimated using a four-point parameterized homography matrix. Feature pyramids and feature correlations are introduced to increase the utilization and receptive field of feature maps. The global homography of the input image is calculated through a regression network, and the image is warped to achieve rough alignment. The rough alignment stage only considers the global homography of the image, which can only represent spatial transformations of the same depth, so the input image cannot be accurately aligned. Specifically, it includes the following:

[0122] (1) Four-point parameterized homography matrix;

[0123] Suppose the two pairs of coordinate points corresponding to a pair of transformed images are x = (u, v) T , x′=(u′,v′) T , after introducing homogeneous coordinates, x = (u, v, 1) T , x′=(u′,v′,1 ) T, then the homography matrix H corresponding to the mapping relationship between the two points is:

[0124]

[0125] Let Δu = Δu′1-u1, Δv = Δv′1-v1, that is, use Δu and Δv to represent the offset of the four point pairs, then (7) can be converted into a four-point parameterized homography matrix H 4pt :

[0126]

[0127] (2) Feature pyramid;

[0128] The image is input to the multi-scale deep homography network and processed by 8 convolutional layers with sizes set to 64, 64, 128, 128, 256, 256, 512, 512, respectively. Figure 7 As shown in the figure. A maximum pooling layer is used between every two convolutional layers, denoted as F and F respectively. 1 / 2 、F 1 / 4 、F 1 / 8 . Select F 1 / 2 、F 1 / 4 、F 1 / 8 A three-layer feature pyramid is formed. The feature map of each layer in the pyramid is used to estimate the homography, and the homography estimated by the upper layer is passed to the lower layer.

[0129] (3) Feature correlation;

[0130] The feature association layer strengthens feature matching and improves the network's receptive field. With target features The correlation coefficient c between them can be calculated as:

[0131]

[0132] in, They are and The two-dimensional spatial position of w (or R h ) represents the search radius on the width (or height) axis, and we can get c∈W l ×H l ×(2R w +1)(2R h +1). Therefore, by setting R w (or R h ) is equal to W l (or H l ) to calculate the global correlation, set R w (or R h ) is less than W l (or H l ) to calculate local correlation. Finally, the global correlation and local correlation are used to predict the homography from global to local.

[0133] (4) predict homography;

[0134] After extracting pyramid features and calculating feature correlations, a regression network consisting of three convolutional layers and two fully connected layers is designed to predict the offsets of the eight vertices of the target image to determine a unique homography matrix, thereby calculating the four-point parameterized homography matrix H 4pt .

[0135] Assume that the residual offset predicted by each layer of the three-layer pyramid is Δ i , i = 1, 2, 3, each layer of the pyramid only calculates the correlation between the distorted target features and the reference features. i The calculation formula is as follows:

[0136]

[0137] Among them, H 4pt The four-point parameterized homography operation to calculate the deformation of the reference feature map to the target feature map. W represents the operation of warping the target feature map using the homography transformation, and the DLJ operation converts the offset to the corresponding homography transformation. Let Δ0 = 0, then the final predicted offset is calculated as follows:

[0138] Δ w×h =Δ1+Δ2+Δ3 (12)

[0139] Finally, the images are coarsely aligned by solving the homography H and warping the input image.

[0140] (5) Objective function;

[0141] The multi-scale depth homography is trained in a supervised manner, considering the ground truth offset as The designed objective function is:

[0142]

[0143] Among them, w1, w2, and w3 represent the weights of each layer in the three-layer pyramid.

[0144] (6) Rough image alignment;

[0145] First, find the coordinates of the four vertices in the distorted target graph. The formula is:

[0146]

[0147] in, Represent the distorted target image and the kth vertex coordinate of the target image respectively. (Δx k , Δy k) is the offset of the kth vertex estimated by the network. Then, the distorted image (H * ×W * ) size:

[0148]

[0149] in, is the vertex coordinate of the reference image, and its value is Finally, the input reference image I A and the target image I B After deformation, I AW ,I BW , which can be expressed as:

[0150]

[0151] Where I and H are homography matrices, the identity matrix and the estimated matrix, respectively. W(·) represents the operation of distorting the image using a 3×3 transformation matrix, and its operation area is (H * ×W * ).

[0152] Step 2.2, unsupervised image reconstruction stage;

[0153] The unsupervised image reconstruction stage is introduced to reconstruct the coarse alignment image from features to pixels to achieve accurate image alignment. The unsupervised image reconstruction framework is shown in Figure 8 , including two branches: low-resolution deformation and high-resolution refinement. For high-resolution images, the low-resolution branch ensures the integrity of the network's receptive field. First, the resolution of the coarsely aligned image is reduced to 256×256. The content mask ensures that the features of the reconstructed image are close to the deformed image, and the seam mask ensures that the edges of the overlapping area have a natural transition. The high-resolution image is input to the high-resolution refinement branch for content mask constraints and seam mask constraints, and finally a high-resolution seamless splicing image is generated.

[0154] (1) Low-resolution deformation branch;

[0155] The first step, such as Figure 8 As shown on the left. First, reduce the distorted image to a lower resolution, defined as 256×256. Then use the codec network consisting of 3 pooling layers and 3 deconvolution layers to reconstruct the spliced ​​image. Set the convolution layer filter numbers to 64, 64, 128, 128, 256, 256, 512, 512, 256, 256, 128, 128, 64, 64, 3.

[0156] In the second step, the content mask and seam mask are used to learn the deformation law of image splicing, such as Figure 8As shown on the left, the content mask is used to constrain the features of the reconstructed image to be close to the deformed image, and the seam mask is used to constrain the edges of the overlapping area to achieve a natural stitching transition.

[0157] In formula (16), I A ,I B Replaced by a matrix E with all 1s H×W The content mask M can be obtained AC 、M BC , the calculation formula is:

[0158]

[0159] Then the seam mask can be calculated by equation (18) and equation (19):

[0160]

[0161] Among them, (i, j) is the coordinate position, * represents the convolution operation, and C is used for all elements between 0 and 1.

[0162] Finally, the content loss and seam loss at low resolution are calculated as follows:

[0163]

[0164] Among them, S LR Represents a low-resolution stitched image. P and L l They represent the estimated content loss and seam loss respectively, and the estimated loss on the 'conv5_3' layer of VGG-19 is used. The total loss function of the unsupervised low-resolution deformation branch can be expressed as:

[0165]

[0166] Among them, λ s , c is the weight of content loss and seam loss in the total loss.

[0167] (2) High-resolution optimization branch;

[0168] After the initial deformation by the low-resolution branch, the high-resolution refinement branch increases the resolution of the spliced ​​image and refines it at the same time. This branch is composed entirely of convolutional layers, such as Figure 8 As shown on the right, there are 3 independent convolutional layers and 8 resblocks. Each resblock consists of a convolutional layer, a summation function, and a relu in sequence. Except for the last layer, which has a convolution kernel of 3, the rest of the layers have a kernel of 64. In order to prevent the underlying information from fading as the convolutional network goes deeper, the features of the second-to-last layer are added based on the features of the first layer of the convolutional network.

[0169] At the same time, S LR The resolution of the image is increased to the resolution of the original image and connected with the output of the high-resolution refinement branch to obtain a high-resolution stitched image S HR . High-resolution refinement branch L HR The loss function is:

[0170]

[0171] in, are the content loss and seam loss at high resolution, respectively. We only need to replace S in (20) HR and high-resolution mask replaced by S LR And the low-resolution mask can be calculated. Calculate the L of the high-resolution branch P The 'conv3_3' layer of VGG-19 is used.

[0172] (3) Objective function;

[0173] The high-resolution branch is designed to refine the spliced ​​image, but the increase in resolution will relatively reduce the network's receptive field, often resulting in artifacts in the spliced ​​image. Here, a content consistency loss is used to allow the network to improve the resolution while eliminating artifacts. Since the resolution of the low-resolution branch output is 256×256, S HR The resolution is also adjusted to 256×256 and used The formula is as follows:

[0174]

[0175] Combining all the above constraints, the objective function of the image reconstruction stage is obtained as follows:

[0176] L R =ω LR L LR +ω HR L HR +ω CS L CS (twenty four)

[0177] Among them, ω LR ,ω HR ,ω CS Indicates the weight of each part.

[0178] Step 3: Image stitching based on traditional key point matching;

[0179] The following problems may occur when unsupervised stitching methods are applied to high-resolution stitching of multiple images:

[0180] (1) The network structure will limit the image resolution. The maximum input and output image resolution is 1024*1024, which is too low for panoramic image stitching.

[0181] (2) The network requires that the input images have the same size. For stitched images with different overlapping areas, the stitched images have different sizes. If this method is used continuously, the problem of image distortion will occur.

[0182] (3) It is impossible to estimate the camera parameters and use them to perform projection correction on the stitched image.

[0183] In order to overcome the limitation of unsupervised network structure and realize high-resolution stitching of multiple images, this solution uses OpenCV-stitching for panoramic stitching. The OpenCV-stitching stitching process is shown in Fig. 9 The estimated camera parameters provide projection transformation parameters for subsequent panoramic correction, including:

[0184] Adjust the unsupervised deep learning image stitching results to medium resolution and low resolution respectively to obtain medium resolution images and low resolution images;

[0185] Perform feature extraction, feature matching, and feature point purification on medium-resolution images to obtain estimated camera parameters and generate transformation parameters based on the camera parameters;

[0186] Performing image transformation on the low-resolution image based on the transformation parameters to obtain an estimated exposure parameter and a first transformed image;

[0187] The first stitching result is transformed based on the transformation parameters to obtain a second transformed image. After compensating for exposure errors in the second transformed image in combination with the exposure parameters, the first transformed image is mixed with the second transformed image to obtain a panoramic stitching result.

[0188] Step 4: Perform multi-image projection correction based on projection transformation;

[0189] Homography describes the transformation of pure rotation or translation of the same plane. When it is only stitched by simple plane projection, the result will show the effect of spatial distortion, see Fig.10 Left picture.

[0190] In order to achieve panoramic correction of multi-image stitching, this solution changes the projection transformation method to cylindrical projection. Cylindrical projection uses the camera as the center point of the cylinder and the cylindrical surface with the focal length as the radius as the projection plane. Its projection image and horizontal direction can meet 360° panoramic view, which can be better applied in complex environments such as lanes.

[0191] First, the camera intrinsic parameters K and rotation matrix R are obtained through camera parameter estimation. Assuming the two-dimensional coordinates of a pixel in the image are (x, y), the corresponding world coordinates can be calculated as:

[0192]

[0193] Let (u,v) be the two-dimensional coordinates after cylindrical projection:

[0194]

[0195] Among them, s is the transformation scale of the stitched image. The panoramic image correction can be realized simply and quickly through cylindrical projection, so that the overall visual effect of the image conforms to the real scene. The effect is shown in Fig.10 Right side.

[0196] Step 5: Rectification of the image based on corrosion subtraction;

[0197] After the panorama stitching is completed, irregular black borders often appear on the edge of the image. This is not visually ideal. Here, a simple and efficient algorithm is used to reduce the irregular black borders. The detailed process framework of the algorithm is shown in Fig.11 .

[0198] First, a 10-pixel black border is extended in the upper, lower, left, and right panorama. Then, the pixel points in the area with pixel values ​​greater than 0 in the image, that is, the pixel points in the image content area, are changed to white, and the pixel points in other positions are changed to black to generate mask A. Then, the minimum rectangular area that can accommodate the outline of the entire panorama is calculated to generate mask B.

[0199] Using the two masks generated in A and B, first perform horizontal and vertical erosion operations on the mask generated in B to eliminate the boundary points of the object and shrink the boundary inward. Then subtract pixels from the mask generated in A. Repeat the erosion-subtraction process to obtain a panoramic internal mask with the largest rectangular area. Multiply the mask with the unrectangularized spliced ​​image to generate the following rectangularized panoramic image.

[0200] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. A high-resolution stitching method for multiple lane images based on unsupervised deep learning, characterized in that: include: Acquiring an image sequence in the lane, wherein the images in the image sequence are acquired from different viewing angles and adjacent images have overlapping areas; Adopting an unsupervised image stitching method to stitch adjacent image sequences to obtain a high-resolution seamless stitching image; Using a key point matching image stitching method to stitch the seamless stitching images to obtain a panoramic image; Performing projection correction and rectangularization processing on the panoramic image to complete high-resolution stitching of multiple images of the lane; The image sequence collected in the lane includes: Arranging a camera device in the lane, and collecting images through the camera device; Based on the arrangement position of the camera device, the images are combined with an image sequence matching algorithm to form an image sequence, wherein the image sequence matching algorithm includes a brute force matching algorithm and a K-nearest neighbor matching algorithm; Combining the images into an image sequence based on the arrangement positions of the camera devices and in combination with an image sequence matching algorithm comprises: Extract features from the image based on scale-invariant feature transformation to obtain feature points; The feature points are processed using the brute force matching algorithm and the K-nearest neighbor matching algorithm to obtain the nearest neighbor feature points and the next nearest neighbor feature points of each feature point, and to construct a data set; Eliminate outliers from the data set using a random sampling consensus algorithm to obtain the image sequence; The unsupervised image stitching method comprises: an unsupervised image coarse alignment algorithm and an unsupervised image reconstruction algorithm, wherein the unsupervised image coarse alignment algorithm performs image enhancement and feature extraction on adjacent image sequences and then inputs the features into a regression network to perform coarse alignment on adjacent image sequences to obtain a coarsely aligned image; The unsupervised image reconstruction algorithm obtains a high-resolution seamless stitching image by sequentially inputting the coarse alignment image into a low-resolution deformation branch and a high-resolution refinement branch, wherein the low-resolution deformation branch is used to receive the coarse alignment image and ensure that the coarse alignment image is complete, and the high-resolution refinement branch is used to process the coarse alignment image through content mask constraints and seam mask constraints to output a high-resolution seamless stitching image; Using a key point matching image stitching method to stitch the seamless stitching images to obtain a panoramic image includes: Adjusting the seamless stitching image to a medium resolution and a low resolution respectively to obtain a medium resolution image and a low resolution image; Performing feature extraction, feature matching, and feature point purification on the medium-resolution image to obtain estimated camera parameters, and generating transformation parameters based on the camera parameters; Performing image transformation on the low-resolution image based on the transformation parameters to obtain estimated exposure parameters and a first transformed image; Performing image transformation on the seamless stitching image based on the transformation parameters to obtain a second transformed image, compensating for exposure error on the second transformed image in combination with the exposure parameters, and then performing image blending on the first transformed image and the second transformed image to obtain the panoramic image; Performing projection correction on the panoramic image includes: Based on the camera parameters, the camera intrinsic parameters and the rotation matrix are obtained, and the world coordinates corresponding to the pixel points in the panoramic image are calculated; Calculate the two-dimensional coordinates after projection transformation through the world coordinates to obtain a cylindrical projection image; The rectangularization process of the panoramic image includes: Extending the frame of the cylindrical projection image to obtain a frame image; The pixel points whose pixel values ​​in the frame image are greater than a preset value are set to white, and the pixel points whose pixel values ​​are less than the preset value are set to black, to generate a first mask; Calculate the minimum rectangular area that accommodates the first mask to obtain the second mask; Cyclic erosion-subtraction operations are performed on the second mask and the first mask until the pixels are subtracted to zero, thereby obtaining a third mask; The third mask is spliced ​​with the cylindrical projection image to obtain a rectangular panoramic image.

2. The method for high-resolution stitching of multiple lane images based on unsupervised deep learning according to claim 1, characterized in that: The method for calculating the world coordinates corresponding to the pixel points in the panoramic image is: Among them, (X, Y, Z) is the world coordinate corresponding to the pixel point, R is the rotation matrix, K is the camera intrinsic parameter, and (x, y) is the two-dimensional coordinate of the pixel point; The method for calculating the two-dimensional coordinates after projection transformation through the world coordinates is: Among them, (u, v) is the two-dimensional coordinate after cylindrical projection, and s is the transformation scale of the stitched image.

Images