An image stitching processing system based on feature matching

Abstract

The technical field of image stitching system of the present invention particularly relates to an image stitching processing system based on feature matching, including a feature detection module, a feature matching module and a stitching seam processing module; the feature detection module is used to extract features of the image to be stitched; the feature matching module is used to determine the feature matching relationship between the images to be stitched; the stitching seam processing module uses the connecting line of the overlapping areas of the two images to be stitched as the stitching seam, and then fuses the stitching seam through the image fusion module. The deep learning method of this system replaces the traditional algorithm, and an efficient network model is obtained through training to detect image features; the feature detection network inputs the image to be stitched, and extracts features through a convolutional neural network. And this system can train the network in combination with the isomorphic adaptation mechanism, so that the network can enhance the data of the feature points and descriptors output by the feature detection network through self-supervision learning.

Description

Technical Field

[0001] The present invention relates to the technical field of image stitching systems, and in particular to an image stitching processing system based on feature matching. Background Art

[0002] Image stitching refers to the process of adjusting and stitching multiple overlapping images taken by cameras at different viewing angles through algorithms, ultimately generating a seamless, high-resolution, large-viewing-angle image that meets visual requirements. Image stitching technology has been widely used in security monitoring, 3D reconstruction, augmented reality, and other fields.

[0003] At present, the image stitching operation has the problem that the image stitching speed is too slow, resulting in a long time consumption, which is not suitable for applications with strong real-time requirements. In addition, most of the image stitching technology is completed by PC-side software, which uses SIFT algorithm, SURF algorithm, etc. to detect image features, and then matches the images to be stitched through the cross-matching algorithm, and finally finds the stitching seams of the images through the point-by-point method, thereby completing the stitching and integration operation of the images to be stitched.

[0004] Although the above operations can stitch images, traditional feature detection algorithms usually rely on manually designed feature points and descriptors, which still have some defects in terms of real-time and accuracy of feature detection; the image matching algorithm has a large amount of calculation and is prone to a certain amount of false matches; and for the algorithm for finding the stitching seam of the image, the stitched image is relatively rough, and it can only segment the overlapping parts, and it is difficult to really find the best stitching seam. After the image is stitched, it simply selects the images on both sides of the stitching seam for stitching. Since the image may have color difference and other conditions, the image stitching result will be discontinuous at the stitching seam, which appears very abrupt and does not conform to the visual perception. Summary of the invention

[0005] The present invention solves the above technical problem and adopts the following technical solution: an image splicing processing system based on feature matching, comprising a feature detection module, a feature matching module and a splicing seam processing module.

[0006] The feature detection module extracts features of the image to be spliced ​​by using a convolutional neural network, and outputs feature points and descriptors of the image to be spliced.

[0007] The feature matching module inputs the feature points and descriptors of the images to be spliced ​​into the SuperGlue feature matching network, uses the self-attention mechanism and the cross-attention mechanism to extract information from the feature points, and then determines the feature matching relationship between the images to be spliced.

[0008] The seam processing module includes a seam finding unit and an image fusion unit. The seam finding module finds and marks the overlapping area between two images to be stitched, and uses the connecting line of the overlapping area as the seam, and then fuses the seam through the image fusion module.

[0009] Furthermore, the convolutional neural network includes a BaseDetector network and a SuperPoint network; the BaseDetector network detects corner points of three-dimensional images in a virtual data set through supervised learning, and performs feature detection on a real scene data set, thereby generating feature points of the image to be spliced, and at the same time performs self-labeling and data enhancement on the data set, thereby using the SuperPoint network for self-supervised learning; the SuperPoint network performs feature detection on a real scene data set through self-supervised learning, thereby generating feature points and descriptors of the image to be spliced.

[0010] Furthermore, the SuperGlue feature matching network consists of an attention GNN layer and an optimal matching layer, wherein: the attention GNN layer encodes feature points and descriptors into a set of feature vectors through a multi-layer perceptron, and uses a self-attention mechanism and a cross-attention mechanism to extract the information of the feature vector to enhance the independence of the feature vector.

[0011] The optimal matching layer uses the hard conditions of feature matching to establish the constraints of the soft assignment matrix, expands the soft assignment matrix by introducing the situation where no match is found, performs inner product operation on the feature vector to obtain the score matrix, brings the score matrix and the soft assignment matrix into the sinkhorn algorithm for iterative solution, and finally obtains the feature matching matrix. After sorting the matrix, the feature matching relationship between the two images to be spliced ​​is obtained.

[0012] Furthermore, the BaseDetector network includes a first encoding network and a first decoding network; the first encoding network is a VGG structure network, which extracts features of the image to be spliced ​​and reduces the dimension of the image information to be spliced ​​by performing a convolution operation on the image to be spliced, thereby obtaining an output tensor of the image to be spliced.

[0013] The first decoding network obtains a feature map by passing the output tensor of the image to be spliced ​​through a convolution layer, calculates the feature map through a softmax function, analyzes the pixels in the feature map, and screens qualified pixels in the feature map, recording them as feature points, and finally restores the size of the feature map to its initial size, thereby generating a heat map of the feature points of the image to be spliced.

[0014] Furthermore, the SuperPoint network includes a second encoding network and a second decoding network; the operation steps of the second encoding network are the same as those of the first encoding network.

[0015] The second decoding network also has a descriptor generation function based on the first decoding network. The second decoding network generates a semi-dense descriptor, then uses an interpolation algorithm to obtain a complete descriptor, and finally obtains a normalized descriptor of the image to be spliced ​​through L2 standardization.

[0016] Furthermore, the seam finding unit is used to find marks and preprocess the seams of the images to be stitched; the seam finding unit includes a mark finding end, a preprocessing end and a seam determination end, and the specific analysis method of the mark finding end is: in the first step, the overlapping parts of the images to be stitched are stitched together, and the overlapping parts of the images to be stitched are recorded as overlapping areas.

[0017] The second step is to calculate the distance between each pixel in the overlapping area and the corresponding image, and select the image closest to each pixel according to the principle of the closest distance, so that the pixels in the overlapping area of ​​the images to be stitched form a stitching area.

[0018] The third step is to mark the raised points on both sides of the splicing area and the endpoints at the upper and lower ends.

[0019] The preprocessing end is used to process the stitching area, thereby reducing the area of ​​the overlapping area of ​​the images to be stitched. The specific working steps of the preprocessing end are as follows: The first step is to screen the salient points on both sides of the stitching area that are farthest from the midline of the overlapping area, and record them as left salient pole points and right salient pole points respectively. The endpoints of the upper end of the stitching area are connected to the left salient pole points and the right salient pole points respectively to obtain the upper left connecting line and the upper right connecting line. Similarly, the endpoints of the lower end of the stitching area are connected to the left salient pole points and the right salient pole points respectively to obtain the lower left connecting line and the lower right connecting line.

[0020] In the second step, the upper integration area is obtained by analyzing the upper left line and the upper right line; the lower integration area is obtained by analyzing the lower left line and the lower right line.

[0021] In the third step, the overlapping part of the upper integrated region and the lower integrated region is recorded as the final integrated region.

[0022] The seam determination end is used to obtain the final seam position according to the final integration area analysis.

[0023] Furthermore, the upper integration area is determined as follows: the positions of the splicing area and the upper left line and the upper right line are judged; if the splicing areas are all located on the right side of the upper left line, the upper left line is recorded as the upper left tangent; if part of the splicing area is located on the right side of the upper left line, the upper left line is translated to the left so that it is tangent to a certain left convex position of the splicing area, and the splicing areas are all located on the right side of the upper left line. At this time, the translated upper left line is recorded as the upper left tangent, and the upper right tangent is obtained in the same way; the area of ​​the upper left tangent and the upper right tangent is recorded as the upper integration area; and the lower integration area is obtained in the same way using the above method.

[0024] Furthermore, the specific analysis method of the stitching seam determination end is as follows: the pixels in the stitching area are sorted from top to bottom, and the pixels at the same height are marked as i, i=1, 2, ..., k, and the midpoint of the overlapping area is recorded as the baseline, the position on the left side of the baseline is recorded as a negative value and the position on the right side is recorded as a positive value, and then the distance between each pixel at the same height and the baseline is recorded as Di, and the value of the pixel at the same height is determined by superposition. The position point of each pixel at the same height can be determined by the value of the pixel at the same height, and it is recorded as the stitching seam preset point; similarly, the stitching seam preset points of each height are obtained by analyzing in sequence from top to bottom, and the stitching seam preset points of each height are added with the corresponding position correction coefficient to obtain each stitching seam point, and each stitching seam point is connected to obtain the stitching seam.

[0025] Furthermore, the position correction coefficient is calculated by selecting the left and right points of the final integrated area corresponding to each pixel point at a certain height, recording them as the left position vertex and the right position fixed point, calculating the minimum distance from the left position vertex to the left side of the overlapping area and recording it as D 左 , calculate the minimum distance from the right position vertex to the right side of the overlapping area and record it as D 右 , through the formula Then we can get the position correction coefficient of a certain height, D 总 represents the distance between the left and right sides of the overlapping area, and e represents a natural constant.

[0026] Furthermore, the image fusion unit is used to stitch and fuse the images to be stitched based on their stitching seams, retain the image of the final integrated area in the overlapping area of ​​the images to be stitched, remove the images at other positions in the overlapping area, neutralize the contrast of the image of the final integrated area of ​​the images to be stitched, and thus complete the stitching operation of the images to be stitched.

[0027] The beneficial effects of this system are as follows: 1. The deep learning method of this system replaces the traditional algorithm, and an efficient network model is obtained through training to detect image features; the feature detection network inputs the image to be spliced, extracts features through the convolutional neural network, and finally outputs feature points and descriptors. In addition, this system can train the network in combination with the isomorphic adaptation mechanism, so that the network can learn through self-supervision, so that the data of feature points and descriptors output by the feature detection network is enhanced, and the generalization ability of the model is improved, thereby improving the real-time and accuracy of feature points and descriptors.

[0028] Second, this system uses the strengths of deep learning to train a reliable feature matching network. The network inputs the feature points and descriptors of the image, and outputs the features to match point pairs, which increases the accuracy of feature matching and greatly reduces the amount of calculation during image matching.

[0029] 3. This system calculates the distance between the pixels in the overlapping area and the corresponding image to find the stitching seam, and then determines the stitching area. The final integrated area is obtained through stitching area analysis, and the final stitching seam position is obtained through the final integrated area analysis, thereby increasing the accuracy of stitching seam finding. The stitching seam not only refers to the position of the final integrated area but also refers to the distance between the overlapping areas of the images, thereby further increasing the accuracy of stitching seam finding.

[0030] Fourth, when stitching images, this system retains the image of the final integrated area and neutralizes the contrast of the image of the final integrated area, thereby reducing the color difference of the stitched image and making the stitched image more coherent. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The present invention is further described using the accompanying drawings, but the embodiments in the accompanying drawings do not constitute any limitation to the present invention. A person skilled in the art can obtain other drawings based on the following drawings without creative work.

[0032] Figure 1 It is a schematic diagram of the module composition of the present invention.

[0033] Figure 2 This is the neural network composition structure intention of the present invention.

[0034] Figure 3 It is a schematic diagram of the image stitching result of the traditional image feature matching algorithm.

[0035] Figure 4 It is a schematic diagram of the image stitching result of the SuperGlue feature matching network image stitching algorithm of the present invention.

[0036] Figure 5It is a schematic diagram of marking the upper left connecting line, the upper right connecting line, the lower left connecting line and the lower right connecting line of the stitching image by the stitching seam finding unit in the stitching seam processing module.

[0037] Figure 6 It is a schematic diagram of the translation of the upper left tangent and the lower right tangent of the image to be stitched by the stitching seam finding unit in the stitching seam processing module.

[0038] Figure 7 It is a schematic diagram of marking the upper left tangent, upper right tangent, lower left tangent and lower right tangent of the image to be stitched by the stitching seam finding unit in the stitching seam processing module.

[0039] Figure 8 It is a schematic diagram of marking the final integrated area of ​​the stitching image by the stitching seam finding unit in the stitching seam processing module. DETAILED DESCRIPTION

[0040] The embodiments of the present invention are described in detail below. The embodiments described below are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention. If no specific techniques or conditions are specified in the embodiments, the techniques or conditions described in the literature in the art or the product specifications are used.

[0041] See also Figure 1 , an image stitching processing system based on feature matching, including a feature detection module, a feature matching module and a stitching seam processing module.

[0042] The feature detection module extracts features of the image to be spliced ​​through a convolutional neural network, and outputs feature points and descriptors of the image to be spliced.

[0043] like Figure 2 As shown in the figure, the convolutional neural network includes the BaseDetector network and the SuperPoint network; the BaseDetector network detects the corner points of the three-dimensional image in the virtual data set through supervised learning, and performs feature detection on the real scene data set, thereby generating feature points of the image to be spliced, and at the same time performs self-labeling and data enhancement on the data set, which is then used for self-supervised learning of the SuperPoint network; the SuperPoint network performs feature detection on the real scene data set through self-supervised learning to generate feature points and descriptors of the image to be spliced. The BaseDetector network and the SuperPoint network in the convolutional neural network obtain feature points and descriptors through supervised learning and self-supervised learning, respectively, making the acquisition of feature points and descriptors more accurate.

[0044] The BaseDetector network includes a first encoding network and a first decoding network; the first encoding network is a VGG structure network, which extracts the features of the images to be spliced ​​and reduces the dimension of the image information to be spliced ​​by performing convolution operations on the images to be spliced, thereby obtaining the output tensor of the images to be spliced, thereby reducing the amount of calculation; the first decoding network obtains the feature map by passing the output tensor of the images to be spliced ​​through a convolution layer, analyzes the pixels in the feature map, and selects qualified pixels in the feature map to record them as feature points, and finally restores the size of the feature map to the initial size, thereby generating a heat map of the feature points of the images to be spliced. The BaseDetector network can reduce the dimension of the image information of the image features to be spliced, thereby reducing the amount of calculation for image decoding and increasing the speed of obtaining image feature information.

[0045] Exemplarily, the first encoding network outputs a tensor of size H / 8*W / 8*128 for an image of size H*W*1, the first decoding network inputs a tensor of size H / 8*W / 8*128, obtains a feature map of H / 8*W / 8*65 through a convolutional layer, removes the 65th layer used to contain the case of no feature points, obtains the probability value of each pixel becoming a feature point, defined as the score of the pixel, and finally stretches the feature map size to H*W*1; compares the score of each pixel with a set threshold, if the score is higher than the threshold, the pixel is a feature point; if the score is lower than the threshold, it is filtered and eliminated, and finally a heat map of image feature points is generated.

[0046] The SuperPoint network includes a second encoding network and a second decoding network. The operation steps of the second encoding network are the same as those of the first encoding network. The second decoding network also has a descriptor generation function based on the first decoding network. The second decoding network generates a semi-dense descriptor, and then uses an interpolation algorithm to obtain a complete descriptor. Finally, the normalized descriptor of the image to be spliced ​​is obtained through L2 normalization.

[0047] In this embodiment, SuperPoint network training is used to implement self-supervised learning.

[0048] The specific steps of SuperPoint network training are as follows: S1, let the BaseDetector network perform supervised learning, that is, build a virtual three-dimensional object image dataset with labels; train the BaseDetector network to extract corner points from the image, which are candidate feature points in the traditional feature detection algorithm.

[0049] S2, combined with the isomorphic adaptation mechanism, allows the BaseDetector network trained in step S1 to perform self-supervised learning in an unlabeled real scene image dataset; first, each input image is geometrically transformed N times to obtain N different images; then these images are input into the BaseDetector network to obtain N different feature point score maps, and the N different feature point score maps are geometrically inverse transformed and unified to finally obtain the feature points of the image; then the image feature points detected in the dataset are annotated as labels for the subsequent SuperPoint network dataset; the BaseDetector network is trained in combination with the isomorphic adaptation mechanism, the data can be enhanced, and the generalization ability of the model is improved.

[0050] S3, self-supervised learning of the SuperPoint network generated in step S2 in a self-annotated real scene image dataset; first, geometric transformation is performed on the image in the dataset to obtain a new image, and then the two images are respectively input into the feature detection network, and two pairs of feature points and descriptors are output; because the geometric transformation relationship between the two images is known, the new image can be inversely transformed using the pose relationship; at the same time, the feature points of the new image will also be inversely transformed, superimposed and integrated with the feature points of the original image, and finally the feature points of the image are obtained. The self-supervised learning is achieved by training the SuperPoint network, thereby further increasing the accuracy of obtaining image feature points and descriptors.

[0051] The feature matching module inputs the feature points and descriptors of the images to be stitched into the SuperGlue feature matching network, uses the self-attention mechanism and cross-attention mechanism to extract information from the feature points, and then determines the feature matching relationship between the images to be stitched.

[0052] The SuperGlue feature matching network consists of an attention GNN layer and an optimal matching layer, where the attention GNN layer encodes feature points and descriptors into a set of feature vectors through a multi-layer perceptron, and uses self-attention mechanism and cross-attention mechanism to extract the information of the feature vector to enhance the independence of the feature vector.

[0053] The optimal matching layer uses the hard conditions of feature matching to establish the constraints of the soft assignment matrix, expands the soft assignment matrix by introducing the situation where no match is found, performs inner product operations on the feature vector to obtain the score matrix, brings the score matrix and the soft assignment matrix into the dual-softmax operator for iterative solution, and finally obtains the feature matching matrix. After sorting the matrix, the feature matching relationship between the two images to be spliced ​​is obtained. The feature matching module continuously extracts the information of the feature vector through the self-attention mechanism and the cross-attention mechanism through the attention GNN layer to enhance the independence of the feature vector. The optimal matching layer can expand the soft assignment matrix when no match is found, thereby increasing the feature matching relationship between the images to be spliced.

[0054] In this specific embodiment, the SuperGlue feature matching network is optimized by using the optimized weight decay method and the dual-softmax operator, and the optimized SuperGlue feature matching network is used to replace the traditional feature matching network to perform image feature matching for image stitching, and a comparative experiment is performed with the traditional feature matching network algorithm; the experimental image obtained, in which the image stitching result of the traditional image feature matching algorithm is as follows Figure 3 As shown, the SuperGlue feature matching network image stitching algorithm stitching result image is as follows Figure 4 As shown, the experimental data in the following Table 1 are obtained.

[0055] Table 1

[0056]

[0057] From the image and data analysis, we can see that when the results of simple image stitching are almost the same, using the optimized SuperGlue feature matching network to replace the traditional feature matching algorithm can significantly improve the speed of image stitching.

[0058] The seam processing module includes a seam finding unit and an image fusion unit. The seam finding module finds and marks the overlapping area between the two images to be stitched, and uses the connecting line of the overlapping area as the seam, and then fuses the seam through the image fusion module.

[0059] The seam finding unit is used to find marks and preprocess the seams of the images to be stitched; the seam finding unit includes a mark finding end, a preprocessing end and a seam determination end. The specific analysis method of the mark finding end is: in the first step, the overlapping parts of the images to be stitched are stitched together, and the overlapping parts of the images to be stitched are recorded as overlapping areas.

[0060] The second step is to calculate the distance between each pixel in the overlapping area and the corresponding image, and select the image closest to each pixel according to the principle of the closest distance, so that the pixels in the overlapping area of ​​the images to be stitched form a stitching area.

[0061] The third step is to mark the raised points on both sides of the splicing area and the endpoints at the upper and lower ends.

[0062] like Figure 1-4 As shown, the preprocessing end is used to process the stitching area, thereby reducing the area of ​​the overlapping area of ​​the images to be stitched. The specific working steps of the preprocessing end are as follows: the first step is to select the salient points on both sides of the stitching area that are farthest from the midline of the overlapping area, and record them as left salient points and right salient points respectively, and connect the endpoints of the upper end of the stitching area with the left salient point and the right salient point respectively to obtain the upper left connection line and the upper right connection line. Similarly, connect the endpoints of the lower end of the stitching area with the left salient point and the right salient point respectively to obtain the lower left connection line and the lower right connection line. For example, Figure 1 As shown, the stitching area of ​​the to-be-stitched images is marked with an upper left connecting line and an upper right connecting line, and a lower left connecting line and a lower right connecting line.

[0063] In the second step, the upper integration area is obtained by analyzing the upper left line and the upper right line; the lower integration area is obtained by analyzing the lower left line and the lower right line.

[0064] In the third step, the overlapped portion of the upper integration region and the lower integration region is recorded as the final integration region. For example, the schematic diagram of the final integration region of the image to be stitched is as follows: Figure 4 shown.

[0065] The seam determination end is used to obtain the final seam position according to the final integration area analysis.

[0066] The upper integration area is determined as follows: the positions of the stitching area and the upper left line and the upper right line are determined. If the stitching areas are all located on the right side of the upper left line, the upper left line is recorded as the upper left tangent. If part of the stitching area is located on the right side of the upper left line, the upper left line is translated to the left so that it is tangent to a certain left convex position of the stitching area, and the stitching areas are all located on the right side of the upper left line. At this time, the translated upper left line is recorded as the upper left tangent, and the upper right tangent is obtained in the same way. For example, the upper left line and the lower right line of the image to be stitched need to be translated, while the lower left line and the upper right line do not need to be translated, and the upper left tangent and the upper right tangent, the lower left tangent and the lower right tangent are obtained. The above state is as shown in FIG. Figure 5-Figure 8 As shown; the area of ​​the upper left tangent and the upper right tangent is recorded as the upper integration area; similarly, the above method is used to obtain the lower integration area.

[0067] The specific analysis method of the seam determination end is as follows: sort the pixels in the seam area from top to bottom, mark the pixels at the same height as i, i=1, 2, ..., k, record the midpoint of the overlapping area as the baseline, record the left position of the baseline as a negative value and the right position as a positive value, and then record the distance between each pixel at the same height and the baseline as Di, determine the value of the pixel at the same height by superposition, and determine the position of each pixel at the same height by the value of the pixel at the same height, which is recorded as the seam preset point; similarly, the seam preset points of each height are obtained by analyzing from top to bottom, and the seam preset points of each height are added with the corresponding position correction coefficient to obtain each seam point, and the seam points are connected to obtain the seam.

[0068] The position correction coefficient is calculated as follows: select the left and right points of the final integrated area corresponding to each pixel point at a certain height, record them as the left position vertex and the right position fixed point, calculate the minimum distance from the left position vertex to the left side of the overlapping area and record it as D 左 , calculate the minimum distance from the right position vertex to the right side of the overlapping area and record it as D 右 , through the formula Then we can get the position correction coefficient of a certain height, D 总 Indicates the distance between the left and right sides of the overlapping area, and e represents a natural constant. This system calculates the distance between the pixels in the overlapping area and the corresponding image to find the stitching seam, and then determines the stitching area, and obtains the final integration area through stitching area analysis, and obtains the final stitching seam position through the final integration area analysis, which increases the accuracy of stitching seam finding. The stitching seam not only refers to the position of the final integration area, but also can refer to the distance between the overlapping areas of the image, thereby further increasing the accuracy of stitching seam finding.

[0069] The image fusion unit is used to stitch and fuse the images to be stitched based on their stitching seams, and retain the image of the final integration area in the overlapping area of ​​the images to be stitched, remove the images at other positions of the overlapping area, and neutralize the contrast of the image of the final integration area of ​​the images to be stitched, thereby completing the stitching operation of the images to be stitched. When stitching images, this system retains the image of the final integration area and neutralizes the contrast of the image of the final integration area, thereby reducing the color difference of the stitched image and making the stitched image more coherent.

[0070] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations on the present invention. A person skilled in the art may make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention and they are still covered by the protection scope of the present invention.

Claims

1. An image stitching processing system based on feature matching, characterized in that: It includes a feature detection module, a feature matching module and a seam processing module; wherein: The feature detection module extracts features of the image to be spliced ​​through a convolutional neural network, and outputs feature points and descriptors of the image to be spliced; the convolutional neural network includes a BaseDetector network and a SuperPoint network; The feature matching module inputs the feature points and descriptors of the images to be stitched into the SuperGlue feature matching network, extracts information from the feature points using the self-attention mechanism and the cross-attention mechanism, and then determines the feature matching relationship between the images to be stitched; The stitching seam processing module includes a stitching seam finding unit and an image fusion unit. The stitching seam finding unit finds and marks the overlapping area between the two images to be stitched, and uses the connecting line of the overlapping area as the stitching seam, and then fuses the stitching seam through the image fusion unit. The seam finding unit is used to find marks and preprocess the seams of the image to be spliced; the seam finding unit includes a mark finding end, a preprocessing end and a seam determination end. The specific analysis method of the mark finding end is: In the first step, the overlapping parts of the images to be stitched are stitched together, and the overlapping parts of the images to be stitched are recorded as overlapping areas; The second step is to calculate the distance between each pixel in the overlapping area and the corresponding image, and select the image closest to each pixel according to the principle of the closest distance, so that the pixels in the overlapping area of ​​the images to be stitched form a stitching area; The third step is to mark the convex points on both sides of the splicing area and the endpoints at the upper and lower ends; The preprocessing end is used to process the stitching area, thereby reducing the area of ​​the overlapping area of ​​the images to be stitched. The specific working steps of the preprocessing end are as follows: The first step is to select the salient points on both sides of the splicing area that are farthest from the midpoint of the overlapping area, record them as left salient points and right salient points, connect the endpoints of the upper end of the splicing area with the left salient points and the right salient points, and obtain the upper left connection line and the upper right connection line. Similarly, connect the endpoints of the lower end of the splicing area with the left salient points and the right salient points, and obtain the lower left connection line and the lower right connection line. The second step is to obtain the upper integration area through the upper left line and the upper right line; and obtain the lower integration area through the lower left line and the lower right line. In the third step, the overlapped part of the upper integration area and the lower integration area is recorded as the final integration area; The seam determination end is used to obtain the final seam position according to the final integration area analysis; The upper integration area is determined by: judging the positions of the splicing area and the upper left connecting line and the upper right connecting line; if the splicing areas are all located on the right side of the upper left connecting line, the upper left connecting line is recorded as the upper left tangent; if the splicing area is partially located on the right side of the upper left connecting line, the upper left connecting line is translated to the left so that it is tangent to a certain left convex position of the splicing area, and the splicing areas are all located on the right side of the upper left connecting line. At this time, the translated upper left connecting line is recorded as the upper left tangent, and the upper right tangent is obtained in the same way; the area of ​​the upper left tangent and the upper right tangent is recorded as the upper integration area; and the lower integration area is obtained in the same way by the above method; The specific analysis method for determining the end of the splicing seam is: sorting the pixels in the splicing area from top to bottom, marking the pixels at the same height as i, , the midline of the overlapping area is recorded as the baseline, the position on the left side of the baseline is recorded as a negative value and the position on the right side is recorded as a positive value, and then the distance between each pixel point at the same height and the baseline is recorded as , the values ​​of the pixels at the same height are determined by superposition, and the positions of the pixels at the same height can be determined by the values ​​of the pixels at the same height, which are recorded as the preset points of the splicing seams; similarly, the preset points of the splicing seams at each height are obtained by sequential analysis from top to bottom, and the preset points of the splicing seams at each height are added with the corresponding position correction coefficients to obtain the splicing seam points, and the splicing seams are connected to obtain the splicing seams; The position correction coefficient is calculated by selecting the left and right points of the final integrated area corresponding to each pixel point at a certain height, recording them as the left position vertex and the right position fixed point, calculating the minimum distance from the left position vertex to the left side of the overlapping area and recording it as , calculate the minimum distance from the right position vertex to the right side of the overlapping area and record it as , through the formula , and then get the position correction coefficient for a certain height, represents the distance between the left and right sides of the overlapping area, and e represents a natural constant.

2. The image stitching processing system based on feature matching according to claim 1, characterized in that: The BaseDetector network detects the corner points of the three-dimensional images in the virtual data set through supervised learning, and performs feature detection on the real scene data set to generate feature points of the image to be spliced. At the same time, the data set is self-labeled and data enhanced, and then used for self-supervised learning of the SuperPoint network; the SuperPoint network performs feature detection on the real scene data set through self-supervised learning to generate feature points and descriptors of the image to be spliced.

3. The image stitching processing system based on feature matching according to claim 1, characterized in that: The SuperGlue feature matching network consists of an attention GNN layer and an optimal matching layer, where: The attention GNN layer encodes the feature points and descriptors into a set of feature vectors through a multi-layer perceptron, and uses the self-attention mechanism and the cross-attention mechanism to extract the information of the feature vector to enhance the independence of the feature vector; The optimal matching layer uses the hard conditions of feature matching to establish the constraints of the soft assignment matrix, expands the soft assignment matrix by introducing the situation where no match is found, performs inner product operation on the feature vector to obtain the score matrix, brings the score matrix and the soft assignment matrix into the sinkhorn algorithm for iterative solution, and finally obtains the feature matching matrix. After sorting the matrix, the feature matching relationship between the two images to be spliced ​​is obtained.

4. The image stitching processing system based on feature matching according to claim 1, characterized in that: The BaseDetector network includes a first encoding network and a first decoding network; The first encoding network is a VGG structure network, which extracts features of the image to be spliced ​​and reduces the dimension of the image information to obtain an output tensor of the image to be spliced ​​by performing a convolution operation on the image to be spliced; The first decoding network obtains a feature map by passing the output tensor of the image to be spliced ​​through a convolution layer, calculates the feature map through a softmax function, analyzes the pixels in the feature map, and screens qualified pixels in the feature map, recording them as feature points, and finally restores the size of the feature map to its initial size, thereby generating a heat map of the feature points of the image to be spliced.

5. The image stitching processing system based on feature matching according to claim 1, characterized in that: The SuperPoint network includes a second encoding network and a second decoding network; The operation steps of the second coding network are the same as those of the first coding network; The second decoding network also has a descriptor generation function based on the first decoding network. The second decoding network generates a semi-dense descriptor, then uses an interpolation algorithm to obtain a complete descriptor, and finally obtains a normalized descriptor of the image to be spliced ​​through L2 standardization.

6. The image stitching processing system based on feature matching according to claim 1, characterized in that: The image fusion unit is used to stitch and fuse the images to be stitched based on their stitching seams, retain the image of the final integrated area in the overlapping area of ​​the images to be stitched, remove the images at other positions of the overlapping area, neutralize the contrast of the image of the final integrated area of ​​the images to be stitched, and then complete the stitching operation of the images to be stitched.

Images