Scene image splicing method and system based on feature point matching and electronic equipment

Abstract

The invention discloses a scene image splicing method and system based on feature point matching and electronic equipment. The method comprises the following steps: acquiring an image set shot by each camera at the current moment under a BEV visual angle; sequentially selecting the next image in the image set as a to-be-spliced image by adopting a traversal splicing mode, and calculating a feature matching point pair between the to-be-spliced image and the spliced image; based on the pose matrix and the feature matching point pairs, performing pose transformation on the feature points in the to-be-spliced image, and updating the feature matching point pairs; calculating matching point transformation loss according to the updated feature matching point pairs, and iteratively updating a pose matrix based on the matching point transformation loss; and performing pose transformation on the to-be-spliced image based on the updated pose matrix, and splicing the transformed to-be-spliced image to the spliced image until splicing of all the images in the image set is completed. Through the technical scheme in the invention, the calculation efficiency of a splicing algorithm and the quality of image splicing can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of binocular vision, and in particular to a scene image stitching method based on feature point matching, a scene image stitching based on feature point matching, and an electronic device. Background Art

[0002] With the rapid development of autonomous driving technology, the intelligence level of vehicles is constantly improving, and autonomous driving technology has become one of the important research directions in the future transportation field. In order to achieve efficient, safe and reliable autonomous driving, vehicles need to have accurate environmental perception capabilities to fully and accurately understand and judge the surrounding environment. Among the many perception technologies, scene perception technology based on Bird's Eye View (BEV) has become an important research direction in the field of autonomous driving because it can provide environmental information from a global perspective.

[0003] The acquisition of BEV images is based on the fusion of data collected by multiple sensors (especially cameras) at a certain moment, and its range is limited. Therefore, in order to obtain a BEV image in a global scene, it is necessary to use scene stitching technology to stitch and fuse BEV images acquired at different locations to obtain a global BEV image.

[0004] Existing scene image stitching methods are mainly divided into two categories: feature point-based methods and direct stitching methods. Feature point-based methods mainly calculate the transformation relationship between images by detecting and matching feature points in the image to achieve stitching; while direct stitching methods achieve stitching by directly optimizing the overlapping area of ​​image pixels. In addition, deep learning technology has also begun to emerge in the field of image stitching, and by using convolutional neural networks (CNNs) for feature extraction and image registration, the accuracy and speed of stitching have been further improved.

[0005] For example, Brown and Lowe proposed an automatic panorama generation system in 2007. The system first uses the SIFT (Scale Invariant Feature Transform) algorithm to extract feature points in the image, then matches the feature points through nearest neighbor search, and then uses the RANSAC (Random Sample Consensus) algorithm to calculate the homography matrix between the images. Finally, the multi-band fusion algorithm is used to process the overlapping areas of the images to generate a seamless panoramic image. This method has high robustness and accuracy in feature point detection, matching and image fusion, and is widely used in actual panorama generation systems.

[0006] In the existing technology, although image stitching technology has made great progress, it still faces some challenges in practical applications. First, the traditional stitching method has high computational complexity when processing large-scale image data, and it is difficult to meet the needs of real-time applications. Second, for scenes with complex geometric relationships, such as scenes with significant depth changes, the existing homography matrix transformation model is difficult to accurately describe the geometric relationship between images (feature points).

[0007] Therefore, how to optimize the computational efficiency of the stitching algorithm and improve the quality of image stitching remains an important topic in the research of image stitching technology. Summary of the invention

[0008] The purpose of this application is to optimize the computational efficiency of the stitching algorithm and improve the quality of image stitching.

[0009] The technical solution of the first aspect of the present application is: a scene image stitching method based on feature point matching is provided, and the method includes: step 1: obtaining a set of images taken by each camera at the current moment under the BEV perspective; step 2: adopting a traversal stitching method, sequentially selecting the next image in the image set as the image to be stitched, and calculating the feature matching point pairs between the image to be stitched and the stitched image, wherein the stitched image is the first image in the image set during the first traversal stitching; step 3: based on the pose matrix and the feature matching point pairs, performing pose transformation on the feature points in the image to be stitched, and updating the feature matching point pairs; step 4: calculating the matching point transformation loss according to the updated feature matching point pairs, and iteratively updating the pose matrix based on the matching point transformation loss; step 5: performing pose transformation on the image to be stitched based on the updated pose matrix, and stitching the transformed image to be stitched to the stitched image, and re-executing step 2 until the stitching of all images in the image set is completed.

[0010] In any of the above technical solutions, further, in step 2, the feature matching point pairs between the image to be spliced ​​and the spliced ​​image are calculated, specifically including: step 21: using a feature map extraction network, respectively obtaining feature maps of the image to be spliced ​​and the spliced ​​image, generating multiple feature map groups, and recording the multiple feature map groups in order of resolution from small to large as a first feature map group, a second feature map group, and a third feature map group; step 22: using a query vector, calculating a first similarity map on the first feature map group; step 23: respectively calculating the two feature points in the first similarity map with the highest similarity to any query vector to form a first similarity point pair; step 24: based on the position of the first similarity point pair, intercepting a second feature map on the second feature map group, and based on the second feature map, calculating a second similarity map to generate a second similarity point pair; step 25: based on the position of the second similarity point pair, intercepting a third feature map on the third feature map group, and based on the third feature map, calculating a third similarity map to generate a third similarity point pair, and recording the third similarity point pair as a feature matching point pair.

[0011] In any of the above technical solutions, further, step 24 also includes: based on the first threshold, judging in turn whether the similarity of the point pairs in the second similarity graph is greater than the first threshold; if not, discarding the point pairs; if so, recording the point pairs as second similar point pairs.

[0012] In any of the above technical solutions, further, step 25 also includes: using a layer of convolution and a sigmoid activation function to calculate an edge degree map corresponding to the third feature map group; weighting the third similarity map based on the edge degree map; based on the second threshold, judging in turn whether the similarity of the point pairs in the weighted third similarity map is greater than the second threshold; if not, discarding the point pair, and if so, recording the point pair as a third similar point pair.

[0013] In any of the above technical solutions, further, step 25 also includes: using a layer of convolution and a sigmoid activation function to calculate a background degree map corresponding to the third feature map group; weighting the third similarity map based on the background degree map; based on a third threshold, judging in turn whether the similarity of the point pairs in the weighted third similarity map is greater than the third threshold; if not, discarding the point pair; if so, recording the point pair as a third similar point pair.

[0014] In any of the above technical solutions, further, the calculation formula of the matching point transformation loss is:

[0015]

[0016] In the formula, is the feature point in the stitched image in the feature matching point pair, is the feature point located in the image to be stitched after the pose transformation, p = 1, 2, …, P, P is the total number of feature matching point pairs.

[0017] The technical solution of the second aspect of the present application is: a scene image stitching system based on feature point matching is provided, the system comprising: an acquisition unit, the acquisition unit is configured to acquire a set of images captured by each camera at the current moment under the BEV perspective; a matching unit, the matching unit is configured to adopt a traversal stitching method, sequentially select the next image in the image set as the image to be stitched, and calculate the feature matching point pair between the image to be stitched and the stitched image, wherein the stitched image is the first image in the image set during the first traversal stitching; a transformation unit, the transformation unit is configured to perform a posture transformation on the feature points in the image to be stitched based on the posture matrix and the feature matching point pair, and update the feature matching point pair; an iteration unit, the iteration unit is configured to calculate the matching point transformation loss according to the updated feature matching point pair, and iteratively update the posture matrix based on the matching point transformation loss; a stitching unit, the stitching unit is configured to perform a posture transformation on the image to be stitched based on the updated posture matrix, and stitch the transformed image to be stitched to the stitched image until the stitching of all images in the image set is completed.

[0018] In any of the above technical solutions, the matching unit is further configured to: use a feature map extraction network to obtain feature maps of the image to be spliced ​​and the spliced ​​image respectively, generate multiple feature map groups, and record the multiple feature map groups in order from small to large resolution as a first feature map group, a second feature map group, and a third feature map group; use a query vector to calculate a first similarity map on the first feature map group; calculate the two feature points in the first similarity map with the highest similarity to any query vector respectively to form a first similarity point pair; based on the position of the first similarity point pair, intercept the second feature map on the second feature map group, and calculate the second similarity map based on the second feature map to generate a second similarity point pair; based on the position of the second similarity point pair, intercept the third feature map on the third feature map group, and calculate the third similarity map based on the third feature map to generate a third similarity point pair, and record the third similarity point pair as a feature matching point pair.

[0019] In any of the above technical solutions, the matching unit is further configured to: based on the first threshold, determine in turn whether the similarity of the point pairs in the second similarity graph is greater than the first threshold; if not, discard the point pairs; if so, record the point pairs as second similar point pairs.

[0020] The technical solution of the third aspect of the present application is: an electronic device is provided, which includes: a processor; a memory for storing processor executable instructions; wherein the processor is configured to implement any scene image stitching method based on feature point matching in the above-mentioned first aspect technical solution when executing the executable instructions.

[0021] The beneficial effects of this application are:

[0022] The technical solution in the present application transforms the pose of feature points in the image to be stitched through a pose matrix, obtains the matching result under the BEV perspective, constructs the matching point transformation loss, and iteratively updates the pose matrix in a manner that maximizes the overlap of feature points, thereby optimizing the computational efficiency of the stitching algorithm and improving the quality of image stitching.

[0023] This application also optimizes the calculation process of feature matching point pairs. In the process of calculating feature matching point pairs between images, the query vector is used as an intermediary to calculate the two feature points with the highest similarity to the query vector in the image group composed of the left and right feature maps, respectively, to form the first similar point pair. Afterwards, the "coarse-grained-fine-grained" iterative update method is used to gradually obtain more accurately located and final feature matching point pairs, rather than directly performing dense matching on the two images. This not only helps to reduce the amount of calculation, but also enables the matching results to have a certain degree of robustness while also having a high degree of accuracy, thereby improving the quality of image stitching. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The advantages of the above and / or additional aspects of the present application will become apparent and easily understood in the description of the embodiments in conjunction with the following drawings, in which:

[0025] Figure 1 is a schematic flow chart of a scene image stitching method based on feature point matching according to an embodiment of the present application;

[0026] Figure 2 is a schematic diagram of a second similarity point pair calculation process according to an embodiment of the present application;

[0027] Figure 3 It is a schematic block diagram of a scene image stitching system based on feature point matching according to an embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0029] In the following description, many specific details are elaborated to facilitate a full understanding of the present application. However, the present application may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present application is not limited to the specific embodiments disclosed below.

[0030] Embodiment 1:

[0031] like Figure 1As shown, this embodiment provides a scene image stitching method based on feature point matching, the method comprising:

[0032] Step 1: Obtain the image set from the BEV perspective captured by each camera at the current moment;

[0033] Specifically, this embodiment takes the vehicle-mounted surround view camera as an example, and sets the number of vehicle-mounted surround view cameras to 6, and numbers them in a clockwise direction. When the vehicle-mounted surround view camera is started, it will obtain 6 images from the BEV perspective centered on the vehicle. 1≤i≤C, C=6, which then constitutes the image set from the BEV perspective

[0034] Step 2: Using the traversal stitching method, select the next image in the image set as the image to be stitched, and calculate the feature matching point pairs between the image to be stitched and the stitched image. In the first traversal stitching, the stitched image is the first image in the image set.

[0035] Specifically, the 6 images acquired under the BEV perspective are stitched in sequence according to the camera numbers. In the stitching process, the first image is first used as the initial stitching image, and then the remaining images are stitched in sequence. Now, any image except the first image is used as an example for explanation, and the label sub is introduced. BEV , Represents the current image to be stitched, I BEV Represents the current stitched image.

[0036] Based on the feature point matching algorithm, the current image to be stitched Current stitched image I BEV The two images are matched with feature points to obtain the matched feature matching point pairs. Among them, P is the total number of successfully matched point pairs, The current image to be stitched The pth matching point in is the current stitched image I BEV Zhongyu Matching points for matching.

[0037] It should be noted that in the feature point matching algorithm, only some feature points in the image can be matched.

[0038] Step 3: Based on the pose matrix and feature matching point pairs, perform pose transformation on the feature points in the stitched image and update the feature matching point pairs;

[0039] Specifically, the pose matrix is ​​set to include the rotation matrix R and the translation matrix T, where the rotation matrix R is initialized to the unit matrix and the translation matrix T is initialized to the unit vector. The pose matrix represents the current image to be spliced. Current stitched image I BEV The relative pose between two images.

[0040] Therefore, based on the pose matrix, the current image to be stitched can be The pth feature point in Transform to the current stitching image I BEV In the above example, we get the updated feature matching point pair

[0041] Step 4: Calculate the matching point transformation loss based on the updated feature matching point pairs, and iteratively update the pose matrix based on the matching point transformation loss;

[0042] Specifically, due to the above characteristics The transformation is based on the initial pose matrix, so the feature points With the transformed feature points There is a deviation between them. Based on this deviation, the matching point transformation loss can be calculated. The corresponding calculation formula can be:

[0043]

[0044] In the formula, is the feature point in the stitched image in the feature matching point pair, is the feature point located in the image to be stitched after the pose transformation, p = 1, 2, …, P, P is the total number of feature matching point pairs.

[0045] After that, an iterative algorithm, such as Newton iteration method, is introduced to perform iterative calculations and update the pose matrix so that the feature points With the transformed feature points The deviation between them is minimal to ensure that the current image to be stitched after the pose transformation Can be transformed to the current stitching image I BEV .

[0046] Step 5: Perform posture transformation on the image to be stitched based on the updated posture matrix, and stitch the transformed image to be stitched into the stitched image, and select the next image labeled sub BEV +1 is used as a new image to be stitched, and step 2 is executed again until all images in the image set are stitched.

[0047] Considering that factors such as perspective differences, lighting changes, and dynamic objects that may exist during image capture may cause the accuracy of feature point matching to decrease, thereby affecting the overall effect of image stitching. Therefore, this embodiment also shows a method for image feature point matching. In this embodiment, the resolution of the image to be stitched and the stitched image is the same, both (W, H). In the above step 2, the feature matching point pairs between the image to be stitched and the stitched image are calculated, specifically including:

[0048] Step 21: using a feature map extraction network, respectively obtaining feature maps of the image to be stitched and the stitched image, generating multiple feature map groups, and recording the multiple feature map groups in order of resolution from small to large as a first feature map group, a second feature map group, and a third feature map group, wherein the resolution of the third feature map group is the same as the resolution (W, H) of the image to be stitched;

[0049] Specifically, based on the existing U-Net network framework, the images to be spliced Stitching images I BEV Two images are input into the network, and three feature maps with different resolutions can be obtained, which are recorded as the first feature map group (F A1 、F B1 ); The second feature map group (F A2 、F B2 ); The third feature map group (F A3 、F B3 ), that is, {F Ai ,F Bi |1≤i≤3}, the resolutions are The feature dimensions are d1, d2, and d3 respectively.

[0050] Step 22: Calculate a first similarity graph on the first feature graph group using the query vector;

[0051] Specifically, initialize M query vectors Respectively with the two feature maps F in the first feature map group A1 、F B1 Calculate the similarity for each point on the query vector. In the calculation process, firstly sort all query vectors in a row-first manner. Stack them up to get the query matrix Then the query matrix Q is respectively combined with the two feature maps F A1 、F B1 Perform matrix multiplication to obtain the first similarity graph Sim A1 、Sim B1 , the corresponding calculation formula is:

[0052]

[0053] in, The Softmax() function can be used to normalize the two similarity graphs.

[0054] Step 23: respectively calculating two feature points in the first similarity graph that have the highest similarity to any query vector to form a first similarity point pair;

[0055] Specifically, the first similarity graph Sim obtained by the above calculation A1 、Sim B1 There are 2M similarity graphs, where the i1th query vector The corresponding first similarity graphs are For the two first similarity graphs, the two feature points with the largest similarity values ​​are calculated respectively, and the corresponding calculation formula is:

[0056]

[0057] In the formula, Representation calculation The horizontal and vertical coordinates corresponding to the maximum value are: The first similarity graph The corresponding horizontal and vertical coordinates are For the calculated The coordinates corresponding to the maximum; similarity graph The calculation in is similar.

[0058] They are respectively represented in the first feature map (F A1 、F B1 ) and the i1th query vector The position of the most similar point, since these two points With the same query vector The similarity of is the highest, so it can be inferred that these two feature points also have a high similarity, and these two points are recorded as the first similar point pair, that is,

[0059] Therefore, this embodiment combines these two feature points into a first similar point pair Through this indirect calculation method, the query vector is an intermediate variable, and searches for the same query vector in the two feature maps The most similar feature points are used to indirectly obtain a pair of similar point pairs. Compared with the dense matching method in the traditional feature point matching process, this method can greatly reduce the amount of calculation and optimize the calculation efficiency.

[0060] Step 24: based on the position of the first similar point pair, intercepting a second feature map on the second feature map group, and based on the second feature map, calculating a second similarity map to generate a second similar point pair;

[0061] Specifically, Figure 2 As shown, due to the second feature map group (F A2 、F B2 ) is lower than the downsampling ratio (resolution) of the first feature map group (F A1 、F B1 ) is twice as much, that is, (F A1 、F B1 ) corresponds to a feature point in (F A2 、F B2 ) in a 2x2 region.

[0062] Considering that the above process has calculated that (\ A1 、F B1 ), but because its downsampling ratio is not high, it is impossible to obtain pixel-level matching. Therefore, based on the position of the first similar point pair From the second feature map group (\ A2 、F B2 ) The corresponding 2x2 area in the image extracts two 2x2 second feature maps The two second feature maps are densely matched to obtain a second similarity map of size 4x4. The corresponding calculation formula is:

[0063]

[0064] In the formula, the Flatten() operation is to convert the original The feature map of size is flattened into The feature map of size is calculated by the two to get R 4×4 Size Similarity Graph Among them, 1≤i1≤M.

[0065] In order to obtain the second similar point pair, it is necessary to find the position with the largest similarity from the second similarity graph. The corresponding calculation formula is:

[0066] x=x A *2+y A

[0067] y=x B *2+y B

[0068]

[0069] Among them, xA ,y A ,x B ,y B ∈[0,2) is the second feature map The corresponding coordinates in x, y∈[0,4) are the second similarity graph The corresponding horizontal and vertical coordinates are calculated by calculating the position corresponding to the maximum similarity in the second similarity graph We can infer that this point corresponds to the position in the second feature map:

[0070]

[0071] Among them, / / represents the integer division operation, % represents the remainder operation, and the position of the feature map can be obtained, and its position in the second feature map (F A2 、F B2 ) in the corresponding coordinates:

[0072]

[0073] At this point, we get the second similar point pair:

[0074] Step 25: Repeat the above process, based on the position of the second similar point pair, intercept the third feature map on the third feature map group, and based on the third feature map, calculate the third similarity map, generate the third similar point pair, and record the third similar point pair as the feature matching point pair.

[0075] Specifically, based on the positions of the two similar points in the second similar point pair pair2 From the third feature map group (F A3 、F B3 ) The corresponding area in the image is intercepted with two 2x2 size third feature maps The two third feature maps are densely matched to obtain a third similarity map of size 4x4. The corresponding calculation formula is:

[0076]

[0077] Through the above method, we get the similarity of the second similarity pair pair2 in the third feature map group (F A3 、F B3 ), the third similarity graph Each point in corresponds to a set of 16 combinations of similar points in the two 2x2-sized third feature maps intercepted above, namely:

[0078]

[0079] In order to obtain the third similar point pair, it is necessary to find the position with the largest similarity from the third similarity graph. The corresponding calculation formula is:

[0080] x=x A *2+y A

[0081] y=x B *2+y B

[0082]

[0083] Among them, x A ,y A ,x B ,y B ∈[0,2) is the third feature map The corresponding coordinates in x, y∈[0,4) are the third similarity graph The corresponding horizontal and vertical coordinates are calculated by calculating the position corresponding to the maximum similarity in the third similarity graph It can be inferred that this point corresponds to the position in the third feature map:

[0084]

[0085] Among them, / / represents the integer division operation, % represents the remainder operation, and the position of the feature map can be obtained, and its position in the second feature map (F A3 、F B3 ) in the corresponding coordinates:

[0086]

[0087]

[0088] At this point, we get the third similarity point pair:

[0089] In this embodiment, an intermediate variable query vector is introduced Find the vector corresponding to any query vector in the two low-resolution feature maps The feature points with the highest similarity form the first similar point pair. Then, by upsampling, dense matching is performed in the (second and third) feature maps with high resolution and small area to obtain the final feature matching point pair, which reduces the amount of calculation in the feature point matching calculation process and optimizes the calculation efficiency of the splicing algorithm.

[0090] In any of the above embodiments, further, after obtaining the second similarity graph After that, in order to obtain more matching point pairs, so that the model can match as many similar points in the two images as possible, rather than being limited to obtaining a pair of matching point pairs on each similarity graph, the step 24 further includes:

[0091] Based on the first threshold θ2, the second similarity graph is judged in turn Whether the similarity of the point pairs in is greater than the first threshold θ2;

[0092] If not, it means that this point pair is very likely not a set of similar point pairs, and the point pair can be discarded; if so, it means that this point pair is very likely a set of similar point pairs, and the point pair is retained and recorded as the second similar point pair.

[0093] In any of the above embodiments, further, considering that edge points in the image, such as corners of a wall, have a higher matching success rate, step 25 further includes:

[0094] Using a layer of convolution Conv1() and a sigmoid activation function, the edge degree map corresponding to the third feature map group is calculated to obtain the probability that each point in the third feature map group is located on the edge, where the resolution of the third feature map group is the same as the resolution of the spliced ​​image;

[0095] Specifically, a layer of convolution Conv1() is used to transform the original third feature map group (F A3 、F B3 ) into two corresponding edge maps edgeMap A 、edgeMap B ∈R H×W , the input channel is d3, the output channel is 1, followed by a pixel-by-pixel Sigmoid() activation function to convert the output into a probability in the range of [0,1]. The calculation formula is:

[0096]

[0097] During the training process of the above-mentioned convolution layer Conv1(), the true value of the edge degree map calculated by the traditional algorithm can be used for supervision, so that the parameters in the convolution here can focus on learning the edge-related information in the image features.

[0098] The third similarity graph is weighted based on the edge graph, and the corresponding calculation formula is:

[0099]

[0100] In the formula, is the third weighted similarity graph.

[0101] Specifically, appropriate weights are added to the point pairs on the third similarity graph based on the edge degree graph. It is more difficult to successfully match the pixel points of non-edge points. Even if the match is successful, the credibility is not high. Therefore, a lower weight is added to them. The credibility of edge points is higher, and their weight can be appropriately increased.

[0102] Multiply the similarity of each position by its corresponding edge degree in images A and B to obtain the weighted third similarity map

[0103] Based on the second threshold θ3, the weighted third similarity graph is determined in turn. Is the similarity of the point pair in greater than the second threshold θ3? If not, it means that this point pair is very likely not a group of similar point pairs, and the point pair can be discarded; if so, it means that this point pair is very likely a group of similar point pairs, and the point pair is retained and recorded as the third similar point pair, that is,

[0104] In any of the above embodiments, further, considering that there may be dynamic objects in the image, even if such objects are successfully matched in two images, since the dynamic objects in the two images are not necessarily in the same position, such point pairs are of no value for pose estimation, the step 25 further includes:

[0105] Using a layer of convolution Conv1() and a sigmoid activation function, the background degree map corresponding to the third feature map group is calculated to obtain the probability that each point in the third feature map group is a background point;

[0106] Specifically, a layer of convolution Conv1() is used to transform the original third feature map (F A3 、F B3 ) is transformed into two corresponding background maps backMap A 、backMap B , the input channel is d3, the output channel is 1, followed by a pixel-by-pixel Sigmoid() function to convert the output into a probability in the range of [0,1]. The calculation formula is:

[0107]

[0108] During the training process of the above-mentioned convolution layer Conv1(), the true value of the background degree map calculated by the existing instance segmentation algorithm can be used for supervision to guide the learning of this layer of convolution, so that the parameters in the convolution here can focus on learning the information related to the background in the image features.

[0109] The third similarity map is weighted based on the background degree map; it should be noted that the third similarity map can be weighted using the background degree map alone, or the third similarity map can be weighted using the edge degree map and the background degree map at the same time, and the corresponding calculation formulas are:

[0110]

[0111] or

[0112]

[0113] Based on the third threshold, determine in turn whether the similarity of the point pairs in the weighted third similarity graph is greater than the third threshold; if not, it means that this point pair is very likely not a group of similar point pairs, and the point pair can be discarded; if so, it means that this point pair is very likely a group of similar point pairs, and the point pair is retained and recorded as the third similar point pair.

[0114] Embodiment 2:

[0115] like Figure 3 As shown, this embodiment provides a scene image stitching system 100 based on feature point matching, and the system 100 includes:

[0116] An acquisition unit 101, the acquisition unit 101 is configured to acquire a set of images captured by each camera at a current moment under the BEV perspective;

[0117] The matching unit 102 is configured to sequentially select the next image in the image set as the image to be stitched by traversing stitching, and calculate the feature matching point pairs between the image to be stitched and the stitched image, wherein the stitched image is the first image in the image set during the first traversal stitching;

[0118] A transformation unit 103, the transformation unit 103 is configured to perform posture transformation on the feature points in the stitched image based on the posture matrix and the feature matching point pairs, and update the feature matching point pairs;

[0119] An iteration unit 104, the iteration unit 104 is configured to calculate a matching point transformation loss according to the updated feature matching point pairs, and iteratively update the pose matrix based on the matching point transformation loss;

[0120] The stitching unit 105 is configured to perform posture transformation on the image to be stitched based on the updated posture matrix, and stitch the transformed image to be stitched into the stitching image until the stitching of all images in the image set is completed.

[0121] In any of the above embodiments, further, the matching unit 102 is also configured to: use a feature map extraction network to obtain feature maps of the image to be spliced ​​and the spliced ​​image, respectively, generate multiple feature map groups, and record the multiple feature map groups in order of resolution from small to large as a first feature map group, a second feature map group, and a third feature map group; use a query vector to calculate a first similarity map on the first feature map group; respectively calculate the two feature points in the first similarity map with the highest similarity to any query vector to form a first similarity point pair; based on the position of the first similarity point pair, intercept the second feature map on the second feature map group, and based on the second feature map, calculate the second similarity map to generate a second similarity point pair; based on the position of the second similarity point pair, intercept the third feature map on the third feature map group, and based on the third feature map, calculate the third similarity map to generate a third similarity point pair, and record the third similarity point pair as a feature matching point pair.

[0122] In any of the above embodiments, further, the matching unit 102 is also configured to: based on the first threshold, determine in turn whether the similarity of the point pairs in the second similarity graph is greater than the first threshold; if not, discard the point pairs; if so, record the point pairs as second similar point pairs.

[0123] In any of the above embodiments, further, the matching unit 102 is also configured to: use a layer of convolution and a sigmoid activation function to calculate an edge degree map corresponding to the third feature map group; weight the third similarity map based on the edge degree map; based on the second threshold, determine in turn whether the similarity of the point pairs in the weighted third similarity map is greater than the second threshold; if not, discard the point pair; if so, record the point pair as a third similar point pair.

[0124] In any of the above embodiments, further, the matching unit 102 is also configured to: use a layer of convolution and a sigmoid activation function to calculate a background degree map corresponding to the third feature map group; weight the third similarity map based on the background degree map; based on a third threshold, determine in turn whether the similarity of the point pairs in the weighted third similarity map is greater than the third threshold; if not, discard the point pair; if so, record the point pair as a third similar point pair.

[0125] In any of the above embodiments, further, the calculation formula of the matching point transformation loss is:

[0126]

[0127] In the formula, is the feature point in the stitched image in the feature matching point pair, is the feature point located in the image to be stitched after the pose transformation, p = 1, 2, …, P, P is the total number of feature matching point pairs.

[0128] Embodiment 3:

[0129] This embodiment provides an electronic device, which includes: a processor; a memory for storing processor executable instructions; wherein the processor is configured to implement any one of the image stitching methods in the above-mentioned embodiment 1 when executing the executable instructions.

[0130] It should be noted that the number of processors may be one or more. At the same time, the electronic device in the embodiment of the present application may also include an input device and an output device. The processor, memory, input device, and output device may be connected via a bus or in other ways, which are not specifically limited here.

[0131] So far, various embodiments of the present application have been described in detail. In order to avoid obscuring the concept of the present application, some details known in the art are not described. Based on the above description, those skilled in the art can fully understand how to implement the technical solution disclosed herein.

[0132] Although some specific embodiments of the present application have been described in detail through examples, those skilled in the art should understand that the above examples are only for illustration rather than for limiting the scope of the present application.

[0133] The steps in this application can be adjusted in order, combined, and deleted according to actual needs.

[0134] Although the present application is disclosed in detail with reference to the accompanying drawings, it should be understood that these descriptions are merely exemplary and are not intended to limit the application of the present application. The scope of protection of the present application is defined by the appended claims and may include various modifications, alterations and equivalents made to the invention without departing from the scope and spirit of the present application.

Claims

1. A scene image stitching method based on feature point matching, characterized in that: The method comprises: Step 1: Obtain the image set from the BEV perspective captured by each camera at the current moment; Step 2: adopting a traversal stitching method, sequentially selecting the next image in the image set as the image to be stitched, and calculating the feature matching point pairs between the image to be stitched and the stitched image, wherein the stitched image is the first image in the image set during the first traversal stitching; Step 3: Based on the pose matrix and the feature matching point pairs, perform pose transformation on the feature points in the image to be stitched, and update the feature matching point pairs; Step 4: Calculate the matching point transformation loss according to the updated feature matching point pair, and iteratively update the pose matrix based on the matching point transformation loss; Step 5: Performing posture transformation on the images to be stitched based on the updated posture matrix, and stitching the transformed images to be stitched into the stitched images, and re-executing step 2 until the stitching of all images in the image set is completed.

2. The scene image stitching method based on feature point matching according to claim 1, characterized in that: In step 2, calculating the feature matching point pairs between the image to be stitched and the stitched image specifically includes: Step 21: using a feature map extraction network, respectively obtaining the feature maps of the image to be stitched and the stitched image, generating a plurality of feature map groups, and recording the plurality of feature map groups in order of resolution from small to large as a first feature map group, a second feature map group, and a third feature map group; Step 22: using the query vector, calculating a first similarity graph on the first feature graph group; Step 23: respectively calculating two feature points in the first similarity graph that are most similar to any of the query vectors to form a first similarity point pair; Step 24: based on the position of the first similar point pair, intercepting a second feature map on the second feature map group, and based on the second feature map, calculating a second similarity map to generate a second similar point pair; Step 25: Based on the position of the second similar point pair, intercept the third feature map on the third feature map group, and based on the third feature map, calculate the third similarity map, generate a third similar point pair, and record the third similar point pair as the feature matching point pair.

3. The scene image stitching method based on feature point matching according to claim 2, characterized in that: The step 24 further includes: Based on a first threshold, determining in sequence whether the similarities of the point pairs in the second similarity graph are greater than the first threshold; If not, discard the point pair. If so, record the point pair as the second similar point pair.

4. The scene image stitching method based on feature point matching according to claim 2, characterized in that: The step 25 further includes: Calculate an edge degree map corresponding to the third feature map group using a layer of convolution and a sigmoid activation function; weighting the third similarity graph based on the edge degree graph; Based on the second threshold, determining in sequence whether the weighted similarities of the point pairs in the third similarity graph are greater than the second threshold; If not, discard the point pair. If so, record the point pair as the third similar point pair.

5. The scene image stitching method based on feature point matching according to claim 2 or 4, characterized in that: The step 25 further includes: Calculate the background degree map corresponding to the third feature map group using a layer of convolution and a sigmoid activation function; weighting the third similarity map based on the background degree map; Based on a third threshold, determining in sequence whether the weighted similarities of the point pairs in the third similarity graph are greater than the third threshold; If not, discard the point pair. If so, record the point pair as the third similar point pair.

6. The scene image stitching method based on feature point matching according to claim 1, characterized in that: The calculation formula of the matching point transformation loss is: In the formula, is a feature point in the feature matching point pair located in the stitched image, is the feature point located in the image to be stitched after the posture transformation, p=1,2,…,P, and P is the total number of the feature matching point pairs.

7. A scene image stitching system based on feature point matching, characterized in that: The system comprises: An acquisition unit, the acquisition unit is configured to acquire a set of images captured by each camera at a current moment under the BEV perspective; A matching unit, wherein the matching unit is configured to sequentially select the next image in the image set as the image to be stitched, and calculate the feature matching point pairs between the image to be stitched and the stitched image, wherein the stitched image is the first image in the image set during the first traversal stitching; A transformation unit, wherein the transformation unit is configured to perform a posture transformation on the feature points in the to-be-joined images based on a posture matrix and the feature matching point pairs, and update the feature matching point pairs; An iteration unit, the iteration unit being configured to calculate a matching point transformation loss according to the updated feature matching point pairs, and iteratively update the pose matrix based on the matching point transformation loss; A stitching unit is configured to perform posture transformation on the images to be stitched based on the updated posture matrix, and stitch the transformed images to be stitched into the stitching image until the stitching of all images in the image set is completed.

8. The scene image stitching system based on feature point matching according to claim 7, characterized in that: The matching unit is further configured to: Using a feature map extraction network, respectively obtaining feature maps of the image to be stitched and the stitched image, generating a plurality of feature map groups, and recording the plurality of feature map groups in order of resolution from small to large as a first feature map group, a second feature map group, and a third feature map group; Using the query vector, calculating a first similarity graph on the first feature graph group; Calculate two feature points in the first similarity graph that are most similar to any of the query vectors to form a first similarity point pair; Based on the position of the first similar point pair, intercepting a second feature map on the second feature map group, and based on the second feature map, calculating a second similarity map to generate a second similar point pair; Based on the position of the second similar point pair, a third feature map is intercepted on the third feature map group, and based on the third feature map, a third similarity map is calculated to generate a third similar point pair, and the third similar point pair is recorded as the feature matching point pair.

9. The scene image stitching system based on feature point matching according to claim 8, characterized in that: The matching unit is further configured to: Based on a first threshold, determining in sequence whether the similarities of the point pairs in the second similarity graph are greater than the first threshold; If not, discard the point pair. If so, record the point pair as the second similar point pair.

10. An electronic device, characterized in that: The electronic device comprises: processor; a memory for storing instructions executable by the processor; Wherein, the processor is configured to implement the scene image stitching method based on feature point matching as described in any one of claims 1 to 6 above when executing the executable instructions.

Images