A method for eliminating local similar false matching of adjacent images in oblique aerial image stitching

Abstract

The application belongs to the technical field of photoelectric detection. The application provides a method for eliminating similar false matching of adjacent image backgrounds in the splicing of oblique aerial images. The disclosed embodiment further filters feature points on the basis of feature point matching, thereby improving matching accuracy. Based on the positional relationship of feature point matching, the overlapping area of two images is obtained. In the case of normal matching, the contents of the overlapping area are consistent, while in the case of abnormal matching, only part of the overlapping area is consistent. Therefore, the gray similarity and structural similarity of the overlapping area image are calculated to eliminate false matching. The matching results of the current image and multiple surrounding images are used to further verify whether there is false matching, thereby finally improving the accuracy of matching and the splicing effect.

Description

TECHNICAL FIELD

[0001] The embodiment of the disclosure relates to the technical field of photoelectric detection, and particularly relates to a method for removing mis-matching of adjacent images in image splicing of oblique aerial images. BACKGROUND

[0002] The imaging field of view of a detector of an aerial image is very small, and can only cover a very small range of an observation area. In order to obtain an overall image of a large range area, there is a position overlap between adjacent imaging areas of the detector, so as to provide the same area features for adjacent images, and the position relationship between the entire scanning sequence images is obtained by using an image processing method, and the adjacent images are spliced into one large image.

[0003] However, in the process of large-scale observation, similar features may exist in adjacent images, but are not in the same area, for example, in a relatively dense building area, the buildings in adjacent images are almost the same, which may cause mis-matching, so that the entire subsequent matching is completely wrong, and the splicing effect is affected.

[0004] From the current research, the main direction of the researchers is how to extract features, match and splice. No research is found on the above-mentioned abnormal situation in splicing.

[0005] Therefore, it is necessary to improve one or more problems in the related technical solutions.

[0006] It should be noted that this section aims to provide background or context to the technical solutions of the disclosure stated in the claims. The description herein is not admitted to be prior art merely because it is included in this section. SUMMARY

[0007] In order to avoid the shortcomings of the prior art, the present application provides a method for removing mis-matching of adjacent images in image splicing of oblique aerial images, which solves the problem that in the prior art, in the process of large-scale observation, similar features may exist in adjacent images, but are not in the same area, which may cause mis-matching, so that the entire subsequent matching is completely wrong, and the splicing effect is affected.

[0008] According to the embodiment of the disclosure, a method for removing mis-matching of adjacent images in image splicing of oblique aerial images is provided, and the method comprises the following steps:

[0009] scanning according to a preset aerial path, and acquiring all feature points of a current image;

[0010] Based on the current image and neighboring images, all the feature points are filtered to obtain a set of feature points to be matched; wherein, the neighboring images include at most the previous image, the first neighboring image adjacent to the previous scan strip, the second neighboring image, and the third neighboring image;

[0011] Feature point matching is performed on the set of feature points to be matched to obtain... i The first feature point pair corresponds one-to-one; where, i ≤4;

[0012] right i The first feature point pair is processed to obtain a first transformation matrix group. Anomaly detection is performed based on the first transformation matrix group to obtain... j A set of effective feature point pairs and their corresponding second transformation matrix sets; wherein... j ≤ i ;

[0013] based on j For each effective feature point pair in the group, the local normalized correlation coefficient of the image containing each matching point pair is calculated. Reliability matching is then performed based on these local normalized correlation coefficients to obtain... j The second feature point pair in the group;

[0014] right j The second set of feature points is processed to obtain a third set of transformation matrices, and the third set of transformation matrices is used to obtain... j Group of overlapping rectangular regions;

[0015] Based on each of the overlapping region rectangles and the third transformation matrix group, the normalized correlation coefficient and structural similarity of each of the overlapping region rectangles are obtained;

[0016] The correct feature point pairs are determined based on the normalized correlation coefficient and structural similarity of each overlapping region; wherein, the correct feature point pairs include a first set of correct feature points corresponding to the current image and a second set of correct feature points corresponding to adjacent images;

[0017] Spatially stitch the first set of correct feature points and the second set of correct feature points in the correct feature points respectively to obtain the fourth transformation matrix of the current image relative to the stitched image space;

[0018] The matching is determined based on the normalized correlation coefficients of all overlapping regions. If the match is correct, the fourth transformation matrix is ​​saved to obtain the final transformation matrix. If not, the third transformation matrix group is anomaly processed to obtain new correct feature point pairs. The new correct feature point pairs are then spatially spliced ​​to update and save the fourth transformation matrix to obtain the final transformation matrix.

[0019] Furthermore, the step of filtering all the feature points to be matched based on the current image and neighboring images to obtain a set of feature points to be matched includes:

[0020] Based on the current image respectively Compared to the previous image The first adjacent image The second adjacent image and the third adjacent image The positional relationship and the preset scan overlap rate are used to select feature points to obtain the set of feature points to be matched. - .

[0021] Furthermore, feature point matching is performed on the set of feature points to be matched to obtain... i The steps for establishing a one-to-one correspondence of the first feature point pair include:

[0022] The current image The set of feature points to be matched is respectively compared with the above image The set of feature points to be matched, the first neighboring image The set of feature points to be matched, the second neighboring image The set of feature points to be matched and the third neighboring image The set of feature points to be matched is used for feature point matching to obtain... i The first feature point pair - .

[0023] Furthermore, regarding i The first feature point pair is processed to obtain a first transformation matrix group. Anomaly detection is performed based on the first transformation matrix group to obtain... j The steps of creating a set of effective feature point pairs and their corresponding second transformation matrix sets include:

[0024] Based on the affine transformation model, the least squares fitting algorithm is used to respectively... i The first feature point pair - Processing is performed to obtain the current image. With the above image The first transformation matrix, the current image The first adjacent image The first transformation matrix, the current image The second adjacent image The first transformation matrix, the current image The third adjacent image The first transformation matrix;

[0025] According to the current image With the above image The first transformation matrix, the current image The first adjacent image The first transformation matrix, the current image The second adjacent image The first transformation matrix, the current image The third adjacent image The first transformation matrix is ​​used to obtain the first transformation matrix group. ;

[0026] like Greater than the first preset value or Less than the second preset value Greater than the first preset value or If the value is less than the second preset value, then delete the corresponding first feature point pair. - In order to obtain j The effective feature point pairs described in the group - and the second transformation matrix group .

[0027] Furthermore, based on j For each effective feature point pair in the group, the local normalized correlation coefficient of the image containing each matching point pair is calculated. Reliability matching is then performed based on these local normalized correlation coefficients to obtain... j The steps for forming the second feature point pair include:

[0028] Based on the effective feature point pairs - The s Point pair, with the coordinates of that point pair , Centered on the image, normalized correlation coefficients are calculated within the range of (±M, ±N) in each image to obtain the local normalized correlation coefficients of each matching point to its respective image.

[0029] If the local normalized correlation coefficient is less than a third preset value, then the matched point pair is discarded to obtain... j Group 2 feature point pair - .

[0030] Furthermore, regarding jprocessing the second feature point pairs to obtain a third transformation matrix group, and obtaining a third transformation matrix group according to the third transformation matrix group j In the step of grouping the overlapping area rectangles, the step comprises:

[0031] Based on the affine transformation model, the least square fitting algorithm is used to respectively group the second feature point pairs - processing to obtain the third transformation matrix of the current image and the upper image , the third transformation matrix of the current image and the first adjacent image , the third transformation matrix of the current image and the second adjacent image , and the third transformation matrix of the current image and the third adjacent image ;

[0032] According to the third transformation matrix of the current image and the upper image , the third transformation matrix of the current image and the first adjacent image , the third transformation matrix of the current image and the second adjacent image , and the third transformation matrix of the current image and the third adjacent image , the third transformation matrix group is obtained ;

[0033] According to the third transformation matrix group , the overlapping area between the current image , the upper image , the first adjacent image , the second adjacent image and the third adjacent image is calculated respectively to obtain j the overlapping area rectangles are grouped.

[0034] Further, according to the third transformation matrix group and the third transformation matrix group, the step of obtaining the normalized correlation coefficient and the structural similarity of each overlapping area rectangle comprises:

[0035] Taking one image in each overlapping area as a reference, the corresponding pixel in the other image is found based on the third transformation matrix group, and the normalized correlation coefficient of each overlapping area rectangle is obtained by using the normalized correlation coefficient equation;

[0036] Taking one image in each of the overlap area rectangles as a reference, corresponding pixels in another image are searched based on the third transformation matrix group, and the structural similarity of each of the overlap area rectangles is obtained by using a structural similarity calculation equation.

[0037] Further, in the step of judging according to the normalized correlation coefficient and the structural similarity of each overlap area respectively to obtain a correct feature point pair, the step comprises:

[0038] If the normalized correlation coefficient is less than a third preset value and the structural similarity is less than a fourth preset value, it is judged as a false match, and the corresponding matching point pair is discarded to obtain k pairs of correct matching images and k groups of the correct feature point pairs. .

[0039] Further, in the step of respectively performing spatial stitching on the first correct feature point set and the second correct feature point set in the correct feature points to obtain a fourth transformation matrix of the current image relative to a stitching image space, the step comprises:

[0040] The coordinates of the first correct feature point set are merged into a first set .

[0041] According to the final transformation matrix of the last image , the coordinates of the second correct feature point set are converted into the stitching image space and merged into a second set .

[0042] Based on the first set and the second set , the affine transformation parameters between are obtained by using a least square fitting algorithm and an affine transformation model, that is, the fourth transformation matrix of the current image relative to the stitching image space. .

[0043] Further, in the step of judging whether the match is correct according to the normalized correlation coefficients of all the overlap areas, if yes, saving the fourth transformation matrix to obtain a final transformation matrix, if no, performing abnormal processing on the third transformation matrix group to obtain new correct feature point pairs, and performing spatial stitching on the new correct feature point pairs to update and save the fourth transformation matrix to obtain a final transformation matrix, the step comprises:

[0044] ​determining whether the normalized correlation coefficients of all the overlapping region images are greater than a fifth preset value, if yes, the matching is correct, the fourth transformation matrix of the current image is saved to obtain a final transformation matrix, and a next image is processed .

[0045] if no, the matching fails, the fourth transformation matrix of the current image is saved to obtain a final transformation matrix, and a next image is processed . abnormal translation amount condition .

[0046] if normal, the fourth transformation matrix of the current image is saved to obtain a final transformation matrix, and a next image is processed .

[0047] if abnormal, and the error absolute value of the translation amount of the current image and that of the previous image is greater than a sixth preset value, or the error absolute value of the rotation angle of the current image and that of the previous image is greater than the sixth preset value, or the error absolute value of the scale factor of the current image and that of the previous image is greater than the sixth preset value, the correct feature point pair is selected from the matching results , other matching feature points in the correct feature point pair are substituted for checking, if the error is greater than half of the feature point number of a pixel, it is considered that the matching result is abnormal, the feature points in the group are removed, all the matching results are traversed, and the normal matching result is reserved . . the normal matching result is spatially spliced to obtain the fourth transformation matrix of the current image relative to a spliced image space , the fourth transformation matrix is saved to obtain a final transformation matrix, and a next image is processed . .

[0048] The technical scheme provided by the embodiments of the present disclosure can have the following beneficial effects:

[0049]

[0050] ​​​In the embodiments of the present disclosure, by using the above-mentioned method for eliminating false matching of adjacent image background local similarity in aerial image stitching, on the one hand, the matching accuracy is improved by further screening the feature points on the basis of feature point matching. Based on the positional relationship of the feature point matching, the overlapping area of the two images is obtained. In the case of normal matching, the contents of the overlapping area are consistent, while in the case of abnormal matching, only the local overlapping area is consistent, so the gray similarity and structural similarity of the overlapping area image are calculated to eliminate false matching. The matching results of the current image and the surrounding multiple images are used to further verify whether there is false matching, so as to finally improve the accuracy of matching and the stitching effect. On the other hand, based on the feature point matching, the gray similarity and structural similarity of the corresponding area after matching are judged, and at the same time, the consistency of the transformation parameters of the current image and the adjacent image is combined to finally confirm whether the current feature point matching result is reliable, and the false matching relationship is eliminated. BRIEF DESCRIPTION OF DRAWINGS

[0051] The accompanying drawings, which are incorporated herein and form a part of the specification, illustrate embodiments consistent with the present disclosure and, together with the description, further serve to explain the principles of the present disclosure. It is to be expressly understood that the drawings are included solely for purposes of illustration and that they are not to be construed as limiting the present disclosure.

[0052] Figure 1 A step diagram of a method for eliminating false matching of adjacent image background local similarity in aerial image stitching is shown in an exemplary embodiment of the present disclosure.

[0053] Figure 2 A schematic diagram of the positional relationship of the current image, the upper image, the first adjacent image, the second adjacent image and the third adjacent image is shown in an exemplary embodiment of the present disclosure.

[0054] Figure 3 A schematic diagram of the overlapping area rectangle and the image coordinate system is shown in an exemplary embodiment of the present disclosure.

[0055] Figure 4 A schematic diagram of the feature point matching of two images is shown in an exemplary embodiment of the present disclosure.

[0056] Figure 5 An abnormal transformation matrix diagram after matching is shown in an exemplary embodiment of the present disclosure.

[0057] Figure 6 A normalized correlation coefficient and structural similarity abnormal matching result diagram is shown in an exemplary embodiment of the present disclosure.

[0058] Figure 7 A matching result diagram of the current image and the left image is shown in an exemplary embodiment of the present disclosure.

[0059] Figure 8 A matching result graph of a current image and a top-left corner image in an example embodiment of the present disclosure is shown;

[0060] Figure 9 A matching result graph of a current image and a bottom-left corner image in an example embodiment of the present disclosure is shown;

[0061] Figure 10 A matching result graph of a current image and a previous image in an example embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0062] Example implementations will now be described more fully with reference to the accompanying drawings. Example implementations may, however, be implemented in many different forms and should not be construed as limited to the implementations set forth herein; rather, these implementations are provided so that this disclosure will be thorough and complete, and will fully convey the inventive aspects to those skilled in the art. Features described in the description, examples, or claims can be combined in any suitable manner in one or more embodiments.

[0063] In addition, the accompanying drawings are included to provide a further understanding of embodiments of the present disclosure and are incorporated in and constitute a part of this specification, illustrate embodiments of the present disclosure and serve to explain the principles of the present disclosure. Like reference numerals refer to like elements throughout the accompanying drawings. It will be described in detail below.

[0064] A method for removing mis-matching of adjacent images in aerial image stitching is provided in the present example implementation. Referring to FIG. 1, the method for removing mis-matching of adjacent images in aerial image stitching can include steps S101-S110. Figure 1

[0065] Step S101: Scanning according to a preset aerial path, and obtaining all feature points of a current image;

[0066] Step S102: Filtering to-be-matched feature points according to all feature points of the current image and adjacent images, to obtain a to-be-matched feature point set; wherein the adjacent images at most include a previous image, a first adjacent image adjacent to a previous scanning strip, a second adjacent image, and a third adjacent image;

[0067] Step S103: Matching the to-be-matched feature point set to obtain a first feature point pair corresponding to a group one; wherein, i <4. i

[0068] Step S104: Matching the to-be-matched feature point set to obtain a second feature point pair corresponding to a group two; wherein, i ​​performing processing on the first feature point pairs to obtain a first transformation matrix group, performing abnormality detection according to the first transformation matrix group to obtain j a group of effective feature point pairs and corresponding second transformation matrix groups; wherein, j ≤ i ;

[0069] Step S105: based on j calculating a local normalized correlation coefficient of an image where each matching point pair is located in each group of the effective feature point pairs, and performing reliability matching according to the local normalized correlation coefficient to obtain j a second feature point pair group;

[0070] Step S106: performing processing on the second feature point pair group to obtain a third transformation matrix group, and obtaining j a group of overlapping area rectangles from the third transformation matrix group; j

[0071] Step S107: obtaining a normalized correlation coefficient and a structural similarity of each overlapping area rectangle according to each overlapping area rectangle and the third transformation matrix group;

[0072] Step S108: determining according to the normalized correlation coefficient and the structural similarity of each overlapping area to obtain a correct feature point pair; wherein, the correct feature point pair includes a first correct feature point set corresponding to a current image and a second correct feature point set corresponding to an adjacent image;

[0073] Step S109: performing spatial stitching on the first correct feature point set and the second correct feature point set in the correct feature point to obtain a fourth transformation matrix of the current image relative to a stitched image space;

[0074] Step S110: determining whether the matching is correct according to the normalized correlation coefficient of all the overlapping areas; if yes, saving the fourth transformation matrix to obtain a final transformation matrix; if no, performing abnormality processing on the third transformation matrix group to obtain a new correct feature point pair, and performing spatial stitching on the new correct feature point pair to update and save the fourth transformation matrix to obtain a final transformation matrix.

[0075] ​By the above-mentioned method for eliminating mis-matching of adjacent images in the background in the stitching of oblique aerial images, on the one hand, the matching precision is improved by further screening of the feature points on the basis of the matching of the feature points. Based on the positional relationship of the matching of the feature points, the overlapping area of the two images is obtained. In the case of normal matching, the contents of the overlapping area are consistent, while in the case of abnormal matching, only the local contents of the overlapping area are consistent, so the gray similarity and the structural similarity of the images in the overlapping area are calculated for eliminating the mis-matching. The matching results of the current image and the surrounding images are used to further verify whether there is mis-matching, so as to finally improve the accuracy of the matching and the stitching effect. On the other hand, based on the matching of the feature points, the gray similarity and the structural similarity of the corresponding area after the matching are judged, and at the same time, the consistency of the transformation parameters of the current image and the neighboring images is combined, so as to finally confirm whether the matching result of the current feature points is reliable, and to eliminate the mis-matching relationship.

[0076] In the following, the above-mentioned method for eliminating mis-matching of adjacent images in the background in the stitching of oblique aerial images in the present example embodiment will be described in more detail. Figures 1 to 10 In the following, the above-mentioned method for eliminating mis-matching of adjacent images in the background in the stitching of oblique aerial images in the present example embodiment will be described in more detail.

[0077] In steps S101 to S103, according to the preset scanning path, the feature points of the current input image are extracted. As described above, the scanning is performed in the order of 1, 2, 3, 4, 5. Figure 2

[0078] The feature points to be matched of the current image and the three images adjacent to the last scanning strip of the current image , , are screened, and at the same time, the feature points to be matched of the current image and the last image of the current strip are screened (see the positional relationship diagram Figure 1 ).

[0079] If the current image is the first image (1 in Figure 2 ) in the first scanning strip, all the feature points thereof are saved, and the processing of the second image is performed.

[0080] If the current image is an image (2, 3 in Figure 2 ) in the first scanning strip, the feature points to be matched of the current image and the last image of the current strip are screened.

[0081] If the current image is the first image (1 in Figure 2 ​If the number of feature points in the group is less than 5, the group is discarded, and i groups of feature point pairs are finally obtained

[0082] During the feature point screening, the feature points are selected according to the positional relationship between the images and the set scanning overlap rate;

[0083] Suppose that the set row and column scanning overlap rates P are both 30%, and the scanning is back and forth in the direction perpendicular to the flight direction of the aircraft, For the up-down relationship, the feature points in the P+10% overlap area are selected; For the up-down relationship, the feature points in the P+10% overlap area are selected; For the left-right relationship, the feature points in the P+10% overlap area are selected; For the left-right relationship, the feature points in the P+10% overlap area are selected; , For the left-right relationship, the feature points in the P+10% overlap area are selected; Only one corner overlaps, the feature points in the P+10% overlap area of the up-down and left-right overlap areas are selected, and n groups of feature point sets are obtained .

[0084] The n groups of feature point sets obtained are matched, and one-to-one corresponding feature point pairs are obtained ; if the number of feature point pairs in the group is less than 5, the group is discarded, and i groups of feature point pairs are finally obtained ; i≤n; In step S104, the i groups of feature point pairs obtained are respectively subjected to affine transformation model, and the least square algorithm is used to obtain

[0085] the transformation matrix of , , , , , ; if >1.4 or <0.7, >1.4 or <0.7, the parameters of the current two images are abnormal, it is considered that the two images are mismatched, and the feature point pairs of the two images are deleted , , and j groups of effective feature point pairs are finally obtained ; j≤i.

[0086] ​The affine transformation model formula is:

[0087] (1)

[0088] wherein,

[0089] (2)

[0090] represents the rotation, scaling, and translation relationship between the two images. The formula (1) is expanded as:

[0091] (3)

[0092] (4)

[0093] For example the point , represents the th point in , after calculation by the matrix , the corresponding matching image feature point pair will be obtained, and the feature point coordinates ; since must satisfy the mapping relationship of all feature points in the point pair - , therefore needs to use the least square method to fit all points in - to obtain

[0094] In step S105, for the j group of valid feature point pairs - , the local normalized correlation coefficient of each corresponding point in each group of feature point pairs is calculated, if the obtained normalized correlation coefficient is <0.65, the point pair in this group is discarded, to further improve the reliability of matching, and finally the feature point pair - is obtained; when calculating, for the th point pair in the feature point pair - s , the normalized correlation coefficient is calculated in the range of (±M, ±N) in the respective image with the point pair coordinates , as the center;

[0095] (5)

[0096] wherein: , is the feature point pair - The local images within the range of (±M, ±N) corresponding to each pair of feature points in , are the corresponding local image coordinates at which the pixel gray levels are , respectively , are the average gray levels of

[0097] In step S106, for the j pairs of feature points obtained after screening - , the affine transformation model is respectively adopted, and using the least squares algorithm, and , , , the transformation matrices are obtained;

[0098] Suppose here (6)

[0099] represents is the mapping relationship to other images.

[0100] In step S107, according to what is obtained in step S106 , the overlapping regions (the overlapping regions are shown in and , , , the shaded region) between each pair of the matching images are respectively calculated, and j groups of overlapping region rectangles are obtained; Figure 3 Taking

[0101] as an example, Figure 3 in Figure 3 1 represents , 2 represents . Then, bringing the upper left corner coordinates (0, 0) of the image into the right side of equation (6), the coordinates in corresponding to the upper left corner coordinates (0, 0) are obtained; calculating the inverse matrix of , and bringing the lower left corner coordinates of the image into , the coordinates corresponding to in the image are obtained. Thus, the coordinates in with (0, 0) as the upper left corner and The region with the right bottom corner, The region with the right bottom corner, The region with the right bottom corner, The region with the right bottom corner is the corresponding overlapping region.

[0102] The image coordinate system is shown in Figure 3 , The right bottom corner coordinate can be obtained from the width and height of the input image.

[0103] For the overlapping region, the normalized correlation coefficient of the two is calculated, as shown in formula (7). The image coordinates are based on one image, and then the corresponding pixel in the other image is found according to the calculated transformation matrix .

[0104] (7)

[0105] Let be a pixel point in the overlapping region of with coordinates , then the point in the matching image overlapping region needs to be converted through to obtain , and the overlapping region gray mean also needs to be calculated according to ; if is located outside the overlapping region, then , do not participate in the operation.

[0106] For the overlapping region, the structural similarity of the two is calculated. When calculating, the image coordinates are based on one image, and the corresponding pixel in the other image is found according to the calculated transformation matrix , the method is the same as above.

[0107] The structural similarity calculation formula is:

[0108] Among them: , are the mean values of the overlapping region, , are the mean square errors of the overlapping region, is the covariance of the overlapping region, , are constants, , .

[0109] In step S108, if the normalized correlation coefficient is less than 0.65 and the structural similarity is less than 0.85, the current two images are mismatched, and the matching point pair of the two images is discarded, and finally the correct matching image k pair and the feature point pair set ​- , transformation matrix ;

[0110] In step S109, the feature points obtained in step S108 are converted to the stitching image space according to the respective transformation matrixes obtained by stitching respectively, and merged into one set , and the feature points are also merged into one set ; An affine transformation model is adopted, and a least square fitting algorithm is used to obtain the affine transformation parameters between - , that is, the transformation matrix of the current image relative to the stitching image space ;

[0111]

[0112] The formula for converting the feature points to the stitching image space is: = × × (8)

[0113] In step S110, if the normalized correlation coefficients of all the overlapping area images in step S109 are greater than 0.9, the matching is correct, and step S1104 is executed; otherwise, step S1101 is executed.

[0114] Step S1101: The translation abnormality of the transformation matrix of the current image and the transformation matrix of the previous image (that is, the fourth transformation matrix of the previous image) is counted:

[0115] For the transformation matrix of the first image after the scanning commutation , if the difference abs( - ), abs( - ) of the translation in the x and y directions between the transformation matrix of the last image of the previous strip is greater than the image width and height respectively, it is considered that the image matching is abnormal.

[0116] For the transformation matrix of other images , the difference abs( - ), abs( - ) of the translation in the x and y directions between the transformation matrix of the previous image in the current strip is greater than the image width and height respectively.respectively, the image matching is considered abnormal.

[0117] Step S1102: If there is abnormal matching in step S1101, and the absolute value of the error of is greater than 0.2, or the absolute value of the error of is greater than 0.2, or the absolute value of the error of is greater than 0.2, the final obtained feature point pairs of each region are further screened:

[0118] Select the matching result of one group of , and substitute the other group of matching feature points in to check if the error is greater than half of the number of feature points with an error of more than pixels, the current matching is considered abnormal, the matching result is removed, and the normal matching result is retained . .

[0119] The substitution checking method is as follows:

[0120] Substitute the coordinates of each feature point in the other into the right side of the formula , where q represents the qth feature point, and the left side obtains the corresponding matching point coordinates calculated according to . Since represents the mapping of all feature points, there is a certain error between and the real extracted feature points in , and is calculated, if > , then the count is +1, and finally if > , the total number of exceeds half of the number of points participating in the checking, and the matching is considered abnormal. .

[0121] Step S1103: The feature points obtained in step S1102 are calculated according to the method of step S109 to obtain the transformation matrix of the current image relative to the stitching image space .

[0122] Step S1104: The transformation matrix of the current image ​​​Save and proceed to step S101 for further processing.

[0123] In one specific embodiment, for example, Figure 4 Feature extraction and feature point matching were performed on the two images shown.

[0124] The stitching uses an image resolution of 1280*1024.

[0125] SIFT feature points were calculated for the first input image, resulting in 2537 points.

[0126] SIFT feature points were calculated for the second input image, resulting in a total of 2013 points.

[0127] The two images are arranged vertically, with an overlap rate of 30%. The first image has 1153 feature points in the lower 40% area (below row 614) and the second image has 1063 feature points in the upper 40% area (above row 410). After matching, there are 150 matching point pairs, and the crosshairs connect them to form a pair of points.

[0128] After matching, the transformation matrices of the two images ,

[0129] in, , All meet the requirements and are considered a normal match.

[0130] In a specific embodiment, such as Figure 5 As shown, for Figure 5 Two images were stitched side-by-side, but due to the lack of overlap, the buildings were stitched incorrectly. The transformation matrix after stitching is... , , The requirement of step S104 is not met, and it is considered a mismatch.

[0131] In a specific embodiment, such as Figure 6 As shown, Figure 6 The matching results of the two images, Figure 6 The top left corner of the left image in the middle is... Figure 6 Matching is performed on the lower right corner of the right image. Based on the distribution of feature points and the positional relationship of the images, the overlapping area is calculated as shown in the white box. It can be seen that the matching is incorrect due to the influence of similar buildings. Although the feature points of the abnormal matches are locally similar, the background of the entire overlapping area is different. The calculated normalized correlation coefficient is 0.180868 and the structural similarity is 0.417972, so this area can be removed as an abnormal match.

[0132] In a specific embodiment, such as Figures 7 to 10 As shown, Figures 7 to 10The matching result of the current image and the image on the left side of the current image and the image on the upper left corner of the current image and the image on the lower left corner of the current image and the image on the upper side of the current image is shown in Figs. 1 to 4, respectively, and the transformation matrix is shown in Figs. 5 to 8, respectively. Figures 7 to 10 The right side image is the current image. After matching, the transformation matrix of each is shown in Figs. 9 to 12, respectively.

[0133] The matching result of the current image and the image on the left side of the current image and the image on the upper left corner of the current image and the image on the lower left corner of the current image and the image on the upper side of the current image is shown in Figs. 1 to 4, respectively, and the transformation matrix is shown in Figs. 5 to 8, respectively. Figure 7

[0134] .

[0135] The matching result of the current image and the image on the left side of the current image and the image on the upper left corner of the current image and the image on the lower left corner of the current image and the image on the upper side of the current image is shown in Figs. 1 to 4, respectively, and the transformation matrix is shown in Figs. 5 to 8, respectively. Figure 8

[0136] .

[0137] The matching result of the current image and the image on the left side of the current image and the image on the upper left corner of the current image and the image on the lower left corner of the current image and the image on the upper side of the current image is shown in Figs. 1 to 4, respectively, and the transformation matrix is shown in Figs. 5 to 8, respectively. Figure 9

[0138] .

[0139] The matching result of the current image and the image on the left side of the current image and the image on the upper left corner of the current image and the image on the lower left corner of the current image and the image on the upper side of the current image is shown in Figs. 1 to 4, respectively, and the transformation matrix is shown in Figs. 5 to 8, respectively. Figure 10

[0140] .

[0141] It can be seen that due to the influence of similar buildings, the matching is wrong, the parameters are abnormal, and the entire image is abnormal. Figure 8

[0142] According to step S1101, the translation of the transformation matrix of the current image and the image on the left side of the current image and the image on the upper left corner of the current image and the image on the lower left corner of the current image and the image on the upper side of the current image is compared.

[0143]

[0144]

[0145] abs( - ) = 342.117304

[0146] abs( - ) = 1480.222986> image height 1024.

[0147] Not meeting the requirements of step S1101, turn to step S1102, , ​​​​​​​​​​The absolute values of the differences are all greater than 0.2, and there is an abnormality. According to the method of step S1102, the abnormal feature points can be finally eliminated Figure 8 The final matching result is:

[0148] Thus, the abnormality of matching is corrected.

[0149] By the method for eliminating false matching between adjacent images in image stitching of oblique aerial photography based on local similarity of background, on the one hand, the matching precision is improved by further screening of feature points based on feature point matching. Based on the positional relationship of feature point matching, the overlapping area of two images is obtained. In the case of normal matching, the contents of the overlapping area are consistent, while in the case of abnormal matching, only part of the overlapping area is consistent. Therefore, the gray similarity and structural similarity of the images in the overlapping area are calculated to eliminate false matching. The matching results of the current image and the surrounding multiple images are used to further verify whether there is false matching, so as to finally improve the accuracy of matching and the stitching effect. On the other hand, based on feature point matching, the gray similarity and structural similarity of the corresponding area after matching are judged, and at the same time, the consistency of the transformation parameters of the current image and the neighboring images is combined to finally confirm whether the matching result of the current feature point is reliable, and to eliminate false matching relationship.

[0150] In addition, the terms "first", "second", "third", etc. are used only to describe and distinguish the corresponding features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second", etc. can explicitly or implicitly include one or more of the features. In the description of the embodiments of the present disclosure, the meaning of "a plurality of" is two or more, unless otherwise explicitly specified.

[0151] In the description of the present disclosure, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example" or "some examples" means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present disclosure. In the present disclosure, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in the present disclosure.

[0152] Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the features disclosed herein. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.

Claims

1. A method for removing false matches of local similarity of backgrounds of adjacent images in oblique aerial image stitching, characterized in that, The method comprises: According to the preset aerial photography path, scanning is performed, and all feature points of a current image are obtained; According to the current image and adjacent images, all the feature points are screened for matching feature points to obtain a matching feature point set; wherein the adjacent images at most include an upper image, a first adjacent image adjacent to a last scanning strip, a second adjacent image, and a third adjacent image; Feature point matching is performed on the set of feature points to be matched to obtain... i The first feature point pair corresponds one-to-one; where, i ≤4; to i perform processing on the first feature point pairs to obtain a first transformation matrix group, perform anomaly detection according to the first transformation matrix group to obtain j a group of effective feature point pairs and corresponding second transformation matrix groups; wherein, j ≤ i ; Based on j The local normalized correlation coefficients of the image where each matched point pair is located in each group of the effective feature point pairs are calculated, and reliability matching is performed according to the local normalized correlation coefficients to obtain j The second feature point pairs are grouped. to j performing processing on the second feature point pair group to obtain a third transformation matrix group, and obtaining a second image group according to the third transformation matrix group j rectangular overlapping regions According to each of the overlapping area rectangles and the third transformation matrix group, a normalized correlation coefficient and a structural similarity of each of the overlapping area rectangles are obtained; According to the normalized correlation coefficient and the structural similarity of each of the overlapping areas, a correct feature point pair is determined to obtain a correct feature point pair; wherein the correct feature point pair includes a first correct feature point set corresponding to the current image and a second correct feature point set corresponding to the adjacent image; The first correct feature point set and the second correct feature point set in the correct feature points are respectively spatially spliced to obtain a fourth transformation matrix of the current image relative to a spliced image space; According to the normalized correlation coefficients of all the overlapping areas, it is determined whether the matching is correct; if yes, the fourth transformation matrix is saved to obtain a final transformation matrix; if not, the third transformation matrix group is abnormally processed to obtain a new correct feature point pair, and the new correct feature point pair is spatially spliced to update and save the fourth transformation matrix to obtain the final transformation matrix.

2. The method of claim 1, wherein the method further comprises: In the step of screening all the feature points for matching feature points to obtain a matching feature point set according to the current image and adjacent images, the step comprises: According to the current image , the first adjacent image , the second adjacent image , the third adjacent image and the position relationship of the third adjacent image , and the preset scanning overlap rate, the feature points are selected to obtain the set of feature points to be matched - .

3. The method for removing mismatches due to local similarity in the background of adjacent images in oblique aerial image stitching according to claim 2, characterized in that, performing feature point matching on the to-be-matched feature point set to obtain a first feature point pair corresponding to the group one-to-one relationship i In the step of obtaining the first feature point pair corresponding to the group one-to-one relationship, the method comprises the following steps. The current image The set of feature points to be matched is respectively compared with the above image The set of feature points to be matched, the first neighboring image The set of feature points to be matched, the second neighboring image The set of feature points to be matched and the third neighboring image The set of feature points to be matched is used for feature point matching to obtain... i The first feature point pair - .

4. The method of claim 3, wherein the method further comprises: to i processing the first feature point pairs to obtain a first transformation matrix group, performing anomaly detection according to the first transformation matrix group to obtain j In the step of grouping the effective feature point pairs and the corresponding second transformation matrix group, the following steps are included: Based on the affine transformation model, the least square fitting algorithm is used to respectively process the first feature point pairs of the first group and the second group i The first transformation matrix of the current image and the first adjacent image - The first transformation matrix of the current image and the second adjacent image The first transformation matrix of the current image and the third adjacent image The first transformation matrix of the current image and the first adjacent image The first transformation matrix of the current image and the second adjacent image The first transformation matrix of the current image and the third adjacent image The first transformation matrix of the current image and the first adjacent image The first transformation matrix of the current image and the second adjacent image The first transformation matrix of the current image and the third adjacent image The first transformation matrix of the current image and the first adjacent image According to the current image a first transformation matrix of the first adjacent image a first transformation matrix of the second adjacent image a first transformation matrix of the third adjacent image a first transformation matrix of the third adjacent image a first transformation matrix of the third adjacent image a first transformation matrix of the third adjacent image a first transformation matrix of the third adjacent image a first transformation matrix of the third adjacent image ; If greater than a first preset value or less than a second preset value, greater than a first preset value or less than a second preset value, the corresponding first feature point pair is deleted - to obtain j a group of effective feature point pairs - and a second group of transformation matrices .

5. The method of claim 4, wherein the method further comprises: Based on j The step of calculating the local normalized correlation coefficient of each image in each group of the effective feature point pairs respectively, and performing reliability matching according to the local normalized correlation coefficient to obtain j In the step of grouping the second feature point pairs, the step includes: According to the effective feature points - The first s point pair, with the point pair coordinates 、 as the center, the normalized correlation coefficient calculation is performed in the range of (±M, ±N) in each image respectively to obtain the local normalized correlation coefficient of the image where each matching point pair is located. If the local normalized correlation coefficient is less than a third preset value, the matching point pair is discarded to obtain j The second feature point pair is grouped - .

6. The method of claim 5, wherein the method further comprises: to j processing the second group of point pairs to obtain a third group of transformation matrices, and obtaining the second image according to the third group of transformation matrices j In the step of grouping the overlapping area rectangles, the step comprises: Based on the affine transformation model, the least square fitting algorithm is used to respectively j groups of the second feature point pairs - Processing is performed to obtain the third transformation matrix of the current image and the first adjacent image , the third transformation matrix of the current image and the second adjacent image , the third transformation matrix of the current image and the third adjacent image , the third transformation matrix of the current image and the fourth adjacent image . According to the current image a third transformation matrix of the current image a third transformation matrix of the current image a third transformation matrix of the current image a third transformation matrix of the current image a third transformation matrix of the current image a third transformation matrix of the current image a third transformation matrix of the current image a third transformation matrix of the current image ; According to the third transformation matrix group The overlapping areas between the current image , the upper image , the first adjacent image , the second adjacent image and the third adjacent image are calculated respectively, to obtain j The overlapping area rectangle group.

7. The method for removing mismatches due to local similarity in the background of adjacent images in oblique aerial image stitching according to claim 6, characterized in that, In the step of obtaining the normalized correlation coefficient and the structural similarity of each of the overlapping area rectangles according to each of the overlapping area rectangles and the third transformation matrix group, the step comprises: Taking one of the images in each of the overlapping areas as a reference, corresponding pixels in the other image are found based on the third transformation matrix group, and the normalized correlation coefficient of each of the overlapping area rectangles is obtained by using a normalized correlation coefficient equation; Taking one of the images in each of the overlapping area rectangles as a reference, corresponding pixels in the other image are found based on the third transformation matrix group, and the structural similarity of each of the overlapping area rectangles is obtained by using a structural similarity calculation equation.

8. The method of claim 7, wherein the method further comprises: In the step of determining the correct feature point pair according to the normalized correlation coefficient and the structural similarity of each of the overlapping areas, the step comprises: If the normalized correlation coefficient is less than a third preset value and the structural similarity is less than a fourth preset value, it is determined that the matching is incorrect, the corresponding matching point pair is discarded, and k pairs of correct matching images and k groups of the correct feature point pairs are obtained - .

9. The method for removing mismatches due to local similarity in the background of adjacent images in oblique aerial image stitching according to claim 8, characterized in that, In the step of spatially splicing the first correct feature point set and the second correct feature point set in the correct feature points to obtain the fourth transformation matrix of the current image relative to the spliced image space, the step comprises: merging coordinates of the first correct feature point set into a first set ; According to the final transformation matrix of the previous figure Transform the coordinates of the second set of correct feature points into the stitching image space and merge into a second set ; based on the first set and the second set , using the affine transformation model, the affine transformation parameters between - the fourth transformation matrix of the current image relative to the mosaic image space are obtained by using a least square fitting algorithm.

10. The method of claim 9, wherein the method further comprises: According to the normalized correlation coefficients of all the overlapping areas, it is determined whether the matching is correct; if yes, the fourth transformation matrix is saved to obtain a final transformation matrix; if not, the third transformation matrix group is abnormally processed to obtain a new correct feature point pair, and the new correct feature point pair is spatially spliced to update and save the fourth transformation matrix to obtain the final transformation matrix. determining whether the normalized correlation coefficients of all the overlapping region images are greater than a fifth preset value, if yes, the matching is correct, saving the fourth transformation matrix of the current image to obtain a final transformation matrix, and processing the next image. If not, the matching fails, and the fourth transformation matrix of the current image is counted the final transformation matrix of the previous frame image translation abnormal case; If normal, save the fourth transformation matrix of the current image and process the next image; wherein, If the error absolute value of the correct feature point pair is greater than the sixth preset value, or If the error absolute value of the correct feature point pair is greater than the sixth preset value, or If the error absolute value of the correct feature point pair is greater than the sixth preset value, or If the error absolute value of the correct feature point pair is greater than the sixth preset value, or If the error absolute value of the correct feature point pair is greater than the sixth preset value, or If the error absolute value of the correct feature point pair is greater than the sixth preset value, or If the error absolute value of the correct feature point pair is greater than the sixth preset value, one group of matching results in the correct feature point pair - The other group of matching feature points in the correct feature point pair - If the error is greater than half of the feature point number of more than one pixel, it is considered that the matching result is abnormal, and the group of feature points is excluded - All matching results are traversed, and normal matching results are retained - ; spatially stitching the normal matching results to obtain the fourth transformation matrix of the current image relative to a stitched image space storing the fourth transformation matrix to obtain a final transformation matrix, and processing the next image.

Images