Method for eliminating background local similarity mismatching of adjacent images in oblique aerial image splicing

Abstract

The invention belongs to the technical field of photoelectric detection. The invention provides a method for eliminating background local similarity mismatching of adjacent images in oblique aerial image splicing. According to the embodiment of the invention, on the basis of feature point matching, the feature points are further screened, and the matching precision is improved. And obtaining an overlapping region of the two images based on a position relation of feature point matching. Under the condition of normal matching, the content of the overlapping area is consistent, and under the condition of abnormal matching, the overlapping area is only locally consistent, so that calculation is carried out by calculating the gray similarity and the structural similarity of the images of the overlapping area, and wrong matching is eliminated. And further verifying whether wrong matching exists or not by adopting the matching result of the current image and the plurality of surrounding images, and finally improving the matching accuracy and the splicing effect.

Description

Technical Field

[0001] The disclosed embodiments relate to the field of photoelectric detection technology, and in particular to a method for eliminating local similarity mismatches of the backgrounds of adjacent images in splicing oblique aerial images. Background Art

[0002] The imaging field of view of the detector of aerial images is very small, and can only cover a small range of the observation area. In order to obtain the overall image of a large area, there is usually overlap in the position of adjacent detector imaging areas, thereby providing features of the same area to adjacent images, and using image processing methods to achieve image matching, and finally obtain the position relationship between the entire scanning sequence images and splice them into a large picture.

[0003] However, in the process of large-scale observation, similar features may exist in adjacent images, but they are not in the same area. For example, in a densely populated building area, the buildings in adjacent images are almost the same, which will lead to mismatching, making the entire subsequent matching completely wrong and affecting the stitching effect. From the current research, the main direction of researchers is to extract features, match and splice. There is no research on the above abnormal situations in splicing.

[0004] Therefore, it is necessary to improve one or more problems existing in the above-mentioned related technical solutions.

[0005] It should be noted that this section is intended to provide background or context for the technical solutions of the present disclosure stated in the claims. The description herein is not admitted to be prior art by virtue of being included in this section. Summary of the invention

[0006] In order to avoid the shortcomings of the prior art, the present invention provides a method for eliminating local similar mismatching of the backgrounds of adjacent images in the stitching of oblique aerial images, so as to solve the problem in the prior art that during a large-scale observation, similar features may exist in adjacent images, but they are not in the same area, which will lead to mismatching, making the subsequent entire matching completely wrong and affecting the stitching effect.

[0007] According to an embodiment of the present disclosure, a method for eliminating local similarity mismatching of backgrounds of adjacent images in tilted aerial image stitching is provided, the method comprising: Scan according to the preset aerial photography path and obtain all feature points of the current image; Screening all the feature points to be matched according to the current image and the adjacent images to obtain a set of feature points to be matched; wherein the adjacent images at most include the previous image, the first adjacent image, the second adjacent image and the third adjacent image adjacent to the previous scanning strip; Perform feature point matching on the feature point set to be matched, and obtaini The first feature point pairs of a group correspond to each other; i ≤4; right i The first feature point pairs are processed to obtain a first transformation matrix group, and anomaly detection is performed according to the first transformation matrix group to obtain j A group of valid feature point pairs and their corresponding second transformation matrix groups; wherein, j ≤ i ; based on j The effective feature point pairs are grouped together to calculate the local normalized correlation coefficient of the image where each matching point pair is located in each group of the effective feature point pairs, and reliability matching is performed according to the local normalized correlation coefficient to obtain j Group the second feature point pair; right j The second feature point pairs are processed to obtain a third transformation matrix group, and a third transformation matrix group is obtained according to the third transformation matrix group. j Group overlapping area rectangle; According to each of the overlapping area rectangles and the third transformation matrix group, a normalized correlation coefficient and a structural similarity of each of the overlapping area rectangles are obtained; Determine according to the normalized correlation coefficient and the structural similarity of each overlapping area respectively to obtain a correct feature point pair; wherein the correct feature point pair includes a first correct feature point set corresponding to the current image and a second correct feature point set corresponding to the adjacent image; spatially stitching the first correct feature point set and the second correct feature point set in the correct feature points respectively, so as to obtain a fourth transformation matrix of the current image relative to the stitched image space; Whether the match is correct is determined based on the normalized correlation coefficients of all the overlapping areas; if so, the fourth transformation matrix is ​​saved to obtain the final transformation matrix; if not, the third transformation matrix group is subjected to exception processing to obtain new correct feature point pairs, and the new correct feature point pairs are spatially spliced ​​to update and save the fourth transformation matrix to obtain the final transformation matrix.

[0008] Furthermore, the step of screening all the feature points to be matched according to the current image and the adjacent images to obtain a set of feature points to be matched includes: According to the current image Compared with the previous image , the first adjacent image , the second adjacent image and the third adjacent image The position relationship and the preset scanning overlap rate are used to select feature points to obtain the feature point set to be matched. - .

[0009] Furthermore, feature point matching is performed on the feature point set to be matched to obtain i The step of forming a one-to-one correspondence between first feature point pairs comprises: The current image The feature point sets to be matched are respectively The set of feature points to be matched, the first adjacent image The set of feature points to be matched, the second adjacent image The feature point set to be matched and the third adjacent image The feature point set to be matched is matched to obtain i The first feature point pair - .

[0010] Further, yes i The first feature point pairs are processed to obtain a first transformation matrix group, and anomaly detection is performed according to the first transformation matrix group to obtain j The step of forming a group of valid feature point pairs and their corresponding second transformation matrix groups includes: Based on the affine transformation model, the least squares fitting algorithm is used to i The first feature point pair - Processing is performed to obtain the current image With the above image The first transformation matrix, the current image The first adjacent image The first transformation matrix, the current image The second adjacent image The first transformation matrix, the current image The third adjacent image The first transformation matrix of According to the current image With the above image The first transformation matrix, the current image The first adjacent image The first transformation matrix, the current image The second adjacent image The first transformation matrix, the current image The third adjacent image The first transformation matrix is ; like Greater than a first preset value or is less than a second preset value, Greater than a first preset value or is less than a second preset value, the corresponding first feature point pair is deleted. - , to get j The valid feature point pairs - And the second transformation matrix group .

[0011] Further, based on j The effective feature point pairs are grouped together to calculate the local normalized correlation coefficient of the image where each matching point pair is located in each group of the effective feature point pairs, and reliability matching is performed according to the local normalized correlation coefficient to obtain j The step of forming a second feature point pair includes: According to the effective feature point pair - No. s Point pair, with the coordinates of the point pair , Centered on the image, the normalized correlation coefficient is calculated in the range of (±M, ±N) in each image to obtain the local normalized correlation coefficient of each matching point to the image. If the local normalized correlation coefficient is less than the third preset value, the matching point pair is discarded to obtain j The second feature point pair - .

[0012] Further, yes j The second feature point pairs are processed to obtain a third transformation matrix group, and a third transformation matrix group is obtained according to the third transformation matrix group. j The step of forming overlapping area rectangles includes: Based on the affine transformation model, the least squares fitting algorithm is used to respectively group the second feature point pairs into j groups. - Processing is performed to obtain the current image With the above image The third transformation matrix, the current image The first adjacent image The third transformation matrix, the current image The second adjacent image The third transformation matrix, the current image The third adjacent image The third transformation matrix of According to the current image With the above image The third transformation matrix, the current image The first adjacent image The third transformation matrix, the current image The second adjacent image The third transformation matrix, the current image The third adjacent image The third transformation matrix is ​​obtained by ; According to the third transformation matrix group Calculate the current image separately , the above image , the first adjacent image , the second adjacent image and the third adjacent image The overlapping area between the two j Group the overlapping area rectangles.

[0013] Furthermore, the step of obtaining the normalized correlation coefficient and the structural similarity of each of the overlapping area rectangles according to each of the overlapping area rectangles and the third transformation matrix group includes: Taking one image in each overlapping area as a reference, based on the third transformation matrix group, searching for corresponding pixels in another image, and using a normalized correlation coefficient equation to obtain the normalized correlation coefficient of each overlapping area rectangle; Taking one image in each overlapping area rectangle as a reference, based on the third transformation matrix group, searching for corresponding pixels in another image, and using a structural similarity calculation equation to obtain the structural similarity of each overlapping area rectangle.

[0014] Furthermore, the step of determining according to the normalized correlation coefficient and the structural similarity of each overlapping area to obtain a correct feature point pair includes: If the normalized correlation coefficient is less than the third preset value, and the structural similarity is less than the fourth preset value, it is judged as a mismatch, and the corresponding matching point pair is discarded to obtain k pairs of correct matching images and k groups of correct feature point pairs. - .

[0015] Further, the step of spatially splicing the first correct feature point set and the second correct feature point set in the correct feature points respectively to obtain a fourth transformation matrix of the current image relative to the spliced ​​image space includes: The first correct feature point set The coordinates of ; According to the final transformation matrix of the previous picture The second correct feature point set The coordinates of are transformed into the stitched image space and merged into the second set ; Based on the first set And the second set , using the affine transformation model and the least squares fitting algorithm, we get - The affine transformation parameters between the current image and the stitched image space, that is, the fourth transformation matrix of the current image relative to the stitched image space .

[0016] Further, judging whether the match is correct according to the normalized correlation coefficient of all the overlapping areas; if so, saving the fourth transformation matrix to obtain the final transformation matrix; if not, performing exception processing on the third transformation matrix group to obtain a new correct feature point pair, and spatially splicing the new correct feature point pair to update and save the fourth transformation matrix to obtain the final transformation matrix, the step includes: Determine whether the normalized correlation coefficients of all overlapping area images are greater than the fifth preset value. If so, the match is correct, and the fourth transformation matrix of the current image is Save to get the final transformation matrix and process the next image; If not, the match fails, and the fourth transformation matrix of the current image is calculated. The final transformation matrix of the previous frame image Abnormal translation amount; If normal, the fourth transformation matrix of the current image is Save and process the next picture; If abnormal, and and The absolute value of the error is greater than the sixth preset value, or and The absolute value of the error is greater than the sixth preset value, or and If the absolute value of the error is greater than the sixth preset value, the correct feature point pair is selected. - A set of matching results in , the correct feature point pair - Substitute the other groups of matching feature points in for verification. If the error is greater than If the number of feature points of pixels is more than half, the current matching result is considered If the matching is abnormal, remove the feature points in this group, traverse all matching results, and keep the normal matching results - ; The normal matching results are spatially stitched to obtain the fourth transformation matrix of the current image relative to the stitched image space. , save the fourth transformation matrix , to get the final transformation matrix and process the next image.

[0017] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects: In the embodiment of the present disclosure, through the above-mentioned method for eliminating local similarity mismatches of the backgrounds of adjacent images in the splicing of oblique aerial images, on the one hand, on the basis of feature point matching, the feature points are further screened to improve the matching accuracy. Based on the positional relationship of feature point matching, the overlapping area of ​​the two images is obtained. In the case of normal matching, the content of the overlapping area is consistent, while in the case of abnormal matching, the overlapping area is only partially consistent. Therefore, the grayscale similarity and structural similarity of the overlapping area images are calculated to eliminate mismatches. The matching results of the current image and multiple surrounding images are used to further verify whether there are mismatches, and finally improve the matching accuracy and splicing effect. On the other hand, based on the feature point matching, by judging the grayscale similarity and structural similarity of the corresponding area after matching, and combining the consistency of the transformation parameters of the current image and the neighborhood image, it is finally confirmed whether the result of the current feature point matching is reliable, and the mismatch relationship is eliminated. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0019] Figure 1 A step diagram showing a method for eliminating local similarity mismatches of the backgrounds of adjacent images in the stitching of oblique aerial images in an exemplary embodiment of the present disclosure is shown; Figure 2A schematic diagram showing the positional relationship among a current image, a previous image, a first adjacent image, a second adjacent image, and a third adjacent image in an exemplary embodiment of the present disclosure; Figure 3 A schematic diagram showing an overlapping area rectangle and an image coordinate system in an exemplary embodiment of the present disclosure is shown; Figure 4 A schematic diagram showing the matching of feature points of two images in an exemplary embodiment of the present disclosure; Figure 5 An abnormal diagram of the transformation matrix after matching in an exemplary embodiment of the present disclosure is shown; Figure 6 A graph showing abnormal matching results of normalized correlation coefficient and structural similarity in an exemplary embodiment of the present disclosure is shown; Figure 7 A diagram showing a matching result of a current image and a left image in an exemplary embodiment of the present disclosure; Figure 8 A matching result diagram of the current image and the upper left corner image in an exemplary embodiment of the present disclosure is shown; Fig. 9 A diagram showing the matching result of the current image and the lower left corner image in an exemplary embodiment of the present disclosure; Fig.10 A diagram showing the matching results of the current image and the previous image in an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0020] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0021] In addition, the accompanying drawings are only schematic illustrations of the embodiments of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated descriptions will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities.

[0022] This example implementation provides a method for eliminating local similarity mismatches of the backgrounds of adjacent images in the splicing of oblique aerial images. Figure 1 As shown in , the method for eliminating local similarity mismatching of backgrounds of adjacent images in the splicing of oblique aerial images may include: steps S101 to S110.

[0023] Step S101: Scanning according to a preset aerial photography path and obtaining all feature points of the current image; Step S102: screening all the feature points for matching according to the current image and the adjacent images to obtain a set of feature points for matching; wherein the adjacent images at most include the previous image, the first adjacent image, the second adjacent image and the third adjacent image adjacent to the previous scanning strip; Step S103: Perform feature point matching on the feature point set to be matched to obtain i The first feature point pairs of a group correspond to each other; i ≤4; Step S104: i The first feature point pairs are processed to obtain a first transformation matrix group, and anomaly detection is performed according to the first transformation matrix group to obtain j A group of valid feature point pairs and their corresponding second transformation matrix groups; wherein, j ≤ i ; Step S105: Based on j The effective feature point pairs are grouped together to calculate the local normalized correlation coefficient of the image where each matching point pair is located in each group of the effective feature point pairs, and reliability matching is performed according to the local normalized correlation coefficient to obtain j Group the second feature point pair; Step S106: j The second feature point pairs are processed to obtain a third transformation matrix group, and a third transformation matrix group is obtained according to the third transformation matrix group. j Group overlapping area rectangle; Step S107: obtaining the normalized correlation coefficient and structural similarity of each of the overlapping area rectangles according to each of the overlapping area rectangles and the third transformation matrix group; Step S108: determining according to the normalized correlation coefficient and the structural similarity of each overlapping area respectively to obtain a correct feature point pair; wherein the correct feature point pair includes a first correct feature point set corresponding to the current image and a second correct feature point set corresponding to the adjacent image; Step S109: spatially stitching the first correct feature point set and the second correct feature point set in the correct feature points respectively, so as to obtain a fourth transformation matrix of the current image relative to the stitched image space; Step S110: Determine whether the match is correct based on the normalized correlation coefficients of all the overlapping areas; if so, save the fourth transformation matrix to obtain the final transformation matrix; if not, perform exception processing on the third transformation matrix group to obtain new correct feature point pairs, and perform spatial splicing on the new correct feature point pairs to update and save the fourth transformation matrix to obtain the final transformation matrix.

[0024] Through the above-mentioned method for eliminating local similarity mismatches of the background of adjacent images in the splicing of oblique aerial images, on the one hand, on the basis of feature point matching, the feature points are further screened to improve the matching accuracy. Based on the positional relationship of feature point matching, the overlapping area of ​​the two images is obtained. In the case of normal matching, the content of the overlapping area is consistent, while in the case of abnormal matching, the overlapping area is only partially consistent. Therefore, the grayscale similarity and structural similarity of the overlapping area images are calculated to eliminate mismatches. The matching results of the current image and multiple surrounding images are used to further verify whether there are mismatches, and finally improve the matching accuracy and splicing effect. On the other hand, based on feature point matching, by judging the grayscale similarity and structural similarity of the corresponding area after matching, and combining the consistency of the transformation parameters of the current image and the neighborhood image, it is finally confirmed whether the result of the current feature point matching is reliable and the mismatch relationship is eliminated.

[0025] Next, we will refer to Figures 1 to 10 The various steps of the method for eliminating local similarity mismatches of the backgrounds of adjacent images in the above-mentioned inclined aerial image stitching in this example implementation are described in more detail.

[0026] In step S101 to step S103, according to the preset scanning path, the current input image Extract the feature points of the entire image .like Figure 2 As described above, scan in the order of 1, 2, 3, 4, 5.

[0027] For the current image The 3 images adjacent to the previous scan strip , , Screen the feature points to be matched and and the previous image of the current strip Screen the feature points to be matched (see the schematic diagram of the position relationship Figure 1 ); If the current image is the first image of the first scanning strip ( Figure 2 1) in the figure, then save all its feature points and proceed to process the second image; If the current image is the image in the first scan strip ( Figure 2 2, 3) in , only the current image and the previous image of the current strip are processed. Screening feature points to be matched; If the current image is the first image of a new strip ( Figure 2 4), only the feature points to be matched are screened for the current image and the two images adjacent to the previous scan strip; When screening feature points, feature points are selected based on the positional relationship between images and the set scanning overlap rate; Assuming that the scan overlap rate P in the row and column directions is set to 30%, take the back and forth sweep perpendicular to the flight direction of the aircraft as an example. and For the up-down relationship, select P+10% of the feature points in the up-down overlapping area; and For left-right relations, select P+10% of the feature points in the left-right overlapping area; , and Only one corner overlaps, select feature points within the upper and lower, left and right overlapping areas P + 10%, and get n sets of feature points - .

[0028] For the obtained n sets of feature points - , n≤4 (the number of n is related to the image position relationship) to match the feature points and obtain the one-to-one corresponding feature point pairs after matching - ;if - If the number of feature point pairs in is less than 5, then the set of point pairs is discarded, and finally i sets of feature point pairs are obtained. - , i≤n; In step S104, the obtained i groups of feature point pairs are - , respectively, using the affine transformation model and the least squares algorithm, we get Respectively , , , The transformation matrix ;if >1.4 or <0.7, >1.4 or <0.7, the parameters of the current two images are abnormal, and the two images are considered to be mismatched. The feature point pairs of the two images are deleted. - , and finally get j groups of effective feature point pairs - , j≤i.

[0029] The affine transformation model formula is: (1) in, (2) It represents the rotation, scaling, and translation relationship between the two images. Formula (1) is expanded to: (3) (4) For example Points in , express The points, through the matrix After calculation, the corresponding matching image feature point pairs will be obtained The feature point coordinates in ;because To satisfy - The mapping relationship of all feature points in the point pair, so We need to use the least squares method to - All points in this group are fitted.

[0030] In step S105, for j groups of valid feature point pairs - , respectively calculate the local normalized correlation coefficient of each corresponding point in each set of feature point pairs. If the normalized correlation coefficient obtained is less than 0.65, then discard this point pair in the group to further improve the matching reliability. Finally, the feature point pair is obtained. - ; When calculating, the feature points are - The s point pairs, with the coordinates of the point pairs , Centered on the image, the normalized correlation coefficient is calculated in the range of (±M, ±N) in each image; (5) in: , The feature point pair - The local image in the range (±M, ±N) corresponding to each pair of feature points, , is the corresponding local image coordinate The pixel gray level at , They are , The grayscale mean.

[0031] In step S106, the j groups of feature point pairs obtained after screening are - , respectively, using the affine transformation model and the least squares algorithm, we get and , , , The transformation matrix ; Set here (6) express yes Mappings to other images.

[0032] In step S107, according to the , respectively calculate and , , , Match the overlapping areas between the two images (see the overlapping area Figure 3 Shaded area), get j groups of overlapping area rectangles; by Figure 3 For example, Figure 3 1 means , 2 means . Then Substitute the coordinates of the upper left corner of the image (0,0) into the right side of equation (6) to obtain The upper left corner coordinates (0,0) correspond to Coordinates in ;calculate The inverse matrix ,Will The coordinates of the lower left corner of the image Bring in ,get exist The corresponding coordinates in the image . Thus we can get (0,0) is the upper left corner, is the area in the lower right corner, China-Israel is the upper left corner, The area in the lower right corner is the corresponding overlapping area; See the image coordinate system diagram for Figure 3 , for The coordinates of the lower right corner can be obtained from the width and height of the input image; For the overlapping area, the normalized correlation coefficient between the two is calculated, see formula (7); the image coordinates are based on one image, and then the calculated transformation matrix is , find the corresponding pixel in the other picture; (7) set up yes The coordinates of the overlapping area are , then the corresponding points in the overlapping area of ​​the matching image need to be Convert it to get , the grayscale mean of the overlapping area It also needs to be based on Calculated; if Outside the overlapping area, , Does not participate in the operation; For the overlapping area, the structural similarity of the two is calculated. When calculating, the image coordinates are based on one image and the calculated transformation matrix is ​​used. , find the corresponding pixel in the other image, the method is the same as above; Structural similarity calculation formula:

[0033] in: , is the mean of the overlapping area, , are the mean square errors of the overlapping areas, is the covariance of the overlapping area, , is a constant, , ; In step S108, if the normalized correlation coefficient is less than 0.65 and the structural similarity is less than 0.85, the two images are currently mismatched, and the matching point pairs of the two images are discarded. Finally, the correct matching image k pairs are obtained, and the feature point pair set is - , the transformation matrix ; In step S109, the feature points obtained in step S108 are According to the transformation matrix obtained by each splicing , transform the coordinates of each feature point into the stitching image space and merge them into a set , and also the feature points Also merged into one set , using the affine transformation model and the least squares fitting algorithm, we get - The affine transformation parameters between the current image Transformation matrix relative to the stitched image space ;

[0034] Feature Points The formula for converting to the stitching image space is: = × (8) In step S110, if the normalized correlation coefficients of all overlapping area images in step S109 are greater than 0.9, the match is correct and step S1104 is executed; otherwise, step S1101 is executed.

[0035] Step S1101: Count the current image transformation matrix Transformation matrix with the previous frame image The translation amount of (that is, the fourth transformation matrix of the previous frame image) is abnormal: Transformation matrix of the first frame image after scanning reversal , if the transformation matrix of the last frame of the previous strip is The difference in translation in the x and y directions abs ( - )、abs( - ) are respectively greater than the image width and height, the image matching is considered abnormal.

[0036] Transformation matrix of other images The transformation matrix of the previous frame image in the current strip is The difference in translation in the x and y directions abs ( - )、abs( - ) are respectively greater than the image width and height, the image matching is considered abnormal.

[0037] Step S1102: If there is an abnormal match in step S1101, and and The absolute value of the error is > 0.2, or and The absolute value of the error is > 0.2, or and If the absolute value of the error is > 0.2, the feature point pairs finally obtained in each region are further screened: choose - A set of matching results in ,Will - Substitute the other groups of matching feature points in for verification. If the error is greater than If the number of feature points of pixels is more than half, then the current If the matching is abnormal, remove the feature points and keep the normal matching results. - .

[0038] The verification method is as follows: Other The coordinates of each feature point in Substitute into the formula On the right side, q represents the qth feature point, then the left side is obtained according to The calculated corresponding matching point coordinates ,because represents the mapping of all feature points, so and The real feature points extracted from There is a certain error ,calculate ,if > , then count +1, and finally if > The total number of points exceeds half of the points involved in the verification, then it is considered Matching exception.

[0039] Step S1103: The feature points obtained in step S1102 are calculated according to the method in step S109 to obtain the current image The final transformation matrix relative to the stitched image space .

[0040] Step S1104: Transform the current image Save and go to step S101 to continue processing.

[0041] In a specific embodiment, Figure 4 The two images shown are subjected to feature extraction and feature point matching.

[0042] The image resolution used for stitching is 1280*1024: Calculate the SIFT feature points of the first input image, and get a total of 2537 points; Calculate the SIFT feature points of the second input image, and get a total of 2013 points; The two images are in an upper and lower relationship, and the overlap rate is set to 30%. The lower 40% area of ​​the first image, that is, the area below line 614, has 1153 feature points in total, and the upper 40% area of ​​the second image, that is, the area above line 410, has 1063 feature points in total. After matching, there are 150 matching point pairs, and the cross line connects a pair of points.

[0043] After matching, the two images are transformed into matrices , in, , All meet the requirements and are normal matches.

[0044] In a specific embodiment, Figure 5 As shown, Figure 5 The two images are stitched together, but because there is no overlapping area, the building stitching is wrong. The transformation matrix after stitching is: , , If the requirement of step S104 is not met, it is considered as a mismatch.

[0045] In a specific embodiment, Figure 6 As shown, Figure 6 is the matching result of two images. Figure 6 The upper left corner of the left image in Figure 6 The lower right corner of the right image in the figure is matched. According to the distribution of feature points and the positional relationship of the images, the overlapping area is calculated and shown in the white box. It can be seen that the matching is wrong due to the influence of similar buildings. Although the feature points of the abnormal match are locally similar, the background of the entire overlapping area is different. The calculated normalized correlation coefficient = 0.180868 and the structural similarity = 0.417972, so the area can be eliminated as an abnormal match.

[0046] In a specific embodiment, Figures 7 to 10 As shown, Figures 7 to 10 The matching result is the current image, the three images on the left, and the image on the current column. Figures 7 to 10 The right image is the current image. After matching, their respective transformation matrices are shown in H1-H4: The matching results of the current image and the left image are as follows: Figure 7 As shown, its transformation matrix is: .

[0047] The matching results of the current image and the upper left corner image are as follows Figure 8 As shown, its transformation matrix is: .

[0048] The matching results of the current image and the lower left corner image are as follows Fig. 9 As shown, its transformation matrix is: .

[0049] The matching results of the current image and the previous image are as follows: Fig.10 As shown, its transformation matrix is: .

[0050] It can be seen that due to Figure 8 Affected by similar buildings, matching errors and abnormal parameters lead to the whole image abnormal.

[0051] According to step S1101, compare the current image Transformation matrix with the previous frame image The amount of translation,

[0052]

[0053] abs( - ) = 342.117304 abs( - ) = 1480.222986 > image height 1024.

[0054] If the requirement of step S1101 is not met, the process goes to step S1102. and , and The absolute values ​​of the differences are all greater than 0.2, indicating an abnormality. According to the method in step S1102, the Figure 8 Abnormal feature points, final matching results: , thus correcting the matching anomaly.

[0055] Through the above-mentioned method for eliminating local similarity mismatches of the background of adjacent images in the splicing of oblique aerial images, on the one hand, on the basis of feature point matching, the feature points are further screened to improve the matching accuracy. Based on the positional relationship of feature point matching, the overlapping area of ​​the two images is obtained. In the case of normal matching, the content of the overlapping area is consistent, while in the case of abnormal matching, the overlapping area is only partially consistent. Therefore, the grayscale similarity and structural similarity of the overlapping area images are calculated to eliminate mismatches. The matching results of the current image and multiple surrounding images are used to further verify whether there are mismatches, and finally improve the matching accuracy and splicing effect. On the other hand, based on feature point matching, by judging the grayscale similarity and structural similarity of the corresponding area after matching, and combining the consistency of the transformation parameters of the current image and the neighborhood image, it is finally confirmed whether the result of the current feature point matching is reliable and the mismatch relationship is eliminated.

[0056] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present disclosure, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0057] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification.

[0058] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any modification, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the appended claims.

Claims

1. A method for eliminating local similarity mismatching of backgrounds of adjacent images in the splicing of oblique aerial images, characterized in that: The method includes: Scan according to the preset aerial photography path and obtain all feature points of the current image; Screening all the feature points to be matched according to the current image and the adjacent images to obtain a set of feature points to be matched; wherein the adjacent images at most include the previous image, the first adjacent image, the second adjacent image and the third adjacent image adjacent to the previous scanning strip; Perform feature point matching on the feature point set to be matched, and obtain i The first feature point pairs of a group correspond to each other; i ≤4; right i The first feature point pairs are processed to obtain a first transformation matrix group, and anomaly detection is performed according to the first transformation matrix group to obtain j A group of valid feature point pairs and their corresponding second transformation matrix groups; wherein, j ≤ i ; based on j The effective feature point pairs are grouped together to calculate the local normalized correlation coefficient of the image where each matching point pair is located in each group of the effective feature point pairs, and reliability matching is performed according to the local normalized correlation coefficient to obtain j Group the second feature point pair; right j The second feature point pairs are processed to obtain a third transformation matrix group, and a third transformation matrix group is obtained according to the third transformation matrix group. j Group overlapping area rectangle; According to each of the overlapping area rectangles and the third transformation matrix group, a normalized correlation coefficient and a structural similarity of each of the overlapping area rectangles are obtained; Determine according to the normalized correlation coefficient and the structural similarity of each overlapping area respectively to obtain a correct feature point pair; wherein the correct feature point pair includes a first correct feature point set corresponding to the current image and a second correct feature point set corresponding to the adjacent image; spatially splicing the first correct feature point set and the second correct feature point set in the correct feature points respectively, so as to obtain a fourth transformation matrix of the current image relative to the spliced ​​image space; Whether the match is correct is determined based on the normalized correlation coefficients of all the overlapping areas; if so, the fourth transformation matrix is ​​saved to obtain the final transformation matrix; if not, the third transformation matrix group is subjected to exception processing to obtain new correct feature point pairs, and the new correct feature point pairs are spatially spliced ​​to update and save the fourth transformation matrix to obtain the final transformation matrix.

2. The method for eliminating local similarity mismatching of backgrounds of adjacent images in the splicing of oblique aerial images according to claim 1, characterized in that: The step of screening all the feature points to be matched according to the current image and the adjacent images to obtain a set of feature points to be matched includes: According to the current image Compared with the previous image , the first adjacent image , the second adjacent image and the third adjacent image The position relationship and the preset scanning overlap rate are used to select feature points to obtain the feature point set to be matched. - .

3. The method for eliminating local similarity mismatching of backgrounds of adjacent images in the splicing of oblique aerial images according to claim 2, characterized in that: Perform feature point matching on the feature point set to be matched, and obtain i The step of forming a one-to-one correspondence between first feature point pairs comprises: The current image The feature point sets to be matched are respectively The set of feature points to be matched, the first adjacent image The set of feature points to be matched, the second adjacent image The feature point set to be matched and the third adjacent image The feature point set to be matched is matched to obtain i The first feature point pair - .

4. The method for eliminating local similarity mismatching of backgrounds of adjacent images in the splicing of oblique aerial images according to claim 3, characterized in that: right i The first feature point pairs are processed to obtain a first transformation matrix group, and anomaly detection is performed according to the first transformation matrix group to obtain j The step of forming a group of valid feature point pairs and their corresponding second transformation matrix groups includes: Based on the affine transformation model, the least squares fitting algorithm is used to i The first feature point pair - Processing is performed to obtain the current image With the above image The first transformation matrix, the current image The first adjacent image The first transformation matrix, the current image The second adjacent image The first transformation matrix, the current image The third adjacent image The first transformation matrix of According to the current image With the above image The first transformation matrix, the current image The first adjacent image The first transformation matrix, the current image The second adjacent image The first transformation matrix, the current image The third adjacent image The first transformation matrix is ; like Greater than a first preset value or is less than a second preset value, Greater than a first preset value or is less than a second preset value, the corresponding first feature point pair is deleted. - , to get j The valid feature point pairs - And the second transformation matrix group .

5. The method for eliminating local similarity mismatching of backgrounds of adjacent images in the splicing of oblique aerial images according to claim 4, characterized in that: based on j The effective feature point pairs are grouped together to calculate the local normalized correlation coefficient of the image where each matching point pair is located in each group of the effective feature point pairs, and reliability matching is performed according to the local normalized correlation coefficient to obtain j The step of forming a second feature point pair includes: According to the effective feature point pair - No. s Point pair, with the coordinates of the point pair , Centered on the image, the normalized correlation coefficient is calculated in the range of (±M, ±N) in each image to obtain the local normalized correlation coefficient of each matching point to the image. If the local normalized correlation coefficient is less than the third preset value, the matching point pair is discarded to obtain j The second feature point pair - .

6. The method for eliminating local similarity mismatching of backgrounds of adjacent images in the splicing of oblique aerial images according to claim 5, characterized in that: right j The second feature point pairs are processed to obtain a third transformation matrix group, and a third transformation matrix group is obtained according to the third transformation matrix group. j The step of forming overlapping area rectangles includes: Based on the affine transformation model, the least squares fitting algorithm is used to respectively group the second feature point pairs into j groups. - Processing is performed to obtain the current image With the above image The third transformation matrix, the current image The first adjacent image The third transformation matrix, the current image The second adjacent image The third transformation matrix, the current image The third adjacent image The third transformation matrix of According to the current image With the above image The third transformation matrix, the current image The first adjacent image The third transformation matrix, the current image The second adjacent image The third transformation matrix, the current image The third adjacent image The third transformation matrix is ​​obtained by ; According to the third transformation matrix group Calculate the current image separately , the above image , the first adjacent image , the second adjacent image and the third adjacent image The overlapping area between the two j Group the overlapping area rectangles.

7. The method for eliminating local similarity mismatching of backgrounds of adjacent images in the splicing of oblique aerial images according to claim 6, characterized in that: The step of obtaining the normalized correlation coefficient and the structural similarity of each of the overlapping area rectangles according to each of the overlapping area rectangles and the third transformation matrix group includes: Taking one image in each overlapping area as a reference, based on the third transformation matrix group, searching for corresponding pixels in another image, and using a normalized correlation coefficient equation to obtain the normalized correlation coefficient of each overlapping area rectangle; Taking one image in each overlapping area rectangle as a reference, based on the third transformation matrix group, searching for corresponding pixels in another image, and using a structural similarity calculation equation to obtain the structural similarity of each overlapping area rectangle.

8. The method for eliminating local similarity mismatches of the backgrounds of adjacent images in the splicing of oblique aerial images according to claim 7, characterized in that: The step of determining according to the normalized correlation coefficient and the structural similarity of each overlapping area to obtain a correct feature point pair includes: If the normalized correlation coefficient is less than the third preset value, and the structural similarity is less than the fourth preset value, it is judged as a mismatch, and the corresponding matching point pair is discarded to obtain k pairs of correct matching images and k groups of correct feature point pairs. - .

9. The method for eliminating local similarity mismatches of the backgrounds of adjacent images in the splicing of oblique aerial images according to claim 8, characterized in that: The step of spatially splicing the first correct feature point set and the second correct feature point set in the correct feature points respectively to obtain a fourth transformation matrix of the current image relative to the spliced ​​image space comprises: The first correct feature point set The coordinates of ; According to the final transformation matrix of the previous picture The second correct feature point set The coordinates of are transformed into the stitched image space and merged into the second set ; Based on the first set And the second set , using the affine transformation model and the least squares fitting algorithm, we get - The affine transformation parameters between the current image and the stitched image space, that is, the fourth transformation matrix of the current image relative to the stitched image space .

10. The method for eliminating local similarity mismatches of the backgrounds of adjacent images in the splicing of oblique aerial images according to claim 9, characterized in that: The step of judging whether the match is correct according to the normalized correlation coefficients of all the overlapping areas; if so, saving the fourth transformation matrix to obtain the final transformation matrix; if not, performing exception processing on the third transformation matrix group to obtain new correct feature point pairs, and spatially splicing the new correct feature point pairs to update and save the fourth transformation matrix to obtain the final transformation matrix includes: Determine whether the normalized correlation coefficients of all overlapping area images are greater than the fifth preset value. If so, the match is correct, and the fourth transformation matrix of the current image is Save to get the final transformation matrix and process the next image; If not, the match fails, and the fourth transformation matrix of the current image is calculated. The final transformation matrix of the previous frame image Abnormal translation amount; If normal, the fourth transformation matrix of the current image is Save and process the next picture; If abnormal, and and The absolute value of the error is greater than the sixth preset value, or and The absolute value of the error is greater than the sixth preset value, or and If the absolute value of the error is greater than the sixth preset value, the correct feature point pair is selected. - A set of matching results in , the correct feature point pair - Substitute the other groups of matching feature points in for verification. If the error is greater than If the number of feature points of pixels is more than half, the current matching result is considered If the matching is abnormal, remove the feature points in this group, traverse all matching results, and keep the normal matching results - ; The normal matching results are spatially stitched to obtain the fourth transformation matrix of the current image relative to the stitched image space. , save the fourth transformation matrix , to get the final transformation matrix and process the next image.

Images