A fast detection method for buildings with changes based on drone images of old and new phases

Abstract

The invention discloses a method for quickly detecting changed buildings based on old and new phase images of unmanned aerial vehicles. The method comprises the following steps: obtaining an old phase image obtained before a current moment and a new phase image obtained at a current moment, performing rough matching based on Kalman filtering to determine a rough matching image region of a new phase image in an old phase image, accurately matching the rough matching image region and the new phase image based on a k‑d index tree to obtain a time-corresponding region of the new phase image in the old phase image, using the time-corresponding region as a reference image, performing geometric correction on the new phase image, performing relative radiation correction between images based on brightness correction model construction and a color balance method, respectively identifying the old phase image and the new phase image after radiation correction using a pre-trained recognition model to determine a suspected changed building region, and realizing rapid detection of changed buildings. The corresponding detection process has high accuracy.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method for quickly detecting changed buildings based on new and old phase images of unmanned aerial vehicles. Background Art

[0002] In response to the current problem of illegal and abusive use of land to build and expand buildings, relevant departments have also taken many measures to conduct comprehensive supervision, increase supervision, and strengthen dynamic supervision through various means to timely discover changes in buildings, improve work efficiency, and achieve "timely discovery, timely reporting, timely prevention, and timely treatment". In order to keep abreast of land changes, land monitoring and control are required. Traditional drone land monitoring requires manual selection and deployment of a large number of control points, large-scale splicing of images, and then change detection, which is time-consuming and labor-intensive. Summary of the invention

[0003] In view of the above problems, the present invention proposes a method for rapid detection of changed buildings based on drone images of old and new phases. The method first constructs a Kalman filter model to predict the position of the center point of the new phase image in the large-scale old phase image to narrow the search range, and then realizes accurate matching of the new and old phase images through the kd tree index, and then performs geometric correction and relative radiation correction on the image, and finally identifies and judges suspicious changed buildings based on deep learning, morphological space position matching, etc.

[0004] To achieve the purpose of the present invention, a method for quickly detecting buildings with changes based on drone images of old and new phases is provided, comprising the following steps:

[0005] S10, obtaining an old phase image captured and stitched before the current moment and a new phase image captured at the current moment, and performing a rough matching of the new phase image and the old phase image based on a Kalman filter to determine a rough matching image region of the new phase image in the old phase image;

[0006] S20, accurately matching the coarse matching image area and the new phase image based on the kd index tree to obtain the corresponding region of the new phase image in the old phase image;

[0007] S30, using the corresponding area of ​​the time phase as a reference image, geometrically correcting the new time phase image, and performing relative radiation correction between images based on brightness correction model construction and color balance method;

[0008] S40, using a pre-trained recognition model to respectively recognize the old phase image and the new phase image after radiation correction to determine the building area suspected of change.

[0009] In one embodiment, the rough matching of the new phase image and the old phase image based on Kalman filtering to determine the rough matching image area of ​​the new phase image in the old phase image includes:

[0010] S11, based on the position coordinates (x t ,y t ) and the pre-built Kalman model state equation predict the point position coordinates (x t+1 ,y t+1 ), where time t is the previous time before the current time, and time t+1 is the current time;

[0011] S12, convert the longitude and latitude coordinates of the new phase image into the projection rectangular coordinates and image coordinates to obtain the measured value Z t+1 (x, y), according to the point position coordinates (x t+1 ,y t+1 ) and the measured value Z t+1 (x, y), count the values ​​in the intersection of the Gaussian distribution areas of the two, and get the estimated point (x' t+1 ,y' t+1 ), the estimated point (x' t+1 ,y' t+1 ) as the optimal valuation X at the current moment t+1|t+1 ;

[0012] S13, with the best valuation X t+1|t+1 With the image as the center, a rectangular area is delineated in the old phase image as the area where the new phase image roughly matches the old phase image and extracted to determine the roughly matched image area of ​​the new phase image in the old phase image.

[0013] In one embodiment, the rough matching image area and the new phase image are accurately matched based on the kd index tree to obtain the corresponding region of the new phase image in the old phase image, including:

[0014] S21, extracting feature points of the rough matching image area and the new phase image respectively, taking the feature points as leaf nodes, establishing a spatial kd index tree, searching downward through the binary tree of feature points to find the feature points stored in the leaf nodes closest to the target feature points, the target feature point to be searched is the center of the circle, and within the radius area formed by the target query point to the neighboring nodes, find the nearest feature node that can be matched, and obtain the matching result;

[0015] S22, construct a projection transformation model based on the matching result to establish a projection transformation relationship, project the new phase image onto the coarse matching image area based on the projection transformation model, set the pixel values ​​of the part of the new phase image projected onto the coarse matching image area to 255, and set the pixel values ​​of other parts of the coarse matching image area to 0, perform contour search through morphological operators, surround the part with the pixel value set to 255 with an external rectangular frame, and map the external rectangular frame back to the same position in the coarse matching image area of ​​the old phase image, thereby extracting the area of ​​the new phase image corresponding to the large-scale old image after matching, and obtaining the corresponding area of ​​the new phase image in the old phase image.

[0016] In one embodiment, using the corresponding region of the time phase as a reference image, geometrically correcting the new time phase image, and performing relative radiation correction between images based on brightness correction model construction and color balance method include:

[0017] S31, taking the time phase corresponding area as the reference image, and the new time phase image after geometric correction of the matching feature point pairs as the image to be corrected, converting the image to be corrected from the RGB space to the HSV space, and separating the V channel, performing difference operation and binarization on the reference image and the image to be corrected to obtain a changed part and an unchanged part, and mapping the contour of the unchanged part to the reference image and the image to be corrected respectively according to the target invariance principle, and taking the points contained in the unchanged contour area on the reference image and the image to be corrected as pseudo-invariant sample points in the brightness channel V;

[0018] S32, based on the least squares fitting pseudo-invariant sample points, substitute the pseudo-invariant sample points into the adjustment model formula to calculate the brightness correction coefficient k and the offset coefficient b, thereby constructing a brightness correction model y i =kx i +b; the adjustment model formula includes: (B T PB)XB T PL = 0; the value of each element in matrix B is Each element in the L matrix represents the brightness value y of the pixel in the reference image. i , X is the coefficient matrix composed of brightness correction coefficient k and offset coefficient b, P is the unit weight matrix, 1≤i≤m, 0≤p≤1,

[0019] S33, using the solved brightness correction model to correct the brightness channel V of the image to be corrected, and after re-merging the corrected brightness channel V with the H channel and the S channel, converting them into an RGB image to achieve relative radiation correction between images.

[0020] In one embodiment, before using the pre-trained recognition model to respectively recognize the old phase image and the new phase image after radiation correction to determine the suspected changed building area, the method further includes:

[0021] A sample data set is collected and labeled, and the sample data set is trained based on a mask-RCNN network to obtain a recognition model for identifying buildings in new phase images and old phase images, so as to use the recognition model to predict and identify buildings in new phase images and old phase images; the sample data set includes multiple orthophotos of buildings with the same resolution, multiple forms, and multiple types.

[0022] Specifically, the pre-trained recognition model is used to identify the old phase images and the new phase images after radiation correction to determine that the building areas suspected of changes include:

[0023] The recognition model is used to recognize the old phase images and the new phase images after radiation correction. The surface area covered by the identified buildings generates a semantic segmentation mask and a minimum bounding rectangle Rect for target detection containing the mask. b , respectively count the number of buildings M and N identified in the new phase image and the old phase image, obtain the overlap IOU of the building objects in the new phase image and the old phase image, construct an M×N matrix according to the overlap IOU, set operation operations for the element values ​​in the M×N matrix to obtain the matrix 1-IOU, and determine the suspected changed building area according to the value of each matrix element in the matrix 1-IOU.

[0024] Specifically, the building areas suspected of being changed are determined according to the values ​​of each element in the matrix 1-IOU, including:

[0025] Determine the building elements corresponding to each matrix element in the matrix 1-IOU, find the minimum value in each row or column in the matrix 1-IOU, draw a cross with the minimum value element as the center, and if no element draws a cross in the rows or columns other than the rows and columns corresponding to the minimum value element, then determine that the rows or columns without any elements drawing a cross are unmatched building elements, so as to count the matching building elements and unmatched building elements, so as to find the newly added and demolished buildings based on the comparison of the unmatched building elements;

[0026] In the row or column that can match the building element, if the value of the matrix element corresponding to the building element that can match the corresponding position of the new phase image and the old phase image is less than the overlap threshold T iou , then it is judged that the building has not changed, otherwise it is judged that the building has changed.

[0027] The above-mentioned method for rapid detection of changed buildings based on new and old phase images of unmanned aerial vehicles obtains the old phase images taken and spliced ​​before the current moment and the new phase images taken at the current moment, and roughly matches the new phase images with the old phase images based on Kalman filtering to determine the roughly matched image area of ​​the new phase image in the old phase image, and accurately matches the roughly matched image area with the new phase image based on the kd index tree to obtain the time-corresponding area of ​​the new phase image in the old phase image, and uses the time-corresponding area as a reference image to perform geometric correction on the new phase image, and performs relative radiation correction between images based on brightness correction model construction and color balance method, and uses a pre-trained recognition model to respectively identify the old phase image and the new phase image after radiation correction to determine the suspected changed building area, thereby realizing rapid detection of changed buildings, and the corresponding detection process has high accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 The flowchart of a method for rapid detection of buildings based on new and old phase images of drones according to an embodiment;

[0029] Figure 2 is a flow chart of a method for rapid detection of buildings with changes based on drone images of old and new phases according to another embodiment;

[0030] Figure 3 is a schematic diagram of a process flow of rapid image matching according to an embodiment;

[0031] Figure 4 The diagram is a schematic diagram illustrating a building change detection process according to an embodiment. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0033] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0034] refer to Figure 1 As shown, Figure 1 The flowchart of a method for quickly detecting buildings with changes based on drone images of old and new phases is an embodiment, comprising the following steps:

[0035] S10, obtaining the old phase image taken and stitched before the current moment and the new phase image taken at the current moment, and performing coarse matching on the new phase image and the old phase image based on Kalman filtering to determine the coarse matching image area of ​​the new phase image in the old phase image.

[0036] The above-mentioned old phase image is a large-scale image taken and spliced ​​before the current moment. The above-mentioned steps complete the rough matching of the new phase image in the large-scale old phase image based on Kalman filtering.

[0037] In one embodiment, the rough matching of the new phase image and the old phase image based on Kalman filtering to determine the rough matching image area of ​​the new phase image in the old phase image includes:

[0038] S11, based on the position coordinates (x t ,y t ) and the pre-built Kalman model state equation predict the point position coordinates (x t+1 ,y t+1 ), where time t is the previous time before the current time, and time t+1 is the current time;

[0039] S12, convert the longitude and latitude coordinates (POS information) of the new phase image into projection rectangular coordinates and image coordinates to obtain the measurement value Z t+1 (x, y), according to the point position coordinates (x t+1 ,y t+1 ) and the measured value Z t+1 (x, y), count the values ​​in the intersection of the Gaussian distribution areas of the two, and get the estimated point (x' t+1 ,y' t+1 ), the estimated point (x' t+1 ,y' t+1 ) as the optimal valuation X at the current moment t+1|t+1 ;

[0040] S13, with the best valuation X t+1|t+1 With the image as the center, a rectangular area is delineated in the old phase image as the area where the new phase image roughly matches the old phase image and extracted to determine the roughly matched image area of ​​the new phase image in the old phase image.

[0041] This embodiment can construct the Kalman model state equation based on the horizontal and vertical speeds of the drone, time interval, etc., to predict the point position coordinates (x t+1 ,y t+1). The POS information (the longitude and latitude coordinates of the new phase image) is converted into the measured value Z after the projection rectangular coordinates and image coordinates are converted. t+1 (x, y), based on the state prediction value of the point position coordinates obtained in step S11 and the system measurement value after coordinate conversion, the common part under the probability distribution of the possibility of both occurring is counted, that is, the value in the intersection after the Gaussian distribution areas of the two intersect, which is the value of the estimated point (x' after correction and optimization) t+1 ,y' t+1 ), which is the optimal valuation X at the current moment t+1|t+1 With the optimal estimated point as the center, a small rectangular area is delineated in the large-scale old image as the area after the rough matching of the new phase image in the old phase image and extracted, thereby completing the rough matching of the new and old phase images. The coverage area after the rough matching in the large-scale old image is used for accurate matching with the new phase image in the next step.

[0042] S20, accurately matching the coarse matching image area and the new phase image based on a kd index tree to obtain a corresponding region of the new phase image in the old phase image.

[0043] The above steps complete the precise matching of old and new phase images based on the kd tree index.

[0044] In one embodiment, the rough matching image area and the new phase image are accurately matched based on the kd index tree to obtain the corresponding region of the new phase image in the old phase image, including:

[0045] S21, extracting feature points of the rough matching image area and the new phase image respectively, taking the feature points as leaf nodes, establishing a spatial kd index tree, searching downward through the binary tree of feature points to find the feature points stored in the leaf nodes closest to the target feature points, the target feature point to be searched is the center of the circle, and within the radius area formed by the target query point to the neighboring nodes, find the nearest feature node that can be matched, and obtain the matching result;

[0046] S22, construct a projection transformation model based on the matching result to establish a projection transformation relationship, project the new phase image onto the coarse matching image area based on the projection transformation model, set the pixel values ​​of the part of the new phase image projected onto the coarse matching image area to 255, and set the pixel values ​​of other parts of the coarse matching image area to 0, perform contour search through morphological operators, surround the part with the pixel value set to 255 with an external rectangular frame, and map the external rectangular frame back to the same position in the coarse matching image area of ​​the old phase image, thereby extracting the area of ​​the new phase image corresponding to the large-scale old image after matching, and obtaining the corresponding area of ​​the new phase image in the old phase image.

[0047] The above kd index tree is a binary tree.

[0048] In this embodiment, feature points of the image area after rough matching and the new phase image are extracted respectively, and a spatial kd index tree is established with the feature points as leaf nodes. The feature points stored in the leaf node closest to the target feature point are searched downward mainly through the binary tree of the feature points. The target feature point to be searched is taken as the center of the circle, and the nearest feature node that can be matched is found within the radius area formed by the target query point to the neighboring nodes, thereby completing the accurate matching of the features. A projection transformation model is constructed according to the matching results to establish a projection transformation relationship. Based on the projection transformation model, the new phase image is projected and mapped to the small-scale old phase image area obtained after rough matching, and the part projected and mapped to the small-scale image area is set to 255 (white), and the other parts of the small-scale image area are set to 0 (black). The contour search is performed through the morphological operator, the white part is searched and surrounded by an external rectangular frame, and the external rectangular frame is mapped back to the same position in the small-scale old phase image area after rough matching, thereby extracting the area of ​​the new phase image corresponding to the old image after matching in the large-scale old image, which is used for comparison and change detection with the new phase image.

[0049] S30, using the corresponding area in the time phase as a reference image, performing geometric correction on the new time phase image, and performing relative radiation correction between images based on brightness correction model construction and color balance method.

[0050] The above steps perform geometric correction on the images, and perform relative radiation correction between images based on the brightness correction model construction and color balance method.

[0051] In one embodiment, using the corresponding region of the time phase as a reference image, geometrically correcting the new time phase image, and performing relative radiation correction between images based on brightness correction model construction and color balance method include:

[0052] S31, taking the time phase corresponding area as the reference image, and the new time phase image after geometric correction of the matching feature point pairs as the image to be corrected, converting the image to be corrected from the RGB space to the HSV space, and separating the V channel, performing difference operation and binarization on the reference image and the image to be corrected to obtain a changed part and an unchanged part, and mapping the contour of the unchanged part to the reference image and the image to be corrected respectively according to the target invariance principle, and taking the points contained in the unchanged contour area on the reference image and the image to be corrected as pseudo-invariant sample points in the brightness channel V;

[0053] S32, based on the least squares fitting pseudo-invariant sample points, substitute the pseudo-invariant sample points into the adjustment model formula to calculate the brightness correction coefficient k and the offset coefficient b, thereby constructing a brightness correction model y i =kx i+b; the adjustment model formula includes: (B T PB)XB T PL = 0; the value of each element in matrix B is Each element in the L matrix represents the brightness value y of the pixel in the reference image. i , X is the coefficient matrix composed of brightness correction coefficient k and offset coefficient b, P is the unit weight matrix, 1≤i≤m, 0≤p≤1,

[0054] S33, using the solved brightness correction model to correct the brightness channel V of the image to be corrected, re-merging the corrected brightness channel V with the H channel (hue channel) and the S channel (saturation channel), and then converting them into an RGB image to achieve relative radiation correction between images.

[0055] In an example, the pseudo-invariant sample points in the brightness channel V can be recorded as:

[0056] (x1,y1),(x2,y2)LL(x n ,y n ).

[0057] In this embodiment, the pseudo-invariant sample points can be fitted based on the least squares method, using the pseudo-invariant sample points: (x1, y1), (x2, y2) LL (x n ,y n ) is substituted into the adjustment model formula to calculate the brightness correction coefficient k and the offset coefficient b, thereby constructing the brightness correction model y i =kx i +b; the adjustment model formula includes: (B T PB)XB T PL = 0, where the value of each element in the matrix B is Each element in the L matrix represents the brightness value y of the pixel in the reference image. i , X is the coefficient matrix composed of unknown parameters (brightness correction coefficient k and offset coefficient b). P is the unit weight matrix. The brightness channel V of the image to be corrected is corrected with the solved brightness correction model, re-merged with the H and S channels, and then converted into an RGB image. The color of the new and old phase images is adjusted based on the color balance method. As a result, after the brightness model construction correction and the color adjustment of the color balance method, the brightness and color between the image to be corrected (new phase image) and the reference image (old phase image) tend to be consistent, thereby reducing the influence of radiation differences caused by external factors such as light and atmosphere on the subsequent building recognition and change detection of the new and old phase images.

[0058] S40, using a pre-trained recognition model to respectively recognize the old phase image and the new phase image after radiation correction to determine the building area suspected of change.

[0059] The above steps identify buildings in the image based on the deep learning model, and detect and extract building areas suspected of changes based on the morphology and spatial position matching relationship of the identified buildings.

[0060] The above-mentioned method for rapid detection of changed buildings based on new and old phase images of unmanned aerial vehicles obtains the old phase images taken and spliced ​​before the current moment and the new phase images taken at the current moment, and roughly matches the new phase images with the old phase images based on Kalman filtering to determine the roughly matched image area of ​​the new phase image in the old phase image, and accurately matches the roughly matched image area with the new phase image based on the kd index tree to obtain the time-corresponding area of ​​the new phase image in the old phase image, and uses the time-corresponding area as a reference image to perform geometric correction on the new phase image, and performs relative radiation correction between images based on brightness correction model construction and color balance method, and uses a pre-trained recognition model to respectively identify the old phase image and the new phase image after radiation correction to determine the suspected changed building area, thereby realizing rapid detection of changed buildings, and the corresponding detection process has high accuracy.

[0061] In one embodiment, before using the pre-trained recognition model to respectively recognize the old phase image and the new phase image after radiation correction to determine the suspected changed building area, the method further includes:

[0062] A sample data set is collected and labeled, and the sample data set is trained based on a mask-RCNN network to obtain a recognition model for identifying buildings in new phase images and old phase images, so as to use the recognition model to predict and identify buildings in new phase images and old phase images; the sample data set includes multiple orthophotos of buildings with the same resolution, multiple forms, and multiple types.

[0063] This embodiment can collect a large number of sample data sets of orthophotos with the same resolution and containing buildings of different shapes and types and label them, and train the sample data sets based on the mask-RCNN network, mainly performing sample feature extraction, pooling candidate areas, prediction and classification, etc. After training, a model file (recognition model) for identifying buildings in new and old phase images is obtained, and the model file is used to predict and identify buildings in new and old phase images.

[0064] Specifically, the pre-trained recognition model is used to identify the old phase images and the new phase images after radiation correction to determine that the building areas suspected of changes include:

[0065] The recognition model is used to recognize the old phase images and the new phase images after radiation correction. The surface area covered by the identified buildings generates a semantic segmentation mask and a minimum bounding rectangle Rect for target detection containing the mask. b , respectively count the number of buildings M and N identified in the new phase image and the old phase image, obtain the overlap IOU of the building objects in the new phase image and the old phase image, construct an M×N matrix according to the overlap IOU, set operation operations for the element values ​​in the M×N matrix to obtain the matrix 1-IOU, and determine the suspected changed building area according to the value of each matrix element in the matrix 1-IOU.

[0066] Specifically, the building areas suspected of being changed are determined according to the values ​​of each element in the matrix 1-IOU, including:

[0067] Determine the building elements corresponding to each matrix element in the matrix 1-IOU, find the minimum value in each row or column in the matrix 1-IOU, draw a cross with the minimum value element as the center, and if no element draws a cross in the rows or columns other than the rows and columns corresponding to the minimum value element, then determine that the rows or columns without any elements drawing a cross are unmatched building elements, so as to count the matching building elements and unmatched building elements, so as to find the newly added and demolished buildings based on the comparison of the unmatched building elements;

[0068] In the row or column that can match the building element, if the value of the matrix element corresponding to the building element that can match the corresponding position of the new phase image and the old phase image is less than the overlap threshold T iou , then it is judged that the building has not changed, otherwise it is judged that the building has changed.

[0069] The above overlap threshold T iou It can be set based on experience, neither too large nor too small; if the 1-IOU element value corresponding to the building element that can be matched at the corresponding position of the new and old phase images in the matrix is ​​less than T iou , then the building is judged to have not changed, otherwise it is judged to have changed its area range (expanded or reduced). The above setting operation includes: subtracting each element in the M×N matrix from 1 to obtain the matrix 1-IOU.

[0070] In one example, the calculation method of the above-mentioned overlap IOU includes:

[0071]

[0072] In the formula, Rect b旧 and Rect b新They are the bounding rectangles of the target buildings on the old phase image and the new phase image respectively.

[0073] This embodiment detects the changed buildings based on the morphological overlap and spatial position relationship matching method for the identified buildings. The deep learning model is used to identify and predict the new and old phase images. After identification, the surface area covered by the building in the image generates a semantic segmentation red mask and the minimum enclosing rectangular box Rect for target detection containing the mask. b , the semantic category of the classified building is determined according to the mask, and the specific location of the building in the image is determined according to the minimum circumscribed rectangular frame. The number of buildings M and N identified on the old and new phase images are counted respectively. The building objects in the old and new phase images are matched by the coincidence IOU (between 0 and 1), and the coincidence calculation result is used as an indicator to determine whether the building has changed. The calculation results constitute an M×N matrix, and the element values ​​in the matrix are set to operate, that is, 1-IOU. The smaller the value, the higher the matching degree. Find the minimum value in each row or column, draw a cross with the minimum element as the center, and the elements in the row and column corresponding to the minimum element are no longer considered. The row or column where the element corresponding to the minimum value is located represents the building element that can be matched. Finally, if no element in the remaining rows or columns draws a cross, it means that the building element cannot be matched, so that the building elements that can be matched are counted and the unmatched building elements are screened out. The main purpose is to find new and demolished buildings based on the comparison of unmatched building elements. Among the building elements that can be matched, a reasonable overlap threshold T is set based on experience. iou (It cannot be too large or too small). Therefore, if the 1-IOU value of the building elements that can be matched at the corresponding positions of the new and old phase images in the matrix is ​​less than T iou , then the building is judged to have not changed, otherwise it is judged to be a building with a changed area range (expansion or reduction).

[0074] The calculation process of the overlap IOU includes:

[0075]

[0076] In the formula, Rect b旧 and Rect b新 They are the bounding rectangles of the target buildings on the old phase image and the new phase image respectively.

[0077] The above-mentioned method for rapid detection of buildings with changes based on drone images of old and new phases has the following beneficial effects:

[0078] 1. A method for quickly discovering changed buildings in drone images of old and new phases was established.

[0079] 2. A method for fast search and matching positioning based on Kalman filtering and kd tree index is proposed. The position of the center point of the new phase image in the old phase image is predicted based on the Kalman filter model to narrow the search range, and the accurate matching and positioning between images are completed based on the k-dtree index. On the basis of geometric correction, the new and old phase images are relatively radiometrically corrected by constructing a brightness correction model and color consistency correction. Finally, the building is identified based on the deep learning method, and the changes in the building are detected based on the building morphology matching, spatial position pattern relationship, etc. This method realizes the fast matching and positioning of the new and old phase images, narrows the regional scope of matching and retrieval of the new and old phase images, reduces the retrieval and matching time, and improves efficiency. At the same time, starting directly from the building, the buildings identified by the model are matched and compared, eliminating the interference of non-building factors such as noise, and improving the accuracy of change detection.

[0080] In one embodiment, the above-mentioned method for rapid detection of buildings based on new and old phase images of drones can also refer to Figure 2 As shown, including:

[0081] Step 1: Based on Kalman filtering, the rough matching of the new phase image in the large-scale old phase image is completed, which specifically includes:

[0082] (1.1) Decompose the drone’s average speed V and yaw angle α to construct the drone’s horizontal and vertical speeds Adjacent t i-1 and t i The position coordinates of the new phase image in the old phase image at time (x t ,y t )、(x t+1 ,y t+1 ), the sampling time interval between images at adjacent moments is △t. Combining the above information, the Kalman state vector is constructed At the same time, the Kalman state transfer matrix A is constructed according to the sampling time interval:

[0083]

[0084] (1.2) Based on the above factor analysis, the state prediction is performed at each moment, and the position coordinates of the center point of the new phase image predicted at each moment in the old phase image are obtained, as shown in formula (3).

[0085]

[0086] Where w is the noise contained in the UAV during flight.

[0087] (1.3) Take the POS information (the longitude and latitude of the geographic coordinate position) as the measurement value. Based on the conversion between coordinate systems, first convert the longitude and latitude coordinates of the point into coordinates in the projection rectangular coordinate system, and then convert them into coordinates in the image coordinate system to obtain the coordinate measurement value Z of the center point of the new phase image in the old phase image. t+1 (x, y). The conversion between latitude and longitude coordinates and projection coordinate system and between projection coordinates and image coordinates is shown in equations (4) and (5).

[0088]

[0089] Where: L is longitude, B is latitude, is the radius of curvature of the Maoyou circle, X0 is the arc length of the meridian from the equator to a certain point, l is the difference between the geodetic longitude of the calculation point and the central meridian longitude of the projection zone, which is the first eccentricity, a is the major semi-axis of the rotating ellipsoid, b is the minor semi-axis, η 2 =e′ 2 cos 2 B, t = tanB.

[0090]

[0091] Where X and Y are coordinates in the plane rectangular coordinate system, x and y are coordinates in the image coordinate system, pixelweight is the size of each pixel in the horizontal direction, and pixelheight is the size of each pixel in the vertical direction. h is the horizontal rotation parameter, which usually defaults to 0, and v is the vertical rotation parameter, which usually defaults to 0. X0 is the projected rectangular abscissa of the upper left corner of the old phase image measurement area, and Y0 is the projected rectangular ordinate of the upper left corner of the old phase image measurement area.

[0092] Construct the observation matrix H as:

[0093]

[0094] The observation equation is:

[0095] Z t+1 =HX t+1 +V t (7)

[0096] Where V t is the system noise.

[0097] Compute the coerror variance matrix of the state vector:

[0098] P t+1|t =AP t|t A T +Q t (8)

[0099] Where P t|t is the state vector X t The covariance matrix, Q t is the covariance matrix of the noise w.

[0100] Calculate the filter gain coefficient K t+1 :

[0101] K t+1 =P t+1|t H T (HP t+1|t H T +R t ) -1 (9)

[0102] In the formula, R t V t The covariance matrix of

[0103] According to the predicted value X t+1 and the measured value Z t+1 Get the optimal estimate X at the current moment t+1|t+1 ,

[0104] X t+1|t+1 =AX t +K t+1 (Z t+1 -HX t+1 ) (10)

[0105] (1.4) Taking the current estimated point as the center, a small rectangular area is intercepted in the large-scale old image, which is the coverage area after rough matching. The coverage area after rough matching in the large-scale old image is used for accurate matching with the new phase image in the next step. Repeat the above steps to predict the position of the point at the next moment.

[0106] Step 2: Complete the accurate matching of old and new phase images based on the kd tree index, including:

[0107] (2.1) Extract the feature points of the roughly matched image area and the new phase image respectively, and establish a spatial kd index tree (a binary tree) with the feature points as leaf nodes. The feature points are mainly found by binary tree search of the feature points in the leaf node closest to the target feature point. With the target feature point as the center of the circle, the nearest feature node that can be matched is found within the radius area formed by the target query point to the neighboring nodes, thereby completing the accurate matching of the features.

[0108] (2.2) Construct a projection transformation model based on the matching results to establish a projection transformation relationship. Based on the projection transformation model, project the new phase image onto the small-scale image area obtained after rough matching. Set the portion projected onto the small-scale image area to 255 (white), and set the rest of the small-scale image area to 0 (black). Search for the white outline portion and surround it with an external rectangular frame, and map the external rectangular frame back to the same position in the small-scale old phase image area after rough matching, thereby extracting the area of ​​the new phase image that corresponds to the old large-scale image after matching, which is used for comparison and change detection with the new phase image.

[0109] Step 3: Perform geometric correction on the image and perform relative radiation correction between images based on the radiation correction model and color balance method:

[0110] (3.1) Registration is performed based on matched feature point pairs to complete geometric correction between images;

[0111] (3.2) The old phase image area obtained after matching is used as the reference image, and the new phase image (after geometric correction of the matching feature point pairs) is used as the image to be corrected. The RGB color space of the new phase image is converted to the HSV color space, and the V (brightness) channel is separated and extracted. The difference between the old and new phase images is calculated and binarized, and divided into the changed (white) and unchanged parts (black). According to the target invariance principle, the unchanged contour part is mapped to the old and new phase images, and the points contained in the unchanged contour area on the old and new phase images are used as the pseudo-invariant sample points (x1, y1), (x2, y2)LL(x n ,y n );

[0112] (3.3) Count the number m of matched pairs of feature sample points with the same name. According to the pixel brightness values ​​y of the matched feature sample points with the same name on the processed reference image and the image to be corrected, i and x i (i is the i-th pixel feature point in the image) and the number of feature sample point pairs m, construct the matrix B m×2 , X 2×1 and L m×1 , the value of each element in matrix B is Each element in the L matrix represents the brightness value y of the pixel in the reference image. i , X is the coefficient matrix composed of unknown parameters k and b. Based on these three elements, a polynomial equation BX=L is constructed. Substitute it into the indirect adjustment formula 11 to solve the coefficient matrix and build a brightness correction model.

[0113] (B T PB)XB T PL=0 (11)

[0114] The solved brightness correction model corrects the brightness channel V of the image to be corrected, re-merges it with the H and S channels, and then converts it into an RGB image.

[0115] (3.4) Set a suitable k×k transfer window to adjust the color of each local area in the image and calculate the pixel value t after color adjustment new (i,j), so as to achieve the effect that the colors of the two pictures are basically the same. The formula is as follows:

[0116]

[0117] Where t(i, j) is the pixel value of the i-th row and j-th column of the original image to be used for color adjustment, μ and σ are the mean and variance respectively, k represents the length of the window used to transmit statistical data around the pixel (i, j), and t new (i,j) is the pixel value at the i-th row and j-th column in the color-adjusted image.

[0118] Step 4: Identify buildings in the image based on the deep learning model, and detect and extract suspected changed building areas based on the morphology and spatial position matching relationship of the identified buildings, including:

[0119] (4.1) Collect enough orthophoto sample data of buildings of the same size and resolution, including buildings of different types and shapes. The data set is divided into two categories: buildings and non-buildings. The outer contours of the buildings are marked and the label data is saved and exported in PascalVOC format. Modify the pre-trained model file parameters (category name replacement, training path, training iteration number, batch size, etc.);

[0120] (4.2) The labeled data set is trained based on the mask-RCNN network, mainly for sample feature extraction, pooling candidate regions, prediction classification, etc., and the trained model file is output.

[0121] (4.3) The trained model is used to predict and identify the old and new phase images. After identification, the surface area covered by the building in the image generates a semantic segmentation mask and an outer rectangular box for target detection. The upper left corner of the box is the building label name "bl" and the confidence value (the probability of belonging to the building category). The semantic category of the classified building is determined based on the mask, and the specific location of the building in the image is determined based on the minimum outer rectangular box.

[0122] (4.4) The buildings in the old and new phase images are matched in terms of overlap and position pattern, and the IOU after matching is calculated. The result of the overlap calculation is used as an indicator to determine whether the building has changed. The overlap is shown in Formula 9.

[0123]

[0124] To indicate Figure 3 For example, buildings a, b, c identified in the old phase image are matched with buildings 1 and 2 identified in the new phase image.

[0125] Count the number of rectangular frames surrounding the buildings and create a 3×2 matrix M:

[0126]

[0127] The matrix indicates that a goes to match 1, a goes to match 2, b goes to match 1, b goes to match 2, c goes to match 1... Each element in the matrix represents the IOU of the rectangular matching overlap between the old image and the corresponding position in the new image, ranging from 0 to 1.

[0128] After recognition, the position coordinates and area size of each rectangle in the image have been determined. The position relationship between a and 1 does not correspond, and the calculated matching overlap between the two is 0, which means they cannot match. The overlap between a and 2 is also 0, so the first row of the matrix is ​​0. Similarly, the building circumscribed rectangular frame b is used to match buildings 1 and 2, and overlaps with rectangular frame 1. The calculated overlap is iou b1 , does not overlap with rectangular frame 2, and the calculation result is 0. And so on. Form the M matrix:

[0129]

[0130] The elements in the matrix are negated by 1 (set operation) to obtain the matrix 1-IOU. In order to more intuitively see the screening statistics of the changed buildings, the matrix is ​​augmented after negation to form a 3×3 matrix with equal number of rows and columns. Generally, the greater the overlap, the better the match, but after negating 1 on the original basis, the smaller the value of the element in the matrix, the better the match of the building at the corresponding position.

[0131]

[0132] In addition to the augmented 0 elements, count the minimum value in each row or column. The minimum value in the first column is 1-iou b1 , indicating that building b has the highest matching degree with building 1, and the two can match. b1 Draw a crosshair at the center. The element in the second row and first column is not considered.

[0133]

[0134] Similarly, among the remaining elements, the minimum value in the second column is 1-iou c2 , indicating that building c is matched with building 2. c2 Draw a crosshair at the center. The element in the third row and second column is no longer considered.

[0135]

[0136] The remaining first row of building a has not been matched, but since the first and second columns have been crossed out and are no longer considered, the first row can only be matched with elements in the third column. The third column does not actually exist, so building a has not been matched to any corresponding elements and is judged to be a changed building (a new building or demolition).

[0137] Among all the buildings that can be matched in the old and new phase images, a reasonable overlap threshold T is set based on experience. iou (It cannot be too large or too small). Generally speaking, the greater the overlap between buildings, the more they match. However, after inverting the original value with 1, the smaller the value of the element in the matrix, the more the building at the corresponding position matches. Therefore, if the 1-IOU element value of the building element that can be matched at the corresponding position of the new and old phase images in the matrix is ​​less than T iou , then the building is judged to have not changed, otherwise the area range has changed and it is judged to be a changed building (building expansion or reduction).

[0138] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0139] It should be noted that the terms "first\second\third" involved in the embodiments of the present application are only used to distinguish similar objects, and do not represent a specific order for the objects. It is understandable that the specific order or sequence of "first\second\third" can be interchanged where permitted. It should be understood that the objects distinguished by "first\second\third" can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein.

[0140] The terms "including" and "having" and any variations thereof in the embodiments of the present application are intended to cover non-exclusive inclusions. For example, a process, method, device, product, or equipment that includes a series of steps or modules is not limited to the listed steps or modules, but may optionally include steps or modules that are not listed, or may optionally include other steps or modules that are inherent to these processes, methods, products, or equipment.

[0141] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A method for rapid detection of buildings with changes based on drone images of old and new phases, characterized in that: The steps include: S10, obtaining an old phase image captured and stitched before the current moment and a new phase image captured at the current moment, and performing a rough matching of the new phase image and the old phase image based on a Kalman filter to determine a rough matching image region of the new phase image in the old phase image; S20, accurately matching the coarse matching image area and the new phase image based on the kd index tree to obtain the corresponding region of the new phase image in the old phase image; S30, using the corresponding area of ​​the time phase as a reference image, geometrically correcting the new time phase image, and performing relative radiation correction between images based on brightness correction model construction and color balance method; S40, using a pre-trained recognition model to respectively recognize the old phase image and the new phase image after radiation correction to determine the building area suspected of change; The new phase image and the old phase image are roughly matched based on Kalman filtering to determine the rough matching image area of ​​the new phase image in the old phase image, including: S11, based on the position coordinates (x t ,y t ) and the pre-built Kalman model state equation predict the point position coordinates (x t+1 ,y t+1 ), where time t is the previous time before the current time, and time t+1 is the current time; S12, convert the longitude and latitude coordinates of the new phase image into the projection rectangular coordinates and image coordinates to obtain the measured value Z t+1 (x, y), according to the point position coordinates (x t+1 ,y t+1 ) and the measured value Z t+1 (x, y), count the values ​​in the intersection of the Gaussian distribution areas of the two, and get the estimated point (x' t+1 ,y' t+1 ), the estimated point (x' t+1 ,y' t+1 ) as the optimal valuation X at the current moment t+1|t+1 ; S13, with the best valuation X t+1|t+1 With the image as the center, a rectangular area is delineated in the old phase image as the area where the new phase image roughly matches the old phase image and extracted to determine the roughly matched image area of ​​the new phase image in the old phase image.

2. The method for rapid detection of buildings with changes based on drone images of old and new phases according to claim 1 is characterized in that: Based on the kd index tree, the rough matching image area and the new phase image are accurately matched to obtain the corresponding region of the new phase image in the old phase image, including: S21, extracting feature points of the rough matching image area and the new phase image respectively, taking the feature points as leaf nodes, establishing a spatial kd index tree, searching downward through the binary tree of feature points to find the feature points stored in the leaf nodes closest to the target feature points, the target feature point to be searched is the center of the circle, and within the radius area formed by the target query point to the neighboring nodes, find the nearest feature node that can be matched, and obtain the matching result; S22, construct a projection transformation model based on the matching result to establish a projection transformation relationship, project the new phase image onto the coarse matching image area based on the projection transformation model, set the pixel values ​​of the part of the new phase image projected onto the coarse matching image area to 255, and set the pixel values ​​of other parts of the coarse matching image area to 0, perform contour search through morphological operators, surround the part with the pixel value set to 255 with an external rectangular frame, and map the external rectangular frame back to the same position in the coarse matching image area of ​​the old phase image, thereby extracting the area of ​​the new phase image corresponding to the large-scale old image after matching, and obtaining the corresponding area of ​​the new phase image in the old phase image.

3. The method for rapid detection of buildings with changes based on drone images of old and new phases according to claim 1 is characterized in that: Taking the corresponding area of ​​the time phase as the reference image, geometric correction is performed on the new time phase image. Based on the brightness correction model and color balance method, relative radiation correction between images is performed, including: S31, taking the time phase corresponding area as the reference image, and the new time phase image after geometric correction of the matching feature point pairs as the image to be corrected, converting the image to be corrected from the RGB space to the HSV space, and separating the V channel, performing difference operation and binarization on the reference image and the image to be corrected to obtain a changed part and an unchanged part, and mapping the contour of the unchanged part to the reference image and the image to be corrected respectively according to the target invariance principle, and taking the points contained in the unchanged contour area on the reference image and the image to be corrected as pseudo-invariant sample points in the brightness channel V; S32, based on the least squares fitting pseudo-invariant sample points, substitute the pseudo-invariant sample points into the adjustment model formula to calculate the brightness correction coefficient k and the offset coefficient b, thereby constructing a brightness correction model y i =kx i +b; the adjustment model formula includes: (B T PB)XB T PL = 0; the value of each element in matrix B is Each element in the L matrix represents the brightness value y of the pixel in the reference image. i , X is the coefficient matrix composed of brightness correction coefficient k and offset coefficient b, P is the unit weight matrix, 1≤i≤m, 0≤p≤1, S33, using the solved brightness correction model to correct the brightness channel V of the image to be corrected, and after re-merging the corrected brightness channel V with the H channel and the S channel, converting them into an RGB image to achieve relative radiation correction between images.

4. The method for rapid detection of buildings with changes based on drone images of old and new phases according to claim 1 is characterized in that: Before using the pre-trained recognition model to identify the old phase images and the new phase images after radiometric correction to determine the suspected changed building areas, the following steps are also included: A sample data set is collected and labeled, and the sample data set is trained based on a mask-RCNN network to obtain a recognition model for identifying buildings in new phase images and old phase images, so as to use the recognition model to predict and identify buildings in new phase images and old phase images; the sample data set includes multiple orthophotos of buildings with the same resolution, multiple forms, and multiple types.

5. The method for rapid detection of buildings with changes based on drone images of old and new phases according to claim 4 is characterized in that: The pre-trained recognition model is used to identify the old phase images and the new phase images after radiation correction to determine the suspected changed building areas, including: The recognition model is used to recognize the old phase images and the new phase images after radiation correction. The surface area covered by the identified buildings generates a semantic segmentation mask and a minimum bounding rectangle Rect for target detection containing the mask. b , respectively count the number of buildings M and N identified in the new phase image and the old phase image, obtain the overlap IOU of the building objects in the new phase image and the old phase image, construct an M×N matrix according to the overlap IOU, set operation operations for the element values ​​in the M×N matrix to obtain the matrix 1-IOU, and determine the suspected changed building area according to the value of each matrix element in the matrix 1-IOU.

6. The method for rapid detection of buildings with changes based on drone images of old and new phases according to claim 5 is characterized in that: According to the values ​​of each element in the matrix 1-IOU, the suspected building areas include: Determine the building elements corresponding to each matrix element in the matrix 1-IOU, find the minimum value in each row or column in the matrix 1-IOU, draw a cross with the minimum value element as the center, and if no element draws a cross in the rows or columns other than the rows and columns corresponding to the minimum value element, then determine that the rows or columns without any elements drawing a cross are unmatched building elements, so as to count the matching building elements and unmatched building elements, so as to find the newly added and demolished buildings based on the comparison of the unmatched building elements; In the row or column that can match the building element, if the value of the matrix element corresponding to the building element that can match the corresponding position of the new phase image and the old phase image is less than the overlap threshold T iou , then it is judged that the building has not changed, otherwise it is judged that the building has changed.

Images