A three-dimensional change detection method based on UAV video sequences

Abstract

The present invention discloses a three-dimensional change detection method based on drone video sequences. First, a three-dimensional point cloud is reconstructed through the drone video sequence; then two point clouds acquired at different times are registered; secondly, a candidate area is obtained through the difference between the two point clouds; finally, the candidate area is screened through the density, length, width, height and volume of the candidate area. The method is simple to operate and can accurately locate the three-dimensional change area in the face of many interferences.

Description

Technical Field

[0001] The invention belongs to the technical field of unmanned aerial vehicles, and in particular relates to a three-dimensional change detection method based on unmanned aerial vehicle video sequences. Background Art

[0002] Compared with two-dimensional images, three-dimensional point clouds can more completely preserve and display the shape, structure and other characteristics of objects. With the development of three-dimensional reconstruction technology and three-dimensional data acquisition technology, three-dimensional point clouds are gradually being used in many fields such as robots, unmanned driving, and drones. Three-dimensional change detection between scenes has now been widely used in the defense industry, such as whether other targets appear in the field of security monitoring; whether there are new buildings, demolished buildings, or changed buildings in the field of environmental monitoring, etc.

[0003] At present, the two-dimensional change detection method is relatively mature, but it can only detect plane changes and lacks three-dimensional spatial perception capabilities, such as building height. For change detection that requires height information, three-dimensional point clouds have great advantages. It takes a lot of manpower and material resources to obtain point cloud data through traditional aviation airborne lidar. The dense matching technology of multiple images provides a new technology for obtaining three-dimensional information, but there are many erroneous matches. According to the different uses of three-dimensional information, it can be roughly divided into geometric information comparison method and method combining geometric information with spectral information. The geometric information comparison method is easily affected by height data, has high data requirements, and is complex in calculation; the method combining geometric information with spectral information is sensitive to parameters and has high requirements for sample collection and feature extraction.

[0004] In recent years, drone technology has developed rapidly, with the characteristics of low cost, flexibility and high efficiency. Obtaining three-dimensional point clouds through drone video sequence reconstruction has gradually become a new trend. Summary of the invention

[0005] In order to overcome the shortcomings of the prior art, the present invention provides a three-dimensional change detection method based on drone video sequences. First, the three-dimensional point cloud is reconstructed through the drone video sequence; then the two point clouds obtained at different times are registered; secondly, the candidate area is obtained through the difference between the two point clouds; finally, the candidate area is screened through the density and length, width, height and volume of the candidate area. The method is simple to operate and can accurately locate the three-dimensional change area in the face of many interferences.

[0006] The technical solution adopted by the present invention to solve the technical problem includes the following steps:

[0007] Step 1: Reconstruct the 3D point cloud of the scene from the drone video sequence;

[0008] Step 2: Reconstruct the two point clouds P at different timesm and P s Perform registration;

[0009] Step 3: Using point cloud P m From a single point to P s The candidate region is obtained by the minimum Euclidean distance value, and the regional density method is used to remove the noise in the candidate region to obtain the point cloud P c ;

[0010] Step 4: Use the Euclidean clustering method to cluster the point cloud P c The buildings are segmented into several candidate buildings, and then further screened according to the length, width, height and volume multi-level geometric attributes of the candidate buildings to obtain the final result.

[0011] Furthermore, the specific process of reconstructing the three-dimensional point cloud of the scene according to the drone video sequence in step 1 includes:

[0012] Step 1-1: Obtain the image sequence of the drone video by taking frames at intervals;

[0013] Step 1-2: Sparsely reconstruct the image sequence using the motion recovery structure method;

[0014] Step 1-3: Further dense reconstruction is performed through the multi-view stereo algorithm to obtain denser point cloud data and complete the reconstruction.

[0015] Furthermore, in step 2, the two point clouds P reconstructed at different times are m and P s The specific process of registration includes:

[0016] Step 2-1: Randomly select n key points from each of the two point clouds and Get the local area where each key point is located and construct a local feature descriptor and

[0017] Step 2-2: Generate feature matching between two point cloud key points based on feature descriptors and select K feature matchings based on the feature matching scores

[0018] Step 2-3: Select any 3 feature matches from the selected K feature matches Generate rotation and translation transformation (R, t);

[0019]

[0020] In the formula, R represents a 3x3 rotation matrix, and t represents a 1x3 translation transformation;

[0021] Step 2-4: Point cloud P m Downsampling to get point cloud P′ m , calculate the point cloud P′ according to the rotation and translation transformation (R, t) m and P s The overlap rate between them is saved in each iteration. ov The highest rotation-translation transformation is (R result ,t result ), if the iteration reaches a certain number of times or the overlap rate reaches the set threshold threshold_r ov Then stop the iteration;

[0022] Overlap rate r ov The specific steps of the calculation method include:

[0023] 1) Through the rotation and translation transformation (R, t) of the point cloud P′ m Transform to get the new point cloud P″ m ;

[0024] 2) In the point cloud P s Find the distance The nearest point if and The Euclidean distance value Less than the threshold value dis 1 ,Will Put into subcollection

[0025] 3) Overlap rate r ov The calculation is as follows:

[0026]

[0027] in, and P″m are both point sets, |·| represents the size of the set;

[0028] Step 2-5: Point cloud P m Through the rotation and translation transformation (R result , t result ) Get the new point cloud P m′ , so far, point cloud P m and P s Registration completed;

[0029] Furthermore, the specific steps of obtaining the candidate region and removing noise in step 3 are as follows:

[0030] Step 3-1: In the point cloud P s Find the distance The nearest point if and The Euclidean distance value is less than the threshold dis2, Put into subcollection So far, the candidate region is obtained. The candidate region obtained here still has noise. In the next step, the noise in the candidate region is removed.

[0031] Step 3-2: Traversal Every point p in k , Use kdtree to determine the number of points in the spherical neighborhood. If the number of points is greater than the threshold num, p k Insert sub-point cloud P c , so far, the denoising is completed.

[0032] Furthermore, the step 4 is specifically as follows:

[0033] Step 4-1: Select any point from the point cloud Pc Point Add to collection Will From the point cloud P c Removed;

[0034] Step 4-2: Traverse the collection Each point At the same time, traverse the point cloud P c Every point like and If the distance between them is less than the set threshold, the point Add to collection And remove from the point cloud Pc, the already placed points will not be repeated, the collection The points in will not be traversed repeatedly, and the point cloud P c The points that have been removed in the next traversal of P c It will not be traversed again; repeat steps 4-2 until there are no more points in the set

[0035] Step 4-3: If the point cloud P c If there are still points left, continue to repeat steps 4-1 and 4-2 to construct the set Until the collection is constructed After that, point cloud P c There are no remaining points in the set; the number of sets obtained is the number of buildings. So far, all possible areas of each building have been obtained;

[0036] Step 4-4: Calculate the number of points in each candidate building. If the number is less than the preset threshold, the building is considered to be noise that has not been removed and is not a building.

[0037] Step 4-5: After the screening in step 4-4, the bounding box of the remaining candidate buildings is calculated. If the volume of the bounding box is less than the preset volume threshold, the candidate building is determined to be an erroneous result caused by the registration error, and the building is removed. The remaining candidate buildings are the final result.

[0038] The beneficial effects of the present invention are as follows:

[0039] 1) The method of the present invention is highly robust and can still obtain relatively accurate results in the face of many interferences.

[0040] 2) The method of the present invention is simple and effective, and there is no need to detect the area where the key points are located. All operations can be implemented manually without the use of a third-party API. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 Flow chart for implementing the method of the present invention.

[0042] Figure 2 An unmanned aerial vehicle video sequence according to an embodiment of the present invention.

[0043] Figure 3 The embodiment of the present invention reconstructs a three-dimensional point cloud based on the drone video.

[0044] Figure 4 Point clouds after reconstruction of drone video sequences obtained at two different times in an embodiment of the present invention.

[0045] Figure 5 A diagram showing the registration effect of an embodiment of the present invention.

[0046] Figure 6 The embodiment of the present invention generates a candidate region.

[0047] Figure 7 Schematic diagram of noise removal according to an embodiment of the present invention.

[0048] Figure 8 Schematic diagram of segmentation example according to an embodiment of the present invention.

[0049] Fig. 9 The embodiment of the present invention filters schematic diagrams according to the number of instance points.

[0050] Fig.10 Schematic diagram of multi-level geometric attribute screening such as length, width, height and volume in an embodiment of the present invention.

[0051] Fig.11 The final effect diagram of the embodiment of the present invention. DETAILED DESCRIPTION

[0052] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0053] The technical problem to be solved by the present invention is: based on the above existing problems and drone video sequences, a three-dimensional change detection method based on drone video sequences is proposed. The method first reconstructs a three-dimensional point cloud through the drone video sequence; then aligns the two point clouds obtained at different times; secondly, obtains the candidate area through the difference between the two point clouds; finally, the candidate area is screened through the density and length, width, height and volume of the candidate area.

[0054] This method is simple to operate and can accurately locate the three-dimensional change area in the face of many interferences.

[0055] A 3D change detection method based on UAV video sequences.

[0056] The technical solution adopted by the present invention to solve the technical problem includes the following steps:

[0057] Step 1: Reconstruct the 3D point cloud of the scene from the drone video sequence;

[0058] Step 1-1: Obtain the image sequence of the drone video by taking frames at intervals;

[0059] Step 1-2: Sparsely reconstruct the image sequence using the motion recovery structure method;

[0060] Step 1-3: Further dense reconstruction is performed through the multi-view stereo algorithm to obtain denser point cloud data and complete the reconstruction.

[0061] Step 2: Reconstruct the two point clouds P at different times m and P s Perform registration;

[0062] Step 2-1: Randomly select n key points from each of the two point clouds and Get the local area where each key point is located and construct a local feature descriptor and

[0063] Step 2-2: Generate feature matching between two point cloud key points based on feature descriptors and select K feature matchings based on the feature matching scores

[0064] Step 2-3: Select any 3 feature matches from the selected K feature matches Generate rotation and translation transformation (R, t);

[0065]

[0066] Step 2-4: Point cloud P m Downsampling to get point cloud P′ m , calculate the point cloud P′ according to the rotation and translation transformation (R, t) m and P s The overlap rate between them is saved in each iteration. ov The highest rotation-translation transformation is (R result , t result ), if the iteration reaches a certain number of times or the overlap rate reaches the set threshold threshold_r ov Then stop the iteration;

[0067] Overlap rate r ov The specific steps of the calculation method include:

[0068] 1) The point cloud P′ is transformed by rotation and translation (R, t) m Transform to get the new point cloud P″ m ;

[0069] 2) In the point cloud P s Find the distance The nearest point if and The Euclidean distance value Less than the threshold value dis 1 ,Will Put into subcollection

[0070] 3) Overlap rate r ou The calculation is as follows:

[0071]

[0072] Among them, |·| represents the set size;

[0073] Step 2-5: Point cloud P m Through the rotation and translation transformation (R result , t result ) Get the new point cloud P m′ , so far, point cloud P m and P s Registration completed;

[0074] Step 3: Using point cloud P m From a single point to P s The candidate region is obtained by the minimum Euclidean distance value, and the regional density method is used to remove the noise in the candidate region to obtain the point cloud Pc ;

[0075] Step 3-1: In the point cloud P s Find the distance The nearest point if and The Euclidean distance value Less than the threshold value dis 2 ,Will Put into subcollection So far, the candidate region is obtained. The candidate region obtained here still has noise. In the next step, the noise in the candidate region is removed.

[0076] Step 3-2: Traversal Every point p in k , Use kdtree to determine the number of points in the spherical neighborhood. If the number of points is greater than the threshold num, p k Insert sub-point cloud P c , so far, the denoising is completed.

[0077] Step 4: Use the Euclidean clustering method to cluster the point cloud P c The candidate buildings are segmented into several candidate buildings, and then further screened according to their multi-level geometric attributes such as length, width, height and volume to obtain the final result.

[0078] Step 4-1: From the point cloud P c Choose any point Point Add to collection Will From the point cloud P c Removed;

[0079] Step 4-2: Traverse the collection Each point At the same time, traverse the point cloud P c Every point like and If the distance between them is less than the set threshold, the point Add to collection And from the point cloud P c Remove, the points that have been put in will not be put in again, the set The points in will not be traversed repeatedly, and the point cloud P c The points that have been removed in the next traversal of P c It will not be traversed again; repeat steps 4-2 until there are no more points in the set

[0080] Step 4-3: If the point cloud P c If there are still points left, continue to repeat steps 4-1 and 4-2 to construct the set Until the collection is constructed After that, point cloud P c There are no remaining points in the set; the number of sets obtained is the number of buildings. So far, all possible areas of each building have been obtained;

[0081] Step 4-4: Calculate the number of points in each candidate building. If the number is less than the preset threshold, the building is considered to be noise that has not been removed and is not a building.

[0082] Step 4-5: After the screening in step 4-4, the bounding box of the remaining candidate buildings is calculated. If the volume of the bounding box is less than the preset volume threshold, the candidate building is determined to be an erroneous result caused by the registration error, and the building is removed. The remaining candidate buildings are the final result. Specific embodiment:

[0084] The embodiment of the technical solution of the present invention provides a three-dimensional change detection method based on drone video sequence, and its process is as follows: Figure 1 As shown, it includes reconstructing a three-dimensional point cloud through a drone video sequence; selecting key points and constructing local features; registering the point cloud; obtaining a candidate area; and screening the candidate area. The following is a specific description of a three-dimensional change detection method based on a drone video sequence provided by the present invention in combination with an example.

[0085] (1) First, obtain the image sequence of the drone video by taking frames at intervals, such as Figure 2 As shown, the image sequence is then sparsely reconstructed using the structure from motion method. At this point, the reconstructed point cloud data is relatively sparse and cannot fully represent the geometric features of the entire object. It is necessary to further perform dense reconstruction using a multi-view stereo algorithm to obtain denser point cloud data. The reconstruction is now complete, as shown in Figure 3 As shown in the figure, the three-dimensional point cloud reconstructed from the video sequences acquired at two different times is as follows: Figure 4 shown.

[0086] (2) The initial state is as follows Figure 4 As shown in the figure, the two point clouds are not aligned, so the two point clouds need to be aligned before proceeding. The registration process is as follows: First, from the two point clouds P m and P s Randomly select key points and Get the local area where each key point is located and construct a local feature descriptor and Generate feature matching between two point cloud key points and select K feature matchings based on the feature matching scores Randomly rotate 3 from the selected K feature matches Generate a rotation and translation transformation (R, t), where R represents a 3x3 rotation matrix and t represents a 1x3 translation transformation. m Downsampling to get point cloud P′ m , calculate the point cloud P′ according to the rotation and translation transformation (R, t) m and P s The overlap rate between them is saved in each iteration. ov The highest rotation-translation transformation is (R result , t result ), if the iteration reaches a certain number of times or the overlap rate reaches the set threshold threshold_r ov Then stop the iteration. m Through the rotation and translation transformation (R result , t result ) Get the new point cloud P m′ , so far, the point cloud P m and P s The registration is completed, and the result after registration is as follows Figure 5 shown.

[0087] (3) In the point cloud P s Find the distance The nearest point if and The Euclidean distance value Less than a certain threshold value dis 2 ,Will Put into subcollection like Figure 6 As shown. There is a lot of interference noise in the obtained candidate area, which needs to be further removed. Every point p in i , Use kdtree to determine the number of points in the spherical neighborhood. If the number of points is greater than the threshold hum, p i Insert sub-point cloud P c , the results are as follows Figure 7 shown.

[0088] (4) For the point cloud P obtained in (3), c Further split into several candidate regions, the specific process is as follows: From the point cloud P c Choose any point Point Add to collection Initially, j = 0. From the point cloud P c Remove from the collection. Each point At the same time, traverse the point cloud P c Every point like and If the distance between them is less than a certain threshold, the point Add to collection And from the point cloud P c Remove, the points that have been put in will not be put in again, the set The points in will not be traversed repeatedly, and the point cloud P c The points that have been removed in the next traversal of P c Repeat the above steps until no more points are added to the set. If the point cloud P c If there are still points left, continue to repeat the above process to construct the set Until the collection is constructed After that, point cloud P c There are no remaining points in . The number of sets obtained is the number of buildings. So far, all possible areas of each building have been obtained.

[0089] The results are as follows Figure 8 After obtaining all building areas, the obtained areas are further screened using density method and multi-level geometric attributes of length, width, height and volume. The remaining area is the result area, as shown in Fig. 9 and Fig.10 shown.

[0090] The algorithm of the present invention is applied to actual three-dimensional change detection, and the effect is as follows Fig.11 As shown, the final building area, whose point cloud is colored and marked with a bounding box, is Fig.11 It can be seen that the method of the present invention is more accurate in detecting and labeling.

Claims

1. A 3D change detection method based on UAV video sequences, It is characterized in that The steps include: Step 1: Reconstruct the 3D point cloud of the scene from the drone video sequence; Step 2: Reconstruct two point clouds at different times and Perform registration; including: Step 2-1: Randomly select from two point clouds n Key Points and , obtain the local area where each key point is located and construct a local feature descriptor and ; Step 2-2: Generate feature matching between two point cloud key points based on feature descriptors and select based on feature matching scores K Feature matching ; Step 2-3: Select K Select any 3 feature matches from the feature matches Generate rotation and translation transformation ; In the formula, represents a 3x3 rotation matrix, Represents a 1x3 translation transformation; Step 2-4: Point Cloud Downsampling to get point cloud , according to the rotation and translation transformation Calculate point cloud and The overlap rate between them is saved in each iteration. The highest rotation-translation transformation is , if the iteration reaches a certain number of times or the overlap rate reaches the set threshold Then stop the iteration; Overlap rate The specific steps of the calculation method include: 1) Transformation by rotation and translation Point Cloud Transform to get a new point cloud ; 2) In the point cloud Find the distance The nearest point ;if The Euclidean distance value Less than threshold ,Will Put into subcollection ; 3) Overlap rate The calculation is as follows: in, Indicates the size of the collection; Step 2-5: Point Cloud Transform by rotation and translation Get a new point cloud , so far, point cloud and Registration completed; Step 3: Utilizing the Point Cloud From a single point The candidate area is obtained by the minimum Euclidean distance value, and the regional density method is used to remove the noise in the candidate area to obtain the point cloud ; Step 4: Use Euclidean clustering to cluster the point cloud The buildings are segmented into several candidate buildings, and then further screened according to the length, width, height and volume multi-level geometric attributes of the candidate buildings to obtain the final result.

2. According to claim 1, a three-dimensional change detection method based on drone video sequences, It is characterized in that The specific process of reconstructing the scene 3D point cloud according to the drone video sequence in step 1 includes: Step 1-1: Obtain the image sequence of the drone video by taking frames at intervals; Step 1-2: Sparsely reconstruct the image sequence using the motion recovery structure method; Step 1-3: Further dense reconstruction is performed through the multi-view stereo algorithm to obtain denser point cloud data and complete the reconstruction.

3. According to claim 1, a three-dimensional change detection method based on drone video sequences, It is characterized in that The specific steps of obtaining the candidate region and removing noise in step 3 are as follows: Step 3-1: In the point cloud Find the distance The nearest point ;if The Euclidean distance value Less than threshold ,Will Put into subcollection ; So far, the candidate region is obtained. The candidate region obtained here still has noise. The noise in the candidate region will be removed in the next step; Step 3-2: Traversal Every point in , use kdtree to determine the number of points in the spherical neighborhood, if the number of points is greater than the threshold ,Will Insert sub-point cloud , so far, the denoising is completed.

4. According to claim 3, a three-dimensional change detection method based on drone video sequences, It is characterized in that The step 4 is specifically as follows: Step 4-1: From point cloud Choose any point , point Add to collection , ,Will From point cloud Removed; Step 4-2: Traverse the collection Each point , while traversing the point cloud Every point ,like and If the distance between them is less than the set threshold, the point Add to collection And from the point cloud Remove, the points that have been put in will not be put in again, the set The points in will not be traversed repeatedly, the point cloud The points that have been removed in the next traversal It will not be traversed again; repeat steps 4-2 until there are no more points in the set ; Step 4-3: If the point cloud If there are still points left, continue to repeat steps 4-1 and 4-2 to construct the set , until the set is constructed Afterwards point cloud There are no remaining points in the set; the number of sets obtained is the number of buildings. So far, all possible areas of each building have been obtained. Step 4-4: Calculate the number of points in each candidate building. If the number is less than the preset threshold, the building is considered to be noise that has not been removed and is not a building. Step 4-5: After the screening in step 4-4, the bounding box of the remaining candidate buildings is calculated. If the volume of the bounding box is less than the preset volume threshold, the candidate building is determined to be an erroneous result caused by the registration error, and the building is removed. The remaining candidate buildings are the final result.

Images