Building change detection method and system based on unmanned aerial vehicle video comparison

Abstract

The invention discloses a building change detection method and system based on unmanned aerial vehicle video comparison, and relates to the technical field of unmanned aerial vehicles, and the method comprises the following steps: S10, setting an air route path of a to-be-detected area and an unmanned aerial vehicle patrol period; s20, acquiring a reference video along the set route by using the unmanned aerial vehicle; s30, using the unmanned aerial vehicle to collect a to-be-detected video along the set route path according to the set patrol period; s40, extracting a video key frame to be detected; s50, extracting a reference video key frame corresponding to the video key frame to be detected; s60, performing image correction and alignment on the reference video key frame and the video key frame to be detected, and obtaining a building target in an aligned image; and S70, extracting and comparing the building targets at the same position in the reference video key frame and the to-be-detected video key frame so as to identify the changed building targets. The method has the beneficial effects that the detection efficiency can be optimized, and the detection cost can be reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicles, and more specifically, to a method and system for detecting building changes based on unmanned aerial vehicle video comparison. Background Art

[0002] With the acceleration of urbanization, changes such as the addition, reconstruction, expansion, partial demolition or collapse of buildings occur frequently. Therefore, timely and accurate grasp of the changes in these buildings is of great significance to urban planning, safety monitoring, etc. Traditional methods often rely on manual field surveys or analysis based on satellite remote sensing images. These methods have problems such as low efficiency, high labor costs, slow update speed, and low resolution. In recent years, drones, with their high efficiency and flexibility, can regularly shoot the same area to obtain high-resolution image data, and then use automatic image block comparison technology to provide a new solution for building change detection.

[0003] However, the current detection method of building changes based on drones mainly uses the method of taking photos at fixed points on fixed routes for comparison. That is, on a fixed route, a group of photos are taken according to the pre-set waypoints as the reference map. After a period of time (according to the set inspection frequency), another group of photos are taken at the same waypoint position on the same route as the image to be detected. Next, compare the two sets of photos to ensure that they can correspond one by one. It is necessary to mark the position of the building image blocks in the photo in the reference map in advance, and then extract the building image blocks at the corresponding positions in the reference map and the image to be detected. Finally, use the automatic comparison algorithm to analyze whether the building has changed.

[0004] The main defects of this method are as follows: 1. Under the premise of ensuring the clarity and resolution of the photos, if the drone wants to take pictures of buildings covering the entire area in one flight, it often needs to take hundreds of photos. This requires the drone to set hundreds of waypoints in the route for fixed-point shooting. In order to obtain clear photos, the drone often has to slow down frequently or even hover. This method will greatly affect the operating efficiency of the drone. An area that originally took only 5 minutes to complete the inspection and record the video at the same time under uniform flight now takes more than half an hour to complete the inspection. In addition, the drone's battery life limits its ability to fly continuously in a large area. At the same time, this frequent deceleration and hovering method of taking pictures will reduce the battery life, resulting in a significant reduction in the cost-effectiveness of the detection method. 2. The calibration of buildings in the reference photos taken is done manually, which consumes human resources.

[0005] To this end, the present invention provides a building change detection method and system based on drone video comparison, which can make full use of the video data collected by the drone for comparison, intelligently detect changes in buildings, reduce manual participation, optimize detection efficiency, and reduce detection costs. Summary of the invention

[0006] In order to overcome the shortcomings of the prior art, the present invention provides a building change detection method and system based on drone video comparison, which can make full use of video data collected by drones for comparison, intelligently detect changes in buildings, reduce manual participation, optimize detection efficiency, and reduce detection costs.

[0007] The technical solution adopted by the present invention to solve the technical problem is: a building change detection method based on drone video comparison, the improvement of which is that the building change detection method based on drone video includes the following steps: S10, setting the route of the area to be inspected and the inspection cycle of the drone; S20, using the drone to collect reference videos along the set route path; S30, using a drone to collect videos to be inspected along a set route according to a set inspection cycle; S40, extracting key frames of the video to be detected; S50, extracting a reference video key frame corresponding to the video key frame to be detected; S60, performing image correction and alignment on the reference video key frame and the to-be-detected video key frame, and obtaining a building target in the aligned image; S70, extracting and comparing building targets at the same position in the reference video key frame and the key frame of the video to be detected to identify changed building targets.

[0008] Furthermore, in step S20, during the process of collecting the reference video, it is also necessary to collect and integrate the fixed-frequency data of the UAV to generate the first trajectory file of the UAV; wherein the fixed-frequency data includes the GPS information, flight altitude, flight speed, flight attitude, gimbal angle, camera parameters, video timestamp and battery power of the UAV.

[0009] Furthermore, in step S30, in the process of collecting the video to be detected, it is also necessary to collect and integrate the fixed-frequency data of the UAV to generate a second trajectory file of the UAV, and the flight parameters of the UAV are kept as consistent as possible with those in the process of collecting the reference video; wherein the flight parameters of the UAV include the flight altitude, flight speed, flight attitude and gimbal angle of the UAV.

[0010] Furthermore, in step S40, the specific method of extracting the key frames of the video to be detected is: according to the flight speed of the drone, frames are evenly extracted at fixed time intervals to serve as the key frames of the video to be detected, and a small overlap between adjacent key frame images is allowed.

[0011] Furthermore, in the process of extracting the key frame of the video to be detected, it is also necessary to parse the GPS information of the key frame of the video to be detected. The specific method of parsing the GPS information of the key frame of the video to be detected is: Extract the first target key frame, the frame number of the target key frame in the video to be detected is , the video timestamp is ; Determine that the two times when the fixed-frequency data are recorded before and after the first target key frame are and , and read the drones respectively and GPS information, where the drone is at the time The GPS information is recorded as , Indicates that the drone is at time The longitude, Indicates that the drone is at time The latitude, Indicates that the drone is at time The altitude of the drone at time The GPS information is recorded as , Indicates that the drone is at time The longitude, Indicates that the drone is at time The latitude, Indicates that the drone is at time Height; Calculate the GPS information of the first target key frame: The GPS information of the first target key frame is recorded as ,in: represents the longitude of the drone in the first target keyframe, The calculation expression is: ; represents the latitude of the drone in the first target keyframe, The calculation expression is: ; Indicates the height of the drone in the first target keyframe, The calculation expression is: ; In the above formula, .

[0012] Furthermore, in step S50, the specific method of extracting the reference video key frame corresponding to the video key frame to be detected is: S501, determining a search range of a second target key frame corresponding to a first target key frame in a reference video; specifically, the frame number of the second target key frame in the reference video is , the video timestamp of the second target key frame is recorded as ,and The value range is set in In which, Set to 10s; S502, parsing the reference video at the timestamp Target trajectory file within the range, and obtain the GPS information recorded in the target trajectory file , where n represents The number of times the fixed-frequency data of the drone within the range is collected; S503, in Take out and The GPS information of the closest second target key frame is recorded as ; and determine The corresponding video timestamp , The corresponding frame number of the second target key frame is ,in, , Indicates the video frame rate, The value of is 30; S504, according to the frame sequence number Extract frames from the reference video and add them in the frame number Additional extraction before and after frames, and select the frame with the highest similarity to the first target key frame as the final second target key frame; wherein the value of M is 30.

[0013] Furthermore, in step S60, the specific method of performing image correction and alignment on the reference video key frame and the key frame of the video to be detected is: Extract feature vectors of the first target key frame and the second target key frame using SIFT algorithm; Match feature points based on feature vectors and calculate the affine transformation matrix; The first target key frame and the second target key frame are accurately aligned with each other by means of an affine transformation matrix to obtain an aligned image of the first target key frame relative to the second target key frame.

[0014] Furthermore, the specific method of obtaining the building target in the alignment image is: using the YOLO-NAS detection algorithm to train the building recognition model, and positioning the building target in the alignment image of the first target key frame relative to the second target key frame through the trained building recognition model.

[0015] Furthermore, in step S70, the specific method of extracting and comparing the building targets at the same position in the reference video key frame and the key frame of the video to be detected to identify the changed building targets is as follows: S701, using a building recognition model to extract building target image blocks at the same position in the reference video key frame and the key frame of the video to be detected, wherein the building target image block of the reference video key frame is denoted as A, and the building target image block of the key frame of the video to be detected is denoted as B; S702, determine a local window, slide the local window on A and B, and calculate the mean value of each window in image block A during the sliding process ,variance , the mean value in each window of image block B ,variance And the covariance of image block A and image block B ; S703, calculating the brightness contrast items of image block A and image block B respectively according to the mean, variance, and covariance obtained in each image , Contrast comparison items , Structural comparison items , to measure the similarity between image block A and image block B in brightness, contrast and structure, where: ; ; ; In the above formula, , , are constants, The calculation expression is , The calculation expression is , The calculation expression is , Indicates the pixel value range; S704, according to the formula Calculate the SSIM value of each window, and then take the average of the SSIM values ​​of all windows as the global SSIM value to indicate the similarity between image block A and image block B in the overall structure. S705, set the SSIM threshold according to application requirements, and compare the global SSIM value with the SSIM threshold. If the global SSIM value is greater than or equal to the SSIM threshold, image block A and image block B are similar in structure and the building target has not changed; if the global SSIM value is less than the SSIM threshold, image block A and image block B are different in structure and the building target has changed.

[0016] A building change detection system based on drone video comparison is applied to the building change detection method based on drone video comparison as described above, wherein the improvement lies in that the building change detection system based on drone video comparison comprises: The flight mission planning module is used to set the flight path of the area to be inspected and the drone inspection cycle; A video acquisition module, used to acquire reference videos along a set route path and to acquire videos to be detected along a set route path according to a set inspection cycle; A video frame extraction module, used to extract the key frames of the video to be detected and the reference video key frames corresponding to the key frames of the video to be detected; An image processing module is used to perform image correction and alignment on the reference video key frame and the key frame of the video to be detected, and obtain the building target in the aligned image; The building target change comparison module is used to extract and compare the building targets at the same position in the reference video key frame and the key frame of the video to be detected to identify the changed building targets.

[0017] The beneficial effects of the present invention are as follows: the present invention successively collects a reference video and a video to be detected of the area to be inspected, extracts and corrects the alignment of the corresponding reference video key frames and the video key frames to be detected, detects the building targets in the alignment diagram of the reference video key frames and the video key frames to be detected, extracts and compares the building targets at the same position in the reference video key frames and the video key frames to be detected, so as to identify the changed building targets; in this process, when the drone is patrolling, there is no need to stop at a fixed point to take pictures, and there is no need to manually calibrate the buildings in the reference photos. Therefore, compared with the prior art, the present invention can make full use of the video data collected by the drone for comparison, intelligently detect changes in buildings, reduce manual participation, so as to optimize detection efficiency and reduce detection costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is an overall flow chart of a building change detection method based on drone video comparison of the present invention; Figure 2 A block diagram of a building change detection system based on drone video comparison of the present invention; Figure 3A hardware structure diagram of an electronic device is shown in an exemplary embodiment; Figure 4 The present invention is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0019] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0020] The following will clearly and completely describe the concept, specific structure and technical effects of the present invention in combination with the embodiments and drawings, so as to fully understand the purpose, characteristics and effects of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, other embodiments obtained by technicians in this field without creative work are all within the scope of protection of the present invention. In addition, all the connection / connection relationships involved in the patent do not refer to the direct connection of components, but refer to the formation of a better connection structure by adding or reducing connection accessories according to the specific implementation situation. The various technical features in the invention can be combined interchangeably without conflicting with each other.

[0021] Reference Figure 1 As shown, the present invention discloses a building change detection method based on drone video comparison, and the building change detection method based on drone video includes the following steps: S10, setting the flight path of the area to be inspected and the inspection cycle of the drone; in this embodiment, during the specific application of the drone, the ground control platform of the drone sets the flight path of the area to be inspected and the inspection cycle of the drone. After the setting is completed, the ground control platform controls the drone to perform the inspection task; specifically, the setting of the flight path needs to take into account the distribution of buildings in the area to be inspected, ensuring that the drone can cover all building targets in the area to be inspected during the flight mission; the inspection cycle determines how often the drone needs to perform a flight mission to collect new video data, and the setting of the inspection cycle should be determined according to the speed of building changes. For example, some areas may need to be monitored every week, while some areas may only need to be monitored once a month; S20, using the drone to collect a reference video along the set route; in the process of collecting the reference video, it is also necessary to collect and integrate the fixed-frequency data of the drone to generate a first trajectory file of the drone; wherein the fixed-frequency data includes GPS information, flight altitude, flight speed, flight attitude, gimbal angle, camera parameters, video timestamp, and battery power of the drone; It should be noted that, in this embodiment, before conducting building monitoring, the ground control platform controls the UAV to conduct a first inspection along a set route path to collect a baseline video of the area to be inspected. The baseline video records the initial state of the building in the area to be inspected, which serves as a comparison standard for whether the subsequent building has changed; and in the process of collecting the baseline video, the UAV will monitor its own fixed-frequency data and send it to the ground control platform, which will then be integrated into a first trajectory file by the ground control platform as a supplement to the accuracy and completeness of the baseline video data; specifically, the UAV end is equipped with a high-definition camera and various functional sensors. In the specific process of collecting the baseline video of the area to be inspected, the UAV end flies at a uniform speed along the set route path, and uses a high-definition camera to shoot video in a vertical overhead manner as a baseline video of the area to be inspected. In the process of collecting the video, the UAV will collect the fixed-frequency data of the UAV through various functional sensors. Specifically, the fixed-frequency data includes The GPS information, flight altitude, flight speed, flight attitude, gimbal angle, camera parameters, video timestamp and battery power of the drone, wherein the GPS information can help locate the position of the building in the video relative to the geographic coordinates; the flight altitude, flight speed, flight attitude and gimbal angle are used to display the current flight status of the drone, which can provide a reference for the drone when performing subsequent inspection tasks; the camera parameters include focal length, exposure time, ISO, etc., which can determine the clarity and brightness of the captured video; the video timestamp is used to record the shooting time of the video to ensure that the video files can be sorted in chronological order; the battery power monitoring can help determine whether the drone will interrupt the mission due to insufficient power; in addition, most drone manufacturers will provide APIs for developers to access some data, but the data push frequency is usually limited by the SDK design and cannot reach the high frequency of the original sensor, which is generally 0.5Hz, that is, the current data is transmitted every 2s; S30, using the drone to collect the video to be detected along the set route according to the set inspection cycle; in the process of collecting the video to be detected, it is also necessary to collect and integrate the fixed frequency data of the drone to generate a second trajectory file of the drone, and the flight parameters of the drone are kept consistent with those in the process of collecting the reference video as much as possible; wherein the flight parameters of the drone include the flight altitude, flight speed, flight attitude and gimbal angle of the drone; In this embodiment, the principle of the drone collecting the video to be detected and generating the second trajectory file is the same as that of collecting the reference video and generating the first trajectory file, which will not be described in detail here; in addition, the flight parameters of the reference video and the video to be detected are kept consistent as much as possible, which can ensure that the accuracy of subsequent building change detection will not be affected by the difference in flight parameters, thereby ensuring the reliability of the detection result; S40, extracting key frames of the video to be detected; specifically, the specific method for extracting the key frames of the video to be detected is: according to the flight speed of the drone, frames are evenly extracted at fixed time intervals to serve as key frames of the video to be detected, and a small overlap between adjacent key frame images is allowed (for example, when the drone is shooting, the entire screen can be captured in 5 seconds, and the frame extraction time interval can be set to 4.5 seconds); in addition, in the process of extracting the key frames of the video to be detected, the GPS information of the key frames of the video to be detected needs to be parsed, and the specific method for parsing the GPS information of the key frames of the video to be detected is: Extract the first target key frame, the frame number of the target key frame in the video to be detected is , the video timestamp is ; Determine that the two times when the fixed-frequency data are recorded before and after the first target key frame are and , and read the drones respectively and GPS information, where the drone is at the time The GPS information is recorded as , Indicates that the drone is at time Longitude, Indicates that the drone is at time The latitude, Indicates that the drone is at time The altitude of the drone at time The GPS information is recorded as , Indicates that the drone is at time Longitude, Indicates that the drone is at time The latitude, Indicates that the drone is at time Height; Calculate the GPS information of the first target key frame: The GPS information of the first target key frame is recorded as ,in: represents the longitude of the drone in the first target keyframe, The calculation expression is: ; represents the latitude of the drone in the first target keyframe, The calculation expression is: ; Indicates the height of the drone in the first target keyframe, The calculation expression is: ; In the above formula, ; It should be noted that, in this embodiment, since the fixed-frequency data in the second trajectory file is recorded once every 2 seconds, it means that the GPS information is also recorded once every 2 seconds. It is known that the frame rate of the video to be detected is fps (fps is usually 30), and there are 2fps frames in a 2s time period. Therefore, the probability that the currently extracted key frame can directly obtain its GPS information is only 1 / 2fps, and in most cases it is missing. It is necessary to calculate the approximate GPS information of the key frame through the above process; S50, extracting a reference video key frame corresponding to the key frame of the video to be detected; specifically, the specific method of extracting the reference video key frame corresponding to the key frame of the video to be detected is: S501, determining a search range of a second target key frame corresponding to a first target key frame in a reference video; specifically, the frame number of the second target key frame in the reference video is , the video timestamp of the second target key frame is recorded as ,and The value range is set in In which, Set to 10s; It should be noted that, in this embodiment, due to the limitations of the drone's own performance, the wind speed on the day of flight, and operational factors such as turns, even if the same flight parameters as the reference video are used when shooting the video to be detected, the two videos cannot have exactly the same duration and number of frames. Moreover, under the same video timestamp or the same frame number, the image screens corresponding to the two videos are also different and are not in the same position. Therefore, in the process of extracting frames from the reference video, the range of the video timestamp of the second target key frame is first determined to provide a basis for subsequently selecting the second target key frame that is most similar to the first target key frame within the range; S502, parsing the reference video at the timestamp Target trajectory file within the range, and obtain the GPS information recorded in the target trajectory file , where n represents The number of times the fixed-frequency data of the drone within the range is collected; S503, in Take out and The GPS information of the closest second target key frame is recorded as ; and determine The corresponding video timestamp , The corresponding frame number of the second target key frame is ,in, , Indicates the video frame rate, The value of is 30; S504, according to the frame sequence number Extract frames from the reference video and add them in the frame number Additional extraction before and after frames, and select the frame with the highest similarity to the first target key frame as the final second target key frame; wherein the value of M is 30; It should be noted that, in this embodiment, in order to ensure that the image that best matches the key frame of the video to be detected is obtained, when extracting the key frame of the reference video, in addition to extracting the key frame with the sequence number In addition to the key frame of the video to be detected, the M frames before and after are additionally extracted, and the total of 2M+1 frames are matched with the key frame of the video to be detected for image features, and the frame with the highest similarity to the key frame of the video to be detected is selected and used as the frame that best matches the key frame of the video to be detected, that is, the final reference video key frame (the second target key frame); S60, performing image correction alignment on the reference video key frame and the key frame of the video to be detected, and obtaining the building target in the aligned image; specifically, the specific manner of performing image correction alignment on the reference video key frame and the key frame of the video to be detected is: Extract feature vectors of the first target key frame and the second target key frame using SIFT algorithm; Match feature points based on feature vectors and calculate the affine transformation matrix; Accurately aligning the first target key frame and the second target key frame by using an affine transformation matrix to obtain an aligned image of the first target key frame relative to the second target key frame; It should be noted that, in this embodiment, the SIFT algorithm is an algorithm for extracting key feature points from an image. These feature points are points in the image with strong distinguishability and stability (such as corner points or edges). By applying the algorithm to the first target key frame and the second target key frame, respective feature vectors can be extracted. The feature vector image contains key information of the first target key frame / the second target key frame, and this information is used to compare the images in the subsequent matching process; the feature point matching means that by comparing the feature vectors of the first target key frame and the second target key frame, the same or similar feature points in the first target key frame and the second target key frame can be found. The position comparison of these feature points in space helps to determine the relative relationship between the images; the affine transformation matrix can describe how an image is transformed from one coordinate system to another, including transformations such as translation, rotation, and scaling, so as to ensure that the first target key frame and the second target key frame can be accurately aligned. Specifically, through the affine transformation matrix, the feature points of the first target key frame will be accurately matched to the corresponding feature points of the second target key frame, thereby obtaining an aligned image, that is, an image that is relatively aligned with the second target key frame after transformation; In addition, the specific method of obtaining the building target in the alignment image is: using the yolo-nas detection algorithm to train the building recognition model, and positioning the building target in the alignment image of the first target key frame relative to the second target key frame through the trained building recognition model; It should be noted that, in this embodiment, since the comparison is performed by automatically capturing images from the video, which is different from directly taking photos to obtain the reference map, it is difficult to manually mark the location of the building in the reference map in advance using annotation tools; therefore, it is necessary to use a deep learning target detection method to automatically detect the building targets in the alignment map of the reference video key frame and the key frame of the video to be detected; specifically, it is necessary to pre-train a building recognition model and load it on the drone end or the ground system end, and when conducting inspection operations, use the reasoning ability of the model to locate the building targets in the alignment map of the reference video key frame and the key frame of the video to be detected; specifically, the training method of the building recognition model includes the following steps: First, a building dataset of the area to be inspected is constructed. Each sample data contains an image and a label file. The label file marks the location of the building in the image and the category label. Then, the dataset is divided into a training set and a validation set in a ratio of 4:1. The training set is used for model parameter learning, and the validation set is used to evaluate the performance of the model during training. Next, load the pre-trained model of yolo-nas (which can help the model converge faster), and set hyperparameters such as learning rate, batch size, and number of training rounds. During training, the model will continuously read data from the training set for forward propagation to calculate losses, and update parameters through back propagation. Loading the pre-trained model of yolo-nas means importing a yolo-nas model that has been trained on a large-scale data set into the current environment for inference, fine-tuning, or further training. Finally, after the training is completed, the model performance is evaluated and the model with the best performance is selected as the final building recognition model; It should also be noted that the YOLO-NAS detection algorithm is a target detection algorithm based on YOLO (You Only Look Once), which combines the Neural Architecture Search (NAS) technology to optimize the model structure through an automated method to achieve higher detection accuracy and efficiency. S70, extracting and comparing the building targets at the same position in the reference video key frame and the key frame of the video to be detected to identify the changed building targets; specifically, the extracting and comparing the building targets at the same position in the reference video key frame and the key frame of the video to be detected to identify the changed building targets is as follows: S701, using a building recognition model to extract building target image blocks at the same position in the reference video key frame and the key frame of the video to be detected, wherein the building target image block of the reference video key frame is denoted as A, and the building target image block of the key frame of the video to be detected is denoted as B; S702, determine a local window, slide the local window on A and B, and calculate the mean value of each window in image block A during the sliding process ,variance , the mean value in each window of image block B ,variance And the covariance of image block A and image block B ; S703, calculating the brightness contrast items of image block A and image block B respectively according to the mean, variance, and covariance obtained in each image , Contrast comparison items , Structural comparison items , to measure the similarity between image block A and image block B in brightness, contrast and structure, where: ; ; ; In the above formula, , , are constants, The calculation expression is , The calculation expression is , The calculation expression is , Indicates the pixel value range; S704, according to the formula Calculate the SSIM value of each window, and then take the average of the SSIM values ​​of all windows as the global SSIM value to indicate the similarity between image block A and image block B in the overall structure. S705, set the SSIM threshold according to application requirements, and compare the global SSIM value with the SSIM threshold. If the global SSIM value is greater than or equal to the SSIM threshold, image block A and image block B are similar in structure and the building target has not changed; if the global SSIM value is less than the SSIM threshold, image block A and image block B are different in structure and the building target has changed.

[0022] It should be noted that, in this embodiment, the similarity between image block A and image block B in the overall structure is represented by the global SSIM value, which is more in line with the judgment of the human visual system on image similarity; in addition, when a change in a building target is detected, the changed building image block needs to be marked on the reference video key frame and the key frame of the video to be detected, and a json file of the video timestamp and GPS information corresponding to these key frames is generated at the same time. Subsequently, the image information and text information are integrated together as an early warning information package and sent to relevant personnel for further verification and processing, so as to provide timely decision-making basis for the maintenance and management of the building.

[0023] Reference Figure 2 As shown, the present invention also discloses a building change detection system 600 based on drone video comparison, which is applied to a building change detection method based on drone video comparison as described in the above embodiment. The building change detection system 600 based on drone video comparison includes: The flight mission planning module 601 is used to set the flight path of the area to be inspected and the inspection cycle of the drone; The video acquisition module 602 is used to acquire a reference video along a set route path and to acquire a video to be detected along a set route path according to a set inspection cycle; The video frame extraction module 603 is used to extract the key frames of the video to be detected and the reference video key frames corresponding to the key frames of the video to be detected; The image processing module 604 is used to perform image correction and alignment on the reference video key frame and the key frame of the video to be detected, and obtain the building target in the aligned image; The building target change comparison module 605 is used to extract and compare the building targets at the same position in the reference video key frame and the key frame of the video to be detected, so as to identify the changed building targets.

[0024] It should be noted that the building change detection strategy based on drone video comparison provided in the above embodiment is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the building change detection system 600 based on drone video comparison will be divided into different functional modules to complete all or part of the functions described above. In addition, the building change detection system 600 based on drone video comparison provided in the above embodiment and the embodiment of the building change detection method based on drone video comparison belong to the same concept, and the specific way in which each module performs the operation has been described in detail in the method embodiment and will not be repeated here.

[0025] Figure 3 The present invention schematically illustrates the structure of an electronic device according to an exemplary embodiment.

[0026] It should be noted that the electronic device is only an example adapted to the present invention and cannot be considered to provide any limitation on the scope of use of the present invention. The electronic device cannot be interpreted as requiring dependence on or having Figure 3 One or more components of exemplary electronic device 2000 are shown.

[0027] The hardware structure of the electronic device 2000 may vary greatly due to differences in configuration or performance, such as Figure 3 As shown, the electronic device 2000 includes: a power supply 210 , an interface 230 , at least one memory 250 , and at least one central processing unit (CPU) 270 .

[0028] Specifically, the power supply 210 is used to provide operating voltage for each hardware device on the electronic device 2000 .

[0029] The interface 230 includes at least one wired or wireless network interface 231 for interacting with external devices. Of course, in other examples adapted by the present invention, the interface 230 may further include at least one serial-to-parallel conversion interface 233, at least one input-output interface 235, and at least one USB interface 237, etc. Figure 3 As shown, this is not intended to be a specific limitation.

[0030] The memory 250 is a carrier for storing resources, which may be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon include an operating system 251, an application 253 and data 255, etc. The storage method may be temporary storage or permanent storage.

[0031] Among them, the operating system 251 is used to manage and control various hardware devices and application programs 253 on the electronic device 2000 to enable the central processor 270 to calculate and process the massive data 255 in the memory 250. It can be WindowsServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0032] The application program 253 is a computer-readable instruction that performs at least one specific task based on the operating system 251, and may include at least one module ( Figure 3 (not shown), each module may respectively include computer-readable instructions for the electronic device 2000. For example, the device for assessing task priority may be regarded as an application 253 deployed on the electronic device 2000.

[0033] The data 255 may be signal information, etc., stored in the memory 250 .

[0034] The central processor 270 may include one or more processors and is configured to communicate with the memory 250 through at least one communication bus to read the computer-readable instructions stored in the memory 250, thereby realizing the operation and processing of the mass data 255 in the memory 250. For example, the central processor 270 reads a series of computer-readable instructions stored in the memory 250 to complete the building change detection method based on drone video comparison.

[0035] In addition, the present invention can also be implemented by hardware circuits or hardware circuits combined with software. Therefore, the implementation of the present invention is not limited to any specific hardware circuits, software, or a combination of the two.

[0036] See also Figure 4 In an embodiment of the present invention, an electronic device 4000 is provided. The electronic device 4000 may include: a desktop computer, a laptop computer, a server, etc. with sensor recognition capability.

[0037] exist Figure 4 In the embodiment, the electronic device 4000 includes at least one processor 4001 and at least one memory 4003.

[0038] The data interaction between the processor 4001 and the memory 4003 can be realized through at least one communication bus 4002. The communication bus 4002 may include a path for transmitting data between the processor 4001 and the memory 4003. The communication bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The communication bus 4002 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0039] Optionally, the electronic device 4000 may further include a transceiver 4004, which may be used for data interaction between the electronic device and other electronic devices, such as data transmission and / or data reception, etc. It should be noted that in actual applications, the transceiver 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present invention.

[0040] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of the present invention. Processor 4001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0041] The memory 4003 can be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program instructions or codes in the form of instructions or data structures and can be accessed by the electronic device 4000, but is not limited to this.

[0042] Computer-readable instructions are stored in the memory 4003 , and the processor 4001 can read the computer-readable instructions stored in the memory 4003 through the communication bus 4002 .

[0043] The computer-readable instructions are executed by one or more processors 4001 to implement the building change detection method based on drone video comparison in the above embodiments.

[0044] In addition, a storage medium is provided in an embodiment of the present invention, on which computer-readable instructions are stored, and the computer-readable instructions are executed by one or more processors to implement the building change detection method based on drone video comparison as described above.

[0045] Compared with the prior art, the present invention has the following beneficial effects: 1. The present invention successively collects the reference video and the video to be detected of the area to be inspected, extracts and corrects the alignment of the corresponding reference video key frames and the key frames of the video to be detected, detects the building targets in the alignment diagram of the reference video key frames and the key frames of the video to be detected, and extracts and compares the building targets at the same position in the reference video key frames and the key frames of the video to be detected, so as to identify the changed building targets; in this process, when the drone is patrolling, there is no need to stop at a fixed point to take pictures, and there is no need to manually calibrate the buildings in the reference photos, so that the video data collected by the drone can be fully utilized for comparison, and the changes of the buildings can be intelligently detected, reducing manual participation, so as to optimize the detection efficiency and reduce the detection cost; 2. The present invention utilizes the fast and flexible characteristics of drones to collect video data of buildings in real time, and analyzes it through intelligent algorithms to promptly discover changes in buildings, providing timely decision-making basis for building maintenance and management, greatly improving detection efficiency, and is particularly suitable for monitoring buildings in large areas.

[0046] The above is a specific description of the preferred implementation of the present invention, but the invention is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A building change detection method based on drone video comparison, characterized in that: The method for detecting building changes in drone videos comprises the following steps: S10, setting the route of the area to be inspected and the inspection cycle of the drone; S20, using the drone to collect reference videos along the set route path; S30, using a drone to collect videos to be inspected along a set route according to a set inspection cycle; S40, extracting key frames of the video to be detected; S50, extracting a reference video key frame corresponding to the video key frame to be detected; S60, performing image correction and alignment on the reference video key frame and the to-be-detected video key frame, and obtaining a building target in the aligned image; S70, extracting and comparing building targets at the same position in the reference video key frame and the key frame of the video to be detected to identify changed building targets.

2. The building change detection method based on drone video comparison according to claim 1 is characterized in that: In step S20, during the process of collecting the reference video, the fixed-frequency data of the drone needs to be collected and integrated to generate the first trajectory file of the drone; wherein the fixed-frequency data includes the drone's GPS information, flight altitude, flight speed, flight attitude, gimbal angle, camera parameters, video timestamp, and battery power.

3. The building change detection method based on drone video comparison according to claim 2 is characterized in that: In step S30, in the process of collecting the video to be detected, it is also necessary to collect and integrate the fixed-frequency data of the drone to generate a second trajectory file of the drone, and the flight parameters of the drone should be kept as consistent as possible with those in the process of collecting the reference video; wherein the flight parameters of the drone include the flight altitude, flight speed, flight attitude and gimbal angle of the drone.

4. The building change detection method based on drone video comparison according to claim 3 is characterized in that: In step S40, the specific method of extracting the key frames of the video to be detected is: according to the flight speed of the drone, frames are evenly extracted at fixed time intervals to serve as the key frames of the video to be detected, and a small overlap between adjacent key frame images is allowed.

5. The building change detection method based on drone video comparison according to claim 4 is characterized in that: In the process of extracting the key frame of the video to be detected, it is also necessary to parse the GPS information of the key frame of the video to be detected. The specific method of parsing the GPS information of the key frame of the video to be detected is: Extract the first target key frame, the frame number of the target key frame in the video to be detected is , the video timestamp is ; Determine that the two times when the fixed-frequency data are recorded before and after the first target key frame are and , and read the drones respectively and GPS information, where the drone is at the time The GPS information is recorded as , Indicates that the drone is at time Longitude, Indicates that the drone is at time The latitude, Indicates that the drone is at time The altitude of the drone at time The GPS information is recorded as , Indicates that the drone is at time Longitude, Indicates that the drone is at time The latitude, Indicates that the drone is at time Height; Calculate the GPS information of the first target key frame: The GPS information of the first target key frame is recorded as ,in: represents the longitude of the drone in the first target keyframe, The calculation expression is: ; represents the latitude of the drone in the first target keyframe, The calculation expression is: ; Indicates the height of the drone in the first target keyframe, The calculation expression is: ; In the above formula, .

6. The building change detection method based on drone video comparison according to claim 5 is characterized in that: In step S50, the specific method of extracting the reference video key frame corresponding to the video key frame to be detected is: S501, determining a search range of a second target key frame corresponding to a first target key frame in a reference video; specifically, the frame number of the second target key frame in the reference video is , the video timestamp of the second target key frame is recorded as ,and The value range is set in In which, Set to 10s; S502, parsing the reference video at the timestamp Target trajectory file within the range, and obtain the GPS information recorded in the target trajectory file , where n represents The number of times the fixed-frequency data of the drone within the range is collected; S503, in Take out and The GPS information of the closest second target key frame is recorded as ; and determine The corresponding video timestamp , The corresponding frame number of the second target key frame is ,in, , Indicates the video frame rate, The value of is 30; S504, according to the frame sequence number Extract frames from the reference video and add them in the frame number Additional extraction before and after frames, and select the frame with the highest similarity to the first target key frame as the final second target key frame; wherein the value of M is 30.

7. The building change detection method based on drone video comparison according to claim 6 is characterized in that: In step S60, the specific method of performing image correction and alignment on the reference video key frame and the key frame of the video to be detected is: Extract feature vectors of the first target key frame and the second target key frame using SIFT algorithm; Match feature points based on feature vectors and calculate the affine transformation matrix; The first target key frame and the second target key frame are accurately aligned with each other by means of an affine transformation matrix to obtain an aligned image of the first target key frame relative to the second target key frame.

8. The building change detection method based on drone video comparison according to claim 7 is characterized in that: The specific method of obtaining the building target in the alignment image is: using the YOLO-NAS detection algorithm to train the building recognition model, and positioning the building target in the alignment image of the first target key frame relative to the second target key frame through the trained building recognition model.

9. The building change detection method based on drone video comparison according to claim 8 is characterized in that: In step S70, the method of extracting and comparing the building targets at the same position in the reference video key frame and the key frame of the video to be detected to identify the changed building targets is as follows: S701, using a building recognition model to extract building target image blocks at the same position in the reference video key frame and the key frame of the video to be detected, wherein the building target image block of the reference video key frame is denoted as A, and the building target image block of the key frame of the video to be detected is denoted as B; S702, determine a local window, slide the local window on A and B, and calculate the mean value of each window in image block A during the sliding process ,variance , the mean value in each window of image block B ,variance And the covariance of image block A and image block B ; S703, calculating the brightness contrast items of image block A and image block B respectively according to the mean, variance, and covariance obtained in each image , Contrast comparison items , Structural comparison items , to measure the similarity between image block A and image block B in brightness, contrast and structure, where: ； ； ； In the above formula, , , are constants, The calculation expression is , The calculation expression is , The calculation expression is , Indicates the pixel value range; S704, according to the formula Calculate the SSIM value of each window, and then take the average of the SSIM values ​​of all windows as the global SSIM value to indicate the similarity between image block A and image block B in the overall structure. S705, set the SSIM threshold according to application requirements, and compare the global SSIM value with the SSIM threshold. If the global SSIM value is greater than or equal to the SSIM threshold, image block A and image block B are similar in structure and the building target has not changed; if the global SSIM value is less than the SSIM threshold, image block A and image block B are different in structure and the building target has changed.

10. A building change detection system based on drone video comparison, applied to a building change detection method based on drone video comparison according to any one of claims 1 to 9, characterized in that: The building change detection system based on drone video comparison includes: The flight mission planning module is used to set the flight path of the area to be inspected and the drone inspection cycle; A video acquisition module, used to acquire reference videos along a set route path and to acquire videos to be detected along a set route path according to a set inspection cycle; A video frame extraction module, used to extract the key frames of the video to be detected and the reference video key frames corresponding to the key frames of the video to be detected; An image processing module is used to perform image correction and alignment on the reference video key frame and the key frame of the video to be detected, and obtain the building target in the aligned image; The building target change comparison module is used to extract and compare the building targets at the same position in the reference video key frame and the key frame of the video to be detected to identify the changed building targets.

Images