A Method and System for Detecting Building Changes Based on UAV Video Comparison

Abstract

The present invention discloses a building change detection method and system based on drone video comparison, which relates to the field of drone technology and includes the following steps: S10, setting a flight path and drone inspection cycle for the area to be inspected; S20, using the drone to collect a reference video along the set flight path; S30, using the drone to collect the video to be detected along the set flight path according to the set inspection cycle; S40, extracting key frames of the video to be detected; S50, extracting key frames of the reference video corresponding to the key frames of the video to be detected; S60, performing image correction and alignment on the reference video key frames and the key frames of the video to be detected, and obtaining building targets in the aligned images; S70, extracting and comparing building targets at the same position in the reference video key frames and the key frames of the video to be detected to identify changed building targets. The present invention has the beneficial effects of optimizing detection efficiency and reducing detection costs.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) technology, and more specifically, to a method and system for detecting building changes based on UAV video comparison. Background Technology

[0002] With the acceleration of urbanization, changes such as additions, alterations, expansions, partial demolitions, and collapses of buildings occur frequently. Therefore, timely and accurate monitoring of these building changes is of great significance for urban planning and safety monitoring. Traditional methods often rely on manual on-site surveys or analysis based on satellite remote sensing imagery, which suffer from low efficiency, high labor costs, slow update speed, and low resolution. In recent years, drones, with their high efficiency and flexibility, can periodically photograph the same area to acquire high-resolution image data. Then, through automatic image patch comparison technology, a new solution for building change detection has been provided.

[0003] However, current methods for detecting building changes using drones primarily rely on comparing images taken at fixed points along a fixed flight path. This involves taking a set of photos along a fixed flight path at pre-defined waypoints to serve as a baseline image. After a period of time (based on a set patrol frequency), another set of photos is taken at the same waypoints along the same flight path to serve as the detection image. The two sets of photos are then compared to ensure a one-to-one correspondence. This requires pre-marking the locations of building image blocks in the photos on the baseline image, and then extracting the corresponding building image blocks from both the baseline and detection images. Finally, an automatic comparison algorithm is used to analyze and determine whether the building has changed.

[0004] The main drawbacks of this method are as follows: 1. To ensure image clarity and resolution, hundreds of photos are often needed to cover the entire area of ​​buildings in a single drone flight. This requires setting hundreds of waypoints along the drone's flight path for fixed-point shooting. To obtain clear photos, the drone often needs to frequently decelerate or even hover. This significantly impacts the drone's operational efficiency. An area that could be inspected and recorded simultaneously in just 5 minutes at a constant speed now takes more than half an hour to inspect. Furthermore, the drone's flight time already limits its ability to fly continuously over large areas. Additionally, this frequent deceleration and hovering for photography reduces battery life, significantly diminishing the cost-effectiveness of this inspection method. 2. The labeling of buildings in the reference photos is done manually, which consumes considerable manpower.

[0005] To address this, the present invention provides a method and system for detecting building changes based on UAV video comparison. This method can fully utilize video data collected by UAVs for comparison, intelligently detect changes in buildings, reduce human intervention, optimize detection efficiency, and lower detection costs. Summary of the Invention

[0006] To overcome the shortcomings of existing technologies, this invention provides a method and system for detecting building changes based on UAV video comparison. This method can make full use of video data collected by UAVs for comparison, intelligently detect changes in buildings, reduce human intervention, optimize detection efficiency, and reduce detection costs.

[0007] The technical solution adopted by this invention to solve its technical problem is: a method for detecting building changes based on UAV video comparison, wherein the improvement is that the method for detecting building changes based on UAV video includes the following steps:

[0008] S10, set the flight path and drone patrol cycle for the area to be surveyed;

[0009] S20 uses drones to collect baseline video along a set flight path;

[0010] S30 uses drones to collect videos to be inspected along a set route according to the set inspection cycle;

[0011] S40, extract key frames from the video to be detected;

[0012] S50, extract the reference video keyframes corresponding to the keyframes of the video to be detected;

[0013] S60, perform image correction and alignment on the reference video keyframes and the video keyframes to be detected, and obtain the building targets in the aligned image;

[0014] S70: Extract and compare building targets at the same location in the reference video keyframe and the video keyframe to be detected to identify building targets that have changed.

[0015] Furthermore, in step S20, during the process of acquiring the reference video, it is also necessary to acquire and integrate the fixed-frequency data of the UAV to generate the first trajectory file of the UAV; wherein, the fixed-frequency data includes the UAV's GPS information, flight altitude, flight speed, flight attitude, gimbal angle, camera parameters, video timestamp, and battery level.

[0016] Furthermore, in step S30, during the process of acquiring the video to be detected, it is also necessary to acquire and integrate the fixed-frequency data of the UAV to generate the second trajectory file of the UAV, and the flight parameters of the UAV should be kept as consistent as possible with those acquired during the acquisition of the reference video; wherein, the flight parameters of the UAV include the flight altitude, flight speed, flight attitude and gimbal angle of the UAV.

[0017] Furthermore, in step S40, the specific method for extracting the key frames of the video to be detected is as follows: according to the flight speed of the drone, frames are extracted evenly at fixed time intervals to serve as key frames of the video to be detected, and a small amount of overlap is allowed between adjacent key frame images.

[0018] Furthermore, during the extraction of keyframes from the video to be detected, it is also necessary to parse the GPS information of the keyframes. The specific method for parsing the GPS information of the keyframes is as follows:

[0019] Extract the first target keyframe, the frame number of which is [frame number missing] in the video to be detected. The video timestamp is ;

[0020] The times when the fixed-frequency data was recorded before and after the first target keyframe were determined as follows: and and read the drones respectively in and GPS information, in which the drone is in time GPS information recorded as , Indicates the drone at a certain time longitude, Indicates the drone at a certain time latitude, Indicates the drone at a certain time The altitude; the drone at all times GPS information recorded as , Indicates the drone at a certain time longitude, Indicates the drone at a certain time latitude, Indicates the drone at a certain time Height;

[0021] Calculate the GPS information of the first target keyframe: The GPS information of the first target keyframe is denoted as... ,in:

[0022] This indicates the longitude of the drone in the first target keyframe. The calculation expression is: ;

[0023] This indicates the latitude of the drone in the first target keyframe. The calculation expression is: ;

[0024] This indicates the altitude of the drone in the first target keyframe. The calculation expression is: ;

[0025] In the above formula, .

[0026] Furthermore, in step S50, the specific method for extracting the reference video keyframe corresponding to the keyframe of the video to be detected is as follows:

[0027] S501, determine the search range of the second target keyframe corresponding to the first target keyframe in the reference video; specifically, the frame number of the second target keyframe in the reference video is... The video timestamp of the second target keyframe is recorded as ,and The value range is set at Inside, among them, Set to 10s;

[0028] S502, analyze the reference video at the timestamp Obtain the target trajectory file within the specified range and retrieve the GPS information recorded in the target trajectory file. Where n represents the number of times n is in the equation. The number of times the fixed-frequency data of the drone within the range was collected;

[0029] S503, in Take out and The GPS information of the closest second target keyframe is denoted as ; and determine Corresponding video timestamp , The corresponding frame number of the second target keyframe is ,in, , Indicates the video frame rate. The value is 30;

[0030] S504, according to frame sequence number Frames are extracted from the reference video and the frame sequence number is... Additional extractions before and after The frame is selected from the first target keyframe, and the frame with the highest similarity to the first target keyframe is selected as the final second target keyframe; where M is 30.

[0031] Furthermore, in step S60, the specific method for performing image correction and alignment on the reference video keyframe and the video keyframe to be detected is as follows:

[0032] The SIFT algorithm is used to extract the feature vectors of the first target keyframe and the second target keyframe.

[0033] Feature point matching is performed based on eigenvectors, and the affine transformation matrix is ​​calculated.

[0034] The first target keyframe and the second target keyframe are precisely aligned using an affine transformation matrix to obtain an aligned image of the first target keyframe relative to the second target keyframe.

[0035] Furthermore, the specific method for obtaining the building targets in the alignment image is as follows: a building recognition model is trained using the YOLO-NAS detection algorithm, and the trained building recognition model is used to locate the building targets in the alignment image of the first target keyframe relative to the second target keyframe.

[0036] Furthermore, in step S70, the specific method for extracting and comparing building targets at the same location in the reference video keyframe and the video keyframe to be detected, in order to identify changed building targets, is as follows:

[0037] S701, using a building recognition model, extract building target image blocks at the same position in the reference video keyframe and the video keyframe to be detected, respectively. The building target image block in the reference video keyframe is denoted as A, and the building target image block in the video keyframe to be detected is denoted as B.

[0038] S702, define a local window and slide the local window on A and B, and calculate the mean value within each window in image patch A during the sliding process. ,variance The mean value within each window of image patch B ,variance and the covariance of image patch A and image patch B ;

[0039] S703, calculate the brightness contrast term for image block A and image block B based on the mean, variance, and covariance obtained in each image. Contrast comparison item Structural comparison items To measure the similarity between image patch A and image patch B in terms of brightness, contrast, and structure, where:

[0040] ;

[0041] ;

[0042] ;

[0043] In the above formula, , , All are constants. The calculation expression is as follows , The calculation expression is , The calculation expression is as follows , Indicates the range of pixel values;

[0044] S704, according to the formula Calculate the SSIM value for each window, and then take the average of the SSIM values ​​of all windows as the global SSIM value to represent the similarity between image patch A and image patch B in the overall structure.

[0045] S705 sets an SSIM threshold according to application requirements, compares the global SSIM value with the SSIM threshold. If the global SSIM value is greater than or equal to the SSIM threshold, then image patch A and image patch B are structurally similar and the building target has not changed; if the global SSIM value is less than the SSIM threshold, then image patch A and image patch B are structurally different and the building target has changed.

[0046] A building change detection system based on UAV video comparison is applied to the building change detection method based on UAV video comparison described above. The improvement lies in that the building change detection system based on UAV video comparison includes:

[0047] The flight mission planning module is used to set the flight path and drone patrol cycle for the area to be surveyed.

[0048] The video acquisition module is used to acquire baseline video along a set route and to acquire video to be inspected along a set route according to a set inspection cycle.

[0049] The video frame extraction module is used to extract key frames of the video to be detected and to extract reference video key frames corresponding to the key frames of the video to be detected.

[0050] The image processing module is used to perform image correction and alignment on the reference video keyframes and the video keyframes to be detected, and to obtain the building targets in the aligned image;

[0051] The building target change comparison module is used to extract and compare building targets at the same location in the reference video keyframe and the video keyframe to be detected, in order to identify building targets that have changed.

[0052] The beneficial effects of this invention are as follows: This invention identifies changed building targets by sequentially acquiring reference videos and videos to be inspected in the area to be inspected, extracting and correcting the corresponding key frames of the reference videos and the videos to be inspected, detecting building targets in the alignment diagram of the reference videos and the videos to be inspected, and extracting and comparing building targets at the same position in the reference videos and the videos to be inspected. During this process, when the drone is patrolling, there is no need to stop and take pictures at fixed points, nor is there any need for manual calibration of buildings in the reference photos. Therefore, compared with the prior art, this invention can make full use of the video data collected by the drone for comparison, intelligently detect changes in buildings, reduce manual intervention, optimize detection efficiency, and reduce detection costs. Attached Figure Description

[0053] Figure 1 This is an overall flowchart of a building change detection method based on UAV video comparison according to the present invention;

[0054] Figure 2 This is a block diagram of a building change detection system based on UAV video comparison according to the present invention;

[0055] Figure 3 This is a hardware structure diagram of an electronic device as an example embodiment;

[0056] Figure 4 This is a block diagram illustrating an electronic device as an example embodiment. Detailed Implementation

[0057] The present invention will be further described below with reference to the accompanying drawings and examples.

[0058] The following will clearly and completely describe the concept, specific structure, and technical effects of the present invention in conjunction with embodiments and accompanying drawings, so as to fully understand the purpose, features, and effects of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of them. Other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are all within the scope of protection of the present invention. Furthermore, all connections / linkages involved in the patent do not simply refer to direct contact between components, but rather to the ability to form a better connection structure by adding or reducing connecting accessories according to specific implementation conditions. The various technical features in this invention can be combined interactively without contradicting each other.

[0059] Reference Figure 1As shown, this invention discloses a method for detecting building changes based on drone video comparison. The method for detecting building changes in drone video includes the following steps:

[0060] S10, Set the flight path and UAV patrol cycle for the area to be surveyed; In this embodiment, during the specific application of the UAV, the ground control platform of the UAV sets the flight path and UAV patrol cycle for the area to be surveyed. After the setting is completed, the ground control platform controls the UAV to perform the patrol task; Specifically, the setting of the flight path needs to take into account the distribution of buildings in the area to be surveyed, to ensure that the UAV can cover all building targets in the area to be surveyed during the flight mission; The patrol cycle determines how often the UAV needs to perform a flight mission to collect new video data. The setting of the patrol cycle should be determined according to the speed of building changes. For example, some areas may need to be monitored weekly, while some areas may only need to be monitored once a month.

[0061] S20, use the drone to collect reference video along the set flight path; during the process of collecting reference video, it is also necessary to collect and integrate the drone's fixed frequency data to generate the drone's first trajectory file; wherein, the fixed frequency data includes the drone's GPS information, flight altitude, flight speed, flight attitude, gimbal angle, camera parameters, video timestamp, and battery level.

[0062] It should be noted that, in this embodiment, before building monitoring, the ground control platform controls the UAV to conduct an initial patrol along a set flight path to collect reference video of the area to be monitored. This reference video records the initial state of the buildings in the area to be monitored, serving as a comparison standard for whether subsequent changes have occurred. During the collection of reference video, the UAV monitors its own frequency data and sends it to the ground control platform, which then integrates it into a first trajectory file to supplement the accuracy and completeness of the reference video data. Specifically, the UAV is equipped with a high-definition camera and various functional sensors. During the specific collection of reference video of the area to be monitored, the UAV flies at a constant speed along the set flight path and uses the high-definition camera to capture video in a vertical overhead view, serving as the reference video of the area to be monitored. During video collection, the UAV collects its own frequency data through various functional sensors. Specifically, the frequency data includes… The drone's GPS information, flight altitude, flight speed, flight attitude, gimbal angle, camera parameters, video timestamp, and battery level are monitored. GPS information helps locate the position of buildings relative to geographic coordinates in the video. Flight altitude, flight speed, flight attitude, and gimbal angle indicate the drone's current flight status, providing a reference for subsequent patrol missions. Camera parameters, including focal length, exposure time, and ISO, determine the clarity and brightness of the captured video. The video timestamp records the video's recording time, ensuring video files can be sorted chronologically. Battery level monitoring helps determine if the drone will interrupt its mission due to insufficient power. Additionally, most drone manufacturers provide APIs for developers to access some data; however, the data transmission frequency is usually limited by the SDK design and cannot reach the high frequency of the original sensors, typically around 0.5Hz, meaning the current data is transmitted every 2 seconds.

[0063] S30, according to the set inspection cycle, the drone is used to collect the video to be detected along the set flight path; during the process of collecting the video to be detected, it is also necessary to collect and integrate the fixed frequency data of the drone to generate the second trajectory file of the drone, and the flight parameters of the drone are kept as consistent as possible with those of the reference video; wherein, the flight parameters of the drone include the drone's flight altitude, flight speed, flight attitude and gimbal angle.

[0064] In this embodiment, the principle of the UAV acquiring the video to be detected and generating the second trajectory file is the same as that of acquiring the reference video and generating the first trajectory file, and will not be described in detail here. In addition, the flight parameters of the acquired reference video and the video to be detected are kept as consistent as possible to ensure that the accuracy of subsequent building change detection is not affected by the difference in flight parameters, thereby ensuring the reliability of the detection results.

[0065] S40, extract key frames from the video to be detected; specifically, the extraction method for the key frames is as follows: based on the drone's flight speed, frames are extracted evenly at fixed time intervals to serve as key frames for the video to be detected, and a small overlap between adjacent key frame images is allowed (for example, if the drone can capture the entire scene in 5 seconds, the frame extraction time interval can be set to 4.5 seconds); in addition, during the extraction of key frames, the GPS information of the key frames also needs to be parsed. The specific method for parsing the GPS information of the key frames is as follows:

[0066] Extract the first target keyframe, the frame number of which is [frame number missing] in the video to be detected. The video timestamp is ;

[0067] The times when the fixed-frequency data was recorded before and after the first target keyframe were determined as follows: and and read the drones respectively in and GPS information, in which the drone is in time GPS information recorded as , Indicates the drone at a certain time longitude, Indicates the drone at a certain time latitude, Indicates the drone at a certain time The altitude; the drone at all times GPS information recorded as , Indicates the drone at a certain time longitude, Indicates the drone at a certain time latitude, Indicates the drone at a certain time Height;

[0068] Calculate the GPS information of the first target keyframe: The GPS information of the first target keyframe is denoted as... ,in:

[0069] This indicates the longitude of the drone in the first target keyframe. The calculation expression is: ;

[0070] This indicates the latitude of the drone in the first target keyframe. The calculation expression is: ;

[0071] This indicates the altitude of the drone in the first target keyframe. The calculation expression is: ;

[0072] In the above formula, ;

[0073] It should be noted that in this embodiment, since the fixed frequency data in the second trajectory file is recorded only once every 2 seconds, it means that the GPS information is also recorded once every 2 seconds. It is known that the frame rate of the video to be detected is fps (fps is usually 30). There are a total of 2fps frames in the 2-second time period. Therefore, the probability that the GPS information of the currently extracted key frame can be directly obtained is only 1 / 2fps. In most cases, it is missing. It is necessary to calculate the approximate GPS information of the key frame through the above process.

[0074] S50, extract the reference video keyframe corresponding to the keyframe of the video to be detected; specifically, the specific extraction method of the reference video keyframe corresponding to the keyframe of the video to be detected is as follows:

[0075] S501, determine the search range of the second target keyframe corresponding to the first target keyframe in the reference video; specifically, the frame number of the second target keyframe in the reference video is... The video timestamp of the second target keyframe is recorded as ,and The value range is set at Inside, among them, Set to 10s;

[0076] It should be noted that, in this embodiment, due to the limitations of the drone's own performance, the wind speed on the day of flight, and operational factors such as turning and U-turns, even if the same flight parameters as the reference video are used when shooting the video to be detected, it is impossible for the two videos to have exactly the same duration and frame number. Furthermore, under the same video timestamp or the same frame number, the images corresponding to the two videos are also different and not in the same position. Therefore, in the process of extracting frames from the reference video, the range of the video timestamp of the second target key frame is first determined to provide a basis for subsequently selecting the second target key frame that is most similar to the first target key frame within this range.

[0077] S502, analyze the reference video at the timestamp Obtain the target trajectory file within the specified range and retrieve the GPS information recorded in the target trajectory file. Where n represents the number of times n is in the equation. The number of times the fixed-frequency data of the drone within the range was collected;

[0078] S503, in Take out and The GPS information of the closest second target keyframe is denoted as ; and determine Corresponding video timestamp , The corresponding frame number of the second target keyframe is ,in, , Indicates the video frame rate. The value is 30;

[0079] S504, according to frame sequence number Frames are extracted from the reference video and the frame sequence number is... Additional extractions before and after The frame is selected from the first target keyframe, and the frame with the highest similarity to the first target keyframe is selected as the final second target keyframe; where M is 30.

[0080] It should be noted that, in this embodiment, in order to ensure that the image that best matches the keyframe of the video to be detected is obtained, when extracting the reference video keyframe, in addition to extracting the image with the sequence number 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 19, 19, In addition to the keyframe, an additional M frames before and after it are extracted. These 2M+1 frames are then matched with the keyframe of the video to be detected. The frame with the highest similarity to the keyframe of the video to be detected is selected as the frame that best matches the keyframe of the video to be detected, which is the final reference video keyframe (second target keyframe).

[0081] S60, perform image correction and alignment on the reference video keyframes and the video keyframes to be detected, and obtain the building targets in the aligned image; specifically, the specific method for performing image correction and alignment on the reference video keyframes and the video keyframes to be detected is as follows:

[0082] The SIFT algorithm is used to extract the feature vectors of the first target keyframe and the second target keyframe.

[0083] Feature point matching is performed based on eigenvectors, and the affine transformation matrix is ​​calculated.

[0084] The first target keyframe and the second target keyframe are precisely aligned using an affine transformation matrix to obtain an aligned image of the first target keyframe relative to the second target keyframe.

[0085] It should be noted that, in this embodiment, the SIFT algorithm is an algorithm for extracting key feature points from an image. These feature points are points in the image with strong discriminative power and stability (such as corners or edges). By applying this algorithm to the first target keyframe and the second target keyframe, their respective feature vectors can be extracted. The feature vector image contains key information of the first target keyframe / second target keyframe, which is used to compare the images in the subsequent matching process. Feature point matching refers to finding the same or similar feature points in the first and second target keyframes by comparing their feature vectors. The comparison of the spatial positions of these feature points helps to determine the relative relationship between the images. The affine transformation matrix can describe how an image is transformed from one coordinate system to another, including transformations such as translation, rotation, and scaling, to ensure that the first and second target keyframes can be accurately aligned. Specifically, through the affine transformation matrix, the feature points of the first target keyframe will be accurately matched to the corresponding feature points of the second target keyframe, thereby obtaining an aligned image, that is, an image that is relatively aligned with the second target keyframe after transformation.

[0086] In addition, the specific method for obtaining the building targets in the alignment image is as follows: a building recognition model is trained using the YOLO-NAS detection algorithm, and the trained building recognition model is used to locate the building targets in the alignment image of the first target keyframe relative to the second target keyframe.

[0087] It should be noted that in this embodiment, since the comparison is performed by automatically extracting images from the video, this differs from obtaining a baseline image by directly taking photos. It is difficult to manually mark the location of buildings in the baseline image using annotation tools beforehand. Therefore, a deep learning object detection method is needed to automatically detect building targets in the alignment image of the baseline video keyframes and the video keyframes to be detected. Specifically, a building recognition model needs to be pre-trained and loaded onto the UAV or ground system. During patrol operations, the model's inference capabilities are used to locate building targets in the alignment image of the baseline video keyframes and the video keyframes to be detected. Specifically, the training method for the building recognition model includes the following steps:

[0088] First, a building dataset of the area to be surveyed is constructed. Each sample dataset contains an image and a label file. The label file indicates the location of the building in the image and its category label.

[0089] Then, the dataset is divided into a training set and a validation set in a 4:1 ratio. The training set is used for learning the model's parameters, and the validation set is used to evaluate the model's performance during training.

[0090] Next, a pre-trained YOLOvNAS model is loaded (which helps the model converge faster), and hyperparameters such as learning rate, batch size, and number of training epochs are set. During training, the model continuously reads data from the training set to calculate the loss through forward propagation and updates the parameters through backpropagation. Loading a pre-trained YOLOvNAS model means importing a YOLOvNAS model that has already been trained on a large-scale dataset into the current environment for inference, fine-tuning, or further training.

[0091] Finally, after training is completed, the model performance is evaluated, and the best-performing model is selected as the final building recognition model.

[0092] It should also be noted that the YOLO-NAS detection algorithm is a YOLO (You Only Look Once) based object detection algorithm that combines Neural Architecture Search (NAS) technology. It aims to optimize the model structure through automated methods to achieve higher detection accuracy and efficiency.

[0093] S70, extract and compare building targets at the same position in the reference video keyframe and the video keyframe to be detected to identify building targets that have changed; specifically, the method for extracting and comparing building targets at the same position in the reference video keyframe and the video keyframe to be detected to identify building targets that have changed is as follows:

[0094] S701, using a building recognition model, extract building target image blocks at the same position in the reference video keyframe and the video keyframe to be detected, respectively. The building target image block in the reference video keyframe is denoted as A, and the building target image block in the video keyframe to be detected is denoted as B.

[0095] S702, define a local window and slide the local window on A and B, and calculate the mean value within each window in image patch A during the sliding process. ,variance The mean value within each window of image patch B ,variance and the covariance of image patch A and image patch B ;

[0096] S703, calculate the brightness contrast term for image block A and image block B based on the mean, variance, and covariance obtained in each image. Contrast comparison item Structural comparison items To measure the similarity between image patch A and image patch B in terms of brightness, contrast, and structure, where:

[0097] ;

[0098] ;

[0099] ;

[0100] In the above formula, , , All are constants. The calculation expression is as follows , The calculation expression is , The calculation expression is as follows , Indicates the range of pixel values;

[0101] S704, according to the formula Calculate the SSIM value for each window, and then take the average of the SSIM values ​​of all windows as the global SSIM value to represent the similarity between image patch A and image patch B in the overall structure.

[0102] S705 sets an SSIM threshold according to application requirements, compares the global SSIM value with the SSIM threshold. If the global SSIM value is greater than or equal to the SSIM threshold, then image patch A and image patch B are structurally similar and the building target has not changed; if the global SSIM value is less than the SSIM threshold, then image patch A and image patch B are structurally different and the building target has changed.

[0103] It should be noted that in this embodiment, the global SSIM value is used to represent the similarity of image block A and image block B in terms of overall structure, which is more in line with the human visual system's judgment of image similarity. In addition, when a change in the building target is detected, the changed building image block needs to be marked on the reference video keyframe and the video keyframe to be detected. At the same time, a JSON file containing video timestamps and GPS information corresponding to these keyframes is generated. Subsequently, the image information and text information are integrated together as an early warning information package and sent to relevant personnel for further verification and processing, so as to provide timely decision-making basis for the maintenance and management of the building.

[0104] Reference Figure 2 As shown, the present invention also discloses a building change detection system 600 based on UAV video comparison, applied to a building change detection method based on UAV video comparison as described in the above embodiments. The building change detection system 600 based on UAV video comparison includes:

[0105] Flight mission planning module 601 is used to set the flight path and UAV patrol cycle for the area to be surveyed.

[0106] The video acquisition module 602 is used to acquire reference video along a set route path and to acquire video to be inspected along a set route path according to a set inspection cycle.

[0107] The video frame extraction module 603 is used to extract key frames of the video to be detected and to extract reference video key frames corresponding to the key frames of the video to be detected.

[0108] The image processing module 604 is used to perform image correction and alignment on the reference video keyframes and the video keyframes to be detected, and to obtain the building targets in the aligned image;

[0109] The building target change comparison module 605 is used to extract and compare building targets at the same position in the reference video keyframe and the video keyframe to be detected, so as to identify building targets that have changed.

[0110] It should be noted that the building change detection strategy based on UAV video comparison provided in the above embodiments is only an example of the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed. That is, the internal structure of the building change detection system 600 based on UAV video comparison will be divided into different functional modules to complete all or part of the functions described above.

[0111] Furthermore, the building change detection system 600 based on UAV video comparison provided in the above embodiments and the building change detection method based on UAV video comparison belong to the same concept. The specific way each module performs its operation has been described in detail in the method embodiments, and will not be repeated here.

[0112] Figure 3 A schematic diagram of the structure of an electronic device according to an exemplary embodiment is shown.

[0113] It should be noted that this electronic device is merely an example adapted to the present invention and should not be construed as providing any limitation on the scope of use of the present invention. Furthermore, this electronic device should not be interpreted as requiring or depending on any particular feature. Figure 3 One or more components of the exemplary electronic device 2000 shown.

[0114] The hardware structure of electronic devices 2000 can vary significantly due to differences in configuration or performance, such as... Figure 3As shown, the electronic device 2000 includes: a power supply 210, an interface 230, at least one memory 250, and at least one central processing unit (CPU) 270.

[0115] Specifically, power supply 210 is used to provide operating voltage for various hardware devices on electronic device 2000.

[0116] Interface 230 includes at least one wired or wireless network interface 231 for interacting with external devices. Of course, in other examples adapted to this invention, interface 230 may further include at least one serial-to-parallel conversion interface 233, at least one input / output interface 235, and at least one USB interface 237, etc. Figure 3 As shown, this does not constitute a specific limitation.

[0117] The memory 250 serves as a carrier for resource storage and can be a read-only memory, random access memory, disk, or optical disk, etc. The resources stored on it include the operating system 251, application programs 253, and data 255, etc., and the storage method can be temporary storage or permanent storage.

[0118] The operating system 251 is used to manage and control the various hardware devices and application programs 253 on the electronic device 2000, so as to enable the central processing unit 270 to perform calculations and processing on the massive data 255 in the memory 250. It can be Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0119] Application 253 is a computer-readable instruction based on operating system 251 that performs at least one specific task, and may include at least one module ( Figure 3 (Not shown), each module may contain computer-readable instructions for the electronic device 2000. For example, a device for prioritizing tasks may be considered as an application program 253 deployed on the electronic device 2000.

[0120] Data 255 may be signal information, etc., and is stored in memory 250.

[0121] The central processing unit 270 may include one or more processors and is configured to communicate with the memory 250 via at least one communication bus to read computer-readable instructions stored in the memory 250, thereby enabling the computation and processing of massive amounts of data 255 in the memory 250. For example, a building change detection method based on UAV video comparison can be implemented by the central processing unit 270 reading a series of computer-readable instructions stored in the memory 250.

[0122] Furthermore, the present invention can also be implemented through hardware circuits or a combination of hardware circuits and software. Therefore, the implementation of the present invention is not limited to any specific hardware circuit, software, or combination thereof.

[0123] Please see Figure 4 This invention provides an electronic device 4000, which may include: a desktop computer, a laptop computer, a server, etc., with sensor recognition capabilities.

[0124] exist Figure 4 In this context, the electronic device 4000 includes at least one processor 4001 and at least one memory 4003.

[0125] The data interaction between the processor 4001 and the memory 4003 can be achieved through at least one communication bus 4002. This communication bus 4002 may include a path for transmitting data between the processor 4001 and the memory 4003. The communication bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The communication bus 4002 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0126] Optionally, the electronic device 4000 may further include a transceiver 4004, which can be used for data interaction between the electronic device and other electronic devices, such as sending and / or receiving data. It should be noted that in practical applications, the transceiver 4004 is not limited to one type, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present invention.

[0127] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this invention. Processor 4001 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0128] The memory 4003 may be a ROM (Read Only Memory) or other type of static storage device capable of storing static information and instructions, RAM (Random Access Memory) or other type of dynamic storage device capable of storing information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program instructions or code in the form of instructions or data structures and accessible by the electronic device 4000, but not limited thereto.

[0129] The memory 4003 stores computer-readable instructions, and the processor 4001 can read the computer-readable instructions stored in the memory 4003 through the communication bus 4002.

[0130] The computer-readable instructions are executed by one or more processors 4001 to implement the building change detection method based on UAV video comparison in the above embodiments.

[0131] Furthermore, this embodiment of the invention provides a storage medium storing computer-readable instructions, which are executed by one or more processors to implement the building change detection method based on UAV video comparison as described above.

[0132] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0133] 1. This invention identifies changed building targets by sequentially acquiring a reference video and a video to be inspected of the area to be inspected, extracting and correcting the corresponding key frames of the reference video and the video to be inspected, detecting building targets in the alignment diagram of the reference video key frames and the video to be inspected, and extracting and comparing building targets at the same position in the reference video key frames and the video to be inspected. During this process, the drone does not need to stop and take pictures at fixed points, nor does it require manual calibration of buildings in the reference photos. This allows for full utilization of the video data collected by the drone for comparison, intelligent detection of changes in buildings, reduced manual intervention, optimized detection efficiency, and reduced detection costs.

[0134] 2. This invention utilizes the speed and flexibility of drones to collect video data of buildings in real time and analyze it through intelligent algorithms to promptly detect changes in buildings, providing timely decision-making basis for building maintenance and management, greatly improving detection efficiency, and is especially suitable for monitoring buildings in large-scale areas.

[0135] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A method for detecting building changes based on UAV video comparison, characterized in that, The method for detecting changes in buildings based on UAV video includes the following steps: S10, set the flight path and drone patrol cycle for the area to be surveyed; S20, the drone is used to collect reference video along the set flight path; and during the process of collecting reference video, the drone's fixed frequency data also needs to be collected and integrated to generate the drone's first trajectory file; wherein, the fixed frequency data includes the drone's GPS information, flight altitude, flight speed, flight attitude, gimbal angle, camera parameters, video timestamp and battery level. S30, according to the set inspection cycle, the drone is used to collect the video to be detected along the set flight path; and in the process of collecting the video to be detected, it is also necessary to collect and integrate the fixed frequency data of the drone to generate the second trajectory file of the drone, and the flight parameters of the drone are kept as consistent as possible with those of the reference video; wherein, the flight parameters of the drone include the drone's flight altitude, flight speed, flight attitude and gimbal angle. S40, extract key frames from the video to be detected; specifically, the extraction method for the key frames is as follows: based on the flight speed of the UAV, frames are extracted evenly at fixed time intervals to serve as key frames for the video to be detected, and a small overlap between adjacent key frame images is allowed; furthermore, during the extraction of key frames, the GPS information of the key frames also needs to be parsed, and the specific method for parsing the GPS information of the key frames is as follows: Extract the first target keyframe, the frame number of which is [frame number missing] in the video to be detected. The video timestamp is ; The times when the fixed-frequency data was recorded before and after the first target keyframe were determined as follows: and and read the drones respectively in and GPS information, in which the drone is in time GPS information recorded as , Indicates the drone at a certain time longitude, Indicates the drone at a certain time latitude, Indicates the drone at a certain time The altitude; the drone at all times GPS information recorded as , Indicates the drone at a certain time longitude, Indicates the drone at a certain time latitude, Indicates the drone at a certain time Height; Calculate the GPS information of the first target keyframe: The GPS information of the first target keyframe is denoted as... ,in: This indicates the longitude of the drone in the first target keyframe. The calculation expression is: ; This indicates the latitude of the drone in the first target keyframe. The calculation expression is: ; This indicates the altitude of the drone in the first target keyframe. The calculation expression is: ; In the above formula, ; S50, extract the reference video keyframe corresponding to the keyframe of the video to be detected; specifically, the specific extraction method of the reference video keyframe corresponding to the keyframe of the video to be detected is as follows: S501, determine the search range of the second target keyframe corresponding to the first target keyframe in the reference video; specifically, the frame number of the second target keyframe in the reference video is... The video timestamp of the second target keyframe is recorded as ,and The value range is set at Inside, among them, Set to 10s; S502, analyze the reference video at the timestamp Obtain the target trajectory file within the specified range and retrieve the GPS information recorded in the target trajectory file. Where n represents the number of times n is in the equation. The number of times the fixed-frequency data of the drone within the range was collected; S503, in Take out and The GPS information of the closest second target keyframe is denoted as ; and determine Corresponding video timestamp , The corresponding frame number of the second target keyframe is ,in, , Indicates the video frame rate. The value is 30; S504, according to frame sequence number Frames are extracted from the reference video and the frame sequence number is... Additional extractions before and after The frame is selected from the first target keyframe, and the frame with the highest similarity to the first target keyframe is selected as the final second target keyframe; where M is 30. S60, perform image correction and alignment on the reference video keyframes and the video keyframes to be detected, and obtain the building targets in the aligned image; S70: Extract and compare building targets at the same location in the reference video keyframe and the video keyframe to be detected to identify building targets that have changed.

2. The method for detecting building changes based on UAV video comparison according to claim 1, characterized in that, In step S60, the specific method for image correction and alignment of the reference video keyframe and the video keyframe to be detected is as follows: The SIFT algorithm is used to extract the feature vectors of the first target keyframe and the second target keyframe. Feature point matching is performed based on eigenvectors, and the affine transformation matrix is ​​calculated. The first target keyframe and the second target keyframe are precisely aligned using an affine transformation matrix to obtain an aligned image of the first target keyframe relative to the second target keyframe.

3. The method for detecting building changes based on UAV video comparison according to claim 2, characterized in that, The specific method for obtaining building targets in the alignment image is as follows: a building recognition model is trained using the YOLO-NAS detection algorithm, and the trained building recognition model is used to locate building targets in the alignment image of the first target keyframe relative to the second target keyframe.

4. The method for detecting building changes based on UAV video comparison according to claim 3, characterized in that, In step S70, the specific method for extracting and comparing building targets at the same location in the reference video keyframe and the video keyframe to be detected, in order to identify changed building targets, is as follows: S701, using a building recognition model, extract building target image blocks at the same position in the reference video keyframe and the video keyframe to be detected, respectively. The building target image block in the reference video keyframe is denoted as A, and the building target image block in the video keyframe to be detected is denoted as B. S702, define a local window and slide the local window on A and B, and calculate the mean value within each window in image patch A during the sliding process. ,variance The mean value within each window of image patch B ,variance and the covariance of image patch A and image patch B ; S703, calculate the brightness contrast term for image block A and image block B based on the mean, variance, and covariance obtained in each image. Contrast comparison item Structural comparison items To measure the similarity between image patch A and image patch B in terms of brightness, contrast, and structure, where: ； ； ； In the above formula, , , All are constants. The calculation expression is as follows , The calculation expression is , The calculation expression is as follows , Indicates the range of pixel values; S704, according to the formula Calculate the SSIM value for each window, and then take the average of the SSIM values ​​of all windows as the global SSIM value to represent the similarity between image patch A and image patch B in the overall structure. S705 sets an SSIM threshold according to application requirements, compares the global SSIM value with the SSIM threshold. If the global SSIM value is greater than or equal to the SSIM threshold, then image patch A and image patch B are structurally similar and the building target has not changed; if the global SSIM value is less than the SSIM threshold, then image patch A and image patch B are structurally different and the building target has changed.

5. A building change detection system based on UAV video comparison, applied to the building change detection method based on UAV video comparison as described in any one of claims 1-4, characterized in that, The building change detection system based on UAV video comparison includes: The flight mission planning module is used to set the flight path and drone patrol cycle for the area to be surveyed. The video acquisition module is used to acquire baseline video along a set route and to acquire video to be inspected along a set route according to a set inspection cycle. The video frame extraction module is used to extract key frames of the video to be detected and to extract reference video key frames corresponding to the key frames of the video to be detected. The image processing module is used to perform image correction and alignment on the reference video keyframes and the video keyframes to be detected, and to obtain the building targets in the aligned image; The building target change comparison module is used to extract and compare building targets at the same location in the reference video keyframe and the video keyframe to be detected, in order to identify building targets that have changed.

Images