Change detection method and device based on unmanned aerial vehicle scene

Abstract

The invention discloses a change detection method and device based on an unmanned aerial vehicle scene, and the method comprises the steps: obtaining unmanned aerial vehicle flight remote sensing images in different periods, and carrying out the splicing of the remote sensing images in different periods, and obtaining a to-be-detected region image; inputting the to-be-detected area images in different periods into the trained deep learning model so as to detect change areas in different periods; wherein the trained deep learning model comprises feature extraction networks and region candidate networks corresponding to the feature extraction networks; the feature extraction network is composed of a plurality of feature pyramid networks, and different scale features are extracted from different feature pyramid networks; each region candidate network processes the extracted different scale features; and the candidate regions output by the different region candidate networks are fused and then used for change detection. The efficiency and precision of change detection and recognition are improved, and the defect that manual detection is poor in efficiency and precision is overcome.

Description

Technical Field

[0001] The present invention relates to the field of image detection technology, and in particular to a change detection method and device based on an unmanned aerial vehicle scene. Background Art

[0002] Drones are increasingly being used in environmental monitoring, urban planning, agricultural management, and other fields. Using remote sensing images collected by drones for change detection and identification can effectively monitor land use changes, disaster assessments, and ecological environment changes.

[0003] In related technologies, traditional change detection methods usually rely on manual feature extraction, and the efficiency and accuracy need to be further improved. In addition, the commonly used data in related technologies is optical satellite data, accounting for 48.77%, which is the mainstream, and drone data accounts for 9.82%, with a small amount of data; the resolution of optical satellite data is not as good as drone data, and the detailed features of small targets are not enough, and manual extraction of image features is required. For example, the recognition and change detection accuracy of fine seedlings and forest tree species is low, and it cannot be promoted for use. Summary of the invention

[0004] In view of this, the present invention proposes a change detection method and device based on drone scenes to solve the technical problems existing in the related technologies.

[0005] In order to achieve the above-mentioned purpose, according to a first aspect of the present invention, a change detection method based on a drone scene is provided, comprising: acquiring drone flight remote sensing images of different periods, and splicing the remote sensing images of different periods to obtain images of the area to be detected; inputting the images of the area to be detected of different periods into a trained deep learning model to detect the changed areas of different periods; wherein the trained deep learning model comprises a feature extraction network and a region candidate network corresponding to each feature extraction network; the feature extraction network is composed of a plurality of feature pyramid networks, and different feature pyramid networks extract features of different scales; each of the region candidate networks processes the extracted features of different scales; and the candidate regions output by different region candidate networks are fused for change detection.

[0006] Optionally, each of the region candidate networks processes the extracted features of different scales, including: each region candidate network does not process the scale features that do not match the scale it is facing; and if the overlap degree between the anchor frame and the invalid real area in the feature map after processing is greater than a preset threshold, it is judged as an invalid anchor frame, otherwise it is a valid anchor frame; the area selected by the valid anchor frame is output.

[0007] Optionally, the deep learning model includes the following during training: obtaining a training set, wherein the training set is images of an area of ​​interest at different periods, the areas of interest that have changed are annotated in the images of interest, and the images of interest at different periods are formed by merging multiple UAV flight remote sensing images; inputting the training set into a pre-built deep learning model, and training the model with the changed areas as input, wherein the output results are compared with the annotated content to determine the data under each preset indicator; and adjusting the parameters of the deep learning model based on the data under the preset indicators.

[0008] Optionally, training is performed based on a preset loss function: Where N is the number of image pixels, is the predicted value of each pixel, y is the value of each pixel label, i,j represents the pixel row and column number, n,m represents the total number of rows and columns, y ij represents the true value of this pixel, Represents the model prediction value.

[0009] Optionally, the candidate regions output by different region candidate networks are upsampled respectively and then fused.

[0010] According to a second aspect of the present invention, a training method for a drone scene change detection model is provided, comprising: obtaining a training set, wherein the training set is images of regions of interest at different periods, the regions of interest that have changed are annotated in the images of interest, and the images of interest at different periods are formed by merging multiple drone flight remote sensing images; inputting the training set into a pre-built deep learning model, and training the model with the changed regions as input, wherein the output results are compared with the annotated content to determine the data under each preset indicator; and adjusting the parameters of the deep learning model based on the data under the preset indicators.

[0011] Optionally, training is performed based on a preset loss function: Among them, N is the number of image pixels, is the predicted value of each pixel, y is the value of each pixel label, i,j represents the pixel row and column number, n,m represents the total number of rows and columns, y ij represents the true value of this pixel, Represents the model prediction value.

[0012] According to a third aspect of the present invention, a change detection device based on a drone scene is provided, comprising: a preprocessing unit, configured to obtain drone flight remote sensing images of different periods, and respectively splice the remote sensing images of different periods to obtain images of the area to be detected; a detection unit, configured to input the images of the area to be detected of different periods into a trained deep learning model to detect the changed areas of different periods, wherein the trained deep learning model comprises a plurality of feature extraction networks and a region candidate network corresponding to each feature extraction network, each of the feature extraction networks extracts multi-scale features, and each of the region candidate networks processes the extracted features of different scales; the candidate areas output by different region candidate networks are fused for change detection.

[0013] According to a fourth aspect of the present invention, there is provided a computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the method described in any one of the implementations of the first aspect or the second aspect.

[0014] According to a fifth aspect of the present invention, there is provided an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor executes the method described in any one of the implementations of the first aspect or the second aspect.

[0015] This embodiment is based on a change detection method and device for drone scenes, wherein the method includes obtaining drone flight remote sensing images of different periods, and splicing the remote sensing images of different periods to obtain images of the area to be detected; inputting the images of the area to be detected of different periods into a trained deep learning model to detect the changed areas of different periods; wherein the trained deep learning model includes a feature extraction network and a region candidate network corresponding to each feature extraction network; the feature extraction network is composed of multiple feature pyramid networks, and different feature pyramid networks extract features of different scales; each of the region candidate networks processes the extracted features of different scales; and the candidate regions output by different region candidate networks are fused for change detection. By adopting scale normalization and multi-feature extraction fusion strategies, the efficiency and accuracy of change detection and recognition are improved, and the defects of poor efficiency and accuracy of manual detection are overcome. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 is a flow chart of a change detection method based on a drone scene according to an embodiment of the present invention;

[0018] Figure 2 is a schematic diagram of a system architecture corresponding to a method for detecting changes in a drone scene according to an embodiment of the present invention;

[0019] Figure 3 It is a schematic diagram of the composition structure of the deep learning model of an embodiment of the present invention;

[0020] Figure 4 is a logical schematic diagram of a local structure of a deep learning model according to an embodiment of the present invention;

[0021] Figure 5 is a schematic diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0022] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the implementation mode of the present application will be clearly and completely described below in conjunction with the drawings in the implementation mode of the present application. Obviously, the described implementation mode is only a part of the implementation mode of the present application, not all the implementation modes. Based on the implementation mode in the present application, all other implementation modes obtained by ordinary technicians in the field without creative work should fall within the scope of protection of the present application.

[0023] It should be noted that the terms "first", "second", etc. in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0024] For those skilled in the art, the specific meanings of these terms in this application can be understood according to specific circumstances. In addition, the term "plurality" should mean two or more than two.

[0025] According to an embodiment of the present invention, a change detection method based on a drone scene is provided. Figure 1 As shown, it includes the following steps 101 to 103:

[0026] Step 101: Obtain UAV flight remote sensing images at different periods, and stitch the remote sensing images at different periods to obtain images of the area to be detected.

[0027] In the present embodiment of the present application, an unmanned aerial vehicle (UAV) with a camera and a sensor captures aerial images, so the present embodiment uses drones to collect multi-temporal remote sensing images. Each aerial platform is connected to a system responsible for collecting captured images and data. The images are associated with geographic coordinates to ensure accurate spatial representation, and multiple images are merged into a seamless large map covering the entire area of ​​interest. The merged images are preprocessed, including but not limited to radiation correction, geometric correction, and noise removal operations to obtain images of the area to be detected.

[0028] Step 102: Input the images of the area to be detected at different periods into the trained deep learning model to detect the changed areas at different periods, wherein the trained deep learning model includes multiple feature extraction networks and region candidate networks corresponding to each feature extraction network, each feature extraction network extracts multi-scale features, and each region candidate network processes the extracted features of different scales; the candidate areas output by different region candidate networks are fused for change detection.

[0029] Change detection and recognition are performed through a deep learning model (CDNet), which uses a scale normalization strategy (Scale Normalization for Image Pyramids, SNIP) to handle objects of different scales. In particular, a region proposal network (RPN) module is constructed at different scales; a feature pyramid network (FPN) is constructed for each feature of different scales to extract multi-level features of the target, and the features of different levels are fused to obtain multi-level feature information. Finally, the fused features are used for subsequent change detection extraction of the model.

[0030] refer to Figure 2 The schematic diagram of the change detection system architecture based on drone scenes in , multiple drones can collect scene data to obtain drone scene data sets, and then change detection can be implemented based on the deep learning model, and the changed areas can be marked, and finally an intuitive drone scene change monitoring map can be obtained. The details of the deep learning model are in Figure 4 Specific presentation is given in the

[0031] Further, refer to Figure 3 The schematic diagram of the deep learning model architecture in the figure shows that the deep learning model can be fed with influences from different periods, such as images from two periods. After convolution processing, images of different scales are fed into the corresponding RPN modules. After the output features are fused and analyzed for differences, the change detection results can be obtained. Figure 3 In the description, three scales are used as an example, which is only exemplary and can be increased or decreased as needed.

[0032] This embodiment generates a change detection map based on the model output results according to deep learning and marks the changed areas.

[0033] As an optional implementation method of the present application, each of the region candidate networks processes the extracted features of different scales, including: each region candidate network does not process the scale features that do not match the scale it is facing; and if the overlap degree between the anchor frame and the invalid real area in the feature map after processing is greater than a preset threshold, it is judged as an invalid anchor frame, otherwise it is a valid anchor frame; the area selected by the valid anchor frame is output.

[0034] In this optional implementation, the scale normalization (SNIP) strategy aims to construct image pyramids at different scales to handle objects of different scales. A region proposal network (RPN) module is trained at each scale to improve the accuracy of small object detection. Figure 4 The schematic diagram of the SNIP network structure is shown in Figure 4 In the example, three scales have their own RPN modules, and each scale predicts objects within a specified range. For large-scale feature maps, its RPN is only responsible for predicting small objects that are enlarged; for small-scale feature maps, its RPN is only responsible for predicting large objects that are reduced. In this way, the scale of real objects is distributed in a smaller range, avoiding extremely large or small objects. In the RPN stage, if the real object is not within the RPN prediction range, it will be judged as invalid, and the anchor whose IoU with the invalid object is greater than 0.3 will also be judged as invalid.

[0035] Furthermore, the FPN structure is used to extract multi-scale features from different levels, and a shared region proposal network (RPN) module is applied at each scale to generate candidate regions. Specifically, feature maps of different sizes are obtained through forward propagation to construct a feature map pyramid architecture, and each layer of feature maps is changed through 1×1 convolution. The bottom layer feature map is upsampled and added to the element of the previous layer feature map until the first layer feature is obtained to obtain the fused feature map.

[0036] As an optional implementation method, the candidate regions output by different region candidate networks are upsampled respectively and then fused.

[0037] In this optional implementation, reference Figure 3 , after upsampling, feature fusion is performed to obtain the fused feature map, and finally the change detection map with marked change areas is obtained.

[0038] As an optional implementation method, the deep learning model includes the following steps during training: obtaining a training set, wherein the training set is images of an area of ​​interest at different periods, the areas of interest that have changed are annotated in the images of interest, and the images of interest at different periods are formed by merging multiple UAV flight remote sensing images; inputting the training set into a pre-built deep learning model, and training the model with the changed areas as input, wherein the output results are compared with the annotated content to determine the data under each preset indicator; and adjusting the parameters of the deep learning model based on the data under the preset indicators.

[0039] In this optional implementation, when constructing a training data set, the changed areas in the image are annotated to construct a training data set for the change detection model. The training data set is divided into a training set, a validation set, and a test set. The detection results are compared with the true annotations, and evaluation indicators such as accuracy, recall rate, and F1-score are calculated. The false detection and missed detection areas are analyzed to optimize the model or data processing flow.

[0040] As an optional implementation of this embodiment, training is performed based on a preset loss function during training: Where N is the number of image pixels, is the predicted value of each pixel, y is the value of each pixel label, i,j represents the pixel row and column number, n,m represents the total number of rows and columns, y ij represents the true value of this pixel, Represents the model prediction value.

[0041] In this optional implementation, the model uses the cross entropy loss function (CrossEntropyLoss) to evaluate the stability of the training results. The cross entropy loss function formula is:

[0042] Among them: y is the true value, is the predicted value, and N is the number of samples. For an image, N here can be regarded as the number of pixels in the image. is the predicted value of each pixel, y is the value of each pixel label, and the cross entropy of an image is actually the average of the difference between the predicted value and the label of each pixel. Furthermore, data enhancement techniques (such as rotation, translation, scaling, etc.) are used to improve the robustness of the model. The model parameters are adjusted through the validation set to optimize the model performance.

[0043] The trained deep learning model is applied to the test set for change detection and recognition. A change detection map is generated based on the model output results, and the changed areas are marked. The deep learning model for change detection and recognition of small target objects based on drone images adopts scale normalization and multi-feature extraction fusion strategies to improve the efficiency and accuracy of change detection and recognition.

[0044] Furthermore, the operating environment and application of the deep learning model can be packaged into a container using Docker technology to achieve cross-platform calls.

[0045] The above implementation of the present application builds a change detection model based on the drone image scene. The model uses a scale normalization strategy (SNIP) to construct an image pyramid. When processing targets at different scales, a region candidate network (RPN) is designed for each scale. At the same time, a feature pyramid network (FPN) is constructed for each feature of different scales to extract multi-level features of the target, and the features at different levels are fused to obtain multi-level feature information. Finally, the fused features are used for subsequent change detection extraction of the model. Through the above method, deep learning change detection and recognition based on drone scenes is realized, especially the detection and recognition of small targets, which effectively improves the efficiency and accuracy of change detection and has broad application prospects.

[0046] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0047] According to an embodiment of the present invention, a training method based on a drone scene change detection model is also provided, including: obtaining a training set, wherein the training set is an image of an area of ​​interest at different periods, the changed area in the image of interest is marked, and the images of interest at different periods are formed by merging multiple drone flight remote sensing images; inputting the training set into a pre-built deep learning model, and training the model with the changed area as input, wherein the output result is compared with the marked content to determine the data under each preset indicator; and adjusting the parameters of the deep learning model based on the data under the preset indicators.

[0048] During training, training is performed based on a preset loss function: Where N is the number of image pixels, is the predicted value of each pixel, y is the value of each pixel label, i,j represents the pixel row and column number, n,m represents the total number of rows and columns, y ij represents the true value of this pixel, Represents the model prediction value.

[0049] According to an embodiment of the present invention, there is also provided a change detection device based on a drone scene, comprising a preprocessing unit configured to obtain drone flight remote sensing images of different periods, and respectively splice the remote sensing images of different periods to obtain an image of a region to be detected;

[0050] The detection unit is configured to input images of the area to be detected at different periods into a trained deep learning model to detect the changed areas at different periods; wherein the trained deep learning model includes multiple feature extraction networks and region candidate networks corresponding to each feature extraction network, each of the feature extraction networks extracts multi-scale features, and each of the region candidate networks processes the extracted features of different scales; the candidate areas output by different region candidate networks are fused and used for change detection.

[0051] As an optional implementation method, each of the region candidate networks processes the extracted features of different scales, including: each region candidate network does not process the scale features that do not match the scale it is facing; and if the overlap degree between the anchor box and the invalid real area in the feature map after processing is greater than a preset threshold, it is judged as an invalid anchor box, otherwise it is a valid anchor box; the area selected by the valid anchor box is output.

[0052] As an optional implementation method, the deep learning model includes the following steps during training: obtaining a training set, wherein the training set is images of an area of ​​interest at different periods, the areas of interest that have changed are annotated in the images of interest, and the images of interest at different periods are formed by merging multiple UAV flight remote sensing images; inputting the training set into a pre-built deep learning model, and training the model with the changed areas as input, wherein the output results are compared with the annotated content to determine the data under each preset indicator; and adjusting the parameters of the deep learning model based on the data under the preset indicators.

[0053] As an optional implementation, training is performed based on a preset loss function during training: Among them, N is the number of image pixels, is the predicted value of each pixel, y is the value of each pixel label, i,j represents the pixel row and column number, n,m represents the total number of rows and columns, y ij represents the true value of this pixel, Represents the model prediction value.

[0054] As an optional implementation method, the candidate regions output by different region candidate networks are upsampled respectively and then fused.

[0055] An embodiment of the present invention provides an electronic device, such as Figure 5 As shown, the electronic device includes one or more processors 31 and a memory 32. Figure 5 A processor 31 is taken as an example.

[0056] The controller may further include: an input device 33 and an output device 34 .

[0057] The processor 31, the memory 32, the input device 33 and the output device 34 may be connected via a bus or other means. Figure 5 The example of connecting through bus is taken in the following.

[0058] The processor 31 may be a central processing unit (CPU). The processor 31 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips. A general-purpose processor may be a microprocessor or the processor may be any conventional processor.

[0059] The memory 32 is a non-transitory computer-readable storage medium that can be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as program instructions / modules corresponding to the control method in the embodiment of the present invention. The processor 31 executes various functional applications and data processing of the server by running the non-transitory software programs, instructions and modules stored in the memory 32, that is, the method of implementing the above method embodiment.

[0060] The memory 32 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of a processing device operated by the server, etc. In addition, the memory 32 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 32 may optionally include a memory remotely arranged relative to the processor 31, and these remote memories may be connected to a network connection device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0061] The input device 33 can receive input digital or character information, and generate key signal input related to user settings and function control of the processing device of the server. The output device 34 can include display devices such as a display screen.

[0062] One or more modules are stored in the memory 32, and when executed by one or more processors 31, the execution is as follows: Figure 1 The method shown.

[0063] Those skilled in the art can understand that the implementation of all or part of the processes in the above-mentioned embodiment method is a program that can be completed by instructing related hardware through a computer program, which can be stored in a computer-readable storage medium. When the program is executed, it can include the process of the embodiment of the above-mentioned method. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory (Flash Memory), a hard disk (Hard Disk Drive, abbreviated: HDD) or a solid-state drive (SSD), etc.; the storage medium can also include a combination of the above-mentioned types of memory.

[0064] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A change detection method based on drone scenes, characterized in that: include: Obtain UAV flight remote sensing images from different periods, and stitch the remote sensing images from different periods to obtain images of the area to be detected; Input the images of the area to be detected at different periods into the trained deep learning model to detect the changed areas at different periods; Among them, the trained deep learning model includes a feature extraction network and a region candidate network corresponding to each feature extraction network; the feature extraction network is composed of multiple feature pyramid networks, and different feature pyramid networks extract features of different scales; each of the region candidate networks processes the extracted features of different scales; the candidate regions output by different region candidate networks are fused and used for change detection.

2. The change detection method based on drone scene according to claim 1 is characterized in that: The processing of the extracted features of different scales by each of the region candidate networks includes: each region candidate network does not process the scale features that do not match the scale it faces; and if the overlap degree between the anchor frame and the invalid real area in the feature map after processing is greater than a preset threshold, it is judged as an invalid anchor frame, otherwise it is a valid anchor frame; and the area selected by the valid anchor frame is output.

3. The change detection method based on drone scene according to claim 1 is characterized in that: The deep learning model includes: Acquire a training set, wherein the training set is images of regions of interest at different periods, the regions of interest that have changed are marked in the images of interest, and the images of interest at different periods are formed by merging multiple UAV flight remote sensing images; The training set is input into a pre-built deep learning model, and the model is trained with the changed area as input, wherein the output result is compared with the annotated content to determine the data under each preset indicator; and the deep learning model is adjusted based on the data under the preset indicators.

4. The change detection method based on drone scene according to claim 3 is characterized in that: During training, training is performed based on a preset loss function: Where N is the number of image pixels, is the predicted value of each pixel, y is the value of each pixel label, i,j represents the pixel row and column number, n,m represents the total number of rows and columns, y ij represents the true value of this pixel, Represents the model prediction value.

5. The change detection method based on drone scene according to claim 1 is characterized in that: The candidate regions output by the different region candidate networks are upsampled respectively and then fused.

6. A method for training a drone scene change detection model, characterized in that: include: Acquire a training set, wherein the training set is images of regions of interest at different periods, the regions of interest that have changed are marked in the images of interest, and the images of interest at different periods are formed by merging multiple UAV flight remote sensing images; The training set is input into a pre-built deep learning model, and the model is trained with the changed area as input, wherein the output result is compared with the annotated content to determine the data under each preset indicator; and the deep learning model is adjusted based on the data under the preset indicators.

7. The method for training a drone scene change detection model according to claim 6, characterized in that: During training, training is performed based on a preset loss function: Where N is the number of image pixels, is the predicted value of each pixel, y is the value of each pixel label, i,j represents the pixel row and column number, n,m represents the total number of rows and columns, y ij represents the true value of this pixel, Represents the model prediction value.

8. A change detection device based on drone scenes, characterized in that: include: The preprocessing unit is configured to obtain the remote sensing images of the UAV flight at different periods, and respectively stitch the remote sensing images at different periods to obtain the image of the area to be detected; A detection unit is configured to input images of the area to be detected at different periods into the trained deep learning model to detect the changed areas at different periods; The trained deep learning model includes multiple feature extraction networks and region candidate networks corresponding to each feature extraction network, each feature extraction network extracts multi-scale features, and each region candidate network processes the extracted features of different scales; The candidate regions output by different region candidate networks are fused for change detection.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the method according to any one of claims 1 to 7.

10. An electronic device, characterized in that: include: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor executes the method described in any one of claims 1 to 7.

Images