Method, device and equipment for detecting changes in homestead buildings

Abstract

The present invention discloses a method, device and equipment for detecting changes in homestead buildings. The method comprises: determining a target area; obtaining two remote sensing images of the target area; marking the homestead area in the two remote sensing images to obtain a homestead vector file; aligning the two remote sensing images, and cropping the aligned two remote sensing images based on the homestead vector file to obtain a number of image blocks, each of which contains only one homestead building; generating a pairing table according to the several image blocks, and randomly allocating a training set and a validation set in proportion based on the pairing table; using the training set to train a twin neural network model to obtain a homestead building change detection model, and realizing homestead building change detection. The present invention can realize monitoring of building changes in a known homestead, and can be well coordinated with an existing geographic database.

Description

Technical Field

[0001] The present invention relates to remote sensing technology, and in particular to a method, device and equipment for detecting changes in homestead buildings. Background Art

[0002] Regarding building change detection, many scholars and research institutions at home and abroad have proposed various ideas and methods from different perspectives and for different field conditions.

[0003] Change detection based on remote sensing images has always been a hot research topic and has developed rapidly under the continuous research of many scholars. After years of development in multiple stages, the analysis unit has evolved from the initial pixel-based to object-based, scene-based and hybrid analysis based on the above analysis units.

[0004] 1) Pixel-based

[0005] When remote sensing image change detection was first developed, the change detection technology based on pixel units was studied. After decades of research and development, a large number of pixel-based detection technologies and algorithms have emerged. For example, Rawashdeh et al. used pixel-by-pixel difference method to detect the change of land use types in a large arid area over 21 years (Rawashdeh et al, 2011). Due to the great limitations of the single pixel detection method, in order to better deal with the relationship and influence between adjacent pixels, the concept of pixel sliding window (Kernel) was proposed later (Tewkesbury, Comber and Tate et al., 2015). Pixel-based detection technology can detect obvious changes through relatively low-complexity algorithms. Even the introduction of pixel sliding window does not integrate the information between pixels well. There is still a lack of information extraction and context information mixed analysis of the complete image map, so it is not possible to detect subtle changes well, and it is also greatly affected by the environment and image quality. However, to this day, many scholars are still constantly researching and developing pixel-based detection technology.

[0006] 2) Object-based

[0007] The core of the object-based detection method is to first segment the object to be detected from the complete image, and then superimpose and compare it with the image to be compared. Since the object-based method was first proposed, it has begun to develop rapidly. The object-based detection method can simultaneously compare and analyze remote sensing images of multiple phases, and different comparison and superposition methods have different accuracies (Hu Yongyue, Xiao Pengfeng, Feng Xuezhi, etc., 2015). Compared with the pixel-based detection method, the object-based detection method can better utilize the texture and shape features of the detection object, which can improve the detection accuracy to a certain extent (Desclée, Bogaert and Defourny, 2006), but it is heavily dependent on the accuracy of object segmentation. Once the remote sensing images of multiple phases have registration accuracy problems, it will greatly affect the accuracy of the object-based detection method.

[0008] 3) Scenario-based

[0009] The scene-based detection method uses the scene as the detection unit. Its core is to detect changes by learning the regularity of changes in multi-temporal remote sensing images, and then detecting the content that does not conform to the regularity from other scenes. Scene-based change detection not only focuses on whether the surface has changed, but can even identify the type of change (Wu, Zhang and Du, 2017). The emergence of scene-based change detection analysis indicates that change detection based on remote sensing images has begun to enter the field of machine learning. Some scholars have implemented this regularity learning for changes using machine learning methods. However, the scene-based detection method will be affected by the changes in the entire scene. The different states of the same object in two scenes will lead to pseudo-changes.

[0010] As for the change detection of buildings, different scholars have published many different processing methods over the years, but the core of the detection is mainly focused on extracting buildings from images of two phases and performing overlay analysis to determine whether the buildings have changed.

[0011] Many scholars at home and abroad have proposed various specific detection schemes. He Hao et al. proposed a detection method that extracts the outline of the building roof based on parallax and building shadows, and then uses superposition analysis to judge the reconstruction and new addition of the building. The detection result has high accuracy, but it is not suitable for uneven spectrum of the building roof or ground spectrum and roof spectrum (He Hao, Liu Xiuguo, Shen Yonglin, 2019). Li Changpeng et al. introduced an object-oriented detection method of building change based on statistical region merging and active contour model, which can obtain better accuracy than traditional methods, but the error rate is greatly affected by the parameters of the algorithm (Li Changpeng, 2019). Lu Lichen et al. also segmented objects based on the object-oriented detection method through the fractal network evolution multi-scale segmentation algorithm, and then used the EM algorithm combined with the BMI algorithm and CVA algorithm to detect building changes. The detection accuracy is better than the pixel-based detection method (Lu Lichen, Hong Liang, 2021). There are also detection methods that take into account multiple features. The building features, spectral features, shadow features, etc. are extracted by algorithms, and then fused and extracted by specific algorithms. Finally, the building change detection results are obtained through the decision model (Wang Huixian, 2020). In addition, there is also a building change detection method based on morphological building index that is specially improved for high-resolution and high-precision remote sensing images from drones. The detection accuracy of the improved algorithm has decreased significantly in terms of missed detection rate and false detection rate compared to before the improvement, and the overall accuracy has also been improved to a certain extent (Li Baolin, 2020). This proves that the old method cannot fully utilize the advantages brought by high-resolution drone images, and there is still a lot of room for improvement.

[0012] As for deep learning, the existing main idea is to use deep learning to extract buildings from known and labeled remote sensing images through deep model training, and then use the deep network model to perform image semantic segmentation of buildings on the two-phase remote sensing images of the same area to obtain binary images of buildings, and then perform overlay analysis to detect changes. Depending on the model used, the accuracy and speed of the results are also different. For example, based on the FPN Res-Unet structure, the detection accuracy, recall rate and F1 index have all reached more than 0.90 (Wang Mingchang, Zhu Chunyu, Chen Xueye, etc., 2021); based on the random patch and DeepLabv3+ detection scheme, compared with the Mahalanobis distance supervised learning and U-Net semantic segmentation, the pixel accuracy of this method can reach 99.17% at best, and the average pixel accuracy can reach 93.24% (Wang Minshui, Kong Xiangming, Chen Xueye, etc., 2020). SiamResNet, which is also based on twin neural networks, has surpassed traditional methods in various detection accuracy indicators in two different regions to varying degrees, with accuracy and Kappa index both reaching 0.90 and 0.80 or above (Ni Liangbo, Lu Hanyu, Lu Tianjian, et al., 2020). Foreign scholars have also studied building change detection based on twin neural networks, and the detection accuracy is better than the mainstream detection method (Moyang, Kun and Xiuping et al., 2020). At the same time, some scholars have proposed a research plan of combining multiple features with deep learning, such as fusing texture features, morphological features, spectral features, etc. to generate a feature structure map, and then using deep learning or machine learning to perform classification learning based on the feature structure map, and finally obtaining a high-precision building change binary map. The partial detection accuracy can reach a recall rate of 82.94%, a precision rate of 99.10%, and an F1 index of 90.30%, and minimize the impact of environmental conditions (Wang Chang, Zhang Yongsheng, Ji Song, et al., 2021).

[0013] Although there are many solutions and good results for building change detection based on remote sensing images, the following problems still exist:

[0014] 1) Since the core of change detection is mainly to extract buildings from images, different extraction schemes have their own limitations. For example, when the area or contour features of the building do not change significantly after the building changes, or when the spectral features of the building are similar to the spectral features of the surface, it will be difficult to extract the building and it will be impossible to accurately judge the changes of the building.

[0015] 2) For the accuracy assessment of change detection, most methods use pixel-based assessment methods, which may not be suitable for detection of building objects.

[0016] 3) The final output of most solutions is based on the comparison results of binary or contour maps of buildings, and the results are difficult to work directly with vectorized and information-based cadastral management systems. Summary of the invention

[0017] The purpose of the present invention is to overcome the deficiencies of the above-mentioned prior art and to provide a method, device and equipment for detecting changes in homestead buildings, so as to monitor the changes in buildings on known homesteads, and to cooperate well with the existing geographic database to build an automated and intelligent homestead building change detection system.

[0018] To achieve the above object, the technical solution of the present invention is:

[0019] In a first aspect, the present invention provides a method for detecting changes in homestead buildings, comprising:

[0020] Identify target areas;

[0021] Acquire two remote sensing images of the target area, wherein the two remote sensing images are remote sensing images of the same target area at different time points;

[0022] Marking the homestead areas in the two remote sensing images to obtain homestead vector files;

[0023] The two remote sensing images are aligned, and the two aligned remote sensing images are cropped based on the homestead vector file to obtain a plurality of image blocks, each of which contains only one building of the homestead;

[0024] Generate a pairing table according to the plurality of image blocks, and randomly allocate a training set and a validation set in proportion based on the pairing table; the pairing table includes a pairing table for changes in homestead buildings and a pairing table for unchanged homestead buildings;

[0025] The training set is used to train the twin neural network model, and the verification set is used to verify the accuracy of the model in real time. During the training process, the training parameters are continuously adjusted according to the accuracy evaluation and the iterative training is repeated to obtain a homestead building change detection model and realize homestead building change detection.

[0026] Preferably, before generating the pairing table according to the plurality of image blocks, the method further includes:

[0027] Each cropped image block is normalized so that each image block has the same size and resolution.

[0028] Preferably, the twin neural network includes two U-Net subnetworks, a fully connected layer and a decision layer; the two U-Net subnetworks share the same parameters and are used to extract building features from two remote sensing image blocks containing a single independent building, and obtain two building feature maps as inputs to the fully connected layer. The fully connected layer maps the two building feature maps into two feature vectors and calculates the distance between the two feature vectors to obtain a distance vector as input to the decision layer. The decision layer outputs the feature vector value to represent the similarity between image blocks to determine whether the homestead building has changed.

[0029] Preferably, the decision layer adopts a Sigmoid classifier; the decision layer outputs a feature vector value to represent the similarity between image blocks to determine whether the homestead building has changed, including:

[0030] The Sigmoid classifier maps the distance vector to obtain a feature vector value, and compares the feature vector value with a preset threshold. When the feature vector value is lower than the threshold, it is considered that the two image blocks belong to the same homestead building and no change has occurred; when the feature vector value is higher than the threshold, it is considered that the two image blocks do not belong to the homestead building and a change has occurred.

[0031] Preferably, before using the training set to perform model training on the twin neural network, an image enhancement operation is also performed on the training set, and the image enhancement operation includes flipping, rotating and mirroring;

[0032] The two remote sensing images are aligned using control point matching and geometric transformation methods; and before the alignment, the two remote sensing images are also subjected to white balance and histogram equalization operations. Preferably, the twin neural network uses Binary Cross Entropy as a loss function, and the Binary Cross Entropy loss function is weighted.

[0033] Preferably, the pairing table is generated in the following manner, including:

[0034] Feature point detection and description step: perform feature point detection on each image block, identify key points in the image, and calculate a local feature descriptor for each key point, wherein the local feature descriptor represents local feature information of the key point;

[0035] Feature matching and screening step: Match the local feature descriptors of the image blocks at different time points, and screen the matching results using a random sampling consistency method to find feature point pairs corresponding to the same object to pair the image blocks;

[0036] Pairing table forming step: the successfully paired image blocks form several pairing tables, each pairing table contains two images, one is the image before the change, and the other is the image after the change, the two images correspond to the same homestead, but the shooting time is different; the pairing table for the homestead building changes in the pairing table contains pairing samples of both changed and unchanged situations, which are used to train the twin neural network model to distinguish between changed and unchanged situations; the pairing table for the homestead building does not change in the pairing table only contains pairing samples of unchanged situations, which are used to train the twin neural network model to identify unchanged situations.

[0037] Preferably, the remote sensing image is an orthophoto image obtained by taking pictures with an unmanned aerial vehicle.

[0038] In a second aspect, the present invention provides a device for detecting changes in homestead buildings, comprising:

[0039] An acquisition module is used to acquire two remote sensing images of a target area, wherein the two remote sensing images are remote sensing images of the same target area at different time points;

[0040] A marking module, used for marking the homestead area in the two remote sensing images to obtain a homestead vector file;

[0041] A cropping module is used to align the two remote sensing images, and crop the aligned two remote sensing images based on the homestead vector file to obtain a plurality of image blocks, each of which contains only one building of the homestead;

[0042] A pairing table generation module, used to generate a pairing table according to the plurality of image blocks, and randomly allocate a training set and a validation set in proportion based on the pairing table; the pairing table includes two pairing tables for a homestead building change and a homestead building no change;

[0043] The training module uses the training set to train the twin neural network model, and uses the validation set to test the accuracy of the model in real time. During the training process, the training parameters are continuously adjusted according to the accuracy evaluation and the training is repeated to obtain the homestead building change detection model and realize the homestead building change detection.

[0044] In a third aspect, the present invention provides an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0045] Memory, used to store computer programs;

[0046] The processor is used to implement the steps of the homestead building change detection method as described above when executing the program stored in the memory.

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] The homestead building change detection method provided in the embodiment of the present application directly compares the two-phase remote sensing images after the building is cropped, and directly obtains the Boolean value result of whether the change has occurred. Because the image cropping is based on the vector geographic elements (i.e., the homestead vector file), the change situation can be easily written into the geographic element file (or geographic element database) through the vector geographic element attribute field, and there is no need to consider a series of work related to the post-processing of the raster pixel binary image, which will greatly simplify the construction of the automatic intelligent building detection system and avoid the introduction of other complex analysis and processing processes. At the same time, the homestead building change detection based on the cropped image map, because each image block only contains the buildings of a single homestead, there is no need to consider the influence of background information such as roads, woods, and non-building areas, thereby simplifying the design and training process of the model; in addition, the model directly outputs the change detection results, without the need for complex post-processing, such as filtering, elimination, etc., which improves the detection efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0050] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0051] Figure 1 A flow chart of a method for detecting changes in homestead buildings provided in an embodiment of the present application;

[0052] Figure 2 Schematic diagram of the twin neural network structure;

[0053] Figure 3 This is the drone orthophoto of target area 1 in 2018;

[0054] Figure 4 This is the drone orthophoto of target area 1 in 2019;

[0055] Figure 5 This is the vector file map of the homestead in target area 1;

[0056] Figure 6 A schematic diagram is shown for some samples;

[0057] Figure 7 This is the loss rate result graph of the training process;

[0058] Figure 8 This is the accuracy result diagram of the training process;

[0059] Fig. 9 The following figure shows some of the verification results;

[0060] Fig.10 A schematic diagram of the composition of a homestead building change detection device provided in an embodiment of the present application;

[0061] Fig.11 A schematic diagram of the composition of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0062] Example:

[0063] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0064] To facilitate the understanding of the embodiments of the present application, further explanation will be given below in conjunction with the drawings and specific embodiments. The embodiments do not constitute a limitation on the embodiments of the present application.

[0065] Traditional building change detection has many relatively mature solutions and high-precision results. Many different analysis solutions have emerged based on mathematical analysis, building morphology analysis, shadow analysis, and deep learning. However, since the final output of these methods is a binary pixel map, considering the actual needs of homestead building management, these results all need to consider an inevitable problem: how to link the analysis results with practical applications. Considering a series of objective factors and analysis and processing such as raster resolution, georeferencing, vectorization, and hybrid analysis, new post-processing problems will arise. This series of post-processing has undoubtedly become another thorny problem that needs to be solved for automated and intelligent building detection solutions and systems.

[0066] To this end, the present application embodiment provides a flow chart of a method for detecting changes in homestead buildings. Figure 1 As shown, the homestead building change detection method provided in the embodiment of the present application may specifically include the following steps:

[0067] Step 110, determining the target area;

[0068] In this embodiment, the target area refers to the area including the homestead buildings. It should be noted that the embodiment of the present application does not limit the size, topography, etc. of the target area. A certain area can be selected as the target area according to actual needs to detect changes in the homestead buildings in the target area.

[0069] Step 120, obtaining two remote sensing images of the target area, wherein the two remote sensing images are remote sensing images of the same target area at different time points.

[0070] Step 130, marking the homestead areas in the two remote sensing images to obtain a homestead vector file.

[0071] In this step, the homestead areas in the two remote sensing images of the target area are marked, including the boundaries, area and other information of the homesteads, which serve as the basis for subsequent cropping operations to ensure that each image block only contains buildings on a single homestead.

[0072] Step 140, aligning the two remote sensing images, and cropping the aligned two remote sensing images based on the homestead vector file to obtain a plurality of image blocks, each of which contains only one building of the homestead.

[0073] Specifically, in this step, the two remote sensing images are accurately registered using control point matching and geometric transformation methods to ensure that the spatial position relationship between the images is accurate, laying the foundation for subsequent cropping operations. At the same time, cropping the two aligned remote sensing images based on the homestead vector file can ensure that each image block only contains buildings on a single homestead, so there is no need to consider the impact of roads, woods, and non-building areas, and there is no need to filter and remove the results. Since image cropping is based on vector geographic elements (that is, homestead vector files), the changes can be easily written into the geographic element file (or geographic element database) through the vector geographic element attribute field, and there is no need to consider a series of work related to the post-processing of raster pixel binary images, which will greatly simplify the construction of an automatic intelligent building detection system and avoid the introduction of other complex analysis and processing processes.

[0074] Step 150, generating a pairing table according to the plurality of image blocks, and randomly allocating a training set and a validation set in proportion based on the pairing table; the pairing table includes a pairing table for homestead buildings that have changed and a pairing table for homestead buildings that have not changed.

[0075] In this step, a pairing table is generated based on the cropped image blocks. Since the pairing table includes two copies, one for the homestead buildings that have changed and one for the homestead buildings that have not changed, the image pairs can be read quickly during the subsequent training of the twin neural network model, thereby accelerating the training process.

[0076] Step 160, use the training set to train the twin neural network model, and use the verification set to verify the accuracy of the model in real time. During the training process, the training parameters are continuously adjusted according to the accuracy evaluation and the iterative training is repeated to obtain a homestead building change detection model to realize homestead building change detection.

[0077] In this step, deep twin neural network training is performed based on image pairs of homestead buildings that have changed and homestead buildings that have not changed. The network model that can directly determine whether the building on a single homestead has changed is trained, and the output of the binary map of building changes is no longer used. At the same time, the characteristics of deep learning are used to fully utilize the texture features and spectral features of buildings in high-resolution remote sensing images, overcome the influence of illumination differences in remote sensing images of different periods, and realize building change detection for segmented paired building image pairs. At the same time, the model is evaluated using a validation set to ensure the accuracy and robustness of the model.

[0078] It can be seen that the homestead building change detection method provided in the embodiment of the present application directly compares the two-time phase remote sensing images after the building is cropped, and directly obtains the Boolean value result of whether the change has occurred. Because the image cropping is based on the vector geographic element (that is, the homestead vector file), the change situation can be easily written into the geographic element file (or geographic element database) through the vector geographic element attribute field, and there is no need to consider a series of work related to the post-processing of the raster pixel binary image, which will greatly simplify the construction of the automatic intelligent building detection system and avoid the introduction of other complex analysis and processing processes. At the same time, the homestead building change detection based on the cropped image map, because each image block only contains the building of a single homestead, there is no need to consider the influence of background information such as roads, woods, and non-building areas, thereby simplifying the design and training process of the model; in addition, the model directly outputs the change detection result, without the need for complex post-processing, such as filtering, elimination, etc., which improves the detection efficiency.

[0079] Since the area and shape of each homestead are different, the size of the cropped image blocks is also irregular. In order to meet the input requirements of the deep learning model, all image blocks need to be standardized, for example, all image blocks are scaled and filled into a square of a fixed size to ensure the consistency of the model input. To this end, in a preferred embodiment, before generating a pairing table based on the plurality of image blocks, the following is also included:

[0080] Each cropped image block is normalized so that each image block has the same size and resolution.

[0081] In this way, by standardizing each cropped image block, the image standardization process will not only standardize the image size, but also standardize the image accuracy and resolution. Therefore, there is no need to resample the orthophoto in advance to ensure consistent accuracy and resolution.

[0082] In a specific embodiment, if Figure 2 As shown, the twin neural network includes two U-Net sub-networks, a fully connected layer and a decision layer; the two U-Net sub-networks share the same parameters and are used to extract building features from two remote sensing image blocks containing a single independent building, and obtain two building feature maps as the input of the fully connected layer. The fully connected layer maps the two building feature maps into two feature vectors and calculates the distance between the two feature vectors to obtain the distance vector as the input of the decision layer. The decision layer outputs the feature vector value to represent the similarity between image blocks to determine whether the homestead building has changed. The U-Net sub-network can effectively extract image features and fuse feature information of different scales, thereby overcoming the influence of light and shadow to a certain extent. The twin neural network can accurately judge the similarity between image blocks, thereby realizing high-precision homestead building change detection; at the same time, U-Net has a strong ability to extract image features, which can overcome the influence of light and shadow to a certain extent and improve the robustness of detection.

[0083] It can be seen that by adopting Figure 2 The twin neural network model shown in the figure extracts features from the images at two time points respectively, and calculates the distance between the two feature vectors through a distance measurement layer to determine whether the image has changed. Its advantages are:

[0084] 1. Directly judge changes: The twin neural network directly outputs Boolean values ​​(changed / unchanged) without the need for image segmentation, and the output results are concise and clear.

[0085] 2. Make full use of texture features: The twin neural network can better capture the texture features of the image, thereby overcoming the influence of factors such as shadows and color differences.

[0086] 3. The twin neural network has a relatively simple structure, fewer parameters, and higher training and reasoning efficiency.

[0087] In this way, by combining the powerful feature extraction ability of U-Net and the similarity judgment ability of the twin neural network, high-precision and high-robustness detection of homestead building changes can be achieved. It also has the advantages of strong scalability and no need for post-processing, providing an effective technical means for homestead management.

[0088] In a specific embodiment, the decision layer uses a Sigmoid classifier to better adapt to the characteristics of the homestead image. The decision layer outputs the similarity between image blocks represented by the feature vector value to determine whether the homestead building has changed, including:

[0089] The Sigmoid classifier maps the distance vector to the interval [0, 1] to obtain a feature vector value. The larger the value, the more similar the two image blocks are. The feature vector value is compared with the preset threshold. When the feature vector value is lower than the threshold, it is considered that the two image blocks belong to the same homestead building and have not changed, and 0 is output; when the feature vector value is higher than the threshold, it is considered that the two image blocks do not belong to the homestead building and have changed, and 1 is output. At the same time, the size of the threshold can be dynamically adjusted by the verification accuracy of the model, so that the model can better adapt to changes in lighting and shadow conditions.

[0090] In addition, in order to further overcome the influence of illumination, shadow and color difference of images between two phases, in a preferred embodiment, before using the training set to train the twin neural network model, the training set is also subjected to image enhancement operation, and the image enhancement operation includes flipping, rotation and mirroring. In this way, by performing image enhancement operation on the training set, the diversity of training data is increased, and the generalization ability of the model is improved, so that it can adapt to images under different illumination and shadow conditions. Before the two remote sensing images are aligned by using control point matching and geometric transformation, it also includes: white balance and histogram equalization operation processing are performed on the two remote sensing images, and the image is adjusted to ensure image intensity and color consistency. The image is enhanced by further histogram equalization operation, which provides an accurate basis for subsequent alignment and change detection, and can reduce the influence of image color difference, so that the image has better comparability under different illumination conditions. The remote sensing image is an orthophoto image obtained by drone photography. By using high-resolution drone orthophoto images, more detailed building texture and structural information can be obtained, thereby reducing the influence of illumination and shadow.

[0091] Like other neural networks, the twin neural network also uses the back-propagation algorithm and loss function to continuously correct the model's weight parameters during the training process. For the twin neural network, its loss function is quite different from the general convolutional neural network function. Along with the first generation of binary twin neural networks, the Contrastive Loss loss function (Formula 1) was published. This loss function can well express the matching degree of paired samples. When the Euclidean distance of the samples increases, the loss function also increases; conversely, the loss function decreases.

[0092]

[0093] In the formula , represents the Euclidean distance between two samples; A label indicating whether two samples match; is the preset threshold.

[0094] At the same time, binary classification problems are not limited to using the Contrastive Loss loss function, but can also be calculated using the Binary Cross Entropy loss function. For multi-classification problems, Contrastive Loss often exhibits overfitting problems. To solve this problem, the ternary twin neural network proposes the Triplet Loss loss function to adapt to multi-classification problems.

[0095] In the embodiment of the present application, in the detection of homesteads, the number of homesteads that have not changed is usually much greater than the number of homesteads that have changed. This will cause the model to tend to predict that it has not changed, thereby reducing the ability to detect changed homesteads. In order to solve this problem, the twin neural network of this embodiment adopts the Binary Cross Entropy loss function, and weights the Binary Cross Entropy loss function, giving a higher weight to the change category, so that the model pays more attention to the detection of changed homesteads.

[0096] In a specific embodiment, the pairing table is generated in the following manner, including:

[0097] Feature point detection and description steps: The SURF algorithm is used to detect feature points for each image block and identify key points in the image. Key points are usually areas with unique shapes and textures in the image, such as corners and edges of buildings. Then, the SURF algorithm calculates the local feature descriptor of each key point, which represents the local feature information of the key point. Specifically, the SURF algorithm uses the determinant approximation of the Hessian matrix to detect key points, and uses the integral image technology to quickly calculate the Hessian matrix, thereby improving the efficiency of the algorithm.

[0098] Feature matching and screening steps: Match the local feature descriptors of the image blocks at different time points to find the feature point pairs corresponding to the same object. The matching method selects the nearest neighbor matching to find the nearest matching point for each feature point. Through feature matching, feature point pairs corresponding to the same object can be found, so that the image blocks can be paired. However, due to factors such as noise and occlusion, there may be mismatched feature point pairs in the matching results. Therefore, it is necessary to screen the matching results to remove mismatched feature point pairs. Use the random sampling consensus (RANSAC) method for screening to improve the accuracy of the matching results.

[0099] Pairing table forming step: the successfully paired image blocks form several pairing tables, each pairing table contains two images, one is the image before the change, and the other is the image after the change, the two images correspond to the same homestead, but the shooting time is different; the pairing table for the homestead building changes in the pairing table contains pairing samples of both changed and unchanged situations, which are used to train the twin neural network model to distinguish between changed and unchanged situations; the pairing table for the homestead building does not change in the pairing table only contains pairing samples of unchanged situations, which are used to train the twin neural network model to identify unchanged situations.

[0100] In this way, a pairing table is generated through the above steps to form changed and unchanged training sample pairs, which supports the training process of the twin neural network and realizes high-precision and high-robustness homestead building change detection.

[0101] The following is a further description of the homestead building change detection method provided in the embodiment of the present application in conjunction with an application scenario example:

[0102] Identify target areas:

[0103] The experimental data for this scenario example comes from a drone high-resolution orthophoto map of a target area in a rural area in Dongguan City, Guangdong Province, with a total coverage area of ​​about 3.8 square kilometers, covering the period from 2018 to 2020. Figure 3 ) and 2019 ( Figure 4 ) and its corresponding homestead vector file ( Figure 5 ). The homestead vector file has been labeled with corresponding changes in buildings. A twin neural network model will be trained using deep learning based on two years of image data to detect changes in homestead buildings.

[0104] Crop and standardize homestead imagery:

[0105] GeoTools is an open source Java development library that implements the OGC (Open Geospatial Consortium) specification. It is affiliated with the OSGeo Project and is open source based on the LGPL (GUN Lesser General Public License) open source agreement. It provides a large number of interfaces related to geographic data processing, allowing developers to easily develop related GIS (Geographic Information System) functions based on GeoTools to complete specific tasks. Many large-scale GIS software are developed based on GeoTools, such as SNAP, a dedicated processing software for the European Space Agency's Sentinel satellite series.

[0106] OpenCV is a cross-platform computer vision software library with source code based on C++ and interfaces for other programming languages ​​such as Python and Java. It is open source based on the BSD license. After years of development, OpenCV has become a very mature computer vision development library and has been widely used in machine learning, object recognition, image analysis and processing, and other fields.

[0107] The cropping and enhancement of the homestead image map used in this example is based on the programming development of GeoTools and OpenCV development library, and is developed in Java language. Each cut orthophoto map is distinguished and marked with specific information using a specific naming format.

[0108] Since the area and shape of each homestead are different, the cropped homestead image is also irregular. However, deep learning requires the input image size to be strictly consistent, so the cropped image still needs to be standardized. The standardization is still developed using GeoTools programming, and all image samples are standardized to a size of 256×256 by scaling and filling the blanks.

[0109] It should be noted that the image standardization process not only standardizes the image size, but also standardizes the image accuracy and resolution. Therefore, there is no need to resample the orthophoto in advance to ensure consistent accuracy and resolution.

[0110] Make a pairing table and sample library:

[0111] After completing the cropping of the homestead image map, it is necessary to make a corresponding pairing table so that the image pairs can be quickly read when training the twin neural network model to speed up the training process. The pairing table will be automatically generated based on Java programming, and two pairing tables will be generated for changes and unchanged ones respectively. After completing the pairing table, the training set and the validation set will be randomly allocated in proportion. At the same time, image enhancement will be performed on the training set, and the enhancement operations include flipping, rotating and mirroring. The sample library data details and some samples are shown in Table 1 and Figure 6 shown.

[0112] Table 1 Sample library details

[0113]

[0114] Training the Siamese Neural Network:

[0115] The twin neural network model is programmed to build, train the model and verify the accuracy. The deep learning training is performed on a computer with an Intel Xeon W-2145 CPU, 32G memory, and a graphics card of Nvidia Quadro P5000. The GPU is used to accelerate the training process. The batch size is set to 15 during training, the ADAM optimization algorithm is used, the basic learning rate is set to 0.00006, the gradient moving average specifies an exponential decay rate of 0.9, and the squared gradient moving average specifies an exponential decay rate of 0.99. The training is iterated 10,000 times in total.

[0116] The loss rate and accuracy during training are as follows Figure 7 and Figure 8 As shown in the figure, it can be seen that the accuracy of the 6000th training has stabilized at around 0.93, and the loss rate has also dropped to 0.1, but there are still fluctuations in some cases.

[0117] Homestead building detection accuracy inspection and analysis:

[0118] Based on the trained twin neural network model, the validation set is validated and the accuracy is evaluated. The validation process is a normal prediction process, which does not trigger the loss function of the model and does not affect the weight parameters in the model. Some of the validation results are shown below: Fig. 9 The confusion matrix is ​​shown in Table 2. From the accuracy results, it can be seen that the overall accuracy of the model can be maintained at 93.17%, and the Kappa coefficient has reached 0.7315. The detection accuracy of the unchanged and changed buildings can reach 97.84% and 69.57%, the recall rate is 94.19% and 86.49%, and the F1Score is 95.98% and 77.11%, respectively, which are relatively good accuracy results (Table 3). For homestead change detection, it is more targeted to require a higher recall rate for the changed buildings in order to capture more building changes and illegal buildings, so as to make more selections rather than miss them.

[0119] Table 2 Confusion matrix of model prediction results

[0120]

[0121] Table 3 Model accuracy evaluation

[0122]

[0123] The following is an introduction to the accuracy criteria involved above:

[0124] Accuracy

[0125] Accuracy refers to the ratio of the number of correctly classified samples in the classification results to the total number of samples (Formula 2). It can roughly determine whether a classifier is effective in certain scenarios, but it is not completely applicable to all scenarios and scenarios with targeted indicators.

[0126]

[0127] In the formula, Refers to the positive class being judged as the positive class; It means that the negative class is judged as the positive class; Refers to the positive class being judged as the negative class; Refers to the negative class being judged as the negative class.

[0128] Accuracy

[0129] Precision refers to the percentage of the number of samples that the classifier correctly judges when classifying a certain category to the total number of samples predicted for that category (Formula 3), also known as the precision rate. It is used to evaluate the classifier's ability to classify a certain category from the selected samples. It should be noted that precision and accuracy are two completely different concepts.

[0130]

[0131] Refers to the positive class being judged as the positive class; It means that the negative class is judged as the positive class; Refers to the negative class being judged as the negative class.

[0132] Recall

[0133] Recall refers to the percentage of samples successfully classified by a classifier to the total number of actual samples in a certain category, also known as recall rate. It is used to evaluate the ability of a classifier to correctly find a category from all data when classifying a category. Precision and recall affect each other. Generally speaking, the higher the precision, the lower the recall. Different scenarios have different requirements for precision and recall. In some scenarios that focus on finding all but don't care whether the classification is completely correct, more attention is paid to improving recall; conversely, attention is paid to improving precision. For scenarios that require both, F-Measure is required for comprehensive evaluation, among which F1 Score is a more commonly used one.

[0134]

[0135] Refers to the positive class being judged as the positive class; Refers to the positive class being judged as the negative class.

[0136] F1 Score

[0137] F1 Score (Formula 5) is based on precision and recall, and is used to evaluate the comprehensive ability of a classifier in classifying a certain category. The higher the F1 Score, the stronger the comprehensive ability of the classifier in classifying a certain category.

[0138]

[0139] In the formula, Precision is the precision rate; Recall is the recall rate.

[0140] Kappa coefficient

[0141] The Kappa coefficient is mainly used for consistency testing and can also be used to evaluate classification accuracy (Formula 6), which is calculated based on the confusion matrix. Its value range is [0, 1], and it is divided into 5 levels, among which 0.61~0.80 can be considered as high consistency, and 0.81~1 can be considered as almost complete consistency.

[0142]

[0143] In the formula, is the accuracy rate (Accuracy); .

[0144] In summary, this application conducted a study on homestead identification based on high-resolution two-phase RGB UAV remote sensing image data pairs and homestead vector files, using a twin neural network model, and obtained the following conclusions: The method based on this study can monitor the changes in buildings on known homesteads, and can cooperate well with existing geographic databases to build an automated and intelligent homestead building change detection system, and can realize regular low-cost remote sensing image acquisition tasks and building change detection based on UAV aerial photogrammetry.

[0145] It should be noted that, for the purpose of simple description, the method embodiments are expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present application are not limited to the described order of actions, because according to the embodiments of the present application, certain steps may be performed in other orders or simultaneously.

[0146] like Fig.10 As shown, the embodiment of the present application also provides a homestead building change detection device 200, comprising:

[0147] An acquisition module 210 is used to acquire two remote sensing images of a target area, wherein the two remote sensing images are remote sensing images of the same target area at different time points;

[0148] A marking module 220 is used to mark the homestead area in the two remote sensing images to obtain a homestead vector file;

[0149] A cropping module 230 is used to align the two remote sensing images, and crop the aligned two remote sensing images based on the homestead vector file to obtain a plurality of image blocks, each of which contains only one building of the homestead;

[0150] A pairing table generating module 240 is used to generate a pairing table according to the plurality of image blocks, and randomly allocate a training set and a validation set in proportion based on the pairing table; the pairing table includes two pairing tables for a homestead building change and a homestead building no change;

[0151] The training module 250 uses a training set to train the twin neural network model, and uses a validation set to test the accuracy of the model in real time. During the training process, the training parameters are continuously adjusted according to the accuracy evaluation and the training is repeated to obtain a homestead building change detection model and realize homestead building change detection.

[0152] Optionally, the homestead building change detection device further includes:

[0153] The preprocessing module is used to perform white balance and histogram equalization operations on the two remote sensing images.

[0154] The standardization processing module is used to perform standardization processing on each cropped image block so that each image block has the same size and resolution.

[0155] The image enhancement module is used to perform image enhancement operations on the training set, and the image enhancement operations include flipping, rotating and mirroring.

[0156] Specifically, the twin neural network includes two U-Net sub-networks, a fully connected layer and a decision layer; the two U-Net sub-networks share the same parameters and are used to extract building features from two remote sensing image blocks containing a single independent building, and obtain two building feature maps as input to the fully connected layer. The fully connected layer maps the two building feature maps into two feature vectors and calculates the distance between the two feature vectors to obtain a distance vector as input to the decision layer. The decision layer outputs the feature vector value to represent the similarity between image blocks to determine whether the homestead building has changed.

[0157] Specifically, the decision layer adopts a Sigmoid classifier; the decision layer outputs a feature vector value to represent the similarity between image blocks to determine whether the homestead building has changed, including:

[0158] The Sigmoid classifier maps the distance vector to obtain a feature vector value, and compares the feature vector value with a preset threshold. When the feature vector value is lower than the threshold, it is considered that the two image blocks belong to the same homestead building and no change has occurred; when the feature vector value is higher than the threshold, it is considered that the two image blocks do not belong to the homestead building and a change has occurred.

[0159] Specifically, the twin neural network adopts Binary Cross Entropy as the loss function and weights the Binary Cross Entropy loss function.

[0160] Specifically, the remote sensing image is an orthophoto image obtained by taking pictures with an unmanned aerial vehicle.

[0161] Specifically, the pairing table generating module includes:

[0162] A feature point detection and description unit is used to perform feature point detection on each image block, identify key points in the image, and calculate a local feature descriptor for each key point, wherein the local feature descriptor represents local feature information of the key point;

[0163] A feature matching and screening unit is used to match the local feature descriptors of the image blocks at different time points, and screen the matching results using a random sampling consistency method to find feature point pairs corresponding to the same object to pair the image blocks;

[0164] A pairing table forming unit is used to form a number of pairing tables from the successfully paired image blocks, each pairing table contains two images, one is the image before the change, and the other is the image after the change, the two images correspond to the same homestead, but the shooting time is different; the pairing table for the homestead building changes in the pairing table contains pairing samples of both changed and unchanged situations, which are used to train the twin neural network model to distinguish between changed and unchanged situations; the pairing table for the homestead building does not change in the pairing table only contains pairing samples of unchanged situations, which are used to train the twin neural network model to identify unchanged situations.

[0165] It should be noted that the homestead building change detection device provided in the embodiments of the present application can execute the homestead building change detection method provided in any embodiment of the present application, and has the corresponding functions and beneficial effects of the execution method.

[0166] In a specific implementation, the above-mentioned homestead building change detection device can be integrated into a device, and the electronic device can be composed of two or more physical entities, or it can be composed of one physical entity. For example, the electronic device can be a personal computer (PC), a computer, a server, etc. The embodiment of the present application does not impose specific restrictions on this.

[0167] like Fig.11 As shown, an embodiment of the present application provides an electronic device, including a processor 111, a communication interface 112, a memory 113 and a communication bus 114, wherein the processor 111, the communication interface 112, and the memory 113 communicate with each other through the communication bus 114; the memory 113 is used to store computer programs; the processor 111 is used to implement the steps of the homestead building change detection method provided by any of the aforementioned method embodiments when executing the program stored in the memory 113. Exemplarily, the steps of the method for detecting changes in homestead buildings may include the following steps: determining a target area; obtaining two remote sensing images of the target area, the two remote sensing images being remote sensing images at different time points; marking the homestead area in the two remote sensing images to obtain a homestead vector file; aligning the two remote sensing images, and cropping the aligned two remote sensing images based on the homestead vector file to obtain a number of image blocks, each image block containing only one homestead building; generating a pairing table based on the number of image blocks, and randomly allocating a training set and a validation set in proportion based on the pairing table; the pairing table includes two pairing tables for homestead buildings that have changed and for homestead buildings that have not changed; using the training set to train a twin neural network model, and using the validation set to verify the accuracy of the model in real time, continuously adjusting the training parameters according to the accuracy evaluation during the training process and repeating the iterative training to obtain a homestead building change detection model and realize homestead building change detection.

[0168] An embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the method for detecting changes in homestead buildings provided in any of the aforementioned method embodiments are implemented.

[0169] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0170] The above description is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest range consistent with the principles and novel features applied for herein.

Claims

1. A method for detecting changes in homestead buildings, characterized in that: include: Identify target areas; Acquire two remote sensing images of the target area, wherein the two remote sensing images are remote sensing images of the same target area at different time points; Marking the homestead areas in the two remote sensing images to obtain homestead vector files; The two remote sensing images are aligned, and the two aligned remote sensing images are cropped based on the homestead vector file to obtain a plurality of image blocks, each of which contains only one building of the homestead; Generate a pairing table according to the plurality of image blocks, and randomly allocate a training set and a validation set in proportion based on the pairing table; the pairing table includes a pairing table for changes in homestead buildings and a pairing table for unchanged homestead buildings; The training set is used to train the twin neural network model, and the verification set is used to test the accuracy of the model in real time. During the training process, the training parameters are continuously adjusted according to the accuracy evaluation and the iterative training is repeated to obtain a homestead building change detection model, thereby realizing homestead building change detection; The twin neural network includes two U-Net subnetworks, a fully connected layer and a decision layer; the two U-Net subnetworks share the same parameters and are used to extract building features from two remote sensing image blocks containing a single independent building, and obtain two building feature maps as inputs of the fully connected layer. The fully connected layer maps the two building feature maps into two feature vectors and calculates the distance between the two feature vectors to obtain a distance vector as input of the decision layer. The decision layer outputs the similarity between the image blocks represented by the feature vector value to determine whether the homestead building has changed; The pairing table is generated in the following manner, including: Feature point detection and description step: perform feature point detection on each image block, identify key points in the image, and calculate a local feature descriptor for each key point, wherein the local feature descriptor represents local feature information of the key point; Feature matching and screening step: Match the local feature descriptors of the image blocks at different time points, and screen the matching results using a random sampling consistency method to find feature point pairs corresponding to the same object to pair the image blocks; Pairing table forming step: the successfully paired image blocks form several pairing tables, each pairing table contains two images, one is the image before the change, and the other is the image after the change, the two images correspond to the same homestead, but the shooting time is different; the pairing table for the homestead building changes in the pairing table contains pairing samples of both changed and unchanged situations, which are used to train the twin neural network model to distinguish between changed and unchanged situations; the pairing table for the homestead building does not change in the pairing table only contains pairing samples of unchanged situations, which are used to train the twin neural network model to identify unchanged situations.

2. The method for detecting changes in homestead buildings as claimed in claim 1, characterized in that: Before generating a pairing table according to the plurality of image blocks, the method further includes: Each cropped image block is normalized so that each image block has the same size and resolution.

3. The method for detecting changes in homestead buildings as claimed in claim 1, characterized in that: The decision layer uses a Sigmoid classifier; the decision layer outputs a feature vector value to represent the similarity between image blocks to determine whether the homestead building has changed, including: The Sigmoid classifier maps the distance vector to obtain a feature vector value, and compares the feature vector value with a preset threshold. When the feature vector value is lower than the threshold, it is considered that the two image blocks belong to the same homestead building and no change has occurred; when the feature vector value is higher than the threshold, it is considered that the two image blocks do not belong to the homestead building and a change has occurred.

4. The method for detecting changes in homestead buildings as claimed in claim 1, characterized in that: Before using the training set to perform model training on the twin neural network, an image enhancement operation is also performed on the training set, and the image enhancement operation includes flipping, rotating and mirroring; Using control point matching and geometric transformation methods to align the two remote sensing images; Before the registration is performed, the method further includes: performing white balance and histogram equalization operations on the two remote sensing images.

5. The method for detecting changes in homestead buildings as claimed in claim 1, characterized in that: The twin neural network adopts Binary Cross Entropy as a loss function and weights the Binary Cross Entropy loss function.

6. The method for detecting changes in homestead buildings as claimed in claim 1, characterized in that: The remote sensing image is an orthophoto image obtained by taking pictures with an unmanned aerial vehicle.

7. A device for detecting changes in homestead buildings, characterized in that: include: An acquisition module is used to acquire two remote sensing images of a target area, wherein the two remote sensing images are remote sensing images of the same target area at different time points; A marking module, used for marking the homestead area in the two remote sensing images to obtain a homestead vector file; A cropping module is used to align the two remote sensing images, and crop the aligned two remote sensing images based on the homestead vector file to obtain a plurality of image blocks, each of which contains only one building of the homestead; A pairing table generation module, used to generate a pairing table according to the plurality of image blocks, and randomly allocate a training set and a validation set in proportion based on the pairing table; the pairing table includes two pairing tables for a homestead building change and a homestead building no change; The training module uses the training set to train the twin neural network model, and uses the validation set to test the accuracy of the model in real time. During the training process, the training parameters are continuously adjusted according to the accuracy evaluation and the training is repeated to obtain the homestead building change detection model and realize the homestead building change detection; The twin neural network includes two U-Net subnetworks, a fully connected layer and a decision layer; the two U-Net subnetworks share the same parameters and are used to extract building features from two remote sensing image blocks containing a single independent building, and obtain two building feature maps as inputs of the fully connected layer. The fully connected layer maps the two building feature maps into two feature vectors and calculates the distance between the two feature vectors to obtain a distance vector as input of the decision layer. The decision layer outputs the similarity between the image blocks represented by the feature vector value to determine whether the homestead building has changed; The pairing table generating module comprises: A feature point detection and description unit is used to perform feature point detection on each image block, identify key points in the image, and calculate a local feature descriptor for each key point, wherein the local feature descriptor represents local feature information of the key point; A feature matching and screening unit is used to match the local feature descriptors of the image blocks at different time points, and screen the matching results using a random sampling consistency method to find feature point pairs corresponding to the same object to pair the image blocks; A pairing table forming unit is used to form several pairing tables from the successfully paired image blocks, each pairing table contains two images, one is the image before the change, and the other is the image after the change, the two images correspond to the same homestead, but the shooting time is different; the pairing table for the homestead building changes in the pairing table contains pairing samples of both changed and unchanged situations, which are used to train the twin neural network model to distinguish between changed and unchanged situations; the pairing table for the homestead building does not change in the pairing table only contains pairing samples of unchanged situations, which are used to train the twin neural network model to identify unchanged situations.

8. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; The processor is used to implement the steps of the homestead building change detection method described in any one of claims 1-6 when executing the program stored in the memory.

Images