Unmanned aerial vehicle-based illegal land use change detection method and device, equipment and medium

Abstract

The invention provides an illegal land use change detection method and device based on an unmanned aerial vehicle, equipment and a medium. Obtaining image difference features of the plurality of channels according to the image pairs and the mask binary image, obtaining text sequence features according to the labeled text, and fusing the image difference features and the text sequence features to obtain first fusion features; aggregating the first fusion features to obtain multi-scale fusion features; fusing the multi-scale fusion feature and the first fusion feature to obtain a second fusion feature; fusing the second fusion feature and the text sequence feature to obtain a third fusion feature; decoding the third fusion feature to obtain a change area graph; performing illegal land use change detection according to the change area map to obtain a detection result; according to the method, multi-type and multi-channel data are fused for multiple times by using the mask binary image containing the image pair and the change region and the multi-modal data of the annotation text, richer feature information is provided for the model, and the generalization ability and prediction accuracy of the model are improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of land surveying, and in particular to a method, device, equipment and medium for detecting changes in illegal land use based on drones. Background Art

[0002] At present, the method of inspecting illegal land use mainly relies on manual visits and inspections. Workers visit the city and then identify illegal land use with the naked eye, which requires a lot of manpower and is very inefficient. In addition, manual inspections can be carried out based on drones. The drones are equipped with cameras, and operators use real-time screen videos to identify illegal land use. In this process, workers also need to rely on manual visual identification, which is inefficient and easy to miss. In addition, the planning department needs to provide original land and building planning data for comparison, and planning data is difficult to obtain conveniently. Summary of the invention

[0003] The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.

[0004] The purpose of this application is to solve at least one of the technical problems existing in the relevant technology to a certain extent. The embodiments of this application provide a method, device, equipment and medium for detecting changes in illegal land use based on drones, which can improve the generalization ability and prediction accuracy of the model and improve the detection efficiency.

[0005] An embodiment of the first aspect of the present application is a method for detecting changes in illegal land use based on a drone, comprising:

[0006] Constructing multimodal data, and inputting the multimodal data into a change detection model, the multimodal data including an image pair consisting of a first land image and a second land image, a mask binary image of a changed region of the image pair, and annotated text describing a change of the image pair, wherein the first land image and the second land image are images taken of the same land at different times;

[0007] Performing feature extraction on the image pair and the mask binary image to obtain image difference features of multiple channels, performing feature extraction on the annotated text to obtain text sequence features, and fusing the image difference features and the text sequence features to obtain a first fused feature;

[0008] Aggregating the first fusion features of multiple channels to obtain multi-scale fusion features of multiple channels;

[0009] Fusing the multi-scale fusion features of multiple channels and the first fusion features to obtain second fusion features of multiple channels;

[0010] Fusing the second fused features of the multiple channels and the text sequence features to obtain third fused features of the multiple channels;

[0011] Decoding the third fusion features of multiple channels to obtain a change area map;

[0012] The illegal land use change detection is performed according to the change area map to obtain the detection result.

[0013] According to certain embodiments of the first aspect of the present application, the step of extracting features according to the image pair and the mask binary image to obtain image difference features of multiple channels includes:

[0014] Extracting features from the first land image of each channel by using a first image encoder to obtain first image features;

[0015] Extracting features from the second land image of each channel by using a second image encoder to obtain second image features;

[0016] Subtracting the first image feature from the second image feature of the same channel to obtain an image difference feature;

[0017] The first image encoder and the second image encoder share parameters.

[0018] According to certain embodiments of the first aspect of the present application, the fusing the image difference feature and the text sequence feature to obtain a first fused feature includes:

[0019] The image difference feature is embedded in the result of the first position encoding to perform attention calculation to obtain a first attention feature;

[0020] The result of embedding the text sequence feature into the second position encoding is used as a query for attention calculation, the first attention feature is used as a key and value for attention calculation, and attention calculation is performed according to the query, key and value to obtain a second attention feature;

[0021] Residual connection processing and normalization processing are performed according to the second attention feature to obtain the first fusion feature.

[0022] According to certain embodiments of the first aspect of the present application, aggregating the first fusion features of the multiple channels to obtain multi-scale fusion features of the multiple channels includes:

[0023] The first fusion features of multiple channels are averagely pooled and combined to obtain dimensional features;

[0024] Extracting features from the dimensional features to obtain a plurality of first branch image features of different types;

[0025] Performing dimensionality reduction processing and vector conversion processing on the result of adding the multiple first branch image features to obtain multiple first feature vectors;

[0026] Multiplying the first feature vectors by the first branch image features to obtain the second branch image features;

[0027] The result of adding multiple second branch image features is split and up-sampled to obtain multi-scale fusion features of multiple channels.

[0028] According to some embodiments of the first aspect of the present application, the fusing of the second fused features of the multiple channels and the text sequence features to obtain the third fused features of the multiple channels includes:

[0029] The second fusion feature of each channel is embedded into the result of the third position encoding to perform attention calculation to obtain a third attention feature;

[0030] The result of embedding the text sequence feature into the fourth position encoding is used as a query for attention calculation, the third attention feature is used as a key and value for attention calculation, and attention calculation is performed according to the query, key and value to obtain a fourth attention feature;

[0031] Residual connection processing and normalization processing are performed according to the fourth attention feature to obtain the third fusion feature.

[0032] According to certain embodiments of the first aspect of the present application, decoding the third fusion features of the multiple channels to obtain a change region map includes:

[0033] The third fusion feature of each channel is gradually upsampled to obtain an upsampled feature;

[0034] The up-sampled features are convolved to obtain a change region map.

[0035] According to certain embodiments of the first aspect of the present application, the method further comprises:

[0036] Obtaining a loss value according to the detection result, and adjusting parameters of the change detection model according to the loss value;

[0037] Wherein, the loss value is expressed as: L bce (Y,S(t1,t2,T))=Y×log(S(t1,t2,T))+(1-Y)×log(1-S(t1,t2,T)); L bce (Y,S(t1,t2,T)) is the loss value, Y is the mask binary image, S(t1,t2,T) is the detection result, t1 is the first land image, t2 is the second land image, and T is the labeled text.

[0038] According to an embodiment of the second aspect of the present application, a device for detecting changes in illegal land use includes:

[0039] An input module is used to construct multimodal data and input the multimodal data into a change detection model, wherein the multimodal data includes an image pair consisting of a first land image and a second land image, a mask binary image of a changed region of the image pair, and an annotation text describing the change of the image pair, wherein the first land image and the second land image are images taken of the same land at different times;

[0040] A multimodal encoding module, used for performing feature extraction based on the image pair and the mask binary image to obtain image difference features of multiple channels, and performing feature extraction based on the annotated text to obtain text sequence features;

[0041] A multimodal feature fusion module, used for fusing the image difference feature and the text sequence feature to obtain a first fusion feature;

[0042] An adaptive aggregation module, used for aggregating the first fusion features of multiple channels to obtain multi-scale fusion features of multiple channels;

[0043] A first fusion module, used for fusing the multi-scale fusion features of multiple channels and the first fusion features to obtain second fusion features of multiple channels;

[0044] A second fusion module, used for fusing the second fusion features of multiple channels and the text sequence features to obtain third fusion features of multiple channels;

[0045] A decoding module, used for decoding the third fusion features of multiple channels to obtain a change area map;

[0046] The detection module is used to detect changes in illegal land use according to the change area map to obtain detection results.

[0047] An embodiment of the third aspect of the present application is an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for detecting changes in illegal land use based on a drone as described in the embodiment of the first aspect of the present application is implemented.

[0048] An embodiment of the fourth aspect of the present application is a computer storage medium storing computer executable instructions, wherein the computer executable instructions are used to execute the drone-based illegal land use change detection method as described in the embodiment of the first aspect of the present application.

[0049] The above scheme has at least the following beneficial effects: by obtaining image difference features of multiple channels according to image pairs and mask binary images, obtaining text sequence features according to annotated texts, fusing image difference features and text sequence features to obtain a first fusion feature; aggregating image difference features to obtain multi-scale fusion features; fusing multi-scale fusion features and first fusion features to obtain a second fusion feature; fusing the second fusion feature and text sequence features to obtain a third fusion feature; decoding the third fusion feature to obtain a change area map; performing illegal land use change detection according to the change area map to obtain a detection result; utilizing multi-modal data including image pairs, mask binary images of change areas and annotated texts to fuse multi-type and multi-channel data multiple times to provide the model with richer feature information, which helps to improve the generalization ability and prediction accuracy of the model; improving the automation and intelligence of land change detection, improving the efficiency of illegal land use detection, and saving labor costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The accompanying drawings are used to provide further understanding of the technical solution of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the technical solution of the present application and do not constitute a limitation on the technical solution of the present application.

[0051] Figure 1 is a step diagram of a method for detecting illegal land use changes based on a drone provided in an embodiment of the present application;

[0052] Figure 2 It is a sub-step diagram of obtaining image difference features of multiple channels by extracting features based on image pairs and mask binary images through a multi-modal encoder module;

[0053] Figure 3 It is a sub-step diagram of step S300 provided in an embodiment of the present application;

[0054] Figure 4 It is a sub-step diagram of step S400 provided in an embodiment of the present application;

[0055] Figure 5 It is a sub-step diagram of step S700 provided in an embodiment of the present application;

[0056] Figure 6 is a structural diagram of an illegal land use change detection device provided in an embodiment of the present application;

[0057] Figure 7 is a schematic diagram of a change detection model provided in an embodiment of the present application;

[0058] Figure 8 is a schematic diagram of a multimodal encoder module provided in an embodiment of the present application;

[0059] Fig. 9is a schematic diagram of a multimodal feature fusion module provided in an embodiment of the present application;

[0060] Fig.10 is a schematic diagram of an adaptive aggregation module provided in an embodiment of the present application;

[0061] Fig.11 is a schematic diagram of a Mamba feature extraction network provided in an embodiment of the present application;

[0062] Fig.12 It is a schematic diagram of a decoding module provided in an embodiment of the present application. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0064] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims or the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0065] The embodiments of the present application are further described below in conjunction with the accompanying drawings.

[0066] An embodiment of the present application provides a method for detecting changes in illegal land use based on a drone.

[0067] Reference Figure 1 and Figure 7 ,The illegal land use change detection method based on UAV includes the following steps:

[0068] Step S100, constructing multimodal data, and inputting the multimodal data into a change detection model; the multimodal data includes an image pair consisting of a first land image and a second land image, a mask binary image of a changed area of ​​the image pair, and annotated text describing the change of the image pair, wherein the first land image and the second land image are images taken of the same land at different times.

[0069] Step S200, performing feature extraction based on the image pair and the mask binary image to obtain image difference features of multiple channels, and performing feature extraction based on the annotated text to obtain text sequence features.

[0070] Step S300: fusing the image difference feature and the text sequence feature to obtain a first fused feature.

[0071] Step S400: Aggregate the first fusion features of multiple channels to obtain multi-scale fusion features of multiple channels.

[0072] Step S500, fusing the multi-scale fusion features of multiple channels and the first fusion features to obtain second fusion features of multiple channels.

[0073] Step S600: fusing the second fused features of multiple channels and the text sequence features to obtain the third fused features of multiple channels.

[0074] Step S700, decoding the third fusion features of multiple channels to obtain a change area map.

[0075] Step S800, performing illegal land use change detection according to the change area map to obtain a detection result.

[0076] By utilizing multimodal data including image pairs, masked binary images of changed areas, and annotated text, multi-type and multi-channel data are fused multiple times to provide the model with richer feature information, which helps to improve the model's generalization ability and prediction accuracy.

[0077] For step S100, a remote sensing device is mounted on a drone, and a remote sensing image is collected once at each node that the drone passes through. After a period of time, remote sensing images are collected again. The collected remote sensing images are manually annotated, and images of the same land taken at different times are repeatedly obtained in this way to form a data set. A first land image and a second land image are selected from the data set, so that the image pair formed by the first land image and the second land image is a dual-phase image.

[0078] For each image pair, the first land image and the second land image are spatially aligned. The difference between the aligned first land image and the second land image is calculated, for example, by subtracting the pixel values ​​of the first land image and the second land image to obtain a difference map, and setting a mask on the change area in the difference map. The difference map including the mask is converted into a binary image.

[0079] The mask binary images of the changed areas are annotated by labeling software such as labe lme to obtain the annotation text. As for the annotation text describing the differences between images, since the dual-temporal remote sensing images in the dataset have a long time span and irrelevant changes often occur in remote sensing images, a pair of remote sensing images cannot be exactly the same. In the text annotation process, only the image pairs with irrelevant changes are the unchanged image pairs, and the following principles are followed: describe important changes (for example, changes in ground objects), ignore irrelevant changes and unimportant interference factors (for example, changes in light). Describe the objects of change and the type of change, such as appearance and disappearance. The following five sentences are used as invariant annotations: "The scenes of these two pictures seem to be the same", "The scene of this picture is the same as the scene of the previous picture", "No difference", "No change", "Almost no change".

[0080] A collection of multimodal data is constructed using bi-temporal images, masked binary images, and annotated text.

[0081] Preprocess the collection of multimodal data. Remote sensing change detection requires high degree of registration of multi-temporal images. However, remote sensing images are inevitably affected by differences in sensor posture, distance, quality, and weather. Therefore, before change detection, it is an essential step to perform corresponding precise geometric correction, atmospheric correction, mutual registration and other image preprocessing on remote sensing images of different periods. The benefit of doing so is to improve the accuracy of change detection. The purpose of image registration is to find the geometric transformation relationship between two images so that their corresponding feature points or feature areas in space can overlap. The optimal transformation parameters are determined by calculation and optimization to minimize the differences between images, improve the accuracy and efficiency of image analysis, and provide a basis for subsequent image processing and analysis tasks.

[0082] Reference Figure 2 For step S200, the multimodal encoder module 20 performs feature extraction based on the image pair and the mask binary image to obtain image difference features of multiple channels, including the following steps:

[0083] Step S210, extracting features from the first land image of each channel by using a first image encoder to obtain first image features;

[0084] Step S220, extracting features from the second land image of each channel by using a second image encoder to obtain second image features;

[0085] Step S230: subtract the first image feature from the second image feature of the same channel to obtain an image difference feature.

[0086] The first image encoder and the second image encoder share parameters.

[0087] Feature extraction is performed based on the annotated text to obtain text sequence features.

[0088] Reference Figure 8 ,The multimodal encoder module adopts a parameter-sharing twin image encoder and a text encoder as the basic ,bitemporal feature extractor for the multimodal remote sensing change ,detection task. The twin image encoder includes two image encoders, namely the ,first image encoder and the second image encoder.

[0089] The dual-phase images are respectively passed through the twin image encoder to generate image encoding results, which are subtracted to obtain image difference features; the text information is input into the text encoder to generate a sequence feature vector, i.e., the text sequence feature. Starting from the feature extraction stage, the dual-phase image encoder obtains two sets of feature pyramids, which represent dual-phase feature mappings at different levels. The difference features obtained by subtracting the two sets of features are used as part of the subsequent feature fusion.

[0090] Reference Figure 3 For step S300, the multimodal feature fusion module 30 fuses the image difference feature and the text sequence feature to obtain a first fusion feature, including the following steps:

[0091] Step S310, embedding the image difference feature into the result of the first position encoding to perform attention calculation to obtain a first attention feature;

[0092] Step S320, embedding the text sequence feature into the result of the second position encoding as a query for attention calculation, using the first attention feature as a key and value for attention calculation, performing attention calculation according to the query, key and value, and obtaining a second attention feature;

[0093] Step S330, performing residual connection processing and normalization processing according to the second attention feature to obtain the first fusion feature.

[0094] The multimodal feature fusion module consists of four parts: input, encoding, decoding and output.

[0095] Exemplarily, the image difference features include image difference feature VF1, image difference feature VF2, image difference feature VF3, and image difference feature VF4. The text sequence feature is LF.

[0096] The image difference features VF1, VF2, VF3, VF4 and text sequence features LF of different channels are input into the embedding position one by one for encoding. In the encoder and decoder parts, they are composed of two sub-layer connection structures. The first sub-layer is a multi-head self-attention sub-layer, and the second sub-layer is a feed-forward fully connected sub-layer. Each sub-layer is followed by a normalization layer and a residual connection. Finally, the first fusion feature is output.

[0097] Reference Fig. 9 Specifically, the image difference feature is input into the embedding layer and embedded into the first position code; then the image difference feature embedded into the first position code is subjected to multi-head attention calculation to obtain the first attention feature. The first attention feature and the image difference feature embedded into the first position code are residually connected and normalized. The normalized result is input into the feedforward network, and the output of the feedforward network is residually connected and normalized with the normalized result to obtain the feature output corresponding to the image difference feature. The text sequence feature is input into the embedding layer and embedded into the second position code; then the text sequence feature embedded into the second position code is used as the query of the attention calculation, and the feature output corresponding to the image difference feature obtained from the first attention feature is used as the key and value of the attention calculation, and the attention calculation is performed according to the query, the key and the value to obtain the second attention feature; the second attention feature and the text sequence feature embedded into the second position code are residually connected and normalized. The normalized result is input into the feedforward network, and the output of the feedforward network is residually connected and normalized with the normalized result to obtain the first fusion feature.

[0098] Reference Figure 4 For step S400, the first fusion features of multiple channels are aggregated by the adaptive aggregation module 40 to obtain multi-scale fusion features of multiple channels, including the following steps:

[0099] Step S410, average pooling and splicing the first fusion features of multiple channels to obtain dimensional features;

[0100] Step S420, extracting the dimensional features to obtain a plurality of first branch image features of different types;

[0101] Step S430, performing dimensionality reduction processing and vector conversion processing on the result of adding the multiple first branch image features to obtain multiple first feature vectors;

[0102] Step S440, multiplying the plurality of first feature vectors by the plurality of first branch image features correspondingly to obtain a plurality of second branch image features;

[0103] Step S450 , performing splitting processing and upsampling processing on the result of adding the multiple second branch image features to obtain multi-scale fusion features of multiple channels.

[0104] Reference Fig.10The adaptive aggregation module 40 is mainly composed of three parts: multi-branch feature extraction, multi-branch feature fusion and dynamic weighted fusion. The first fusion feature is averaged and spliced ​​to obtain dimensional features of the same dimension, and multi-branch feature extraction is performed through Transformer, convolution and Mamba feature extraction networks respectively to obtain multiple first branch image features of different types. Then the first branch image features of multiple branches are added, and the fully connected layer is used to perform dimensionality reduction processing and the Softmax vector conversion processing is used to obtain the first feature vector. The result of adding multiple first branch image features is subjected to dimensionality reduction processing and vector conversion processing to obtain the first feature vector. Finally, the first feature vectors are added, split, and upsampled into a feature pyramid to obtain multi-scale fusion features of multiple channels.

[0105] Among them, the Mamba feature extraction network is built on the basis of the state space model (State Space Model l, SSM). By introducing the concept of selective state space, the model can process long sequences more efficiently. Unlike the standard SSM, the Mamba feature extraction network allows the state transition to partially depend on the current input, thereby increasing the flexibility of the model while maintaining computational efficiency. In traditional SSM, the state transition is usually fixed and the input is unchanged, that is, a set of fixed parameters (matrices A and B) is applied to the entire sequence. The Mamba feature extraction network uses input-related transformations to make these parameters adaptively adjusted according to the current input at each time step.

[0106] Reference Fig.11 In the Mamba feature extraction network, the input feature obtains the first sub-feature through linear projection, convolution, SiLU function calculation and state space model; the input feature obtains the second sub-feature through linear projection and SiLU function calculation; the results of nonlinear calculation of the first sub-feature and the second sub-feature are linearly projected to obtain the output feature of the Mamba feature extraction network.

[0107] For step S500, the multi-scale fusion features of multiple channels and the first fusion features are fused to obtain the second fusion features of multiple channels.

[0108] For step S600, fusing the second fused features of multiple channels and the text sequence features to obtain the third fused features of multiple channels includes the following steps:

[0109] The second fusion feature of each channel is embedded in the result of the third position encoding to perform attention calculation to obtain the third attention feature; the text sequence feature is embedded in the result of the fourth position encoding as the query of the attention calculation, the third attention feature is used as the key and value of the attention calculation, and attention calculation is performed according to the query, key and value to obtain the fourth attention feature; residual connection processing and normalization processing are performed on the fourth attention feature to obtain the third fusion feature.

[0110] Step S600 is similar to step S300 and will not be described again herein.

[0111] Reference Figure 5 and Fig.12 For step S700, decoding the third fusion features of multiple channels by the decoding module 70 to obtain a change area map includes the following steps:

[0112] Step S710, performing step-by-step upsampling processing on the third fused features of each channel to obtain upsampled features;

[0113] Step S720, performing convolution processing on the upsampled features to obtain a change region map.

[0114] For step S800, illegal land use change detection is performed according to the change area map to obtain a detection result.

[0115] It should be noted that the change detection network needs to be trained before the change detection model is launched. The multimodal dataset is divided into a training set (70%), a validation set (15%), and a test set (15%). The input image is represented as t1 and t2, with dimensions of H×W×3, corresponding to the height, width, and RGB color channels of the image, respectively.

[0116] The training set is input into the change detection model, and the change detection model outputs the detection result. The loss function is calculated based on the detection result to obtain the loss value, and the parameters of the change detection model are adjusted based on the loss value.

[0117] Among them, the loss function is the cross entropy loss function. The loss value is expressed as: L bce (Y,S(t1,t2,T))=Y×log(S(t1,t2,T))+(1-Y)×log(1-S(t1,t2,T)); L bce (Y,S(t1,t2,T)) is the loss value, Y is the mask binary image, S(t1,t2,T) is the detection result, t1 is the first land image, t2 is the second land image, and T is the labeled text.

[0118] The cross entropy loss function is used to calculate the relative entropy between the predicted probability distribution and the true label distribution to quantify the degree of difference between the two. During the model training process, the optimization algorithm continuously adjusts the model parameters to minimize the value of the cross entropy loss function, thereby improving the model's prediction accuracy and generalization ability.

[0119] The trained change detection model is tested and verified through the validation set and test set.

[0120] The change detection model is put online. First, the image pairs taken by the drone are preprocessed, then input into the change detection network, and then the detection results are output. The detection results are then transmitted to the staff's relevant applications through the Internet, and finally the staff further determines whether the land use is illegal.

[0121] An embodiment of the present application provides a device for detecting changes in illegal land use.

[0122] Reference Figure 6 The illegal land use change detection device includes: an input module 10, a multimodal encoding module 20, a multimodal feature fusion module 30, an adaptive aggregation module 40, a first fusion module 50, a second fusion module 60, a decoding module 70 and a detection module 80.

[0123] The input module 10 is used to construct multimodal data and input the multimodal data into the change detection model. The multimodal data includes an image pair consisting of a first land image and a second land image, a mask binary image of a changed area of ​​the image pair, and annotated text describing the change of the image pair. The first land image and the second land image are images taken of the same land at different times.

[0124] The multimodal encoding module 20 is used to perform feature extraction based on the image pairs and the mask binary images to obtain image difference features of multiple channels, and to perform feature extraction based on the annotated text to obtain text sequence features.

[0125] The multimodal feature fusion module 30 is used to fuse the image difference feature and the text sequence feature to obtain a first fused feature.

[0126] The adaptive aggregation module 40 is used to aggregate the first fusion features of multiple channels to obtain multi-scale fusion features of multiple channels.

[0127] The first fusion module 50 is used to fuse the multi-scale fusion features of multiple channels and the first fusion features to obtain the second fusion features of multiple channels.

[0128] The second fusion module 60 is used to fuse the second fusion features of multiple channels and the text sequence features to obtain the third fusion features of multiple channels.

[0129] The decoding module 70 is used to decode the third fusion features of multiple channels to obtain a change area map.

[0130] The detection module 80 is used to detect changes in illegal land use according to the change area map to obtain detection results.

[0131] The illegal land use change detection device adopts the above illegal land use change detection method. Each module of the illegal land use change detection device corresponds to each step of the illegal land use change detection method. The illegal land use change detection device and the illegal land use change detection method adopt the same technical means, solve the same technical problems, and have the same technical effects.

[0132] An embodiment of the present application provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned method for detecting illegal land use changes based on a drone when executing the computer program.

[0133] The electronic device may be any intelligent terminal including a computer.

[0134] In general, for the hardware structure of the electronic device, the processor can be implemented using a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, etc., to execute related programs to implement the technical solutions provided in the embodiments of the present application.

[0135] The memory can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory can store an operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory, and the processor is called to execute the methods of the embodiments of this application.

[0136] The input / output interface is used to realize information input and output.

[0137] The communication interface is used to realize the communication interaction between this device and other devices. Communication can be achieved through wired methods (such as USB, network cable, etc.) or wireless methods (such as mobile network, WIFI, Bluetooth, etc.).

[0138] The bus transmits information between the various components of the device (such as the processor, memory, input / output interface, and communication interface). The processor, memory, input / output interface, and communication interface communicate with each other within the device through the bus.

[0139] An embodiment of the present application provides a computer storage medium storing computer executable instructions for executing the above-mentioned method for detecting illegal land use changes based on drones.

[0140] It will be appreciated by those skilled in the art that all or some of the steps and systems in the disclosed method above may be implemented as software, firmware, hardware and appropriate combinations thereof. Some physical components or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or may be implemented as hardware, or may be implemented as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or a non-transitory medium) and a communication medium (or a temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that may be used to store desired information and may be accessed by a computer. In addition, it is well known to those of ordinary skill in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium. In the above description of this specification, the description of reference terms "one embodiment / example", "another embodiment / example" or "certain embodiments / examples" and the like means that the specific features, structures, materials or characteristics described in conjunction with the embodiments or examples are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0141] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0142] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0143] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0144] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0145] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above-mentioned units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms. Although the embodiments of the present application have been shown and described, it can be understood by those of ordinary skill in the art that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and purposes of the present application, and the scope of the present application is defined by the claims and their equivalents.

[0146] The above is a specific description of the preferred implementation of the present application, but the present application is not limited to the embodiments. Technical personnel familiar with the field can make various equivalent modifications or substitutions without violating the spirit of the present application. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present application.

Claims

1. A method for detecting illegal land use changes based on drones, characterized in that: include: Constructing multimodal data, and inputting the multimodal data into a change detection model, the multimodal data including an image pair consisting of a first land image and a second land image, a mask binary image of a changed region of the image pair, and annotated text describing a change of the image pair, wherein the first land image and the second land image are images taken of the same land at different times; Performing feature extraction on the image pair and the mask binary image to obtain image difference features of multiple channels, performing feature extraction on the annotated text to obtain text sequence features, and fusing the image difference features and the text sequence features to obtain a first fused feature; Aggregating the first fusion features of multiple channels to obtain multi-scale fusion features of multiple channels; Fusing the multi-scale fusion features of multiple channels and the first fusion features to obtain second fusion features of multiple channels; Fusing the second fused features of the multiple channels and the text sequence features to obtain third fused features of the multiple channels; Decoding the third fusion features of multiple channels to obtain a change area map; The illegal land use change detection is performed according to the change area map to obtain the detection result.

2. The method for detecting illegal land use changes based on drones according to claim 1 is characterized in that: The step of extracting features from the image pair and the mask binary image to obtain image difference features of multiple channels includes: Extracting features from the first land image of each channel by using a first image encoder to obtain first image features; Extracting features from the second land image of each channel by using a second image encoder to obtain second image features; Subtracting the first image feature from the second image feature of the same channel to obtain an image difference feature; The first image encoder and the second image encoder share parameters.

3. The method for detecting illegal land use changes based on drones according to claim 1 is characterized in that: The fusing the image difference feature and the text sequence feature to obtain a first fused feature includes: The image difference feature is embedded in the result of the first position encoding to perform attention calculation to obtain a first attention feature; The result of embedding the text sequence feature into the second position encoding is used as a query for attention calculation, the first attention feature is used as a key and value for attention calculation, and attention calculation is performed according to the query, key and value to obtain a second attention feature; Residual connection processing and normalization processing are performed according to the second attention feature to obtain the first fusion feature.

4. The method for detecting illegal land use changes based on drones according to claim 1 is characterized in that: The step of aggregating the first fusion features of the multiple channels to obtain multi-scale fusion features of the multiple channels includes: The first fusion features of multiple channels are averagely pooled and combined to obtain dimensional features; Extracting features from the dimensional features to obtain a plurality of first branch image features of different types; Performing dimensionality reduction processing and vector conversion processing on the result of adding the multiple first branch image features to obtain multiple first feature vectors; Multiplying the first feature vectors by the first branch image features to obtain the second branch image features; The result of adding multiple second branch image features is split and up-sampled to obtain multi-scale fusion features of multiple channels.

5. The method for detecting illegal land use changes based on drones according to claim 1 is characterized in that: The fusing of the second fused features of the multiple channels and the text sequence features to obtain the third fused features of the multiple channels includes: The second fusion feature of each channel is embedded into the result of the third position encoding to perform attention calculation to obtain a third attention feature; The result of embedding the text sequence feature into the fourth position encoding is used as a query for attention calculation, the third attention feature is used as a key and value for attention calculation, and attention calculation is performed according to the query, key and value to obtain a fourth attention feature; Residual connection processing and normalization processing are performed according to the fourth attention feature to obtain the third fusion feature.

6. The method for detecting illegal land use changes based on drones according to claim 1, characterized in that: The decoding of the third fusion features of the multiple channels to obtain a change area map includes: The third fusion feature of each channel is gradually upsampled to obtain an upsampled feature; The up-sampled features are convolved to obtain a change region map.

7. The method for detecting illegal land use changes based on drones according to claim 1, characterized in that: The method further comprises: Obtaining a loss value according to the detection result, and adjusting parameters of the change detection model according to the loss value; Wherein, the loss value is expressed as: L bce (Y,S(t1,t2,T))=Y×log(S(t1,t2,T))+(1-Y)×log(1-S(t1,t2,T)); L bce (Y,S(t1,t2,T)) is the loss value, Y is the mask binary image, S(t1,t2,T) is the detection result, t1 is the first land image, t2 is the second land image, and T is the labeled text.

8. A device for detecting changes in illegal land use, characterized in that: include: An input module is used to construct multimodal data and input the multimodal data into a change detection model, wherein the multimodal data includes an image pair consisting of a first land image and a second land image, a mask binary image of a changed region of the image pair, and an annotation text describing the change of the image pair, wherein the first land image and the second land image are images taken of the same land at different times; A multimodal encoding module, used for performing feature extraction based on the image pair and the mask binary image to obtain image difference features of multiple channels, and performing feature extraction based on the annotated text to obtain text sequence features; A multimodal feature fusion module, used for fusing the image difference feature and the text sequence feature to obtain a first fusion feature; An adaptive aggregation module, used for aggregating the first fusion features of multiple channels to obtain multi-scale fusion features of multiple channels; A first fusion module, used for fusing the multi-scale fusion features of multiple channels and the first fusion features to obtain second fusion features of multiple channels; A second fusion module, used for fusing the second fusion features of multiple channels and the text sequence features to obtain third fusion features of multiple channels; A decoding module, used for decoding the third fusion features of multiple channels to obtain a change area map; The detection module is used to detect changes in illegal land use according to the change area map to obtain detection results.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for detecting changes in illegal land use based on a drone as described in any one of claims 1 to 7 is implemented.

10. A computer storage medium, characterized in that: Computer executable instructions are stored, and the computer executable instructions are used to execute the illegal land use change detection method based on a drone as described in any one of claims 1 to 7.

Images