Unmanned aerial vehicle-based illegal land use change detection method, device, equipment and medium

Abstract

The application provides a UAV-based illegal land use change detection method and device, equipment and medium; image difference features of multiple channels are obtained according to an image pair and a mask binary image, text sequence features are obtained according to a labeled text, first fusion features are obtained by fusing the image difference features and the text sequence features; multi-scale fusion features are obtained by aggregating the first fusion features; second fusion features are obtained by fusing the multi-scale fusion features and the first fusion features; third fusion features are obtained by fusing the second fusion features and the text sequence features; a change area map is obtained by decoding the third fusion features; a detection result is obtained by performing illegal land use change detection according to the change area map; multi-modal data including an image pair, a mask binary image of a change area and a labeled text are used to fuse multiple types and multiple channels of data multiple times, so that more rich feature information is provided for a model, and the generalization ability and prediction accuracy of the model are improved.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the field of land surveying, and particularly to a method and device for detecting changes in illegal land use based on a UAV, equipment and medium. BACKGROUND

[0002] Currently, the method of inspecting illegal land use mainly relies on manual visits and investigations. The staff visits in the city and then identifies illegal land use by naked eyes, which requires a lot of manpower and is very inefficient. In addition, manual inspection can be carried out based on a UAV. A camera is mounted on the UAV, and the operator identifies illegal land use by real-time screen video. In this process, the staff also needs to rely on naked eye identification, which is inefficient and easy to miss. In addition, the original land and building planning data need to be provided by the planning department for comparison, and the planning data is difficult to obtain conveniently. SUMMARY

[0003] The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.

[0004] The purpose of the present application is to at least partially solve one of the technical problems existing in the related art. The embodiments of the present application provide a method and device for detecting changes in illegal land use based on a UAV, which can improve the generalization ability and prediction accuracy of the model and improve the detection efficiency.

[0005] In an embodiment of the first aspect of the present application, a method for detecting changes in illegal land use based on a UAV includes:

[0006] Constructing multi-modal data, inputting the multi-modal data into a change detection model, the multi-modal data including an image pair composed of a first land image and a second land image, a mask binary image of a change area of the image pair, and a labeled text describing the changes of the image pair, the first land image and the second land image being images taken of the same land at different times;

[0007] According to the image pair and the mask binary image, feature extraction is performed to obtain image difference features of multiple channels. According to the labeled text, feature extraction is performed to obtain text sequence features. The image difference features and the text sequence features are fused to obtain first fusion features;

[0008] The first fusion features of multiple channels are aggregated to obtain multi-scale fusion features of multiple channels;

[0009] The multi-scale fusion features of multiple channels and the first fusion features are fused to obtain second fusion features of multiple channels;

[0010] fusing the second fusion features of the plurality of channels and the text sequence features to obtain third fusion features of the plurality of channels;

[0011] decoding the third fusion features of the plurality of channels to obtain a change region map;

[0012] detecting illegal land use change according to the change region map to obtain a detection result.

[0013] According to certain embodiments of the first aspect of the application, the feature extraction according to the image pair and the mask binary image to obtain image difference features of a plurality of channels comprises:

[0014] feature extraction of the first land image of each channel by a first image encoder to obtain first image features;

[0015] feature extraction of the second land image of each channel by a second image encoder to obtain second image features;

[0016] subtracting the first image features and the second image features of the same channel to obtain image difference features;

[0017] wherein the first image encoder and the second image encoder share parameters.

[0018] According to certain embodiments of the first aspect of the application, the fusion of the image difference features and the text sequence features to obtain first fusion features comprises:

[0019] attention calculation of the result of embedding the image difference features into first position encoding to obtain first attention features;

[0020] embedding the result of embedding the text sequence features into second position encoding as query for attention calculation, embedding the first attention features as key and value for attention calculation, and performing attention calculation according to the query, the key and the value to obtain second attention features;

[0021] residual connection processing and normalization processing according to the second attention features to obtain first fusion features.

[0022] According to certain embodiments of the first aspect of the application, the aggregation of the first fusion features of the plurality of channels to obtain multi-scale fusion features of the plurality of channels comprises:

[0023] average pooling and splicing of the first fusion features of the plurality of channels to obtain dimensional features;

[0024] feature extraction of the dimensional features to obtain a plurality of first branch image features of different types;

[0025] The results of adding the plurality of first branch image features are subjected to dimension reduction processing and vector conversion processing to obtain a plurality of first feature vectors;

[0026] The plurality of first feature vectors are multiplied with the plurality of first branch image features to obtain a plurality of second branch image features.

[0027] The results of adding the plurality of second branch image features are subjected to splitting processing and up-sampling processing to obtain a plurality of channel multi-scale fusion features.

[0028] According to certain embodiments of the first aspect of the present application, the fusion of the plurality of channel second fusion features and the text sequence feature obtains a plurality of channel third fusion features, comprising:

[0029] The results of embedding each channel second fusion feature into third position encoding are subjected to attention calculation to obtain third attention features;

[0030] The results of embedding the text sequence feature into fourth position encoding are used as the query of attention calculation, the third attention features are used as the key and value of attention calculation, and attention calculation is performed according to the query, key and value to obtain fourth attention features;

[0031] Residual connection processing and normalization processing are performed according to the fourth attention features to obtain third fusion features.

[0032] According to certain embodiments of the first aspect of the present application, the decoding of the plurality of channel third fusion features to obtain a change area map comprises:

[0033] The third fusion features of each channel are subjected to step-up sampling processing to obtain up-sampling features;

[0034] The up-sampling features are subjected to convolution processing to obtain a change area map.

[0035] According to certain embodiments of the first aspect of the present application, the method further comprises:

[0036] According to the detection result, a loss value is obtained, and the parameters of the change detection model are adjusted according to the loss value;

[0037] Wherein, the loss value is represented as: L bce (Y, S(t1, t2, T)) = Y x log(S(t1, t2, T)) + (1-Y) x log(1-S(t1, t2, T)); L bce (Y, S(t1, t2, T)) is a loss value, Y is a mask binary map, S(t1, t2, T) is a detection result, t1 is a first land image, t2 is a second land image, and T is a labeled text.

[0038] In an embodiment of the second aspect of the present application, a device for illegal land use change detection comprises:

[0039] An input module is configured to construct multi-modal data, input the multi-modal data into a change detection model, and the multi-modal data comprises an image pair composed of a first land image and a second land image, a mask binary image of a change region of the image pair, and a labeled text describing changes of the image pair, the first land image and the second land image are images of the same land taken at different times;

[0040] A multi-modal encoding module is configured to extract image difference features of multiple channels according to the image pair and the mask binary image, and extract text sequence features according to the labeled text;

[0041] A multi-modal feature fusion module is configured to fuse the image difference features and the text sequence features to obtain first fusion features;

[0042] An adaptive aggregation module is configured to aggregate the first fusion features of multiple channels to obtain multi-scale fusion features of multiple channels;

[0043] A first fusion module is configured to fuse the multi-scale fusion features of multiple channels and the first fusion features to obtain second fusion features of multiple channels;

[0044] A second fusion module is configured to fuse the second fusion features of multiple channels and the text sequence features to obtain third fusion features of multiple channels;

[0045] A decoding module is configured to decode the third fusion features of multiple channels to obtain a change region map;

[0046] A detection module is configured to perform illegal land use change detection according to the change region map to obtain a detection result.

[0047] In an embodiment of the third aspect of the present application, an electronic device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor executes the computer program to implement the method for illegal land use change detection based on a UAV as described in the embodiment of the first aspect of the present application.

[0048] In an embodiment of the fourth aspect of the present application, a computer storage medium stores computer executable instructions for executing the method for illegal land use change detection based on a UAV as described in the embodiment of the first aspect of the present application.

[0049] The scheme has at least the following beneficial effects: obtaining image difference features of multiple channels from the image pair and the mask binary graph, obtaining text sequence features from the labeled text, fusing the image difference features and the text sequence features to obtain first fusion features, aggregating the image difference features to obtain multi-scale fusion features, fusing the multi-scale fusion features and the first fusion features to obtain second fusion features, fusing the second fusion features and the text sequence features to obtain third fusion features, decoding the third fusion features to obtain a change area graph, performing illegal land use change detection according to the change area graph to obtain a detection result, using multi-modal data including the image pair, the mask binary graph of the change area and the labeled text, fusing multiple types and multiple channels of data multiple times to provide richer feature information for the model, which helps to improve the generalization ability and prediction accuracy of the model, and improves the automation and intelligence of land change detection, improves the illegal land use detection efficiency, and saves the labor cost. BRIEF DESCRIPTION OF DRAWINGS

[0050] The accompanying drawings are used to provide a further understanding of the technical scheme of the present application, and constitute a part of the specification, and are used together with embodiments of the present application to explain the technical scheme of the present application, and do not constitute a limitation on the technical scheme of the present application.

[0051] Figure 1 is a step diagram of the illegal land use change detection method based on a UAV provided by an embodiment of the present application;

[0052] Figure 2 is a sub-step diagram of obtaining image difference features of multiple channels by a multi-modal encoder module according to an image pair and a mask binary graph;

[0053] Figure 3 is a sub-step diagram of step S300 provided by an embodiment of the present application;

[0054] Figure 4 is a sub-step diagram of step S400 provided by an embodiment of the present application;

[0055] Figure 5 is a sub-step diagram of step S700 provided by an embodiment of the present application;

[0056] Figure 6 is a structure diagram of an illegal land use change detection device provided by an embodiment of the present application;

[0057] Figure 7 is a schematic diagram of a change detection model provided by an embodiment of the present application;

[0058] Figure 8 is a schematic diagram of a multi-modal encoder module provided by an embodiment of the present application;

[0059] Figure 9FIG. 1 is a schematic diagram of a multi-modal feature fusion module provided by an embodiment of the present application;

[0060] Figure 10 FIG. 2 is a schematic diagram of an adaptive aggregation module provided by an embodiment of the present application;

[0061] Figure 11 FIG. 3 is a schematic diagram of a Mamba feature extraction network provided by an embodiment of the present application;

[0062] Figure 12 FIG. 4 is a schematic diagram of a decoding module provided by an embodiment of the present application. DETAILED DESCRIPTION

[0063] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0064] It should be noted that although the functional modules are divided in the device schematic diagram, and the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a manner different from the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims or above-described drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence.

[0065] The embodiments of the present application are further described below in combination with the drawings.

[0066] An illegal land use change detection method based on a UAV is provided by the embodiments of the present application.

[0067] Referring to Figure 1 and Figure 7 , the illegal land use change detection method based on a UAV includes the following steps:

[0068] Step S100, constructing multi-modal data, inputting the multi-modal data into a change detection model; the multi-modal data includes an image pair composed of a first land image and a second land image, a mask binary image of a change region of the image pair, and a labeled text describing changes of the image pair, the first land image and the second land image being images obtained by photographing the same land at different times.

[0069] Step S200, performing feature extraction according to the image pair and the mask binary image to obtain image difference features of multiple channels, and performing feature extraction according to the labeled text to obtain text sequence features.

[0070] Step S300, fusing the image difference features and the text sequence features to obtain first fusion features.

[0071] Step S400, the first fusion features of multiple channels are aggregated to obtain multi-scale fusion features of multiple channels.

[0072] Step S500, the multi-scale fusion features of multiple channels and the first fusion features are fused to obtain second fusion features of multiple channels.

[0073] Step S600, the second fusion features of multiple channels and the text sequence features are fused to obtain third fusion features of multiple channels.

[0074] Step S700, the third fusion features of multiple channels are decoded to obtain a change area map.

[0075] Step S800, according to the change area map, illegal land use change detection is performed to obtain a detection result.

[0076] The multi-modal data containing image pairs, mask binary maps of change areas and annotated texts are used to perform multiple fusions of multiple types and multiple channels of data, to provide richer feature information for the model, which helps to improve the generalization ability and prediction accuracy of the model.

[0077] For step S100, a remote sensing device is carried by a drone, and a remote sensing image is collected by the remote sensing device once in each node process of the drone. After a period of time, the remote sensing image is collected again. The collected remote sensing images are manually annotated, and the images obtained by repeatedly shooting the same land at different times are combined to form a data set. A first land image and a second land image are selected from the data set, so that the first land image and the second land image form an image pair. The image pair formed by the first land image and the second land image is a double-time-phase image.

[0078] For each image pair, the first land image and the second land image are spatially aligned. The difference between the aligned first land image and the second land image is calculated, for example, the difference map can be obtained by subtracting the pixel values of the first land image and the second land image, and the mask is set on the change area of the difference map. The difference map containing the mask is converted into a binary map.

[0079] The mask binary image of the change area is annotated by a labelme or other annotation software to obtain annotated text. For the annotated text describing the difference between images, since the dual-time remote sensing images in the data set have a long time span, and irrelevant changes often occur in remote sensing images, a pair of remote sensing images cannot be completely the same. In the text annotation process, only the image pair with irrelevant changes is the unchanged image pair, and the following principles are followed: describing important changes (for example, feature changes), ignoring irrelevant changes and unimportant interference factors (for example, light changes). Describe the object and type of change, such as appearance and disappearance. The following five sentences are used for unchanged annotation: "The scenes of the two pictures seem to be the same", "The scene of this picture is the same as that of the last picture", "There is no difference", "No change has occurred", "There is almost no change".

[0080] A set of multi-modal data is constructed from dual-time images, mask binary images, and annotated texts.

[0081] The set of multi-modal data is preprocessed. Remote sensing change detection requires high registration of multi-temporal images. However, remote sensing images are inevitably affected by different sensors, poses, distances, qualities, and weather. Therefore, before change detection, corresponding accurate geometric correction, atmospheric correction, mutual registration, and other image preprocessing of remote sensing images at different times are essential steps. This improves the accuracy of change detection. The purpose of image registration is to find the geometric transformation relationship between two images, so that the corresponding feature points or feature regions in space can be overlapped. By calculating and optimizing, the best transformation parameters are determined to minimize the difference between images and improve the accuracy and efficiency of image analysis, providing a basis for subsequent image processing and analysis tasks.

[0082] Reference Figure 2 For step S200, the multi-modal encoder module 20 extracts features from the image pair and the mask binary image to obtain image difference features in multiple channels, including the following steps:

[0083] Step S210, the first image encoder extracts features from each channel of the first land image to obtain first image features;

[0084] Step S220, the second image encoder extracts features from each channel of the second land image to obtain second image features;

[0085] Step S230, subtract the first image features and the second image features in the same channel to obtain image difference features.

[0086] Wherein, the first image encoder and the second image encoder share parameters.

[0087] The text sequence features are extracted according to the annotated text.

[0088] Referring to Figure 8 The multi-modal encoder module adopts a parameter-shared twin image encoder and a text encoder as a basic dual-time feature extractor for the multi-modal remote sensing change detection task. The twin image encoder includes two image encoders, i.e., a first image encoder and a second image encoder.

[0089] The dual-time images are respectively input into the twin image encoders to generate image encoding results, and the image difference features are obtained by subtracting the image encoding results. The text information is input into the text encoder to generate a sequence feature vector, i.e., a text sequence feature. Starting from the feature extraction stage, the dual-time image encoders obtain two groups of feature pyramids, which represent different levels of dual-time feature mappings. The difference features obtained by subtracting the two groups of features are used as part of the subsequent feature fusion.

[0090] Referring to Figure 3 For step S300, the image difference features and the text sequence features are fused by the multi-modal feature fusion module 30 to obtain first fusion features, including the following steps:

[0091] In step S310, the image difference features are embedded into the first positional encoding result for attention calculation to obtain first attention features.

[0092] In step S320, the text sequence features are embedded into the second positional encoding result as a query for attention calculation, the first attention features are used as keys and values for attention calculation, and attention calculation is performed according to the query, the keys and the values to obtain second attention features.

[0093] In step S330, residual connection processing and normalization processing are performed according to the second attention features to obtain the first fusion features.

[0094] The multi-modal feature fusion module includes four parts of input, encoding, decoding and output.

[0095] Exemplarily, the image difference features include image difference features VF1, image difference features VF2, image difference features VF3 and image difference features VF4. The text sequence features are LF.

[0096] The image difference features VF1, VF2, VF3 and VF4 of different channels and the text sequence features LF are respectively input into embedding positions for encoding. In the encoder and the decoder, both are composed of two sub-layer connection structures. The first sub-layer is a multi-head self-attention sub-layer, and the second sub-layer is a feed-forward full connection sub-layer. Each sub-layer is connected with a normalization layer and a residual connection. Finally, the first fusion features are output.

[0097] With reference to Figure 9 Specifically, the image difference feature is input into an embedding layer to embed a first position encoding; then the image difference feature with the embedded first position encoding is subjected to multi-head attention calculation to obtain a first attention feature. The first attention feature and the image difference feature with the embedded first position encoding are subjected to residual connection and normalization. The normalized result is input into a feedforward network, and the output of the feedforward network is subjected to residual connection and normalization with the normalized result to obtain a feature output corresponding to the image difference feature. The text sequence feature is input into an embedding layer to embed a second position encoding; then the text sequence feature with the embedded second position encoding is taken as a query for attention calculation, and the feature output corresponding to the image difference feature obtained from the first attention feature is taken as a key and a value for attention calculation, and attention calculation is performed according to the query, the key and the value to obtain a second attention feature; the second attention feature and the text sequence feature with the embedded second position encoding are subjected to residual connection and normalization. The normalized result is input into a feedforward network, and the output of the feedforward network is subjected to residual connection and normalization with the normalized result to obtain a first fusion feature.

[0098] With reference to Figure 4 For step S400, the first fusion features of the multiple channels are aggregated by the adaptive aggregation module 40 to obtain the multi-scale fusion features of the multiple channels, including the following steps:

[0099] Step S410, the first fusion features of the multiple channels are subjected to average pooling and splicing to obtain a dimensional feature;

[0100] Step S420, the dimensional feature is subjected to feature extraction to obtain multiple first branch image features of different types;

[0101] Step S430, the result of adding the multiple first branch image features is subjected to dimension reduction processing and vector conversion processing to obtain multiple first feature vectors;

[0102] Step S440, the multiple first feature vectors are multiplied with the multiple first branch image features corresponding to the multiple first feature vectors to obtain multiple second branch image features;

[0103] Step S450, the result of adding the multiple second branch image features is subjected to splitting processing and up-sampling processing to obtain the multi-scale fusion features of the multiple channels.

[0104] With reference to Figure 10The adaptive aggregation module 40 mainly consists of three parts, multi-branch feature extraction, multi-branch feature fusion and dynamic weighted fusion. The first fusion feature is obtained by average pooling and splicing to obtain the same dimension feature. The multi-branch feature extraction is performed by the Transformer, convolution and Mamba feature extraction network to obtain multiple first branch image features of different types. Then the first branch image features of multiple branches are added, and the first feature vector is obtained by using the full connection layer for dimension reduction processing and using the Softmax vector conversion processing. The result of adding multiple first branch image features is processed by dimension reduction and vector conversion to obtain the first feature vector. Finally, the first feature vector is added, split and up-sampled into a feature pyramid to obtain multiple channel multi-scale fusion features.

[0105] The Mamba feature extraction network is established on the basis of a state space model (SSM), and the concept of selective state space is introduced to enable the model to more efficiently process long sequences. Unlike the standard SSM, the Mamba feature extraction network allows the state transition part to depend on the current input, thereby increasing the flexibility of the model while maintaining computational efficiency. In the traditional SSM, the state transition is usually fixed and input-invariant, i.e., a set of fixed parameters (matrices A and B) is applied to the entire sequence. The Mamba feature extraction network, however, adjusts these parameters adaptively at each time step according to the current input through input-dependent transitions.

[0106] Referring to Figure 11 In the Mamba feature extraction network, the input feature is calculated to obtain a first sub-feature through linear projection, convolution, SiLU function calculation and state space model; the input feature is calculated to obtain a second sub-feature through linear projection and SiLU function calculation; and the output feature of the Mamba feature extraction network is obtained by linear projection of the nonlinear calculation result of the first sub-feature and the second sub-feature.

[0107] For step S500, multiple channel second fusion features are obtained by fusing the multiple channel multi-scale fusion features and the first fusion features.

[0108] For step S600, multiple channel third fusion features are obtained by fusing the multiple channel second fusion features and the text sequence features, including the following steps:

[0109] The second fusion feature of each channel is embedded into the result of the third position coding, and attention calculation is performed to obtain a third attention feature; the result of embedding the text sequence feature into the fourth position coding is taken as a query for attention calculation, and the third attention feature is taken as a key and a value for attention calculation, and attention calculation is performed according to the query, the key and the value to obtain a fourth attention feature; the fourth attention feature is subjected to residual connection processing and normalization processing to obtain the third fusion feature.

[0110] Step S600 is similar to step S300, and will not be described here.

[0111] Referring to Figure 5 and Figure 12 For step S700, the third fusion features of the plurality of channels are decoded by the decoding module 70 to obtain a change area map, including the following steps:

[0112] Step S710, the third fusion features of each channel are subjected to step-by-step up-sampling processing to obtain up-sampling features;

[0113] Step S720, the up-sampling features are subjected to convolution processing to obtain the change area map.

[0114] For step S800, the change area map is used to detect illegal land use changes to obtain a detection result.

[0115] It should be noted that the change detection network needs to be trained before the change detection model goes online. The multi-modal data set is divided into a training set (70%), a validation set (15%) and a test set (15%). The input image is represented as t1, t2, and the dimension is HxWx3, which corresponds to the height, width and RGB three color channels of the image respectively.

[0116] The training set is input into the change detection model, and the change detection model outputs a detection result. The loss value is calculated according to the detection result, and the parameters of the change detection model are adjusted according to the loss value.

[0117] The loss function is a cross-entropy loss function. The loss value is represented as: L bce (Y,S(t1,t2,T))=Y×log(S(t1,t2,T))+(1-Y)×log(1-S(t1,t2,T));L bce (Y,S(t1,t2,T)) is the loss value, Y is the mask binary map, S(t1,t2,T) is the detection result, t1 is the first land image, t2 is the second land image, and T is the labeled text.

[0118] The cross-entropy loss function calculates the relative entropy between the predicted probability distribution and the true label distribution to quantify the degree of difference between the two. During model training, the optimization algorithm continuously adjusts the model parameters to minimize the value of the cross-entropy loss function, thereby improving the model's prediction accuracy and generalization ability.

[0119] The trained change detection model is tested and verified using the validation set and test set.

[0120] The change detection model is launched online. The images captured by the drone are first pre-processed and then fed into the change detection network. The detection results are then transmitted to the staff's relevant application via the internet. Finally, the staff can further determine whether the land use is illegal.

[0121] An embodiment of the present application provides a device for detecting changes in illegal land use.

[0122] Reference Figure 6 The illegal land use change detection device includes: an input module 10, a multimodal encoding module 20, a multimodal feature fusion module 30, an adaptive aggregation module 40, a first fusion module 50, a second fusion module 60, a decoding module 70 and a detection module 80.

[0123] Input module 10 is used to construct multimodal data and input the multimodal data into the change detection model. The multimodal data includes an image pair consisting of a first land image and a second land image, a masked binary image of the changed area of ​​the image pair, and annotated text describing the changes in the image pair. The first and second land images are images of the same land captured at different times.

[0124] The multimodal encoding module 20 is used to perform feature extraction based on the image pairs and the mask binary images to obtain image difference features of multiple channels, and to perform feature extraction based on the annotated text to obtain text sequence features.

[0125] The multimodal feature fusion module 30 is used to fuse the image difference feature and the text sequence feature to obtain a first fused feature.

[0126] The adaptive aggregation module 40 is used to aggregate the first fusion features of multiple channels to obtain multi-scale fusion features of multiple channels.

[0127] The first fusion module 50 is used to fuse the multi-scale fusion features of multiple channels and the first fusion features to obtain second fusion features of multiple channels.

[0128] The second fusion module 60 is used to fuse the second fusion features of multiple channels and the text sequence features to obtain the third fusion features of multiple channels.

[0129] The decoding module 70 is configured to decode the third fused features of the multiple channels to obtain a change region map.

[0130] The detection module 80 is configured to perform illegal land use change detection according to the change region map to obtain a detection result.

[0131] The illegal land use change detection device adopts the illegal land use change detection method as described above. Each module of the illegal land use change detection device corresponds to each step of the illegal land use change detection method. The illegal land use change detection device and the illegal land use change detection method adopt the same technical means, solve the same technical problems, and have the same technical effects.

[0132] Embodiments of the present application provide an electronic device. The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor executes the computer program to implement the illegal land use change detection method based on a UAV as described above.

[0133] The electronic device can be any intelligent terminal including a computer.

[0134] Generally, for the hardware structure of the electronic device, the processor can be implemented in the form of a general-purpose CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, etc., for executing related programs to implement the technical solutions provided by the embodiments of the present application.

[0135] The memory can be implemented in the form of a ROM (ReadOnly Memory), a static storage device, a dynamic storage device, or a RAM (Random Access Memory), etc. The memory can store an operating system and other application programs, and when the technical solutions provided by the embodiments of the present application are implemented by software or firmware, the related program codes are saved in the memory and executed by the processor to implement the methods of the embodiments of the present application.

[0136] The input / output interface is configured to realize information input and output.

[0137] The communication interface is configured to realize the communication interaction between the device and other devices, which can be realized by wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0138] The bus transmits information between the various components (for example, the processor, the memory, the input / output interface, and the communication interface) of the device. The processor, the memory, the input / output interface, and the communication interface are communicatively connected to each other within the device through the bus.

[0139] Embodiments of the present application provide a computer storage medium. The computer storage medium stores computer executable instructions for performing the method for detecting illegal land use change based on a UAV as described above.

[0140] Those skilled in the art can understand that all or some of the steps in the method disclosed above, the system can be implemented as software, firmware, hardware and their appropriate combinations. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or as hardware, or as an integrated circuit, such as an application specific integrated circuit. Such software can be distributed on a computer readable medium, which can include computer storage media (or non-transitory media) and communication media (or transitory media). As known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, as known to those skilled in the art, communication media generally includes computer readable instructions, data structures, program modules or other data in a modulated data signal carrier or other transport mechanism, and can include any information delivery medium. In the above description of the specification, the description of the terms "one embodiment", "another embodiment" or "some embodiments" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiments or examples are included in at least one embodiment or example of the present application. In the specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in one or more embodiments or examples.

[0141] Those skilled in the art can understand that all or some of the steps in the method disclosed above, the system, the device, the functional modules / units can be implemented as software, firmware, hardware and their appropriate combinations.

[0142] The units described as separate components above can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., can be located in one place, or can be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0143] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present alone, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0144] If the integrated unit is realized in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes multiple instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method of each embodiment of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program storage media.

[0145] In several embodiments provided in the present application, it should be understood that the disclosed apparatus and method can be implemented in other ways. For example, the apparatus embodiment described above is only schematic, for example, the division of the above units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms. Although the embodiments of the present application have been shown and described, those skilled in the art can understand that various changes, modifications, replacements and variations can be made to the embodiments without departing from the principles and spirit of the present application, and the scope of the present application is defined by the claims and their equivalents.

[0146] The above describes the preferred embodiments of the present application, but the present application is not limited to the embodiments, and those skilled in the art can make various equivalent modifications or replacements without departing from the spirit of the present application, and these equivalent modifications or replacements are all included in the scope defined by the claims of the present application.

Claims

1. A method for detecting illegal land use changes based on drones, characterized in that: include: Constructing multimodal data, and inputting the multimodal data into a change detection model, the multimodal data including an image pair consisting of a first land image and a second land image, a masked binary image of a changed region of the image pair, and annotated text describing the change in the image pair, wherein the first land image and the second land image are images of the same land captured at different times; Performing feature extraction on the image pair and the mask binary image to obtain image difference features of multiple channels, performing feature extraction on the annotated text to obtain text sequence features, and fusing the image difference features and the text sequence features to obtain a first fused feature; Aggregating the first fusion features of multiple channels to obtain multi-scale fusion features of multiple channels; Fusing the multi-scale fusion features of multiple channels and the first fusion features to obtain second fusion features of multiple channels; Fusing the second fused features of the multiple channels and the text sequence features to obtain third fused features of the multiple channels; Decoding the third fusion features of multiple channels to obtain a change area map; Illegal land use change detection is performed based on the change area map to obtain a detection result.

2. The method for detecting illegal land use changes based on drones according to claim 1, characterized in that: The extracting features based on the image pair and the mask binary image to obtain image difference features of multiple channels includes: Performing feature extraction on the first land image of each channel by using a first image encoder to obtain first image features; Performing feature extraction on the second land image of each channel by a second image encoder to obtain second image features; subtracting the first image feature from the second image feature of the same channel to obtain an image difference feature; The first image encoder and the second image encoder share parameters.

3. The method for detecting illegal land use changes based on drones according to claim 1, characterized in that: The fusing the image difference feature and the text sequence feature to obtain a first fused feature includes: Embedding the image difference feature into the result of the first position encoding to perform attention calculation to obtain a first attention feature; Embedding the text sequence feature into the result of the second position encoding as a query for attention calculation, using the first attention feature as the key and value for attention calculation, performing attention calculation based on the query, key and value to obtain a second attention feature; Residual connection processing and normalization processing are performed on the second attention feature to obtain the first fusion feature.

4. The method for detecting illegal land use changes based on drones according to claim 1, characterized in that: The step of aggregating the first fusion features of the multiple channels to obtain the multi-scale fusion features of the multiple channels includes: The first fusion features of multiple channels are averagely pooled and combined to obtain dimensional features; Extracting features from the dimensional features to obtain a plurality of first branch image features of different types; Performing dimensionality reduction processing and vector conversion processing on a result of adding the multiple first branch image features to obtain multiple first feature vectors; multiplying the first feature vectors by the first branch image features to obtain the second branch image features; The result of adding multiple second branch image features is split and up-sampled to obtain multi-scale fusion features of multiple channels.

5. The method for detecting illegal land use changes based on drones according to claim 1, characterized in that: The fusing of the second fused features of the multiple channels and the text sequence features to obtain the third fused features of the multiple channels includes: The second fusion feature of each channel is embedded into the result of the third position encoding to perform attention calculation to obtain the third attention feature; Embedding the text sequence feature into the fourth position encoding result as a query for attention calculation, using the third attention feature as a key and value for attention calculation, performing attention calculation based on the query, key and value to obtain a fourth attention feature; Residual connection processing and normalization processing are performed according to the fourth attention feature to obtain a third fusion feature.

6. The method for detecting illegal land use changes based on drones according to claim 1, characterized in that: Decoding the third fusion features of the multiple channels to obtain a change region map includes: The third fusion feature of each channel is gradually upsampled to obtain the upsampled feature; The up-sampled features are convolved to obtain a change region map.

7. The method for detecting illegal land use changes based on drones according to claim 1, characterized in that: The method further comprises: Obtaining a loss value according to the detection result, and adjusting parameters of the change detection model according to the loss value; Wherein, the loss value is expressed as: L bce (Y,S(t1,t2,T))=Y×log(S(t1,t2,T))+(1-Y)×log(1-S(t1,t2,T)); L bce (Y, S (t1, t2, T)) is the loss value, Y is the mask binary image, S (t1, t2, T) is the detection result, t1 is the first land image, t2 is the second land image, and T is the labeled text.

8. A device for detecting changes in illegal land use, characterized in that: include: An input module is configured to construct multimodal data and input the multimodal data into a change detection model. The multimodal data includes an image pair consisting of a first land image and a second land image, a masked binary image of a changed region of the image pair, and annotated text describing the change in the image pair. The first land image and the second land image are images of the same land captured at different times. a multimodal encoding module, configured to extract features from the image pair and the mask binary image to obtain image difference features of multiple channels, and extract features from the annotated text to obtain text sequence features; a multimodal feature fusion module, configured to fuse the image difference feature and the text sequence feature to obtain a first fused feature; An adaptive aggregation module, configured to aggregate the first fusion features of multiple channels to obtain multi-scale fusion features of multiple channels; A first fusion module is used to fuse the multi-scale fusion features of multiple channels and the first fusion features to obtain second fusion features of multiple channels; A second fusion module is used to fuse the second fusion features of multiple channels and the text sequence features to obtain third fusion features of multiple channels; A decoding module, configured to decode the third fusion features of multiple channels to obtain a change region map; The detection module is used to detect changes in illegal land use based on the change area map and obtain detection results.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for detecting changes in illegal land use based on a drone as described in any one of claims 1 to 7 is implemented.

10. A computer storage medium, characterized in that Computer-executable instructions are stored, and the computer-executable instructions are used to execute the illegal land use change detection method based on a drone as described in any one of claims 1 to 7.

Images