Remote sensing image land overrun identification method and system based on deep learning

Abstract

The invention discloses a remote sensing image land use overrun identification method and system based on deep learning, and the method comprises the steps: obtaining a land use remote sensing image, and carrying out the preprocessing of the land use remote sensing image, so as to obtain a to-be-identified image pair; performing feature extraction and matching on a picture pair needing to be recognized by using a pre-trained Siamese-Resnet50 model and a gray-level co-occurrence matrix so as to obtain depth similarity and texture similarity; based on the depth similarity and the texture similarity of the picture pair, performing dynamic weight feature fusion calculation to obtain comprehensive similarity; and comparing the comprehensive similarity with a set judgment threshold, judging whether the land use range of the remote sensing image exceeds the limit or not, and outputting a result. According to the method, the depth similarity, the gray level co-occurrence matrix (GLCM) texture features and the dynamic weight adjustment strategy are combined, high-precision detection and anomaly recognition of the land use range are achieved, the problem that ultralimit recognition is difficult is solved, and the recognition efficiency is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision and remote sensing image processing, and particularly relates to a remote sensing image land use over-limit identification method and system based on deep learning. BACKGROUND

[0002] With the acceleration of urbanization, land use changes frequently, and land resource management faces severe challenges. In the construction process, various types of land use exist non-standard use such as occupying permanent farmland and occupying construction land, that is, "land use over-limit". Traditional land monitoring relies on manual patrol or ground measurement, which is low in efficiency and easily affected by subjective factors.

[0003] Current remote sensing image land monitoring methods mainly rely on traditional computer vision algorithms such as threshold segmentation, histogram-based texture analysis or single deep learning models such as convolutional neural networks, but still have defects. For example, the amount of remote sensing image data is huge, and traditional image processing methods are difficult to efficiently extract effective information; the existing technology lacks dynamic monitoring capability and cannot adapt to large-scale monitoring demand and has weak generalization ability to complex scenes. Therefore, how to effectively extract effective information and dynamically monitor according to the complexity of different scenes has become an important technical problem to be solved in the field of land resource management. SUMMARY

[0004] The present application aims at the deficiencies of the prior art, and provides a remote sensing image land use over-limit identification method and system based on deep learning. The present application realizes high-precision detection and abnormal identification of land use range by combining deep similarity, gray level co-occurrence matrix (GLCM) texture features and dynamic weight adjustment strategy.

[0005] The purpose of the present application is achieved by the following technical solutions: the first aspect of the embodiment of the present application provides a remote sensing image land use over-limit identification method based on deep learning, comprising the following steps:

[0006] S1, acquiring land use remote sensing images and pre-processing the land use remote sensing images to obtain image pairs to be identified;

[0007] S2, using a pre-trained Siamese-Resnet50 model and a gray level co-occurrence matrix to extract and match features of the image pairs to be identified to obtain deep similarity and texture similarity;

[0008] S3, based on the deep similarity and texture similarity of the image pairs, performing dynamic weight feature fusion calculation to obtain comprehensive similarity;

[0009] S4, comparing the comprehensive similarity with a set decision threshold to determine whether the land use range of the remote sensing image is over-limit and outputting the result.

[0010] Further, the remote sensing image of the land use is preprocessed, specifically including the following sub-steps:

[0011] S1.1, registration and orthorectification are performed according to the vector boundary provided by the reference datum, and the HSV color space is used to accurately extract the land use range vector contour on the current phase remote sensing image after registration and orthorectification;

[0012] S1.2, based on the land use range vector contour, an inner and outer expansion region is generated through morphological operation, the inner and outer expansion contours are approximated by polygons to obtain the vertex information of the inner and outer expansion regions, the vertex coordinates are extracted for clipping to obtain image blocks A and B;

[0013] S1.3, the image blocks A and B are subjected to size normalization, data type conversion and pixel value standardization processing to meet the input requirements of the Siamese-Resnet50 model, and the image blocks A and B after standardization processing are taken as the picture pair to be identified.

[0014] Further, the Siamese-Resnet50 model is constructed based on the Siamese network and deep learning, and the Siamese-Resnet50 model includes an input layer, a backbone network and an output layer, wherein the input layer is used to receive the picture pair to be identified; the backbone network adopts two parallel and weight parameter shared Resnet50 networks as encoders to respectively extract features of the picture pair and obtain corresponding high-dimensional feature vectors; the output layer is used to calculate and output the deep semantic similarity between the high-dimensional feature vectors of the picture pair, and the output layer includes a normalization layer, an inner product layer and a linear mapping layer, the high-dimensional feature vectors of the picture pair enter the output layer, first pass through the normalization layer to obtain corresponding normalized feature vectors, then pass through the inner product layer to obtain the front deep similarity, and then pass through the linear mapping layer to shift and scale the original interval of the front deep similarity to [0, 1] to obtain the deep similarity.

[0015] Further, the training process of the Siamese-Resnet50 model specifically includes the following steps:

[0016] S2.1, obtaining a pair of remote sensing images, and using expert knowledge to label them, and constructing a training data set according to the pair of remote sensing images and their labels; wherein the training data set contains similar remote sensing image pairs and dissimilar remote sensing image pairs;

[0017] S2.2, a plurality of enhancement methods are used to enhance the remote sensing image pairs in the training data set to expand the training data set; wherein the enhancement methods include data enhancement methods and radiation enhancement methods; the data enhancement methods include rotation, flipping, random cropping and translation transformation;

[0018] S2.3, pre-training the Siamese-Resnet50 model using the augmented training dataset, during the training process, calculating a contrastive loss function based on the deep similarity of the sample pairs and the labels corresponding to the sample pairs, taking minimizing the contrastive loss function as the optimization objective, adjusting the parameters of the Siamese-Resnet50 model until a preset training round is reached, to obtain a pre-trained Siamese-Resnet50 model.

[0019] Further, the calculation formula of the contrastive loss function is:

[0020]

[0021] In the formula, L contrast represents the contrastive loss function, represents the deep similarity of the i-th sample pair, y i represents the label of the i-th sample pair, m is a marginal hyperparameter, and N is the number of sample pairs in a training batch.

[0022] Further, the texture similarity acquisition method specifically includes:

[0023] Firstly, the picture pair A and B to be recognized are converted into grayscale pictures; then the texture feature vectors of the picture pair are extracted through a gray level co-occurrence matrix, wherein the texture feature vector is represented as T=[E, C, H, p, g] T , E, C, H, p, and g represent energy, contrast, entropy, correlation, and homogeneity, respectively; secondly, the cosine similarity between the two texture feature vectors is calculated as the pre-texture similarity; finally, the original interval of the pre-texture similarity is translated and scaled to [0, 1] using linear mapping to obtain the texture similarity.

[0024] Further, the step S3 specifically includes the following sub-steps:

[0025] S3.1, calculating a texture quality score using the contrast in the texture feature vector of the picture pair, and obtaining a depth quality score by calculating the absolute value of the pre-depth similarity of the picture pair;

[0026] S3.2, calculating a dynamic temperature parameter based on the texture quality score and the depth quality score;

[0027] S3.3, based on the dynamic temperature parameter and the texture quality score and the depth quality score, obtaining the weights corresponding to the texture similarity and the depth similarity through soft-max normalization, and normalizing them to obtain the normalized dynamic weights a and b corresponding to the texture similarity and the depth similarity;

[0028] S3.4. Use the normalized dynamic weights α and β corresponding to the texture similarity and depth similarity to perform feature fusion calculation on the texture similarity and depth similarity to obtain the comprehensive similarity.

[0029] The second aspect of an embodiment of the present invention provides a remote sensing image land use limit identification system based on deep learning, comprising: a processor and a memory for storing a computer program that can be run on the processor; wherein, when the processor is used to run the computer program, it executes the above-mentioned remote sensing image land use limit identification method based on deep learning.

[0030] A third aspect of an embodiment of the present invention provides a storage medium, in which a computer program is stored. When the computer program is executed by a processor, the above-mentioned remote sensing image land use limit identification method based on deep learning is implemented.

[0031] A fourth aspect of an embodiment of the present invention provides a computer program product, which includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the above-mentioned remote sensing image land use limit identification method based on deep learning.

[0032] The application has the beneficial effects that: the application utilizes the characteristics that the features inside and outside the edge of the ultra-limit land are inconsistent, carries out edge segmentation comparison, simplifies the problem of ultra-limit recognition difficulty, and improves the recognition efficiency; the application comprehensively utilizes deep semantic information and texture appearance information, fuses multi-modal information, can more comprehensively depict the land use state, and improves the recognition accuracy; the application introduces a dynamic weight adjustment mechanism based on feature quality, so that the model can adaptively allocate weights according to the reliability of each feature in different scenes, enhances the robustness and generalization ability of the model to complex conditions such as light change, local occlusion and texture difference; the application separates training and inference, the training of the deep model focuses on optimizing semantic feature extraction, and the complex fusion logic is carried out in the inference stage, which simplifies the training process and improves the stability and trainability of the model; the application improves the explainability, the output of the dynamic weight can provide a certain degree of explanation for the judgment result of a specific sample, and understand which or which modal feature plays a leading role in the final decision; the application has wide application, not only suitable for land use monitoring, planning verification, illegal land discovery and other business scenes, but also provides a new technical idea for remote sensing image change detection, target recognition and other related fields; the application solves the problems of high precision and scene adaptability in land use range monitoring through image preprocessing, image automatic segmentation and ultra-limit recognition, not only improves the precision of land use monitoring, but also greatly improves the timeliness of dynamic monitoring and identification, and is suitable for various application scenes such as land monitoring, land management and academic research. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 The flowchart of the land use ultra-limit identification method based on deep learning of the application is shown in the figure.

[0034] Figure 2 The inside and outside feature diagram of the embodiment of the application is shown in the figure.

[0035] Figure 3 The segmentation diagram of the pre-processing part of the remote sensing image of the embodiment of the application is shown in the figure.

[0036] Figure 4 The feature extraction and fusion architecture diagram of the double-branch neural network of the application is shown in the figure.

[0037] Figure 5 The loss change diagram in the training process of the Siamese-Resnet50 model of the application is shown in the figure.

[0038] Figure 6 The PR curve diagram of the Siamese-Resnet50 model of the embodiment of the application is shown in the figure.

[0039] Figure 7 The internal structure example diagram of the computer equipment of the embodiment of the application is shown in the figure. DETAILED DESCRIPTION

[0040] Exemplary embodiments will be described in detail herein, examples of which are illustrated in the accompanying drawings. In the following description, when referring to the drawings, like numbers in different figures represent like or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all possible embodiments consistent with the present invention. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present invention, as detailed in the appended claims.

[0041] The terms used in this invention are for the purpose of describing specific embodiments only and are not intended to limit the invention. The singular forms "a," "the," and "the" used in this invention and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0042] It should be understood that although the terms "first," "second," "third," etc. may be used in the present invention to describe various information, such information should not be limited to these terms. These terms are merely used to distinguish information of the same type from one another. For example, first information may also be referred to as second information, and similarly, second information may also be referred to as first information, without departing from the scope of the present invention. Depending on the context, the term "if" as used herein may be interpreted as "when," "when," or "in response to determining."

[0043] The present invention will be described in detail below with reference to the accompanying drawings. Unless there is any conflict, the features of the following embodiments and implementations may be combined with each other.

[0044] The method for identifying land use restrictions exceeding limits in remote sensing images based on deep learning of the present invention extracts two features, deep semantics and texture, in parallel, and designs a dynamic weighting mechanism to adaptively adjust the contribution of each feature in the final decision according to its reliability (quality) in the current sample, thereby improving the accuracy and robustness of land use restriction identification in complex and changing scenarios. Among them, "land use restrictions exceeding limits" is defined in the present invention as: the current actual land use status of the target plot (RegionA) (interpreted through recent remote sensing images) and its corresponding reference benchmark (RegionB) are significantly inconsistent in terms of spatial range, land feature type, or both. As Figure 1 As shown, the method specifically includes the following steps:

[0045] S1. Data acquisition and preprocessing: Acquire land use remote sensing images and preprocess them to obtain image pairs that need to be identified.

[0046] It should be understood that the pre-processing operation of the land remote sensing image and the automatic identification and extraction of the required information can efficiently realize the land use over-limit identification of the remote sensing image, and has important technical value and application prospect.

[0047] Further, the land remote sensing image is pre-processed, specifically including the following sub-steps:

[0048] S1.1, according to the vector boundary (such as Shapefile, GeoJSON) provided by the reference datum, in detail, when detecting the land over-limit, the standard land range vector can be provided, which can be referred to Figure 2 The vector boundary is the limit for detecting whether the land is over-limit. After obtaining the vector boundary, registration and orthorectification are performed according to the vector boundary, and the HSV (Hue (H), Saturation (S), Value (V)) color space is used to accurately extract the land range vector contour on the current phase remote sensing image after registration and orthorectification.

[0049] S1.2, based on the land range vector contour, the inner and outer expansion regions are generated through morphological operation, the inner and outer expansion contours are approximated by polygon to obtain the inner and outer expansion region vertex information, the vertex coordinates are extracted for cropping to obtain image blocks A and B.

[0050] Specifically, due to the unique characteristics of the land over-limit edge, in detail, the land types inside and outside the land edge often differ greatly, Figure 2 As shown. Therefore, according to the extracted land range vector contour, dilation and erosion operations can be performed for inner and outer expansion, and the expansion size is 2-10m inside and outside, wherein the expansion size can be changed according to actual project requirements, and the expansion distance can be converted to the inner and outer expansion pixel distance according to the image resolution. The inner and outer expansion contours are approximated by polygon to obtain the inner and outer expansion region vertex information, the vertex coordinates are extracted for cropping to obtain image blocks A and B, as shown in Figure 3 .

[0051] S1.3, the image blocks A and B are subjected to size normalization, data type conversion and pixel value standardization processing to meet the hard input requirements of the Siamese-Resnet50 model based on deep learning of the downstream module, so as to improve the training efficiency and stability of the Siamese-Resnet50 model based on deep learning, ensure the effectiveness when using the Siamese-Resnet50 model, and enhance the comparability of different features (such as depth, texture) between different samples. The finally pre-processed image blocks A and B are used as the picture pair to be identified.

[0052] S2, using a pre-trained Siamese-Resnet50 model and a gray level co-occurrence matrix (GLCM) to extract features and match the image pairs to be identified to obtain depth similarity and texture similarity, as shown in Figure 4

[0053] In this embodiment, the Siamese-Resnet50 model is constructed based on a Siamese network and deep learning. The Siamese network is a unique neural network architecture, the core of which consists of two or more identical sub-networks that share the same weights and parameters; each sub-network can be a multi-layer perceptron, a convolutional neural network, or a recurrent neural network, etc. In this embodiment, the Siamese network includes two sub-networks, which use ResNet50. The Siamese-Resnet50 model includes an input layer, a backbone network, and an output layer, as shown in Figure 4

[0054] The input layer is used to receive the image pair A and B to be identified.

[0055] The backbone network uses two parallel and weight parameter shared Resnet50 networks as encoders (Encoder) to extract features from the image pair A and B, respectively, to obtain the high-dimensional feature vectors f A ,f B ∈R D , where D represents the dimension of the high-dimensional feature vectors f A ,f B , and usually takes the value D = 2048 (the output channel number of the last convolutional block in the Resnet50 network); the structures and parameters of the two backbone networks are completely identical, and after inputting different data samples, the corresponding feature vectors are extracted. Further details, ImageA and ImageB are respectively passed through two ResNet50 networks with shared weights. Since the shapes and sizes of the images are different, the sizes of different image pairs are usually inconsistent, but since convolution and pooling are adaptive, they do not require fixed input sizes by themselves. Meanwhile, in the application of Siamese-ResNet50 model, ResNet50 is mainly used as a feature encoder. Instead of using its original fully connected layer for 1000-class ImageNet classification. The output of the deeper layer of the network is extracted as the high-dimensional feature vectors f A and f B ​​Global average pooling processes variable input sizes and produces a fixed-size output vector. Global average pooling averages the spatial dimensions (height H and width W) of each feature map into a single value. Therefore, as long as the number of channels is the same (for example, 2048), the output of global average pooling will be a vector of fixed length, which makes the final output high-dimensional feature vector f A and f B The length is fixed and the same.

[0056] Taking ImageA1 and ImageB1 as examples, their image sizes are both 25x106 pixels. The changes in ResNet50 are as follows: the initial input size is 25×106×3. In the convolutional layer (conv) and pooling layer (pool), the output size of conv1 is 13×53×64, which is obtained by the convolution operation with kernel=7, stride=2, and padding=3; then pool1 (max pooling, kernel=3, stride=2) downsamples it to 7×27×64; then, the output size of conv2_x is 7×27×256; the output size of conv3_x is 4×14×512, which is downsampled by the convolution with stride=2 in the first residual block of this stage; the output size of conv4_x is 2×7×1024, which is downsampled in the same way; the output size of conv5_x is 1×4×2048. In the global average pooling stage, the input size is 1×4×2048, and the operation is to average the 1×4 spatial dimensions of each channel. The final output size is 1×1×2048, that is, a 2048-dimensional vector. Therefore, the high-dimensional feature vectors obtained by the model for ImageA1 and ImageB1 are f respectively. A and f B For ImageA2 and ImageB2, whose image size is 50x31 pixels, the same calculation process will be performed to obtain the high-dimensional feature vector f A and f B .

[0057] The output layer is used to calculate and output the high-dimensional feature vector f corresponding to the image pair A and B A ,f B The deep semantic similarity S between deep , the output layer includes normalization layer, inner product layer and linear mapping layer, the high-dimensional feature vector f corresponding to the image pair A and B A ,f B Entering the output layer, it first passes through the normalization layer to obtain the corresponding normalized feature vector, which is expressed as:

[0058]

[0059] Where, denotes the normalized feature vector corresponding to picture A, f A denotes the high-dimensional feature vector corresponding to picture A; denotes the normalized feature vector corresponding to picture B, f B denotes the high-dimensional feature vector corresponding to picture B; |·| denotes the L2 norm, also known as the Euclidean norm, which is used to measure the length of the high-dimensional feature vector in the 2048-dimensional space; a d denotes the d-th dimension of f A a d denotes the d-th dimension of f B a D denotes the dimension of the high-dimensional feature vector. The normalized feature vectors corresponding to the picture pairs A and B are then taken as the inner product by the inner product layer to obtain the pre-depth similarity, denoted as:

[0060]

[0061] In the formula, S pre-deep denotes the pre-depth similarity, whose value falls within [-1, 1]. Subsequently, the original interval of the pre-depth similarity is translated and scaled to [0, 1] by the linear mapping layer to obtain the depth similarity, denoted as:

[0062]

[0063] In the formula, S deep denotes the depth similarity.

[0064] In this embodiment, when extracting the high-dimensional feature vectors of the picture pairs, each Resnet50 network maps the input data to a feature space (embedding space) such that similar input data are closer in the space and dissimilar input data are farther apart; in the feature space, the similarity of the input data pairs is measured by calculating the distance between the two feature vectors. Therefore, the Siamese-Resnet50 model can be trained based on this characteristic, and a contrastive loss function is usually used for training. The contrastive loss function adjusts the network parameters according to the true label (i.e., similar or dissimilar) of the input picture pair and the distance between the high-dimensional feature vectors, so that the feature vectors of similar picture pairs are closer and the feature vectors of dissimilar picture pairs are farther apart, which can effectively learn the similarity features of the pictures and has strong adaptability for tasks with limited sample numbers (such as few-shot learning). In addition, due to its parameter sharing characteristics, the training process is relatively efficient. The training process specifically includes the following steps:

[0065] S2.1, obtain a remote sensing image pair, specifically, the picture pair obtained by using or not using step S1 can be selected as needed as the remote sensing image pair required by the current step, and expert knowledge is used for labeling, and a training data set is constructed according to the remote sensing image pair and the label. Among them, the training data set contains similar remote sensing image pairs (such as the same type of ground objects or compliance changes, and the label is 1) and dissimilar remote sensing image pairs (such as different types of ground objects or out-of-limit changes, and the label is 0).

[0066] S2.2, since the remote sensing image similar plot data set is rare, in order to maximize the simulation of real land use out-of-limit use, it is necessary to enhance the remote sensing image pair in the training data set, and use various enhancement methods to enhance the remote sensing image pair in the training data set for enhancement processing to expand the training data set. Among them, the enhancement method includes but is not limited to data enhancement method, radiation enhancement method, etc.; the data enhancement method includes but is not limited to rotation, flip, random cropping, translation transformation, etc.

[0067] Exemplarily, in an embodiment, the data enhancement strategy geometric enhancement method includes the following operations in order of priority: first, random rotation, the angle range is-15° to +15°, the application probability is 80%; second, horizontal flip, the application probability is 50%; then, random cropping, 80% to 100% of the central region is reserved, the application probability is 70%; finally, translation transformation, the maximum offset of X direction and Y direction is 10%, the application probability is 60%. In the radiation enhancement method, the brightness is changed by linear adjustment in the range of ±20%, and Gaussian noise (standard deviation σ=0.02) is added to simulate cloud and fog noise points. In terms of class balancing strategy, the oversampling method is used to combine rotation and translation enhancement for out-of-limit samples (Label=0) to make the ratio of the number of out-of-limit samples to compliance samples reach 1:1.2; at the same time, difficult samples (such as high similarity negative sample pairs) misclassified in the training process are dynamically selected and added to the next round of training. It should be noted that when the training data set enhancement operation is performed, a pair of similar or dissimilar images is operated at the same time to match the normal detection scene. Through the above operation, the remote sensing out-of-limit image data set is greatly expanded, and the construction of the model training data set is successfully completed.

[0068] S2.3, pre-train the Siamese-Resnet50 model using the expanded training data set, during the training process, calculate the contrastive loss (ContrastiveLoss) function based on the depth similarity of the sample pair and its corresponding label, take minimizing the contrastive loss function as the optimization objective, adjust the parameters of the Siamese-Resnet50 model, and until the preset training round is reached, to obtain the pre-trained Siamese-Resnet50 model.

[0069] Further, the calculation formula of the contrast loss function is:

[0070]

[0071] wherein, L contrast represents the contrast loss function, represents the depth similarity of the i-th sample pair, y i represents the label of the i-th sample pair (the label of 0 represents that the sample pair is not similar land types, and the label of 1 represents that the sample pair is similar land types, and the training target is S deep close to 1, that is, the contrast loss function value is minimum), m is a margin hyperparameter (which takes a value of 0.2 or 0.5 or other numerical values, and can be set according to actual needs), and N is the number of sample pairs in a training batch.

[0072] Specifically, N = 108 picture pairs are randomly extracted from the expanded training data set to form a training batch, wherein the similar remote sensing image pairs and the dissimilar remote sensing image pairs account for 50% respectively. Two pairs are used as examples: sample pair 1 contains ImageA1 and ImageB1, and the label y1 = 1 (similar); sample pair 2 contains ImageA2 and ImageB2, and the label y2 = 0 (dissimilar). The data of this training batch is input into the Siamese-ResNet50 model, and finally the depth similarity S deep 1 of sample pair 1 can be obtained: S deep 2 of sample pair 2 can be obtained: S deep 1 = 0.9; the depth similarity S contrast 2 = 0.3. The calculation formula of the contrast loss function is used to calculate the contrast loss function of the training batch, and the margin hyperparameter m = 0.5 in the training batch. For sample pair 1, y1 = 1, S deep 1 = 0.9, L contrast 1 = 0.01, because it is a similar pair, the depth similarity predicted by the Siamese-ResNet50 model is 0.9, which is close to 1, so the contrast loss function value is small. For sample pair 2, y2 = 0, S i 2 = 0.3, L deepi > 0.5 (marginal hyperparameter m), the Siamese-ResNet50 model needs to be penalized, resulting in a loss, to encourage the Siamese-ResNet50 model to push the feature vectors of this pair of pictures further apart. After the batch is trained, the loss terms of all sample pairs in the batch are added up, and then divided by the batch size N = 108 to obtain the total loss L of the batch contrast_batch . After obtaining , backpropagation and parameter update are performed, and the weight update uses the Adam (Adaptive Moment Estimation) method.

[0073] It should be noted that the Adam method is an adaptive learning rate optimization algorithm for deep learning model training. Its core idea is to combine the advantages of the momentum gradient descent method and the RMSprop algorithm, and dynamically adjust the learning rate of each parameter by calculating the first and second moment estimates of the gradient, so as to realize efficient update of the weight. The update process of the Adam method is to minimize a given loss function f(θ) by iteratively updating the parameters θ, that is, the above L contrast . In each iteration update process, first calculate the update gradient:

[0074]

[0075] In the formula, g t is the gradient value of the tth update, is the derivative of the parameter θ, f t (θ t-1 0 is the loss function value at the t-1th time using the t-1th time, and θ t-1 is the model parameter of the t-1th update. Then calculate the first moment estimate:

[0076] m t = β1·m t-1 +(1-β1)·g t

[0077] In the formula, m t is the first moment estimate value at the tth update (m0 = 0); β1 is the decay coefficient of the first moment estimate, β1 ∈ [0, 1), the purpose is to prevent the initial estimate from being too small, and the typical value is 0.9. Then calculate the second moment estimate:

[0078]

[0079] In the formula, v t is the second moment estimate value at the tth update (v t = 0); β2 is the decay coefficient of the second moment estimate, β2 ∈ [0, 1), the purpose is to prevent the initial estimate from being too small, and the typical value is 0.999; This means squaring each element, i.e., operating on each element. Next, bias correction is performed:

[0080]

[0081] Where, is the first-order moment estimate after bias correction at the t-th update, is the bias-corrected t-th update moment estimate, and is the t-th power of β1 and β2. Finally update the parameters:

[0082]

[0083] Where θ t Model parameters at the tth update; is the learning rate, which controls the speed of parameter update; ε is a small constant that prevents division by 0.

[0084] Since the two ResNet50 branches of the Siamese network share the same set of weights, during back propagation, the path from image A The gradient and path from image B The gradients of will eventually be accumulated to the same set of shared weights W. That is, for any shared parameter w in the Siamese-ResNet50 model, the total gradient it receives is It is the sum of the gradients propagated back through the two branches. More precisely, it is the average of the sum of the gradients propagated back through the two branches for all pairs of samples in the batch. This ensures that the two branches learn exactly the same feature extraction capabilities.

[0085] Repeat the batch training described above, using different batches of training data for multiple rounds (epochs) of training until the model performance on the validation set converges. The specific loss changes and PR curves during the training process can be referred to Figure 5 and Figure 6 .

[0086] The reasoning process of the Siamese-ResNet50 model is as follows: the image pairs A and B to be identified obtained in step S1 are input into the pre-trained Siamese-ResNet50 model, and finally the depth similarity S deep .

[0087] In this embodiment, the method for obtaining the texture similarity specifically comprises the following steps. First, the picture pair A and B to be identified are converted into gray-scale pictures. Then, the texture feature vectors T = [E, C, H, p, g] of the picture pair A and B are extracted through a gray-level co-occurrence matrix (GLCM). T wherein E, C, H, p, g represent energy, contrast, entropy, correlation and homogeneity respectively. Next, the cosine similarity between the two texture feature vectors is calculated as the pre-texture similarity. Finally, the original interval of the pre-texture similarity is translated and scaled to [0, 1] using linear mapping to obtain the texture similarity.

[0088] Specifically, first, the picture pair A and B to be identified are converted into gray-scale pictures, and the gray-scale number, pixel spacing and direction angle parameters are defined. Then, the gray-level co-occurrence matrix of the picture pair A and B is calculated respectively, wherein the gray-level co-occurrence matrix is a statistical method for image texture analysis, which describes the spatial relationship of gray values in the image, i.e., the frequency of the simultaneous occurrence of a pixel with a certain gray value and a pixel with another gray value in a certain spatial relationship. The gray-level co-occurrence matrix is a two-dimensional matrix, denoted as P(i, j, d, q), each element of which represents the number or probability of the simultaneous occurrence of a pixel with a gray value of i and a pixel with a gray value of j in a relative position with a distance of d and a direction of q. Wherein i, j ∈ {0, 1, …, G-1}, G is the gray-scale number of the image; d ∈ Z + is a preset pixel spacing; q ∈ {0°, 45°, 90°, 135°} is a preset direction angle. Each pixel point (x, y) in the image is traversed, and the neighborhood pixel point (x', y') corresponding to the pixel is determined according to the preset co-occurrence direction q and the spatial distance d. Let g1 and g2 be the gray values of the current pixel (x, y) and the neighborhood pixel (x', y') respectively, then the pixel pair can be represented as the gray pair (g1, g2). The reference pixel point (x, y) is moved in the entire image area, and the joint occurrence frequency of all the gray pairs (g1, g2) that meet the conditions is counted, thereby constructing the initial gray-level co-occurrence matrix P. After normalization, the final gray-level co-occurrence matrix P representing the spatial structure features of the image texture is obtained. n

[0089]

[0090] Based on the above gray-level co-occurrence matrix P n , the energy, contrast, entropy, correlation and homogeneity features are extracted through the following formulas to constitute the texture feature vectors T A and T B :

[0091] ① The formula of the energy feature Energy is:​

[0092]

[0093] The formula of the contrast feature is:

[0094]

[0095] The formula of the entropy feature is:

[0096]

[0097] In the formula, ε is a small constant to prevent log(0).

[0098] The formula of the correlation feature is:

[0099]

[0100] In the formula, μ i , μ j are the mean values of i and j, and σ i , σ j are the standard deviations of i and j.

[0101] The formula of the homogeneity feature is:

[0102]

[0103] A group of standard texture feature descriptors are extracted from the GLCM to form a texture feature vector T A and T B , T = [E, C, H, ρ, γ] Twhere E, C, H, p, g represent the energy, contrast, entropy, correlation and homogeneity in the above formula respectively. Among them, the energy (E) reflects the uniformity and consistency of the texture; the higher the energy value, the more uniform the image texture, and the more concentrated the element distribution in the gray level co-occurrence matrix. When detecting the texture similarity, energy can help distinguish the areas with relatively uniform texture and the areas with complex and variable texture. The contrast (C) measures the sharpness of the texture and the variation range of the texture elements; the texture with high contrast usually has obvious gray level change, while the texture with low contrast is relatively smooth. The contrast can effectively distinguish the roughness of the texture and is very sensitive to the detection of the detail change of the texture. The entropy (H) represents the complexity and information amount of the texture; the higher the entropy value, the more complex the texture and the more abundant the information contained. Entropy can help identify the complexity and diversity of the texture, and has important significance for distinguishing different types of texture. The correlation (p) reflects the linear correlation degree between the pixels and their neighborhood pixels in the image; the texture with high correlation means that there is a strong linear relationship between the pixels, while the texture with low correlation has a relatively loose relationship between the pixels. Correlation can be used to detect the directionality and consistency of the texture. The homogeneity (g) measures the similarity of the pixel pairs in the image; the texture with high homogeneity means that the gray difference between the pixel pairs is small, and the texture is relatively uniform. Homogeneity can help identify the uniformity and consistency of the texture, and is very helpful for detecting the similarity of the texture. Secondly, the cosine similarity between the two texture feature vectors T A and T B is calculated as the pre-texture similarity:

[0104]

[0105] where S pre-GLCM represents the pre-texture similarity, and its value falls in [-1, 1]. Then, the original interval of the pre-texture similarity is translated and scaled to [0, 1] using a linear mapping to obtain the texture similarity, and its expression is:

[0106]

[0107] where S GLCM represents the texture similarity, and the higher the value of the texture similarity indicates the more similar the texture patterns.

[0108] S3, based on the picture pair depth similarity and the texture similarity, dynamic weight feature fusion calculation is performed to obtain the comprehensive similarity.

[0109] In an embodiment, the dynamic weight feature fusion calculation process occurs in the inference stage, and is dynamically controlled by the temperature control soft maximum method. The following is a parameter control method, which specifically includes the following sub-steps:

[0110] S3.1, for both texture and depth similarity, calculate their "quality score" respectively, which reflects the reliability of the current sample to the modal. That is: first, use the contrast in the texture feature vector of the picture pair to calculate the texture quality score Q tex : first, based on the contrast C A and C B in the texture feature vectors T A and T B of the picture pair A and B, use the theoretical maximum contrast C max_theoretical to normalize, get the normalized values corresponding to the picture pair A and B and

[0111]

[0112] In the formula, and respectively represent the normalized contrast feature vectors corresponding to the picture pair A and B, and the normalized contrast feature vectors are used to calculate the texture quality score; C A and C B respectively represent the contrast feature values corresponding to the picture pair A and B; C max_theoretical represents the theoretical maximum contrast of the current picture, which depends on the gray level and window size. In the GLCM calculation in this embodiment, the number of gray levels is compressed to 64 levels to reduce the calculation complexity, and the theoretical maximum contrast is (64-1) 2 = 3969 at this time. Then, the normalized contrast feature vectors and corresponding to the picture pair A and B are used to calculate the texture quality score Q tex , and the calculation formula is:

[0113]

[0114] Then, the depth quality score Q deep is obtained by calculating the absolute value of the pre-depth similarity S pre-deep of the picture pair, and is expressed as:

[0115] Q deep = |S pre-deep | ∈ [0, 1]

[0116] In the formula, the value of Q deep is closer to 1, indicating that the Siamese-Resnet50 model is more confident in judging whether it is similar.

[0117] S3.2, based on the texture quality score Q tex and the depth quality score Q deep , calculate the dynamic temperature parameter τ, which is used to adjust the smoothness of the weight distribution:

[0118] τ = τ0 x (1 + κVar(Q tex ,Q deep ))

[0119] In the formula, τ0>0 represents the basic temperature, κ≥0 represents the variance sensitivity coefficient, τ0 and κ are both hyperparameters, and the best values are obtained through multiple tests; Var() represents the variance between the texture quality score Q tex and the depth quality score Q deep . When the variance Var increases, τ increases, and the weight exp(τQ i ), i∈{tex,deep} is more sensitive to the difference in Q i value, and the modality with the highest score will obtain a greater weight. When the variance Var decreases (i.e., the texture quality score Q tex and the depth quality score Q deep are close), τ approaches τ0, and the weight distribution is more uniform.

[0120] S3.3, based on the dynamic temperature parameter τ and the texture quality score Q tex and the depth quality score Q deep , the weights corresponding to the texture similarity and the depth similarity are obtained through soft maximum normalization, and the weights are normalized to ensure that α+β=1, and the normalized dynamic weights α and β are calculated by applying the Softmax function; the Softmax function is a commonly used activation function, which is particularly suitable for converting a set of numerical values into a probability distribution, so that the sum of the output values is 1, thereby avoiding the deviation caused by the imbalance of the weights when the multi-modal feature is fused. At the same time, the Softmax function emphasizes the relative difference between the input values through exponential operation, so that the weight distribution is more inclined to the modality with higher quality score, and the sensitivity of the Siamese-Resnet50 model to the superior features is improved.

[0121] w tex = exp(τQ tex )

[0122] w deep = exp(τQ deep )

[0123]

[0124] In the formula, w tex represents the non-normalized weight value corresponding to the texture modality, w deep represents the non-normalized weight value corresponding to the deep learning modality, α represents the normalized dynamic weight corresponding to the texture similarity, and β represents the normalized dynamic weight corresponding to the depth similarity.

[0125] S3.4, the normalized dynamic weights a and b corresponding to the texture similarity and the depth similarity are used to calculate the texture similarity S GLCM and the depth similarity S deep to obtain the comprehensive similarity, and the calculation formula is:

[0126] S total (A, B) = a S GLCM + b S deep

[0127] In the formula, S total (A, B) represents the comprehensive similarity between the picture A (the target area) and the picture B (the standard ground object area).

[0128] S4, the comprehensive similarity is compared with the set decision threshold value, and it is judged whether the remote sensing image land use range is over-limit and the result is output.

[0129] Specifically, in an embodiment, the comprehensive similarity S total is compared with the preset decision threshold value T, if the comprehensive similarity S total is greater than the preset decision threshold value T, it is considered that the remote sensing image land use range has land use over-limit situation; otherwise, it is considered that the remote sensing image land use range does not have land use over-limit situation; and the final over-limit identification result (normal land use or suspected over-limit) is output. The decision threshold value T is set and optimized by maximizing the F1-Score, in addition, the evaluation indexes in the training process of the Siamese-ResNet50 model also include the precision (Precision) and the recall (Recall), and the calculation formula is as follows:

[0130]

[0131] In the formula, TP is the number of samples correctly judged as over-limit land use; FP is the number of normal land use samples incorrectly judged as over-limit land use; and FN is the number of over-limit land use samples incorrectly judged as normal land use.

[0132] In order to realize the method of the embodiment of the application, the embodiment of the application also provides a remote sensing image land use over-limit identification system based on deep learning, which comprises a processor and a memory for storing a computer program capable of running on the processor; wherein the processor is used to run the computer program, and executes the steps of the above-mentioned method.

[0133] The above-mentioned system and the method embodiment provided by the embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0134] To implement the method of an embodiment of the present invention, an embodiment of the present invention further provides a computer program product. The computer program product includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps of the above method.

[0135] Based on the hardware implementation of the above program modules, and in order to implement the method of the embodiment of the present invention, the embodiment of the present invention further provides an electronic device (computer device). Specifically, in one embodiment, the computer device can be a terminal, and its internal structure diagram can be as follows: Figure 7 As shown. The computer device includes a processor A01, a network interface A02, a display screen A04, an input device A05 and a memory connected via a system bus. The processor A01 of the computer device is used to provide computing and control capabilities. The memory of the computer device includes an internal memory A03 and a non-volatile storage medium A06. The non-volatile storage medium A06 stores an operating system B01 and a computer program B02. The internal memory A03 provides an environment for the operation of the operating system B01 and the computer program B02 in the non-volatile storage medium A06. The network interface A02 of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor A01, the method of any one of the above embodiments is implemented. The display screen A04 of the computer device can be a liquid crystal display or an electronic ink display screen, and the input device A05 of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad provided on the computer device housing, or an external keyboard, touchpad or mouse.

[0136] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0137] The device provided by an embodiment of the present invention includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, the method of any one of the above embodiments is implemented.

[0138] Those skilled in the art will appreciate that embodiments of the application can be readily used as software, hardware, or a combination of software and hardware. In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0139] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more functions specified in the flowchart block or blocks. Figure 1 means for performing each of the functions specified in the flowchart block or blocks.

[0140] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the functions specified in the flowchart block or blocks. Figure 1 one or more functions specified in the flowchart block or blocks. Figure 1 means for performing each of the functions specified in the flowchart block or blocks.

[0141] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more functions specified in the flowchart block or blocks. Figure 1 means for performing each of the functions specified in the flowchart block or blocks.

[0142] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0143] The memory can include non-persistent memory and / or volatile memory, such as random access memory (RAM) about which the processor can execute instructions. The memory can also include non-volatile memory, such as read only memory (ROM) or flash RAM, about which permanent information can be stored; such information can not change much and / or can only change slowly.

[0144] Computer-readable media includes permanent and non-permanent, moveable and non- moveable media that can be implemented in any method or technology for storage of information, such as computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, without limitation, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disks (DVDs) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information for access by a computing device. According to the definitions herein, computer-readable media does not include transitory media, such as modulated data signals and carrier waves.

[0145] It can be understood that the memory of the embodiments of the present application can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a ferromagnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM). The magnetic surface memory can be a disk memory or a tape memory. The volatile memory can be a random access memory (RAM) used as an external cache. By way of example but not limitation, many forms of RAM can be used, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), sync link dynamic random access memory (SLDRAM), and direct rambus random access memory (DRRAM).The memory described in the embodiments of the present application is intended to include, but not limited to, these and any other suitable types of memory.

[0146] The above embodiments are only used to illustrate the technical solutions of the present application, but not limit the present application; although the present application has been described in detail with reference to the foregoing embodiments, those ordinarily skilled in the art should understand: the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced equivalently; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for identifying excessive land use in remote sensing images based on deep learning, characterized in that: The following steps are involved: S1. Acquire land remote sensing images and pre-process the land remote sensing images to obtain image pairs that need to be identified; S2. Use the pre-trained Siamese-Resnet50 model and gray-level co-occurrence matrix to extract and match the image pairs to be identified to obtain depth similarity and texture similarity; S3. Based on the depth similarity and texture similarity of the image pair, dynamic weight feature fusion calculation is performed to obtain comprehensive similarity; S4. Compare the comprehensive similarity with the set judgment threshold to determine whether the land use range of the remote sensing image exceeds the limit and output the result.

2. The method for identifying excessive land use in remote sensing images based on deep learning according to claim 1, characterized in that: The pre-processing of the land remote sensing image specifically includes the following sub-steps: S1.

1. Perform registration and orthorectification based on the vector boundaries provided by the reference benchmark data, and accurately extract the land use area vector outline using the HSV color space from the registered and orthorectified current phase remote sensing image; S1.

2. Based on the land use area vector outline, generate inner and outer expansion regions through morphological operations. Use polygons to approximate the inner and outer expansion contours to obtain vertex information of the inner and outer expansion regions. Extract the vertex coordinates for cropping to obtain image blocks A and B. S1.

3. Normalize the size, convert the data type, and normalize the pixel values ​​of image blocks A and B to meet the input requirements of the Siamese-Resnet50 model. Use the normalized image blocks A and B as the image pair to be recognized.

3. The method for identifying excessive land use in remote sensing images based on deep learning according to claim 1, characterized in that: The Siamese-Resnet50 model is constructed based on the Siamese network and deep learning. The Siamese-Resnet50 model includes an input layer, a backbone network and an output layer, wherein the input layer is used to receive the image pairs to be identified; the backbone network uses two parallel Resnet50 networks with shared weight parameters as encoders to extract features from the image pairs respectively and obtain corresponding high-dimensional feature vectors; the output layer is used to calculate and output the deep semantic similarity between the high-dimensional feature vectors of the image pairs, and the output layer includes a normalization layer, an inner product layer and a linear mapping layer. The high-dimensional feature vector of the image pair enters the output layer, first passes through the normalization layer to obtain the corresponding normalized feature vector, and then performs an inner product through the inner product layer to obtain the pre-depth similarity, and then translates and scales the original interval of the pre-depth similarity to [0,1] through the linear mapping layer to obtain the depth similarity.

4. The method for identifying excessive land use in remote sensing images based on deep learning according to claim 1, characterized in that: The training process of the Siamese-Resnet50 model specifically includes the following steps: S2.

1. Obtain remote sensing image pairs, annotate them using expert knowledge, and construct a training dataset based on the remote sensing image pairs and their labels. The training dataset contains similar remote sensing image pairs and dissimilar remote sensing image pairs. S2.

2. Enhance the remote sensing image pairs in the training dataset using multiple enhancement methods to expand the training dataset; wherein the enhancement methods include data enhancement methods and radiation enhancement methods; the data enhancement methods include rotation, flipping, random cropping, and translation transformation; S2.

3. Use the expanded training dataset to pre-train the Siamese-Resnet50 model. During the training process, calculate the contrast loss function based on the depth similarity of the sample pairs and their corresponding labels. Minimize the contrast loss function as the optimization goal, and adjust the parameters of the Siamese-Resnet50 model until the preset training rounds are reached to obtain the pre-trained Siamese-Resnet50 model.

5. The method for identifying excessive land use in remote sensing images based on deep learning according to claim 4 is characterized in that: The calculation formula of the contrast loss function is: Where, L contrast represents the contrast loss function, Indicates the depth similarity of the i-th sample pair, y i represents the label of the i-th sample pair, m is the marginal hyperparameter, and N is the number of sample pairs in a training batch.

6. The method for identifying excessive land use in remote sensing images based on deep learning according to claim 1, characterized in that: The method for obtaining the texture similarity specifically includes: First, the image pair A and B to be identified are converted into grayscale images; then the texture feature vector of the image pair is extracted through the gray-level co-occurrence matrix, where the texture feature vector is expressed as T = [E, C, H, ρ, γ] T , E, C, H, ρ, γ represent energy, contrast, entropy, correlation and homogeneity respectively; secondly, the cosine similarity between the two texture feature vectors is calculated as the pre-texture similarity; finally, the original interval of the pre-texture similarity is translated and scaled to [0, 1] using linear mapping to obtain the texture similarity.

7. The method for identifying excessive land use in remote sensing images based on deep learning according to claim 1, characterized in that: The step S3 specifically includes the following sub-steps: S3.

1. Calculate a texture quality score using the contrast in the texture feature vector of the image pair, and obtain a depth quality score by calculating the absolute value of the pre-depth similarity of the image pair; S3.

2. Calculate dynamic temperature parameters based on the texture quality score and the depth quality score; S3.

3. Based on the dynamic temperature parameter and the texture quality score and the depth quality score, obtain weights corresponding to the texture similarity and the depth similarity through soft maximum normalization, and normalize the weights to obtain normalized dynamic weights α and β corresponding to the texture similarity and the depth similarity; S3.

4. Use the normalized dynamic weights α and β corresponding to the texture similarity and depth similarity to perform feature fusion calculation on the texture similarity and depth similarity to obtain the comprehensive similarity.

8. A remote sensing image land use limit identification system based on deep learning, characterized by: include: A processor and a memory for storing a computer program that can be run on the processor; wherein, when the processor is used to run the computer program, it executes the remote sensing image land use limit over-limit identification method based on deep learning described in any one of claims 1-7.

9. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the processor, the method for identifying excessive land use limits based on remote sensing images according to any one of claims 1 to 7 is implemented.

10. A computer program product, characterized in that The computer program product includes computer instructions, which are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the remote sensing image land use limit identification method based on deep learning as described in any one of claims 1 to 7.

Images