Remote sensing scene classification method, system, device and medium based on unsupervised ranking learning
By improving the normalized discounted cumulative gain and unsupervised ranking learning method, multiple front views are generated and mixed images, which solves the problems of front view misjudgment and insufficient information in remote sensing image classification, and improves the classification accuracy and network generalization ability.
Patent Information
- Application Number
- CN202210280507.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-22
- Publication Date
- 2025-10-10
- Estimated Expiration
- 2042-03-22
AI Technical Summary
Existing unsupervised learning methods have the problem of mistaking positive views for negative views in remote sensing image classification, and insufficient information mining affects the effectiveness of feature expression, resulting in insufficient classification accuracy.
By improving the normalized discounted cumulative gain, a differentiable optimization objective is established to generate multiple positive views. The unsupervised ranking learning method is used to predict the ranking of all positive and negative views. At the same time, mixed images are generated by image mixing to enhance the generalization ability of the network.
It improves the accuracy of remote sensing scene classification, avoids the situation where positive images are mistaken for negative images, forces the network to learn more discriminative deep features, and enhances the network's generalization ability.
Smart Images
Figure CN114842327B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a remote sensing scene classification method, system, equipment and medium based on unsupervised ranking learning, and belongs to the field of automatic interpretation of remote sensing images. Background Art
[0002] With their overwhelming performance advantages, deep neural networks have dominated numerous fields in recent years. Convolutional neural networks, a prominent representative of the deep network family, have become a mainstream approach. However, training convolutional neural networks is a supervised learning process that requires massive amounts of labeled samples. This significantly hinders their application, especially in remote sensing, where image annotation requires specialized knowledge. Consequently, unsupervised learning, which eliminates the need for image labels and automatically generates supervisory information, has drawn significant attention.
[0003] Recently, an unsupervised learning method called “contrastive learning” has been shown to be comparable to or even better than supervised learning. This method uses cropping, color distortion, geometric transformation, etc. to create a contrastive image for each original image. Generate two expanded images a i and v i (In this application, all superscripts represent serial numbers rather than indices), which are respectively called "anchor image" and "view image", referred to as "anchor" and "view" for short. i For example, v i It is called "front view", v i′ (i′≠i) is called a “negative view”. Contrastive learning aims to automatically learn an embedding space such that the anchor embedding and the positive view embedding are as close as possible, while the anchor embedding and the negative view embedding are as far apart as possible.
[0004] Record a separately i and v i The normalized embedding of i and v i (they are column vectors), given a mini-batch The loss function for contrastive learning is usually defined as:
[0005]
[0006] Where ω represents the parameters of the convolutional neural network (including weights and biases), <·,·> represents the inner product, and T is an adjustable temperature parameter.
[0007] As can be seen from formula (1), “positive” and “negative” are the supervisory information of the network training process. No image labels are required to generate such supervisory information. Therefore, contrastive learning belongs to the category of unsupervised learning.
[0008] However, contrastive learning methods have the following shortcomings:
[0009] 1) Different original images may have similar appearances, causing the positive image to be mistaken for the negative image. Currently, contrastive learning models are generally trained on labeled image libraries (but the training process does not use any labels), which include a limited number of categories. From formula (1), it can be seen that as long as i≠i′, contrastive learning will treat and As a pair of heterogeneous images, v i′ As a i and try to separate their embeddings. In fact, and may come from the same category (for example, both belong to the "forest" category) and thus have similar appearances. In this case, v i′ It should be a i For positive images rather than negative images, the training goal should be to bring their embeddings closer (rather than push them apart).
[0010] 2) Each original image Only one view is generated, resulting in The information is not fully mined and the training requirements are not strict enough, which affects the expressive power of the features. Generate multiple views, then they usually contain more information than one view (because each view carries Moreover, we can also require that these views are close to the anchor a in the embedding space i . Obviously, "there are multiple front views close to a i ” and “There is a front view close to a i "Compared to the former, it is a more demanding training target, thus forcing the network to learn more powerful features. Summary of the Invention
[0011] In light of this, the present invention provides a remote sensing scene classification method, system, computer device, and storage medium based on unsupervised ranking learning to prevent positive images from being mistaken for negative images. Furthermore, the present invention generates multiple positive images for a single anchor image and establishes a differentiable optimization objective by improving the normalized discounted cumulative gain. This replaces the "positive pull, negative push" approach with unsupervised ranking, allowing the network to predict the ranking of all positive and negative images. Furthermore, the present invention generates a set of mixed images with varying proportions through image blending and instructs the network to predict their ranking based on their similarity to the anchor. This further enhances the network's generalization capabilities.
[0012] The first object of the present invention is to provide a remote sensing scene classification method based on unsupervised ranking learning.
[0013] The second object of the present invention is to provide a remote sensing scene classification system based on unsupervised ranking learning.
[0014] A third object of the present invention is to provide a computer device.
[0015] A fourth object of the present invention is to provide a storage medium.
[0016] The first object of the present invention can be achieved by adopting the following technical solutions:
[0017] A remote sensing scene classification method based on unsupervised ranking learning, the method comprising:
[0018] By improving the normalized discounted cumulative gain, a differentiable optimization objective is established, which is used to construct the view ranking loss and the mixed anchor ranking loss.
[0019] Acquire remote sensing images, and crop images of preset sizes from the remote sensing images to create an original image set;
[0020] Generate anchor images and view images based on the original image set;
[0021] Build anchor branches and view branches;
[0022] The anchor image and view image are input into the anchor branch and view branch respectively, and trained using the view ranking loss to obtain the view ranking network;
[0023] Use anchor images to synthesize mixed anchor images;
[0024] The anchor image and the mixed anchor image are input into the anchor branch and trained using the mixed anchor ranking loss to obtain the mixed anchor ranking network.
[0025] Integrate the view ranking network and the mixed anchor ranking network to obtain the view mixed anchor ranking network;
[0026] Inputting the view image, the anchor image, and the mixed anchor image into a view mixed anchor ranking network, and training the network using a total loss to obtain a trained view mixed anchor ranking network, wherein the total loss includes a view ranking loss and a mixed anchor ranking loss;
[0027] Classify remote sensing scene images using the trained view-mixed anchor ranking network.
[0028] Furthermore, the above method establishes a differentiable optimization objective by improving the normalized discounted cumulative gain, specifically including:
[0029] Let π be any ranking list of K key images, then the cumulative loss gain of π is:
[0030]
[0031] Where π(k) represents the position of the kth key image in π;
[0032] The rankings corresponding to the network's predicted value s of the similarity of the k-th key image and the correlation level r of the k-th key image are π s and π r , then π s The normalized discounted cumulative gain is:
[0033]
[0034] Among them, Ψ(π s ,r)∈(0,1];
[0035] When θ→+∞, the Sigmoid function Infinitely approaching the unit step function
[0036]
[0037] According to π s (k)=1+∑ k′≠k h(s k′ -s k ),have
[0038]
[0039] Let μ be the right side of the equation s (k), using μ s (k) replaces π s (k), we get π s Approximate discounted cumulative gain of:
[0040]
[0041] Then we get π s The approximate normalized discounted cumulative gain of :
[0042]
[0043] in, It is a derivable optimization objective that is only targeted at one query image.
[0044] Furthermore, the definition process of the view ranking loss is as follows:
[0045] The anchor image a i and view image Input the anchor branch and view branch respectively, and obtain the normalized feature embedding α through forward propagation i and Where, i = 1, 2, ..., I; j = 1, 2, ..., J;
[0046] Assume that the key image of the previous batch corresponds to a feature embedding sequence υ=(υ1,υ2,…,υ K ), treat υ as a queue with a queue length of K, and let the oldest IJ embed α i Out of the queue, the IJ embeddings just calculated Joining the team, that is:
[0047]
[0048] After the dequeue and enqueue operations are completed, the last IJ elements of υ K-IJ+1 ,υ K-IJ+2 ,…,υ K Corresponding to the view generated by the current small batch, the previous K-IJ elements υ1,υ2,…,υ K-IJ Corresponding to the old views; these views are the key images of the current mini-batch;
[0049] Each anchor image a i As a query image; for the anchor image a i In terms of K-(I-i+1)J+1 ,υ K-(I-i+1)J+2 ,…,υ K-(I-i)J corresponds to the positive view, and the rest of the embeddings in the queue correspond to the negative views;
[0050] Assume that the anchor image a i The correlation with the positive image is 1, and the correlation with the negative image is 0, that is:
[0051]
[0052] remember but The cumulative loss gain is:
[0053]
[0054] make And remember in, The anchor image a is measured i The predicted similarity between the k-th key image is The approximate cumulative loss gain is:
[0055]
[0056] The approximate normalized discounted cumulative gain is:
[0057]
[0058] Define the view ranking loss as:
[0059]
[0060] Furthermore, the definition process of the mixed anchor ranking loss is as follows:
[0061] The anchor image in As the query image, the anchor image regarded as The key image of
[0062] Mixed anchor images The normalized feature embedding of And order in,
[0063] make in, express and Similarity level, equal exist The position in the ascending ranking list is minus 1, that is,
[0064] Let the normalization constant be:
[0065]
[0066] After substitution, we get:
[0067]
[0068]
[0069] Define the mixed anchor ranking loss as:
[0070]
[0071] Furthermore, the total loss is as follows:
[0072]
[0073] in, is the view ranking loss, is the mixed anchor ranking loss.
[0074] Furthermore, the anchor image and the view image are input into the anchor branch and the view branch respectively. During the training process using the view ranking loss, the network parameters of the anchor branch are updated by backpropagation. The network parameters of the view branch are updated as follows:
[0075] ωv ←τω v +(1-τ)ω a
[0076] Among them, ω a is the network parameter of the anchor branch, ω v is the network parameter of the view branch, and τ∈[0,1) is the momentum coefficient.
[0077] Furthermore, the step of cropping an image of a preset size from the remote sensing image to create an original image set specifically includes:
[0078] According to the principle that the horizontal or vertical center distance is not less than D pixels, randomly cut out images of size W×H from the remote sensing image to create the original image set { |When i≠i′,|X i -X i′ |≥D,|Y i -Y i′ |≥D}, where X i and Y i Respectively The horizontal and vertical coordinates of the center in the downloaded image.
[0079] Generating the anchor image and the view image according to the original image set specifically includes:
[0080] Given a small batch From the original image Cut out a piece of size w×h from the center to get the anchor image a i ;
[0081] According to the principle that the horizontal and vertical center distances between the anchor and the view are no more than d pixels, Randomly crop J (J>1) view images of size w×h from View Image satisfy in, and Respectively The horizontal and vertical coordinates of the center, x i and y i Respectively The horizontal and vertical coordinates of the center;
[0082] Furthermore, synthesizing the mixed anchor image using the anchor image specifically includes:
[0083] From the anchor image set {a i Randomly select images from |i=1,2,…,I} to generate L pairs of anchor image pairs Where (.;.) represents an image pair, l=1,2,…,L; il ,i′ l ∈{1,2,…,I} and i l ≠i′ l ;
[0084] For each l, randomly generate Mixing coefficient To mix and get Mixed anchor images:
[0085]
[0086] The second object of the present invention can be achieved by adopting the following technical solutions:
[0087] A remote sensing scene classification system based on unsupervised ranking learning, the system comprising:
[0088] Establish a module for establishing a differentiable optimization objective by improving the normalized discounted cumulative gain, wherein the optimization objective is used to construct the view ranking loss and the mixed anchor ranking loss;
[0089] Creating a module for acquiring remote sensing images and cropping images of a preset size from the remote sensing images to create an original image set;
[0090] A generation module, used for generating an anchor image and a view image according to an original image set;
[0091] Building modules for building anchor branches and view branches;
[0092] The first training module is configured to input the anchor image and the view image into the anchor branch and the view branch respectively, and train the anchor image and the view image using the view ranking loss to obtain a view ranking network.
[0093] A synthesis module, used for synthesizing a mixed anchor image using the anchor image;
[0094] The second training module is used to input the anchor image and the mixed anchor image into the anchor branch and train using the mixed anchor ranking loss to obtain the mixed anchor ranking network;
[0095] An integration module is used to integrate the view ranking network and the mixed-anchor ranking network to obtain the view mixed-anchor ranking network;
[0096] a third training module, configured to input the view image, the anchor image, and the mixed-anchor image into a view-mixed-anchor ranking network, and perform training using a total loss to obtain a trained view-mixed-anchor ranking network, wherein the total loss includes a view ranking loss and a mixed-anchor ranking loss;
[0097] The classification module is used to classify remote sensing scene images using the trained view-mixed anchor ranking network.
[0098] The third object of the present application can be achieved by adopting the following technical solution:
[0099] A computer device comprises a processor and a memory for storing a program executable by the processor, and the processor implements the remote sensing scene classification method described above when executing the program stored in the memory.
[0100] The fourth object of the present application can be achieved by adopting the following technical solution:
[0101] A storage medium stores a program, and the program is executed by a processor to implement the remote sensing scene classification method described above.
[0102] The present application has the following beneficial effects relative to the prior art:
[0103] According to the characteristics of remote sensing images, the present application proposes a new original image creation method, which to some extent avoids the case that the front view is mistaken for the negative view. At the same time, the present application generates multiple front views for a ship anchor image, predicts the ranking of all front and negative views using an unsupervised ranking method, and forces the network to learn more discriminative deep features. In addition, the present application generates a group of mixed images with different proportions through image mixing, and requires the network to predict the ranking of the mixed images according to their similarity with the ship anchor, further enhancing the generalization ability of the network. Compared with the prior art, the method disclosed by the present application can obtain higher classification accuracy. BRIEF DESCRIPTION OF DRAWINGS
[0104] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from the structures shown in these drawings without creative labor.
[0105] Figure 1 The flowchart of the remote sensing scene classification method based on unsupervised ranking learning of the present application embodiment 1.
[0106] Figure 2 The unit step function and Sigmoid function diagram of the present application embodiment 1.
[0107] Figure 3 The training flowchart of the view mixed anchor ranking network of the present application embodiment 1.
[0108] Figure 4 The image sample diagram of EuroSAT of the present application embodiment 1.
[0109] Figure 5 This is a curve diagram showing the impact of the number of views and queue length on classification accuracy in Example 1 of the present invention.
[0110] Figure 6 1 is a performance curve diagram of the mixed-anchor sorting network with different values of λ and view in Example 1 of the present invention.
[0111] Figure 7 This is a structural block diagram of a remote sensing scene classification system based on unsupervised ranking learning according to Example 2 of the present invention.
[0112] Figure 8 This is a structural block diagram of a computer device according to embodiment 3 of the present invention. DETAILED DESCRIPTION
[0113] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.
[0114] Example 1:
[0115] like Figure 1 As shown, this embodiment provides a remote sensing scene classification method based on unsupervised ranking learning, which includes the following steps:
[0116] S101. Establish a differentiable optimization objective by improving the normalized discounted cumulative gain.
[0117] Among them, the optimization objective lays the foundation for the construction of subsequent loss functions, and is mainly used to construct view ranking loss and mixed anchor ranking loss; normalized discounted cumulative gain is a commonly used evaluation indicator in the field of information retrieval, which is used to measure the quality of ranking.
[0118] Given a query image, the network needs to sort K images (they are called "key images"), and the sorting is based on the similarity between the key image and the query image (calculated by using the feature embedding of the two images). The one with higher similarity is ranked higher. Let s = (s1, s2, ..., s K ), where s k is the network's predicted value of the similarity of the kth key image, which is a derivative function of the network parameter ω. Let r=(r1,r2,…,r K ), where r k It represents the relevance level of the k-th key image, which is a predetermined value that measures the relevance between the k-th key image and the query image.
[0119] Let π be any ranking list of K key images, then the cumulative loss gain of π is:
[0120]
[0121] Wherein, π(k) represents the position (ie, serial number) of the k-th key image in π.
[0122] Let s and r correspond to the ranking list respectively π s and π r , then π s The normalized discounted cumulative gain is:
[0123]
[0124] Among them, Ψ(π s ,r)∈(0,1], the closer to 1, the π s The more perfect.
[0125] When π=π s When π in formula (2) s (k) is not differentiable, so Φ(π s ,r) is not differentiable, and thus Ψ(π s ,r) is also not differentiable. Therefore, if the gradient descent method is used to update the network parameters, Ψ(π s ,r) cannot be used as the optimization target, so this embodiment turns to find a differentiable approximation to replace Ψ(π s ,r), it is easy to see:
[0126] π s (k)=1+∑ k′≠k h(s k′ -s k ) (4)
[0127] where h(Δ) is the unit step function:
[0128]
[0129] like Figure 2 As shown, it can be seen intuitively that when θ→+∞, the Sigmoid function
[0130]
[0131] Infinitely approach the unit step function h(Δ), so for sufficiently large θ, we have
[0132]
[0133] Let μ be the right side of (7) s (k), using μs (k) replaces π s (k), we get π s The approximate discounted cumulative gain of :
[0134]
[0135] Then we get π s The approximate normalized discounted cumulative gain of :
[0136]
[0137] in, It is a derivable optimization objective that is only targeted at one query image.
[0138] S102: Acquire remote sensing images, and crop images of a preset size from the remote sensing images to create an original image set.
[0139] Remote sensing images are a reflection of the ground situation. Each remote sensing image (more precisely, the center of each remote sensing image) corresponds to a geographic location. Generally speaking, if two remote sensing images correspond to geographical locations that are far apart, the two images are likely to look different, and therefore the two augmented images generated from them are likely to have different appearances. Based on this unique property of remote sensing images, this embodiment uses a new method for creating original images to avoid, to a certain extent, the situation where a positive image is mistaken for a negative image.
[0140] Remote sensing image data products can be downloaded free of charge through domestic websites such as Geospatial Data Cloud, International Scientific Data Service Platform, Earth Science Data Sharing Platform, and foreign websites such as GLCF, USGS, LAADS DAAC, etc. Then, according to the principle that the horizontal or vertical center distance is not less than D pixels, a number of images of size W×H are randomly cropped from the downloaded images to form the original image set { |When i≠i′,|X i -X i′ |≥D,|Y i -Y i′ |≥D}, where X i and Y i Respectively The center of the image is in the horizontal and vertical coordinates of the downloaded image. When D is large enough, it can be considered that and (i≠i′) looks quite different.
[0141] In this embodiment, 900 Sentinel-2 Class 1C remote sensing images are downloaded free of charge via the Internet. Then, let D = 800 and W = H = 264, and crop 100,000 original images of size 264 × 264 according to the above method. They constitute the original image set (i.e., the unlabeled pre-training set).
[0142] S103: Generate an anchor image and a view image according to the original image set.
[0143] In a small batch, for a given set of original images This embodiment generates the anchor image and view image online. Specifically, from the original image Cut out a piece of size w×h from the center, which is the anchor a i In addition, this embodiment creates multiple front views for each anchor. Specifically, according to the principle that the horizontal and vertical center distances between the anchor and the view are no more than d pixels, Randomly crop J (J>1) views of size w×h They satisfy in, and Respectively The horizontal and vertical coordinates of the center, x i and y i Respectively In this embodiment, the sizes of the anchor image and the view image are both 64×64, that is, w=h=64; in addition, d=100.
[0144] Obviously, in each small batch, a i The front view is a i The negative view is The original image generation method basically guarantees the reliability of the negative image "identity" - since and (i≠i′) is likely to have a very different appearance, so a i and The appearance of is also likely to be very different. So, let a i keep away This makes sense. It is in this way that this embodiment overcomes the shortcomings of existing positive and negative image discrimination criteria. Furthermore, J>1 means that the positive image here carries more information than a single positive image, thus making up for the shortcomings of the contrastive learning method from another perspective.
[0145] S104. Build anchor branches and view branches.
[0146] The network model of this embodiment includes two branches, namely the anchor branch and the view branch. The anchor branch receives the anchor image and the mixed anchor image, and the view branch receives the view image. The structures of the two branches are exactly the same, and the network parameters are ω a and ω v .
[0147] In this embodiment, ResNet-18 is selected as the backbone of the anchor branch and the view branch.
[0148] S105. Input the anchor image and the view image into the anchor branch and the view branch respectively, and train them using the view ranking loss to obtain a view ranking network.
[0149] In a mini-batch of I original images, each anchor image has J positive views and (I-1)J negative views. The larger I is, the more negative views the network "sees" each time, the richer the information captured from the training data, and the greater the difficulty of sorting, thus improving the network's ultimate generalization ability. However, if I is too large, GPU memory will be insufficient.
[0150] This embodiment draws on the approach of the famous contrastive self-supervised learning model MoCo, and cleverly utilizes previous views - that is, the negative views come not only from the current mini-batch, but also from previous mini-batches. The view branch does not participate in gradient backpropagation, thus solving the problem of insufficient memory. In addition, the embedding of the previous view is calculated by the previous network and retained to this day, while the embedding of the current view is calculated by the current network. As key images (in this embodiment, the key image is the view to be sorted, including the views generated by several previous mini-batches and the view generated by the current mini-batch), the previous view and the current view should have equal status, so the previous network parameters and the current network parameters cannot differ too much. In other words, the network parameter update of the view branch should be slow and steady. This embodiment uses the momentum update method to solve this problem.
[0151] During the training process, ω a Updated by the usual back-propagation method, and ω v The update method is:
[0152] ω v ←τω v +(1-)ω a (10)
[0153] Where τ∈[0,1) is the momentum coefficient. When τ is close to 1, formula (10) ensures that ω v The change ratio ω a Slow, so ω v The previous value and the current value are not much different, so the similarity comparison between the query image and images with different keys is basically fair.
[0154] This embodiment uses the ranking learning technology to rank all views and measures the ranking quality based on the approximate normalized discounted cumulative gain.
[0155] Specifically, the anchor image a generated online i(i=1,2,…,I) and view images (i=1,2,…,I;j=1,2,…,J) are input into the anchor network and the view network respectively, and the normalized feature embedding α is obtained through forward propagation i and
[0156] Assume that the key image of the previous batch corresponds to a feature embedding sequence υ=(υ1,υ2,…,υ K ). Treat υ as a queue (queue length is K), and let the oldest IJ embeddings be dequeued. Then, the IJ embeddings just calculated are Joining the team, that is:
[0157]
[0158] After the dequeue and enqueue operations are completed, the last IJ elements of υ K-IJ+1 ,υ K-IJ+2 ,…,υ K Corresponding to the view generated by the current small batch; and the previous K-IJ elements υ1,υ2,…,υ K-IJ For old views, they are computed in previous mini-batches and retained until now. These views are the key images of the current mini-batch.
[0159] Now, each anchor a i Consider the ordering of key images as query images. The calculation of the approximate normalized discounted cumulative gain in step S101 only involves one query image, so none of the symbols there have superscripts. Where necessary, this embodiment will add a superscript "i" to some symbols to distinguish them.
[0160] From formula (11), we can see that for a i In terms of K-(I-i+1)J+1 ,υ K-(I-i+1)J+2 ,…,υ K-(I-i)J corresponds to the positive view, and the rest of the embeddings in the queue correspond to the negative views. Let a i The correlation with the positive image is 1, and the correlation with the negative image is 0, that is:
[0161]
[0162] remember but The cumulative loss gain is:
[0163]
[0164]
[0165] In fact, this is a normalization constant that has nothing to do with i, so let:
[0166]
[0167] remember in, Measured anchor a i The predicted similarity between the k-th key image is The approximate cumulative loss gain is:
[0168]
[0169] then, The approximate normalized discounted cumulative gain is:
[0170]
[0171] Define the view ranking loss as:
[0172]
[0173] The anchor image and view image are input into the anchor branch and view branch respectively, and the view ranking loss is used for training to obtain the view ranking network. The training process is as follows:
[0174]
[0175]
[0176] S106: Synthesize a mixed anchor image using the anchor image.
[0177] First, from the anchor set {a i Randomly select images from |i=1,2,…,I} to generate L pairs of anchor image pairs (l=1,2,…,L;i l ,i′ l ∈{1,2,…,I} and i l ≠i′ l ), where (.;.) represents an image pair. Then, for each l, randomly generate Mixing coefficient To mix and get Mixed anchor images:
[0178]
[0179] Obviously, The bigger, The more like Like the anchor image, the mixed anchor image is also the input of the anchor branch.
[0180] S107: Input the anchor image and the mixed anchor image into the anchor branch, and train them using the mixed anchor ranking loss to obtain a mixed anchor ranking network.
[0181] The anchor image in As the query image, regarded as The key image of . Note the mixed anchor image The normalized feature embedding of And order in, Finally, let in, express and The similarity level is equal to exist The position in the ascending ranking of is reduced by 1, so Similar to formula (14), the normalization constant is:
[0182]
[0183] After substitution, we can get:
[0184]
[0185]
[0186] Define the mixed anchor ranking loss as:
[0187]
[0188] The anchor image and the mixed anchor image are input into the anchor branch, and the mixed anchor ranking loss is used for training to obtain the mixed anchor ranking network, which is a single-branch network. The training process is as follows:
[0189]
[0190] S108 , integrating the view sorting network and the mixed-anchor sorting network to obtain a view mixed-anchor sorting network.
[0191] Loss function of view mixed anchor ranking network
[0192]
[0193] is called the “total loss”, where λ ≥ 0.
[0194] S109 , inputting the view image, the anchor image, and the mixed-anchor image into a view-mixed-anchor ranking network, and performing training using the total loss to obtain a trained view-mixed-anchor ranking network.
[0195] The training process of the view-mixed-anchor ranking network is shown in FIG. 8 and can also be described by the following algorithm. Figure 3
[0196]
[0197]
[0198] It can be seen that, if lines 8-11, 13, 18, and 19 are removed, the training process of the view ranking network is obtained; if lines 2, 3, 7, 14-17, 19, 21, and 22 are removed, the training process of the mixed-anchor ranking network is obtained.
[0199] S110, using the trained view-mixed-anchor ranking network to classify a remote sensing scene image.
[0200] After the unsupervised pre-training is completed, the view-mixed-anchor ranking network can be applied to many downstream tasks. The downstream task considered in this embodiment is remote sensing scene classification. The application method is as follows: the anchor branch is regarded as an off-the-shelf feature extractor, the features of a remote sensing scene image are extracted, and then a classifier is trained to perform the classification task.
[0201] The downstream image library used in this embodiment is the public multispectral remote sensing scene dataset EuroSAT. It includes 27,000 labeled Sentinel-2 images, the image size is 64x64, the spatial resolution is 10 meters, and the 13 bands cover the wavelength range of 443 to 2,190 nanometers of the electromagnetic spectrum. EuroSAT involves 10 categories: annual crops, forests, herbaceous plants, highways, industrial buildings, pastures, perennial crops, residential buildings, rivers, oceans, and lakes. An image sample is shown in FIG. 9. Figure 4
[0202] In this embodiment, EuroSAT is randomly divided into a training set, a validation set, and a test set in a ratio of 7:1:2. The training set is used to train the classifier, the validation set is used to determine the values of the hyperparameters, and the test set is used to calculate the classification accuracy.
[0203] In this embodiment, the stochastic gradient descent method is selected as the optimization algorithm. The hyperparameter settings of the training process are as follows:
[0204] 1) Epoch: 200;
[0205] 2) Batch size: 32 (i.e., I = 32);
[0206] 3) Momentum: 0.9;
[0207] 4) Weight decay: 0.0001;
[0208] 5) Learning rate: the initial learning rate is 0.01, and the learning rate is halved every 40 epochs.
[0209] The model-related parameters are set as follows: τ = 0.999, θ = 10, L = 50,
[0210] In addition, the classifier used in this embodiment is random forest.
[0211] The following describes how to apply the view-mixed anchor ranking network to remote sensing scene classification.
[0212] First, determine the values of the number of views J and the queue length K. The effects of different values of J and K on the classification performance of the view sorting network are examined on the validation set. The results are as follows: Figure 5 As shown (note: K ≥ IJ). It can be seen that, in general, the larger K, the better. However, considering the computational burden, this embodiment only examines the case where K ≤ 4,096. Based on experimental results, this embodiment sets J = 16 and K = 4,096.
[0213] Secondly, determine the optimal value of λ. In this embodiment, by changing the value of λ, six view mixed anchor sorting networks are trained and their average classification accuracy on the validation set is calculated. The results are as follows: Figure 5 As shown in the figure, as the value of λ increases, classification performance first increases and then decreases. This demonstrates that, with an appropriate balance, the two tasks of view ranking and mixed-anchor ranking can achieve a mutually beneficial and optimal cooperation. Based on experimental results, this example sets λ to 0.4. This value indicates that the importance of the view ranking loss is greater than that of the mixed-anchor ranking loss (the ratio is 1:0.4).
[0214] Finally, a random forest classifier was trained on the training set and tested on the test set. This example performed 100 rounds of experiments on EuroSAT, calculating the average classification accuracy and standard deviation. The results are shown in Table 1. As can be seen, in terms of classification performance, the View Ranking Network outperforms the most advanced unsupervised learning models, while the View Anchor Ranking Network outperforms the View Ranking Network. This demonstrates that the method of this example is effective in practice and that the View Ranking Network and the View Anchor Ranking Network are promising unsupervised learning models in the remote sensing field.
[0215] Table 1 Classification performance comparison
[0216]
[0217] Those skilled in the art will appreciate that all or part of the steps in the above-mentioned embodiment method may be implemented by instructing related hardware through a program, and the corresponding program may be stored in a computer-readable storage medium.
[0218] It should be noted that although the method operations of the above embodiments are described in a particular order in the accompanying drawings, this does not require or imply that the operations must be performed in this particular order, or that all of the illustrated operations must be performed to achieve the desired results. Rather, the steps described may be performed in a different order. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into a single step, and / or a single step may be broken down into multiple steps.
[0219] Example 2:
[0220] like Figure 7 As shown, this embodiment provides a remote sensing scene classification system based on unsupervised ranking learning, which includes an establishment module 701, a creation module 702, a generation module 703, a construction module 704, a first training module 705, a synthesis module 706, a second training module 707, an integration module 708, a third training module 709 and a classification module 710. The specific functions of each module are as follows:
[0221] Establishing module 701 is used to establish a differentiable optimization target by improving the normalized discounted cumulative gain, and the optimization target is used to construct the view ranking loss and the mixed anchor ranking loss.
[0222] The creation module 702 is used to obtain remote sensing images and crop images of a preset size from the remote sensing images to create an original image set.
[0223] The generating module 703 is configured to generate an anchor image and a view image according to the original image set.
[0224] Construction module 704 is used to construct the anchor branch and the view branch.
[0225] The first training module 705 is used to input the anchor image and the view image into the anchor branch and the view branch respectively, and perform training using the view ranking loss to obtain a view ranking network.
[0226] The synthesis module 706 is configured to synthesize a mixed anchor image using the anchor image.
[0227] The second training module 707 is used to input the anchor image and the mixed anchor image into the anchor branch, and perform training using the mixed anchor ranking loss to obtain a mixed anchor ranking network.
[0228] The integration module 708 is configured to integrate the view sorting network and the hybrid anchor sorting network to obtain a view hybrid anchor sorting network.
[0229] The third training module 709 is used to input the view image, anchor image and mixed anchor image into the view mixed anchor ranking network, and train it using the total loss to obtain a trained view mixed anchor ranking network, where the total loss includes the view ranking loss and the mixed anchor ranking loss.
[0230] The classification module 710 is configured to classify the remote sensing scene image by using the trained view mixed anchor ranking network.
[0231] The specific implementation of each module in this embodiment can refer to the above-mentioned embodiment 1, which will not be repeated here; it should be noted that the system provided in this embodiment is only exemplified by the division of the above-mentioned functional modules, and in actual application, the above-mentioned functions can be allocated to different functional modules to complete, that is, the internal structure is divided into different functional modules to complete all or part of the functions described above.
[0232] It can be understood that the terms "first", "second", etc. used in the above system can be used to describe various modules, but these modules are not limited by these terms. These terms are only used to distinguish the first module from another module. For example, without departing from the scope of the present application, the first training module can be called the second training module, and similarly, the second training module can be called the first training module, the first training module and the second training module are both training modules, but they are not the same training module.
[0233] Embodiment 3:
[0234] As shown in Figure 8 The computer device provided in this embodiment can be a server, a computer, etc., which includes a processor 802, a memory, an input device 803, a display 804 and a network interface 805 connected through a system bus 801. The processor 802 is configured to provide computing and control capabilities, the memory includes a non-volatile storage medium 806 and an internal memory 807, the non-volatile storage medium 806 stores an operating system, a computer program and a database, the internal memory 807 provides an environment for the running of the operating system and the computer program in the non-volatile storage medium 806, and the computer program is executed by the processor 802 to implement the remote sensing scene classification method of the above-mentioned embodiment 1, as follows:
[0235] By improving the normalized discounted cumulative gain, a derivable optimization objective is established, which is used to construct a view ranking loss and a mixed anchor ranking loss;
[0236] Obtaining a remote sensing image, and cropping an image of a preset size from the remote sensing image to create an original image set;
[0237] According to the original image set, a ship anchor image and a view image are generated;
[0238] A ship anchor branch and a view branch are constructed;
[0239] The ship anchor image and the view image are input into the ship anchor branch and the view branch respectively, the view sorting loss is used for training, and a view sorting network is obtained;
[0240] The ship anchor image is used to synthesize a mixed anchor image;
[0241] The ship anchor image and the mixed anchor image are input into the ship anchor branch, the mixed anchor sorting loss is used for training, and a mixed anchor sorting network is obtained;
[0242] The view sorting network and the mixed anchor sorting network are integrated to obtain a view mixed anchor sorting network;
[0243] The view image, the ship anchor image and the mixed anchor image are input into the view mixed anchor sorting network, the total loss is used for training, and a trained view mixed anchor sorting network is obtained, wherein the total loss comprises the view sorting loss and the mixed anchor sorting loss;
[0244] The trained view mixed anchor sorting network is used for classifying a remote sensing scene image.
[0245] Embodiment 4:
[0246] The embodiment provides a storage medium, which is a computer readable storage medium, and stores a computer program, wherein the computer program is executed by a processor to realize the remote sensing scene classification method in the above embodiment 1, and the method comprises the following steps:
[0247] By improving the normalized loss accumulation gain, a derivable optimization target is established, and the optimization target is used to construct the view sorting loss and the mixed anchor sorting loss;
[0248] An original image set is created by obtaining a remote sensing image and cropping an image with a preset size from the remote sensing image;
[0249] According to the original image set, a ship anchor image and a view image are generated;
[0250] The ship anchor branch and the view branch are constructed;
[0251] The ship anchor image and the view image are input into the ship anchor branch and the view branch respectively, the view sorting loss is used for training, and a view sorting network is obtained;
[0252] The ship anchor image is used to synthesize a mixed anchor image;
[0253] The ship anchor image and the mixed anchor image are input into the ship anchor branch, the mixed anchor sorting loss is used for training, and a mixed anchor sorting network is obtained;
[0254] The view sorting network and the mixed anchor sorting network are integrated to obtain a view mixed anchor sorting network;
[0255] Inputting the view image, the anchor image, and the mixed anchor image into a view mixed anchor ranking network, and training the network using a total loss to obtain a trained view mixed anchor ranking network, wherein the total loss includes a view ranking loss and a mixed anchor ranking loss;
[0256] Classify remote sensing scene images using the trained view-mixed anchor ranking network.
[0257] It should be noted that the computer-readable storage medium of the present embodiment may be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.
[0258] In this embodiment, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. Furthermore, in this embodiment, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries a computer-readable program. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable storage medium other than a computer-readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device. The computer program contained on a computer-readable storage medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.
[0259] The computer readable storage medium described above can be written in one or more programming languages or combinations of languages for executing the computer program of the present embodiment, including object oriented programming languages such as Java, Python, C++, and conventional procedural programming languages such as C language or similar programming languages. The program can be executed entirely on the user computer, partially on the user computer, as a separate software package, partially on the user computer and partially on a remote computer, or entirely on a remote computer or server. In the case involving a remote computer, the remote computer can be connected to the user computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, through the Internet by using an Internet service provider).
[0260] To sum up, the present application proposes a new original image creation method according to the characteristics of remote sensing images, which avoids the situation that the front view is mistaken for the negative view to a certain extent; at the same time, the present application generates multiple front views for a ship anchor image, predicts the ranking of all front and negative views by using an unsupervised ranking method, and forces the network to learn more discriminative deep features; in addition, the present application generates a group of mixed images with different proportions through image mixing, and requires the network to predict the ranking of the mixed images according to their similarity with the ship anchor, which further enhances the generalization ability of the network; compared with the prior art, the method disclosed by the present application can obtain higher classification accuracy.
[0261] The above is only a preferred embodiment of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can make equivalent replacement or change according to the technical scheme and the inventive concept of the present application within the scope disclosed by the present application, which still belongs to the protection scope of the present application.
Claims
1. A remote sensing scene classification method based on unsupervised ranking learning, characterized in that: The method comprises: By improving the normalized discounted cumulative gain, a differentiable optimization objective is established, which is used to construct the view ranking loss and the mixed anchor ranking loss. Acquire remote sensing images, and crop images of preset sizes from the remote sensing images to create an original image set; Generate anchor images and view images based on the original image set; Build anchor branches and view branches; The anchor image and view image are input into the anchor branch and view branch respectively, and the view ranking loss is used for training to obtain the view ranking network. The view ranking loss is Among them, ω a is the network parameter of the anchor branch, ω v is the network parameter of the view branch, B is the given original image set, For s i The corresponding rankings, Represents the anchor image a i and the kth key image υ k The prediction similarity between Represents the anchor image a i and the kth key image υ k the relevant level of Use anchor images to synthesize mixed anchor images; The anchor image and the mixed anchor image are input into the anchor branch, and the mixed anchor ranking loss is used for training to obtain the mixed anchor ranking network. The mixed anchor ranking loss is in, for The corresponding rankings, Anchor image pair As a query image, Mixed anchor image The normalized features of Represents mixed anchor image With query image the relevant level of Integrate the view ranking network and the mixed anchor ranking network to obtain the view mixed anchor ranking network; Inputting the view image, the anchor image, and the mixed anchor image into a view mixed anchor ranking network, and training the network using a total loss to obtain a trained view mixed anchor ranking network, wherein the total loss includes a view ranking loss and a mixed anchor ranking loss; Classify remote sensing scene images using the trained view-mixed anchor ranking network; Generating the anchor image and the view image according to the original image set specifically includes: Given a small batch From the original image Cut out a piece of size w×h from the center to get the anchor image a i ; Create multiple front views for each anchor: According to the principle that the horizontal and vertical center distances between the anchor and the view are no more than d pixels, Randomly crop J view images of size w×h from As a front view, view image satisfy in, and Respectively The horizontal and vertical coordinates of the center, x i and y i Respectively The horizontal and vertical coordinates of the center.
2. The remote sensing scene classification method according to claim 1, characterized in that: The above method establishes a differentiable optimization objective by improving the normalized discounted cumulative gain, specifically including: Let π be any ranking list of K key images, then the cumulative loss gain of π is: Where π(k) represents the position of the kth key image in π; The rankings corresponding to the network's predicted value s of the similarity of the k-th key image and the correlation level r of the k-th key image are π s and π r , then π s The normalized discounted cumulative gain is: Among them, Ψ(π s ,r)∈(0,1]; When θ→+∞, the Sigmoid function Infinitely approaching the unit step function According to π s (k)=1+∑ k′≠k h(s k′ -s k ),have Let μ be the right side of the equation s (k), using μ s (k) replaces π s (k), we get π s Approximate discounted cumulative gain of: Then we get π s The approximate normalized discounted cumulative gain of : in, It is a derivable optimization objective that is only targeted at one query image.
3. The remote sensing scene classification method according to claim 2, characterized in that: The view ranking loss is defined as follows: The anchor image a i and view image Input the anchor branch and view branch respectively, and obtain the normalized feature embedding α through forward propagation i and Where, i = 1, 2, ..., I; j = 1, 2, ..., J; Assume that the key image of the previous batch corresponds to a feature embedding sequence υ=(υ1,υ2,…,υ K ), treat υ as a queue with a queue length of K, and let the oldest IJ embed α i Out of the queue, the IJ embeddings just calculated Joining the team, that is: After the dequeue and enqueue operations are completed, the last IJ elements of υ K-IJ+1 ,υ K-IJ+2 ,…,υ K Corresponding to the view generated by the current small batch, the previous K-IJ elements υ1,υ2,…,υ K-IJ Corresponding to the old views; these views are the key images of the current mini-batch; Each anchor image a i As a query image; for the anchor image a i In terms of K-(I-i+1)J+1 ,υ K-(I-i+1)J+2 ,…,υ K-(I-i)J corresponds to the positive view, and the rest of the embeddings in the queue correspond to the negative views; Assume that the anchor image a i The correlation with the positive image is 1, and the correlation with the negative image is 0, that is: remember but The cumulative loss gain is: make And remember in, The anchor image a is measured i The predicted similarity between the k-th key image is The approximate cumulative loss gain is: The approximate normalized discounted cumulative gain is: Define the view ranking loss as:
4. The remote sensing scene classification method according to claim 2, characterized in that: The definition process of the mixed anchor ranking loss is as follows: The anchor image in As the query image, the anchor image regarded as The key image of Mixed anchor images The normalized feature embedding of And order in, make in, express and Similar levels, equal exist The position in the ascending ranking list is minus 1, that is, Let the normalization constant be: After substitution, we get: The mixed anchor ranking loss is defined as:
5. The remote sensing scene classification method according to claim 1, characterized in that: The total loss is as follows: in, is the view ranking loss, is the mixed anchor ranking loss.
6. The remote sensing scene classification method according to any one of claims 1 to 5, characterized in that: The anchor image and view image are input into the anchor branch and the view branch respectively. During the training process using the view ranking loss, the network parameters of the anchor branch are updated by backpropagation. The network parameters of the view branch are updated as follows: oh v ←to v +(1-t)ω a Among them, ω a is the network parameter of the anchor branch, ω v is the network parameter of the view branch, and τ∈[0,1) is the momentum coefficient.
7. The remote sensing scene classification method according to any one of claims 1 to 5, characterized in that: The step of cropping an image of a preset size from the remote sensing image to create an original image set specifically includes: According to the principle that the horizontal or vertical center distance is not less than D pixels, randomly cut out images of size W×H from the remote sensing image to create the original image set Among them, X i and Y i Respectively The horizontal and vertical coordinates of the center in the downloaded image.
8. The remote sensing scene classification method according to any one of claims 1 to 5, characterized in that: The method of synthesizing a mixed anchor image by using an anchor image specifically includes: From the anchor image set {a i Randomly select images from |i=1,2,…,I} to generate L pairs of anchor image pairs Where (.;.) represents an image pair, l = 1, 2, …, L; and For each l, randomly generate Mixing coefficient To mix and get Mixed anchor images:
9. A remote sensing scene classification system based on unsupervised ranking learning, characterized in that: The system comprises: Establish a module for establishing a differentiable optimization objective by improving the normalized discounted cumulative gain, wherein the optimization objective is used to construct the view ranking loss and the mixed anchor ranking loss; Creating a module for acquiring remote sensing images and cropping images of a preset size from the remote sensing images to create an original image set; A generation module, used for generating an anchor image and a view image according to an original image set; Building modules for building anchor branches and view branches; The first training module is used to input the anchor image and the view image into the anchor branch and the view branch respectively, and train them using the view ranking loss to obtain a view ranking network. The view ranking loss is Among them, ω a is the network parameter of the anchor branch, ω v is the network parameter of the view branch, B is the given original image set, For s i The corresponding rankings, Represents the anchor image a i and the kth key image υ k The prediction similarity between Represents the anchor image a i and the kth key image υ k the relevant level of A synthesis module, used for synthesizing a mixed anchor image using the anchor image; The second training module is used to input the anchor image and the mixed anchor image into the anchor branch and train using the mixed anchor ranking loss to obtain the mixed anchor ranking network. The mixed anchor ranking loss is in, for The corresponding rankings, Anchor image pair As a query image, Mixed anchor image The normalized features of Represents mixed anchor image With query image the relevant level of An integration module is used to integrate the view ranking network and the mixed-anchor ranking network to obtain the view mixed-anchor ranking network; a third training module, configured to input the view image, the anchor image, and the mixed-anchor image into a view-mixed-anchor ranking network, and perform training using a total loss to obtain a trained view-mixed-anchor ranking network, wherein the total loss includes a view ranking loss and a mixed-anchor ranking loss; The classification module is used to classify remote sensing scene images using the trained view-mixed anchor ranking network; Generating the anchor image and the view image according to the original image set specifically includes: Given a small batch From the original image Cut out a piece of size w×h from the center to get the anchor image a i ; Create multiple front views for each anchor: According to the principle that the horizontal and vertical center distances between the anchor and the view are no more than d pixels, Randomly crop J view images of size w×h from As a front view, view image satisfy in, and Respectively The horizontal and vertical coordinates of the center, x i and y i Respectively The horizontal and vertical coordinates of the center.
10. A storage medium storing a program, characterized in that: When the program is executed by a processor, the remote sensing scene classification method according to any one of claims 1 to 8 is implemented.