Training methods for scene classification models, remote sensing image scene classification methods and equipment

Abstract

This invention provides a training method for a scene classification model, a remote sensing image scene classification method, and an apparatus. The specific implementation includes: acquiring multiple first image blocks and their respective sample labels; using a feature extraction network to extract features from each first image block to obtain local features; generating a visual word histogram for each first image block based on its local features; using a first classification network and a second classification network based on the visual word histogram to obtain a first classification result and a second classification result corresponding to each scene to be classified; determining a first loss value for the first classification result and a second loss value for the second classification result based on the correlation between the first classification result, the second classification result, the sample labels, and the multiple scenes to be classified; and adjusting the parameters of the first classification network and the second classification network according to the first loss value and the second loss value, respectively.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, specifically to the technical fields of image processing, deep learning, and computer vision, and particularly to a training method for a scene classification model, a remote sensing image scene classification method, an electronic device, and a storage medium. Background Technology

[0002] Remote sensing imagery is image data captured by high-altitude sensors scanning and photographing the Earth's surface. It contains detailed information on the composition and distribution of ground features. However, due to the complexity of remote sensing scenes, the small size of ground features, and the chaotic spatial distribution of features, there are significant differences between images of the same type of remote sensing scene, while the differences between images of different types of remote sensing scenes are relatively small. In addition, remote sensing images usually contain multiple categories of scenes, and each category of scene often occupies only a small area in the image, making scene classification even more difficult. Therefore, improving the accuracy of remote sensing image scene classification remains a very challenging task. Summary of the Invention

[0003] This invention provides a training method for a scene classification model, a method for classifying remote sensing image scenes, an electronic device, a storage medium, and a computer program product.

[0004] According to one aspect of the present invention, a training method for a scene classification model is provided. The scene classification model includes a feature extraction network, a first classification network, and a second classification network. The method includes: acquiring a first sample image set, the first sample image set including multiple first image blocks and sample labels for each of the multiple first image blocks, the sample labels being used to characterize the scene category to which multiple scenes to be classified in the corresponding first image block belong; extracting features from each first image block in the first sample image set using the feature extraction network to obtain local features of each first image block; wherein the feature extraction network is obtained by training an autoencoder based on a second sample image set; generating a visual word histogram for each first image block based on the local features of each first image block; obtaining a first classification result and a second classification result corresponding to each scene to be classified using the first classification network and the second classification network, respectively, based on the visual word histogram; determining a first loss value for the first classification result and a second loss value for the second classification result according to the correlation between the first classification result, the second classification result, the sample labels, and the multiple scenes to be classified; and adjusting the parameters of the first classification network and the second classification network, respectively, according to the first loss value and the second loss value.

[0005] According to another aspect of the present invention, a remote sensing image scene classification method is provided, comprising: acquiring multiple third image patches based on a remote sensing image to be classified; extracting features from each third image patch using a feature extraction network in a scene classification model to obtain local features of each third image patch; generating a visual word histogram for each third image patch based on the local features of each third image patch; obtaining a first classification result and a second classification result corresponding to each scene to be classified based on each visual word histogram using a first classification network and a second classification network in the scene classification model; and obtaining a scene classification result for the remote sensing image to be classified based on the first classification result and the second classification result corresponding to each scene to be classified; wherein the scene classification model is trained according to the training method of the scene classification model described in the above embodiments.

[0006] According to another aspect of the present invention, an electronic device is provided, comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors perform the method described above.

[0007] According to another aspect of the present invention, a computer-readable storage medium is provided having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the method described above.

[0008] According to another aspect of the present invention, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the method described above. Attached Figure Description

[0009] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly described below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0010] Figure 1 This is a flowchart of the training method for the scene classification model according to an embodiment of the present invention;

[0011] Figure 2A This is a schematic diagram illustrating the construction of the correlation between multiple scenes to be classified according to an embodiment of the present invention;

[0012] Figure 2B This is a schematic diagram illustrating the confusion relationship between multiple unclassified scenarios in an embodiment of the present invention;

[0013] Figure 3 This is a schematic diagram of the training method of the scene classification model according to an embodiment of the present invention;

[0014] Figure 4 This is a flowchart of the remote sensing image scene classification method according to an embodiment of the present invention;

[0015] Figure 5 This is a block diagram of a training device for a scene classification model according to an embodiment of the present invention;

[0016] Figure 6 This is a block diagram of a remote sensing image scene classification device according to an embodiment of the present invention;

[0017] Figure 7 This is a block diagram of an electronic device applicable to a training method for a scene classification model and a scene classification method for remote sensing images, according to embodiments of the present invention. Detailed Implementation

[0018] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments and accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention are within the scope of protection of the present invention.

[0019] It should be noted that the sequence numbers of the operations in the following methods are for descriptive purposes only and should not be considered as indicating the execution order of the operations. Unless explicitly stated otherwise, the method does not need to be executed in the exact order shown.

[0020] In the technical solution of this invention, the collection, storage, use, processing, transmission, provision, disclosure and application of data (including but not limited to user personal information) all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0021] In the technical solution of this invention, the authorization or consent of the data owner is obtained before acquiring or collecting relevant data.

[0022] Figure 1 This is a flowchart of a training method for a scene classification model according to an embodiment of the present invention.

[0023] Please see Figure 1 The training method for the scene classification model includes operations S110~S160. In this embodiment of the invention, the scene classification model includes, for example, a feature extraction network, a first classification network, and a second classification network.

[0024] In operation S110, a first sample image set is obtained, which includes multiple first image blocks and sample labels for each of the multiple first image blocks.

[0025] In operation S120, a feature extraction network is used to extract features from each first image block in the first sample image set to obtain the local features of each first image block.

[0026] In operation S130, a visual word histogram is generated for each first image block based on the local features of each first image block.

[0027] In operation S140, the first classification network and the second classification network are used respectively based on the visual word histogram to obtain the first classification result and the second classification result corresponding to each scene to be classified.

[0028] In operation S150, based on the first classification result, the second classification result, the sample label, and the correlation between multiple scenarios to be classified, the first loss value for the first classification result and the second loss value for the second classification result are determined.

[0029] In operation S160, the parameters of the first classification network and the second classification network are adjusted according to the first loss value and the second loss value, respectively.

[0030] In this embodiment of the invention, the first sample image set may be obtained, for example, by segmenting multiple remote sensing images carrying scene labels. For example, for each remote sensing image, the remote sensing image is uniformly divided into p*q rows and columns to obtain p*q first image blocks, where p and q are both positive integers. For example, p and q are both 512.

[0031] Of course, the present invention is not limited to the methods described above for segmenting remote sensing images. In another example, different segmentation sizes can be used to segment different remote sensing images. For example, a remote sensing image can be uniformly gridded according to the number of rows and columns p1*q1, and another remote sensing image can be uniformly gridded according to the number of rows and columns p2*q2, where p1≠p2 and / or q1≠q2.

[0032] Each remote sensing image may include multiple scenes to be classified. The scene labels carried by each remote sensing image can be used to characterize the scene category to which each scene to be classified belongs. For example, a remote sensing image may include farmland, grassland, and forest scenes to be classified. The scene labels carried by this remote sensing image may include scene labels corresponding to farmland, grassland, and forest.

[0033] Since the first sample image set is obtained by segmenting multiple remote sensing images carrying scene labels, the sample labels for each first image block can be obtained based on the scene labels carried by the multiple remote sensing images. Accordingly, the sample labels of each first image block are used to characterize the scene category to which one or more scenes to be classified in the corresponding first image block belong.

[0034] After acquiring the first sample image set, a feature extraction network is used to extract features from each first image patch in the first sample image set to obtain the local features of each first image patch. Local features can characterize the low-level visual features of the corresponding image patch, such as color features, texture features, and structural features. In this embodiment of the invention, the feature extraction network is obtained by training the autoencoder using a second sample image set. The method for obtaining the feature extraction network will be described later.

[0035] Next, based on the local features of each first image patch, a visual word histogram is generated for each first image patch.

[0036] In embodiments of the present invention, a clustering algorithm such as K-means can be used to cluster the local features of each first image block to obtain at least one corresponding cluster center. Each cluster center is treated as a visual word, and a visual dictionary is constructed based on the at least one visual word and its corresponding sample label. Then, visual word mapping is performed on each first image block in the first sample image set to generate a visual word distribution map for each first image block. Next, a Local Binary Pattern (LBP) transform is performed on each of the generated visual word distribution maps to obtain a visual word histogram for each image block.

[0037] In some embodiments, a trained visual bag-of-words model can also be used to generate the aforementioned visual word histograms based on the local features of each first image patch. The training process of the visual bag-of-words model is the same as or similar to that described in related technologies, and will not be repeated here.

[0038] In this embodiment of the invention, a first classification network can be used to output a first classification result corresponding to each scene to be classified in the first image block based on the visual word histogram of each first image block. A second classification network can be used to output a second classification result corresponding to each scene to be classified in the first image block based on the visual word histogram of each first image block.

[0039] The first classification result can be used, for example, to represent a first scene category score for each scene to be classified. The second classification result can be used, for example, to represent a second scene category score for each scene to be classified. In embodiments of the present invention, the first scene category score and the second scene category score can be, for example, values ​​between 0 and 1.

[0040] After obtaining the visual word histogram of each first image block, the first classification network and the second classification network can be used respectively to obtain the first classification result and the second classification result corresponding to each scene to be classified in each first image block based on the visual word histogram of each first image block.

[0041] It is understandable that there may be certain relationships between the multiple scenes to be classified included in each first image block. These relationships are mainly reflected in the following two aspects.

[0042] First, there is a certain correlation between multiple scenes to be classified. For example, when classifying a first image patch, if a scene to be classified usually appears together with another scene(s), then it can be considered that the scene to be classified is correlated with the other scene(s). Therefore, if the sample labels of the first image patch contain the label of the scene to be classified, it is highly likely that the sample labels also contain the labels of the other scene(s). For example, if the scenes to be classified, "river" and "forest," usually appear in the same first image patch, then the scenes to be classified, "river," and "forest" can be considered correlated. If the sample labels of the first image patch contain the label of the scene to be classified, it is highly likely that it also contains the label of the scene to be classified, "forest." In this way, the correlation between individual scenes to be classified can be used to construct the correlation between multiple scenes to be classified, so as to use the correlation between multiple scenes to classify related scenes, thereby improving the efficiency and accuracy of scene classification.

[0043] Secondly, there is a certain degree of confusion among multiple scenes to be classified. Due to various factors such as shooting angle, background, lighting, and scale, there may be easily confused scenes among multiple scenes to be classified. The interference from these easily confused scenes can significantly affect the training effect of the classification network, resulting in lower classification accuracy.

[0044] Since the relationships between multiple scenes to be classified can affect the training performance of a classification network, these relationships can be incorporated into the iterative training of the classification network during the classification training process of two networks to improve its performance.

[0045] In this embodiment of the invention, a first loss value for the first classification result and a second loss value for the second classification result can be determined based on the first classification result, the second classification result, the sample labels, and the correlation between multiple scenes to be classified. Then, the parameters of the first classification network and the second classification network are adjusted according to the first and second loss values, respectively. By incorporating the correlation between multiple scenes to be classified into the iterative training of the two classification networks, the classification networks can better learn the features of each scene to be classified, thereby obtaining a classification network with higher classification accuracy, and thus improving the precision and accuracy of the scene classification model.

[0046] According to an embodiment of the present invention, the correlation between multiple scenarios to be classified can be determined by the following operations.

[0047] For example, for any two unclassified scenarios among multiple unclassified scenarios, such as a first unclassified scenario and a second unclassified scenario, the degree of correlation between the first unclassified scenario and the second unclassified scenario can be determined. If the degree of correlation is greater than or equal to a preset threshold, then the first unclassified scenario and the second unclassified scenario are determined to be correlated.

[0048] In embodiments of the present invention, for example, cosine similarity can be used to determine the correlation between a first scene to be classified and a second scene to be classified. In some embodiments, other suitable methods can also be used to determine the correlation between two scenes to be classified, which are not limited here.

[0049] In this embodiment of the invention, the preset threshold can be set according to actual needs, and is not limited here. In one example, the preset threshold can be any value or range of values ​​from 0.5 to 1.0.

[0050] Next, based on the correlation between any two scenarios to be classified, we can construct the correlation between multiple scenarios to be classified.

[0051] Figure 2A This is a schematic diagram illustrating the construction of correlations between multiple scenes to be classified according to an embodiment of the present invention. See below for reference. Figure 2A The process of constructing the correlation between multiple scenarios to be classified is illustrated by example.

[0052] like Figure 2A As shown in 210, multiple scenarios to be classified include, for example, scenario 1 to be classified, scenario 2 to be classified, ..., scenario 6 to be classified (for example only).

[0053] Based on the above method, the correlation between any two unclassified scenarios among multiple unclassified scenarios is determined as follows: Unclassified scenario 1 is correlated with unclassified scenario 2 and unclassified scenario 3 respectively; unclassified scenario 2 is correlated with unclassified scenario 1, unclassified scenario 3 and unclassified scenario 5 respectively; unclassified scenario 3 is correlated with unclassified scenario 1, unclassified scenario 2 and unclassified scenario 4 respectively; unclassified scenario 6 is not correlated with any of unclassified scenarios 1 to unclassified scenario 5.

[0054] Based on the correlation between any two scenarios to be classified, we can construct a model as follows: Figure 2A The diagram in Figure 220 illustrates the correlations between multiple scenes to be classified. In the correlations shown in Figure 220, if two scenes to be classified are directly connected by a line, it indicates that the two scenes are correlated; otherwise, they are considered uncorrelated. For example, a direct line connection between scene 1 and scene 3 indicates that scene 1 and scene 3 are correlated. Conversely, a line connecting scene 1 and scene 4 through scene 3 indicates that scene 1 and scene 4 are uncorrelated.

[0055] In embodiments of the present invention, the confusion relationships between various scenes to be classified can be constructed in advance based on prior information of remote sensing scene classification and statistical results of easily confused remote sensing scenes.

[0056] Figure 2B This is a schematic diagram illustrating the confusion relationships between multiple unclassified scenarios according to an embodiment of the present invention. See below for reference. Figure 2B An example is provided to illustrate the confusion relationship.

[0057] Continue using the above Figure 2A Examples of multiple unclassified scenarios, such as scenario 1, scenario 2, ..., scenario 6. For scenarios 1 through 6, a classification can be constructed as follows: Figure 2B The obfuscated relationship is shown.

[0058] like Figure 2B As shown, each directed edge ( Figure 2B Two nodes connected by an arrow (indicated by direction) (e.g.) Figure 2B The circles representing the unclassified scenarios indicate two easily confused scenarios. The confusion relationship represented by a one-way edge is unidirectional; that is, the starting node of a one-way edge is easily confused with the node it points to, while the node it points to is not easily confused with the starting node. For example, Figure 2BThe directed edge between scenario 1 and scenario 4 to be classified represents that scenario 1 is easily confused with scenario 4, while scenario 4 is not easily confused with scenario 1. Similarly, the directed edge between scenario 4 and scenario 6 to be classified represents that scenario 4 is easily confused with scenario 6, while scenario 6 is not easily confused with scenario 4.

[0059] also, Figure 2B The bidirectional edges shown represent a bidirectional confusion relationship. For example, a bidirectional edge between scenario 4 and scenario 5 to be classified can indicate that scenario 4 is easily confused with scenario 5, and scenario 5 is also easily confused with scenario 4. In this embodiment of the invention, a bidirectional edge can be represented by splitting it into two unidirectional edges.

[0060] It should be noted that, in Figure 2A and Figure 2B The number of scenarios to be classified shown is merely an example, and the technical solution of the present invention is not limited thereto. Specifically, the number of scenarios to be classified in the confusion relationship and the correlation relationship between each scenario to be classified can be set according to actual needs.

[0061] After obtaining the correlation and confusion relationships between the various scenarios to be classified, classification labels can be generated based on the correlation, confusion relationships, and sample labels to train the first and second classification networks.

[0062] According to an embodiment of the present invention, the first classification network includes a first classification sub-network corresponding to each of the plurality of scenes to be classified, and the second classification network includes a second classification sub-network corresponding to each of the plurality of scenes to be classified. In this embodiment of the present invention, each first classification sub-network and each second classification sub-network can be, for example, a binary classification network.

[0063] For example, in an example with multiple scenarios to be classified, including scenarios 1 to 6, a first classification sub-network and a second classification sub-network can be set up corresponding to each of the six scenarios. Each first classification sub-network is used to output a first classification result for one scenario, and each second classification sub-network is used to output a second classification result for one scenario. In this embodiment of the invention, the first classification result and the second classification result are the same as or similar to the definitions described above, and will not be repeated here.

[0064] It should be understood that the number of scenarios to be classified, the number of first classification sub-networks, and the number of second classification sub-networks in this invention are not limited to the above examples, and can be set according to actual needs.

[0065] Furthermore, in some embodiments, for a scenario to be classified that has no correlation or confusion with other scenarios to be classified, only a first classification sub-network corresponding to it can be set, without setting a second classification sub-network corresponding to it. For example, assuming that scenario 6 to be classified has no correlation or confusion with any of scenarios 1 to 5, then for scenario 6, only a first classification sub-network corresponding to it can be set, without setting a second classification sub-network corresponding to it.

[0066] According to embodiments of the present invention, generating classification labels for training a first classification network and a second classification network based on correlation, confusion, and sample labels may include, for example, generating a first classification label for training each first classification sub-network based on sample labels and correlation, and generating a second classification label for training each second classification sub-network based on sample labels and confusion. Wherein, both the first classification label and the second classification label are, for example, binary classification labels.

[0067] In this embodiment of the invention, generating a first classification label for training each first classification sub-network based on sample labels and correlations may include, for example, the following operations.

[0068] For example, in response to the correlation between multiple unclassified scenes, including a second unclassified scene that is correlated with the first unclassified scene, for any first unclassified scene corresponding to a first classification subnetwork, one of the following operations is performed: in response to determining that the sample labels include the first unclassified scene, the first classification label corresponding to the first unclassified scene in the first image block is determined to be 1; in response to determining that the sample labels do not include the first unclassified scene but include the second unclassified scene, the first classification label corresponding to the first unclassified scene in the first image block is determined to be 1; and in response to determining that the sample labels do not include the first unclassified scene and the second unclassified scene, the first classification label corresponding to the first unclassified scene in the first image block is determined to be 0. The first unclassified scene and the second unclassified scene can be different unclassified scenes.

[0069] In this embodiment of the invention, assuming the sample label is represented by Y, for any first classifying scene i corresponding to a first classification sub-network, the first classification label y corresponding to the first classifying scene i is... i It can be expressed using the following formula (1).

[0070] (1)

[0071] In formula (1), y iLet represent the first classification label of the first scene to be classified, Y represent the sample label, X represent the correlation between multiple scenes to be classified, and j represent the second scene to be classified.<i, j> This indicates that there is a correlation between the first scenario i to be classified and the second scenario j to be classified.

[0072] For example, for any first classifying scenario i corresponding to a first classification sub-network, if the first classifying scenario i exists in the sample labels, the first classification label corresponding to the first classifying scenario i can be set to 1. If the first classifying scenario i does not exist in the sample labels, but a second classifying scenario j that is related to it exists in the sample labels, the first classification label corresponding to the first classifying scenario i can be set to 1. If neither the first classifying scenario i nor the aforementioned second classifying scenario j exists in the sample labels, the first classification label of the first classifying scenario i can be set to 0.

[0073] In this embodiment of the invention, a first classification label for training each first classification sub-network is generated based on the correlation and sample labels. By utilizing the correlation between multiple scenes to be classified to train the first classification sub-network, it is beneficial to help the first classification sub-network learn the features of the corresponding scenes to be classified better, thereby improving the classification accuracy and precision of the first classification sub-network.

[0074] In this embodiment of the invention, generating a second classification label for training each second classification sub-network based on sample labels and confusion relationships may include, for example, the following operations.

[0075] For example, in response to the confusion relation that includes a third unclassified scene being easily confused with a first unclassified scene, for any first unclassified scene corresponding to a second classification subnetwork, one of the following operations is performed: in response to determining that the sample labels include the first unclassified scene, the second classification label corresponding to the first unclassified scene in the first image block is determined to be 1; in response to determining that the sample labels include a third unclassified scene that is easily confused with the first unclassified scene, but do not include the first unclassified scene, the second classification label corresponding to the first unclassified scene in the first image block is determined to be -1; and in response to determining that the sample labels do not include the first unclassified scene, and do not include the third unclassified scene, the second classification label corresponding to the first unclassified scene in the first image block is determined to be 0.

[0076] In this embodiment of the invention, for any first unclassified scene k corresponding to a second classification sub-network, the second classification label y corresponding to the first unclassified scene k is... k It can be expressed using the following formula (2).

[0077] (2)

[0078] In formula (2), y k Let Y represent the second classification label of the first unclassified scenario k, H represent the sample label, l represent the confusion relationship between multiple unclassified scenarios, and [l, k] represent the third unclassified scenario. [l, k] indicates that the third unclassified scenario l is easily confused with the first unclassified scenario k.

[0079] For example, for any first unclassified scene k corresponding to a second classification subnetwork, if the first unclassified scene k exists in the sample labels, the second classification label corresponding to the first unclassified scene k can be set to 1. If the first unclassified scene k does not exist in the sample labels, but there is a third unclassified scene l in the sample labels that is easily confused with the first unclassified scene k, the second classification label corresponding to the first unclassified scene k can be set to -1. If there is neither the first unclassified scene k nor the aforementioned third unclassified scene l in the sample labels, the second classification label of the first unclassified scene k can be set to 0.

[0080] In this embodiment of the invention, generating second classification labels for training each second classification sub-network based on sample labels and confusion relationships can avoid interference from easily confused scenarios to be classified, thereby improving the classification accuracy and precision of the second classification sub-network.

[0081] After generating first classification labels for training each first classification sub-network and second classification labels for training each second classification sub-network, a first loss value is determined based on each first classification label and the first classification result output by the corresponding first classification sub-network, and a second loss value is determined based on each second classification label and the second classification result output by the corresponding second classification sub-network. Then, the parameters of the first classification sub-network corresponding to the first scene to be classified are adjusted based on the first loss value, and the parameters of the second classification sub-network corresponding to the first scene to be classified are adjusted based on the second loss value.

[0082] According to an embodiment of the present invention, during the training of each first classification subnetwork and each second classification subnetwork, loss functions such as binary cross-entropy or mean squared error can be used to determine each first loss value and each second loss value.

[0083] It should be noted that if the second classification label corresponding to the first scene to be classified contains -1, then the second loss value of the second classification sub-network corresponding to the second classification label -1 is set to 0. This ensures that the training data corresponding to the second classification label -1 has no effect on the calculation of the corresponding second loss value, meaning that no loss occurs when training the corresponding second classification sub-network using the above training data.

[0084] In an embodiment of the present invention, for example, formula (3) can be used to calculate the first loss value.

[0085] (3)

[0086] In formula (3), loss i y represents the first loss value of the first classification sub-network corresponding to the first scene to be classified, i. i p represents the first category label of the first scene to be classified, i. i This represents the first classification result output by the first classification sub-network corresponding to the first scene to be classified, based on the visual word histogram of each first image block.

[0087] In an embodiment of the present invention, for example, formula (4) can be used to calculate the second loss value.

[0088] (4)

[0089] In formula (4), loss k y represents the second loss value of the second classification sub-network corresponding to the first scene to be classified, k. k p represents the second category label of the first scene to be classified, k. k This represents the second classification result output by the second classification sub-network corresponding to the first scene to be classified, based on the visual word histogram of each first image block.

[0090] In this embodiment of the invention, the autoencoder can be trained using the second sample image set, and the aforementioned feature extraction network can be constructed based on the trained autoencoder.

[0091] In this embodiment of the invention, the autoencoder includes, for example, an input layer, a hidden layer, and an output layer. The number of nodes in the hidden layer can be set to be greater than the number of nodes in the input layer, thereby enabling the autoencoder to better learn the sparse features of the training data.

[0092] According to an embodiment of the present invention, the second sample image set includes a plurality of second image blocks. The plurality of second image blocks in the second sample image set can be obtained, for example, by segmenting multiple remote sensing images. The acquisition method of the second image blocks is similar to that of the first image blocks, and will not be described again here.

[0093] Multiple second image patches from the second sample image set can be used as training data for training the autoencoder. After training the autoencoder, the weight matrix between the input layer and the hidden layer can be extracted, and the weight matrix can be used as the convolution kernel to construct convolutional layers. The feature extraction network described above is then constructed based on these convolutional layers. This feature extraction network is used to extract local features from the image patches.

[0094] Figure 3 This is a schematic diagram of the training method for the scene classification model according to an embodiment of the present invention. See below for reference. Figure 3 The training method of the scene classification model in the embodiments of the present invention is illustrated by example.

[0095] In this embodiment of the invention, a first sample image set can be obtained, for example, by segmenting multiple remote sensing images carrying scene labels. The first sample image set includes multiple first image blocks and sample labels for each of the first image blocks. The sample labels are used to characterize the scene category to which one or more scenes to be classified belong in the corresponding first image block. This first sample image set can be used to train the classification network in a scene classification model.

[0096] like Figure 3 As shown, the scene classification model may include, for example, a feature extraction network 310, a first classification network 320, and a second classification network 330.

[0097] The first classification network 320 includes m first classification subnetworks, such as first classification subnetwork 1, first classification subnetwork 2, ..., first classification subnetwork m. The second classification network 330 includes n second classification subnetworks, such as second classification subnetwork 1, second classification subnetwork 2, ..., second classification subnetwork n. Where m and n are both positive integers.

[0098] In this embodiment of the invention, the number of first classification subnetworks and the number of second classification subnetworks can, for example, be the same as and correspond one-to-one with the number of scenes to be classified. Each first classification subnetwork is used to output a first classification result for a scene to be classified, and each second classification subnetwork is used to output a second classification result for a scene to be classified.

[0099] In some embodiments, for a scenario to be classified that has no correlation with other scenarios to be classified and no confusion with other scenarios to be classified, only a first classification subnetwork corresponding to it can be set, without setting a second classification subnetwork corresponding to it.

[0100] In this embodiment of the invention, the feature extraction network 310 is constructed, for example, by training the autoencoder based on the second sample image set. The specific construction process is the same as or similar to the process described above, and will not be repeated here.

[0101] After acquiring a first sample image set comprising multiple first image blocks 31, feature extraction network 310 can be used to extract features from each first image block to obtain local features 32 for each first image block. Then, based on the local features 32 of each first image block, a visual word histogram 33 can be generated for each first image block.

[0102] Next, each of the first classification subnetworks in the first classification network 320 is used based on the visual word histogram of the first image block to obtain the first classification result corresponding to the scene to be classified in the first image block. For example, using the visual word histogram of the first image block based on the first classification subnetwork 1, the first classification result 1 corresponding to the scene to be classified in the first image block is obtained. Using the visual word histogram of the first image block based on the first classification subnetwork 2, the first classification result 2 corresponding to the scene to be classified in the first image block is obtained. Similarly, using the visual word histogram of the first image block based on the first classification subnetwork m, the first classification result m corresponding to the scene to be classified in the first image block is obtained.

[0103] Furthermore, each second classification subnetwork in the second classification network 330 uses the visual word histogram of the first image patch to obtain a second classification result for the scene to be classified in the first image patch. For example, using the visual word histogram of the first image patch, second classification subnetwork 1 obtains a second classification result 1 corresponding to scene 1 in the first image patch. Using the visual word histogram of the first image patch, second classification subnetwork 2 obtains a second classification result 2 corresponding to scene 2 in the first image patch. Similarly, using the visual word histogram of the first image patch, second classification subnetwork n obtains a second classification result n corresponding to scene n in the first image patch.

[0104] Before training the corresponding classification sub-networks using the first and second classification results, a first classification label (e.g., the first classification label corresponding to each of the first classification sub-networks 1 to m) can be generated based on the pre-constructed correlation between multiple scenes to be classified and the sample labels carried by the first image patch. Similarly, a second classification label (e.g., the second classification label corresponding to each of the second classification sub-networks 1 to n) can be generated based on the pre-constructed confusion relationship between multiple scenes to be classified and the sample labels carried by the first image patch.

[0105] Next, for each scene to be classified, a first loss value is determined based on the first classification result and the corresponding first classification label for that scene. A second loss value is determined based on the second classification result and the corresponding second classification label for that scene. In some embodiments, if -1 exists in the second classification label corresponding to the first scene to be classified, then the second loss value of the second classification subnetwork corresponding to the second classification label -1 is determined to be 0.

[0106] Then, the parameters of the first classification sub-network corresponding to the first scenario to be classified are adjusted according to the first loss value, and the parameters of the second classification sub-network corresponding to the first scenario to be classified are adjusted according to the second loss value.

[0107] For example, consider training the first classification subnetwork 1 and the second classification subnetwork 1. A first loss value 1 is calculated based on the first classification result 1 corresponding to the scene to be classified 1 and the first classification label (e.g., first classification label 1) corresponding to the first classification subnetwork 1. A second loss value 1 is calculated based on the second classification result 1 corresponding to the scene to be classified 1 and the second classification label (e.g., second classification label 1) corresponding to the second classification subnetwork 1. Then, the parameters of the first classification subnetwork 1 are adjusted based on the first loss value 1, and the parameters of the second classification subnetwork 1 are adjusted based on the second loss value 1. Similarly, based on the above method, the parameters of each classification subnetwork in the first classification network 320 and the second classification network 330 can be adjusted accordingly, thereby achieving the training of the first classification network 320 and the second classification network 330.

[0108] Figure 4 This is a flowchart of a remote sensing image scene classification method according to an embodiment of the present invention.

[0109] like Figure 4 As shown, the remote sensing image scene classification method includes operations S410~S450.

[0110] In operation of S410, multiple third image blocks are acquired based on the remote sensing image to be classified.

[0111] In operation of S420, the feature extraction network in the scene classification model is used to extract features from each third image block to obtain the local features of each third image block.

[0112] In operation S430, a visual word histogram is generated for each third image patch based on the local features of each third image patch.

[0113] In operating S440, the first and second classification networks in the scene classification model are used to obtain the first and second classification results corresponding to each scene to be classified based on the histogram of each visual word.

[0114] When operating S450, the scene classification results for the remote sensing image to be classified are obtained based on the first and second classification results corresponding to each scene to be classified.

[0115] In an embodiment of the present invention, the scene classification model is trained according to the training method of the scene classification model in the above embodiments.

[0116] According to embodiments of the present invention, the remote sensing image to be classified may include at least one scene to be classified. The remote sensing image to be classified can be divided into a uniform grid to obtain multiple third image blocks. The specific acquisition method is similar to the process described above and will not be repeated here.

[0117] According to an embodiment of the present invention, the process of generating a visual word histogram for each third image block based on the local features of each third image block is similar to the process described above, and will not be repeated here.

[0118] According to embodiments of the present invention, the first classification network includes, for example, a plurality of first classification subnetworks, and the second classification network includes, for example, a plurality of second classification subnetworks. The number of first classification subnetworks and the number of second classification subnetworks may, for example, be the same as and correspond one-to-one with the number of scenes to be classified. Each first classification subnetwork is used to output a first classification result for one scene to be classified, and each second classification subnetwork is used to output a second classification result for one scene to be classified.

[0119] In this embodiment of the invention, obtaining scene classification results for remote sensing images to be classified based on the first classification result and the second classification result corresponding to each scene to be classified may include the following operations.

[0120] For example, for each scene to be classified, the first classification result corresponding to that scene can be compared with a first threshold. If the first classification result is greater than or equal to the first threshold, the first classification result is considered reliable, and it can be determined that the scene to be classified exists in the remote sensing image. If the first classification result is less than the first threshold, the first classification result is considered questionable, and the scene to be classified may not exist in the remote sensing image. In this case, a second classification result can be used to verify the first classification result in order to further determine whether the scene to be classified truly does not exist in the remote sensing image.

[0121] Next, if the first classification result is determined to be less than the first threshold, the second classification result corresponding to the scene to be classified is compared with the second threshold, and the absolute difference between the second classification result and the first classification result is compared with the third threshold.

[0122] If the second classification result is greater than or equal to the second threshold, and the absolute difference between the second classification result and the first classification result is less than the third threshold, then the first classification result is considered reliable, and it can be determined that the scene to be classified actually exists in the remote sensing image to be classified. Otherwise, it is determined that the scene to be classified does not actually exist in the remote sensing image to be classified.

[0123] Based on the above method, the scene classification result of each scene to be classified in the remote sensing image to be classified can be obtained, thus obtaining the scene classification result for the remote sensing image to be classified.

[0124] In this embodiment of the invention, when it is determined that the first classification result is less than the first threshold, the credibility of the first classification result is verified by using the second classification result. This can avoid misclassifying the scene to be classified, thereby improving the accuracy of the scene classification result.

[0125] In another example, obtaining the scene classification result for the remote sensing image to be classified based on the first classification result and the second classification result corresponding to each scene to be classified may include the following operations.

[0126] For example, for each scene to be classified, the first classification result and the second classification result can be fused to obtain a fused classification result. Then, the fused classification result is compared with a fourth threshold. If the fused classification result is greater than or equal to the fourth threshold, it is determined that the scene to be classified exists in the remote sensing image. Otherwise, it is determined that the scene to be classified does not exist in the remote sensing image.

[0127] In this embodiment of the invention, fusing the first classification result and the second classification result can be performed as follows: Obtain the weight coefficients corresponding to the first and second classification results respectively. These weight coefficients can be used to characterize the degree of fusion of each classification result. Then, based on the first classification result, the second classification result, and their respective weight coefficients, the fused classification result is obtained.

[0128] In this embodiment of the invention, by fusing the first classification result and the second classification result, and determining the scene classification result for the scene to be classified based on the fused classification result, the scene classification result for the scene to be classified takes into account the contributions of the two classification methods, thereby improving the accuracy of the scene classification result.

[0129] It should be noted that the first threshold, second threshold, third threshold, fourth threshold, and the weight coefficients corresponding to the first classification result and the second classification result can be set according to actual needs, and this invention does not limit them.

[0130] Figure 5 This is a block diagram of the training device for the scene classification model according to an embodiment of the present invention.

[0131] like Figure 5As shown, the training device 500 for the scene classification model includes a first acquisition module 510, a first extraction module 520, a first generation module 530, a first classification module 540, a first determination module 550, and an adjustment module 560. In this embodiment of the invention, the scene classification model includes a feature extraction network, a first classification network, and a second classification network.

[0132] The first acquisition module 510 is used to acquire a first sample image set, which includes multiple first image blocks and sample labels for each of the multiple first image blocks. The sample labels are used to characterize the scene category to which multiple scenes to be classified in the corresponding first image blocks belong.

[0133] The first extraction module 520 is used to extract features from each first image block in the first sample image set using a feature extraction network to obtain local features of each first image block; wherein, the feature extraction network is obtained by training an autoencoder based on the second sample image set.

[0134] The first generation module 530 is used to generate a visual word histogram for each first image block based on the local features of each first image block.

[0135] The first classification module 540 is used to obtain the first classification result and the second classification result corresponding to each scene to be classified based on the visual word histogram using the first classification network and the second classification network respectively.

[0136] The first determination module 550 is used to determine a first loss value for the first classification result and a second loss value for the second classification result based on the first classification result, the second classification result, the sample label, and the correlation between multiple scenarios to be classified.

[0137] The adjustment module 560 is used to adjust the parameters of the first classification network and the second classification network respectively based on the first loss value and the second loss value.

[0138] Figure 6 This is a block diagram of a remote sensing image scene classification device according to an embodiment of the present invention.

[0139] like Figure 6 As shown, the remote sensing image scene classification device 600 includes a second acquisition module 610, a second extraction module 620, a second generation module 630, a second classification module 640, and a second determination module 650.

[0140] The second acquisition module 610 is used to acquire multiple third image blocks based on the remote sensing image to be classified.

[0141] The second extraction module 620 is used to extract features from each third image block using the feature extraction network in the scene classification model, so as to obtain the local features of each third image block.

[0142] The second generation module 630 is used to generate a visual word histogram for each third image block based on the local features of each third image block.

[0143] The second classification module 640 is used to obtain the first classification result and the second classification result corresponding to each scene to be classified based on the histogram of each visual word by using the first classification network and the second classification network in the scene classification model.

[0144] The second determining module 650 is used to obtain scene classification results for the remote sensing image to be classified based on the first classification results and the second classification results corresponding to each scene to be classified.

[0145] In this embodiment of the invention, the scene classification model is trained using the training device for the scene classification model described in the above embodiments.

[0146] It should be noted that the implementation methods, technical problems solved, functions achieved, and technical effects of each module / unit / subunit in the device embodiments are the same as or similar to the implementation methods, technical problems solved, functions achieved, and technical effects of each corresponding step in the method embodiments, and will not be repeated here.

[0147] Figure 7 A block diagram of an electronic device suitable for implementing a training method for a scene classification model and a remote sensing image scene classification method according to an embodiment of the present invention is shown schematically.

[0148] like Figure 7 As shown, an electronic device 700 according to an embodiment of the present invention includes a processor 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage portion 708 into a random access memory (RAM) 703. The processor 701 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 701 may also include onboard memory for caching purposes. The processor 701 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0149] RAM 703 stores various programs and data required for the operation of electronic device 700. Processor 701, ROM 702, and RAM 703 are interconnected via bus 704. Processor 701 executes various operations of the method flow according to embodiments of the present invention by executing programs in ROM 702 and / or RAM 703. It should be noted that the programs may also be stored in one or more memories other than ROM 702 and RAM 703. Processor 701 may also execute various operations of the method flow according to embodiments of the present invention by executing programs stored in said one or more memories.

[0150] According to an embodiment of the present invention, the electronic device 700 may further include an input / output (I / O) interface 705, which is also connected to a bus 704. The electronic device 700 may also include one or more of the following components connected to the I / O interface 705: an input section 706 including a keyboard, mouse, etc.; an output section 707 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 710 as needed so that computer programs read from it can be installed into the storage section 708 as needed.

[0151] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of the present invention.

[0152] According to embodiments of the present invention, a computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of the present invention, a computer-readable storage medium may include ROM 702 and / or RAM 703 and / or one or more memories other than ROM 702 and RAM 703 described above.

[0153] Embodiments of the present invention also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code enables the computer system to implement the training method for the scene classification model and the remote sensing image scene classification method provided in the embodiments of the present invention.

[0154] When the computer program is executed by the processor 701, it performs the functions defined in the system / apparatus of this invention. According to embodiments of the invention, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0155] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 709, and / or installed from a removable medium 711. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0156] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 709, and / or installed from the removable medium 711. When the computer program is executed by the processor 701, it performs the functions defined in the system of this embodiment of the invention. According to embodiments of the invention, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0157] According to embodiments of the present invention, program code for executing the computer programs provided in the embodiments of the present invention can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0158] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0159] Those skilled in the art will understand that the features described in the various embodiments of the present invention can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in the present invention. In particular, the features described in the various embodiments of the present invention can be combined and / or combined in various ways without departing from the spirit and teachings of the present invention. All such combinations and / or combinations fall within the scope of the present invention.

[0160] The embodiments of the present invention have been described above. However, these embodiments are merely illustrative and not intended to limit the scope of the invention. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of the invention, and all such substitutions and modifications should fall within the scope of the invention.

Claims

1. A method for training a scene classification model, the scene classification model comprising a feature extraction network, a first classification network and a second classification network, the method comprising: obtaining a first sample image set, the first sample image set comprising a plurality of first image patches and a plurality of sample labels respectively for the plurality of first image patches, the sample labels being used to represent scene categories to which a plurality of scenes to be classified belong in the corresponding first image patches; performing feature extraction on each first image patch in the first sample image set by using the feature extraction network to obtain local features of each first image patch, wherein the feature extraction network is obtained based on training an auto-encoder on a second sample image set; generating a visual word histogram for each first image patch based on the local features of the first image patch; obtaining a first classification result and a second classification result corresponding to each scene to be classified based on the visual word histogram by using the first classification network and the second classification network respectively; determining a first loss value for the first classification result and a second loss value for the second classification result according to the first classification result, the second classification result, the sample labels and an association relationship between the plurality of scenes to be classified; and adjusting parameters of the first classification network and the second classification network respectively according to the first loss value and the second loss value; wherein the association relationship comprises a correlation relationship and a confusion relationship between the plurality of scenes to be classified; the first classification network comprises a first classification sub-network corresponding to each of the plurality of scenes to be classified; the second classification network comprises a second classification sub-network corresponding to each of the plurality of scenes to be classified; the determining of the first loss value for the first classification result and the second loss value for the second classification result according to the first classification result, the second classification result, the sample labels and the association relationship between the plurality of scenes to be classified comprises: generating a first classification label for training each first classification sub-network according to the sample labels and the correlation relationship; generating a second classification label for training each second classification sub-network according to the sample labels and the confusion relationship; determining the first loss value according to the first classification result and the first classification label; and determining the second loss value according to the second classification result and the second classification label; wherein in response to the correlation relationship comprising a second scene to be classified having a correlation with a first scene to be classified, for any first scene to be classified corresponding to any first classification sub-network, one of the following operations is performed: in response to determining that the first scene to be classified is included in the sample labels, determining that the first classification label corresponding to the first scene to be classified in the first image patch is 1; in response to determining that the first scene to be classified is not included in the sample labels and that the second scene to be classified is included, determining that the first classification label corresponding to the first scene to be classified in the first image patch is 1; and ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ in response to determining that the first to-be-classified scene and the second to-be-classified scene are not included in the sample label, determining that a first classification label corresponding to the first to-be-classified scene in the first image block is 0; wherein, in response to the confusion relationship including that the third to-be-classified scene is easily confused with the first to-be-classified scene, for any one second classification sub-network corresponding to the first to-be-classified scene, one of the following operations is performed: in response to determining that the first to-be-classified scene is included in the sample label, determining that a second classification label corresponding to the first to-be-classified scene in the first image block is 1; in response to determining that the third to-be-classified scene which is easily confused with the first to-be-classified scene is included in the sample label and the first to-be-classified scene is not included in the sample label, determining that the second classification label corresponding to the first to-be-classified scene in the first image block is -1; and in response to determining that the first to-be-classified scene is not included in the sample label and the third to-be-classified scene is not included in the sample label, determining that the second classification label corresponding to the first to-be-classified scene in the first image block is 0.

2. The method of claim 1, wherein, the adjusting the parameters of the first classification network and the second classification network respectively according to the first loss value and the second loss value includes: adjusting the parameters of the first classification sub-network corresponding to the first to-be-classified scene according to the first loss value; and adjusting the parameters of the second classification sub-network corresponding to the first to-be-classified scene according to the second loss value.

3. The method of claim 2, wherein, in response to the second classification label corresponding to the first to-be-classified scene including -1, determining that the second loss value of the second classification sub-network corresponding to the second classification label -1 is 0.

4. The method of claim 1, wherein, determining a correlation degree between the first to-be-classified scene and the second to-be-classified scene; and in response to determining that the correlation degree is greater than or equal to a preset threshold, determining that the first to-be-classified scene and the second to-be-classified scene have relevance.

5. The method of any one of claims 1 to 4, further comprising: training the auto-encoder using a second sample image set to obtain a trained auto-encoder; wherein the second sample image set includes a plurality of second image blocks; and constructing the feature extraction network based on an input layer and a hidden layer in the trained auto-encoder.

6. A remote sensing image scene classification method, comprising: obtaining a plurality of third image blocks based on a remote sensing image to be classified; performing feature extraction on each third image block using a feature extraction network in a scene classification model to obtain local features of each third image block; generating a visual word histogram for each third image block based on the local features of each third image block; obtaining first classification results and second classification results corresponding to each to-be-classified scene based on each visual word histogram using a first classification network and a second classification network in the scene classification model; and obtaining a scene classification result for the remote sensing image to be classified according to the first classification results and the second classification results corresponding to each to-be-classified scene. ​ ​ The scene classification model is trained according to the method in any one of claims 1-5. 7.An electronic device, comprising: one or more processors; a memory storing one or more programs, wherein the one or more programs, when executed by the one or more processors, enable the one or more processors to perform the method for training the scene classification model according to any one of claims 1-5, or the method for classifying a scene of a remote sensing image according to claim 6.

Images