A mining area remote sensing scene classification model construction method, device and classification method

Abstract

The present invention provides a method, device and classification method for building a classification model of remote sensing scenes in a mining area, which relates to the field of image processing. The model building method includes: obtaining remote sensing data of the mining area, remote sensing data optical images, radar images and elevation image data; extracting surface data based on elevation image data, and resampling radar images, elevation image data and surface data to the same resolution as the optical image to obtain a temporary data set; fusing the temporary data set with the optical image to obtain an initial multimodal scene data set; training and tuning the initial classification model according to the initial multimodal scene data set to obtain a remote sensing scene classification model, wherein the initial classification model is based on a CNN model and a GCN model, and the remote sensing scene is used to classify remote sensing scenes in the mining area. The problem of low accuracy in classification of remote sensing scenes in the mining area caused by irregular or fuzzy edges of mines and sparse spatial distribution is solved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method and device for constructing a classification model for a mining area remote sensing scene, and a classification method. Background Art

[0002] Mining area land cover refers to the surface cover of the mining area, that is, the study of the distribution and characteristics of different types of land or surface cover in the mining area. In mining area land cover, various geomorphic features, vegetation types, land use conditions and other factors in the mining area are taken into account, as well as the impact of mining activities on land cover.

[0003] Remote sensing scene classification of land cover in mining areas aims to correctly label a given remote sensing image using predefined semantic categories of land cover. Nowadays, deep learning-based methods have achieved great results in the field of scene classification. However, unlike natural images, remote sensing image data is large. For example, there are usually only three bands (RGB) of data in natural images, while there are often more than three bands of data in remote sensing images, and the size of remote sensing images is also larger than that of natural images. In addition, there are a large number of multi-sized objects in remote sensing image data, as well as irregular objects, fuzzy edges or large size differences and sparse spatial distribution caused by shooting angles, terrain slopes, light reflections and other factors. These problems limit the accuracy of mining scene classification. Summary of the invention

[0004] The problem solved by the present invention is how to solve the problem of low accuracy in classification of remote sensing scenes in mining areas caused by problems such as irregular or blurred mine edges and sparse spatial distribution during classification of remote sensing scenes in mining areas.

[0005] In order to solve the above problems, the present invention provides a method for constructing a mining area remote sensing scene classification model, comprising the following steps:

[0006] Acquiring remote sensing data of the mining area, wherein the remote sensing data includes optical images, radar images and elevation image data;

[0007] Extracting surface data based on the elevation image data, and resampling the radar image, the elevation image data, and the surface data to the same resolution as the optical image to obtain a temporary data set;

[0008] fusing the temporary data set with the optical image to obtain an initial multimodal scene data set;

[0009] The initial classification model is trained and tuned according to the initial multimodal scene data set to obtain a remote sensing scene classification model, wherein the initial classification model is based on a CNN model and a GCN model, and the remote sensing scene classification model is used to classify remote sensing scenes in mining areas.

[0010] Optionally, the initial multimodal scene data set includes a plurality of original images and labeled image data corresponding to the original images; the training and tuning of the initial classification model according to the initial multimodal scene data set to obtain the remote sensing scene classification model includes:

[0011] Classify the original image using the initial classification model to obtain a temporary classification result;

[0012] The initial classification model is optimized according to the temporary classification result and the corresponding label image data, and the optimized initial classification model is used as the remote sensing scene classification model.

[0013] Optionally, the classifying the original image by using the initial classification model to obtain a temporary classification result includes:

[0014] Performing a multi-scale feature extraction operation on the original image through the CNN model to obtain multi-scale feature data;

[0015] Performing data processing on the original image through the GCN model to obtain graph convolution feature data;

[0016] The multi-scale feature data and the graph convolution feature data are fused to obtain fused feature data, and the fused feature data are classified to obtain the temporary classification result.

[0017] Optionally, performing a multi-scale feature extraction operation on the original image by using the CNN model to obtain multi-scale feature data includes:

[0018] Performing feature extraction on the original image to obtain a preliminary convolution feature map, and performing multi-scale information extraction on the preliminary convolution feature map to obtain multiple temporary feature data;

[0019] Splicing the temporary feature data, and preprocessing the spliced ​​data to obtain temporary data;

[0020] Obtaining corresponding temporary multi-scale feature data according to the temporary data and each of the temporary feature data;

[0021] Extracting edge information from the preliminary convolution feature map to obtain edge information features;

[0022] All the temporary multi-scale feature data and the edge information features are fused to obtain the multi-scale feature data.

[0023] Optionally, the GCN model includes a multi-layer graph convolutional network; and the processing of the original image by the GCN model to obtain graph convolution feature data includes:

[0024] The original image is topologically processed based on a block-by-block method and a nearest neighbor algorithm to obtain a first graph node feature matrix and a first adjacency matrix;

[0025] Inputting the first graph node feature matrix and the first adjacency matrix into a first-layer graph convolutional network to perform a graph convolution operation to obtain a first graph convolution feature image;

[0026] The first graph convolution feature image is topologically processed based on the block method and the nearest neighbor algorithm to obtain a second graph node feature matrix and a second adjacency matrix. The second graph node feature matrix and the second adjacency matrix are then input into a second-layer graph convolution network for graph convolution operation to obtain a second graph convolution feature image. The iterative process is repeated until all the graph convolution networks complete data processing to obtain the graph convolution feature data.

[0027] Optionally, the topology of the original image is performed based on a block-by-block method and a nearest neighbor algorithm to obtain a first graph node feature matrix and a first adjacency matrix, including:

[0028] Segmenting the original image to obtain a plurality of image blocks, and linearly embedding each of the image blocks to obtain the first graph node feature matrix;

[0029] According to the first graph node feature matrix, the topological structure of the graph is constructed based on a nearest neighbor algorithm to obtain the first adjacency matrix.

[0030] Optionally, the step of constructing a topological structure of a graph based on a nearest neighbor algorithm according to the first graph node feature matrix to obtain the first adjacency matrix includes:

[0031] Determine an adjacency relationship graph of each node in the first graph node feature matrix, wherein the distance between each node in the first graph node feature matrix is ​​calculated to find a nearest neighbor node whose distance to a target node is less than a preset threshold, connect the target node to the nearest neighbor node to establish an edge, and construct an adjacency relationship graph of the target node, wherein the target node is any node in the first graph node feature;

[0032] The topological structure of the graph is constructed according to all the adjacency relationship graphs to obtain the adjacency matrix.

[0033] Optionally, the initial classification model further includes a downsampling module; the fusing all the temporary multi-scale feature data and the edge information features to obtain the multi-scale feature data includes:

[0034] Fusing all the temporary multi-scale feature data and the edge information features to obtain a temporary feature image;

[0035] The temporary feature image is downsampled by the downsampling module to obtain the multi-scale feature data, wherein a first downsampled feature map is obtained by pooling the temporary feature image, the temporary feature image is segmented to obtain a plurality of temporary images, all the temporary images are spliced ​​to obtain a second downsampled feature map, and the first downsampled feature map and the second downsampled feature map are fused to obtain the multi-scale feature data.

[0036] The advantages of the mining area remote sensing scene classification model construction method described in the present invention over the prior art are: first, by integrating optical images, radar images and elevation image data, more comprehensive and diverse surface information can be obtained, and more comprehensive scene information of the mining area can be obtained. Compared with the method of using only a single data source, it can provide more accurate classification results, and extracting surface data based on elevation images can better understand the terrain characteristics, which is helpful to consider the influence of terrain in remote sensing image classification, improve the model's understanding and recognition of landform features, and help improve the accuracy and generalization ability of the classification model. All remote sensing data are adjusted to the same resolution through resampling technology to ensure The consistency of data in spatial scale is conducive to improving the accuracy and reliability of classification. Finally, the initial classification model is constructed based on the CNN model and the GCN model. The CNN model can effectively extract the spatial features of the image, that is, the CNN model can be used to capture the spatial distribution, texture features and edge information of the objects in the mining area image, while the GCN model can process the relationship information of the graph data, and can capture the spatial relationship and terrain information between the objects. The connection relationship between the objects can be obtained through the GCN network to help understand the geographic spatial relationship. The combination of the two models can better comprehensively consider the spectral, spatial and terrain characteristics, and improve the classification effect of the mining area remote sensing scene. Compared with the traditional remote sensing classification method, this comprehensive model has stronger feature expression ability and classification accuracy.

[0037] In order to solve the above technical problems, the present invention also provides a mining area remote sensing scene classification model construction device, comprising:

[0038] An acquisition unit, used to acquire remote sensing data of a mining area, wherein the remote sensing data includes optical images, radar images and elevation image data;

[0039] A processing unit is used to extract surface data based on the elevation image data, and resample the radar image, the elevation image data and the surface data to the same resolution as the optical image to obtain a temporary data set; and fuse the temporary data set with the optical image to obtain an initial multimodal scene data set;

[0040] A construction unit is used to train and tune the initial classification model according to the initial multimodal scene data set to obtain a remote sensing scene classification model, wherein the initial classification model is based on a CNN model and a GCN model, and the remote sensing scene classification model is used to classify remote sensing scenes in mining areas.

[0041] The advantages of the mining area remote sensing scene classification model construction device and the mining area remote sensing scene classification model construction method described in the present invention are the same as those of the prior art, which will not be repeated here.

[0042] In order to solve the above technical problems, the present invention also provides a mining area remote sensing scene classification method, comprising:

[0043] Acquire target remote sensing data of the mining area to be tested, and obtain a target multimodal scene data set based on the target remote sensing data;

[0044] The target multimodal scene data set is input into the remote sensing scene classification model obtained by the mining area remote sensing scene classification model construction method to obtain the remote sensing scene classification result of the mining area to be tested.

[0045] The advantages of the mining area remote sensing scene classification method described in the present invention and the mining area remote sensing scene classification model construction method described in the present invention are the same as those of the prior art, which will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 Flow chart of a method for constructing a mining area remote sensing scene classification model in an embodiment of the present invention;

[0047] Figure 2 Schematic diagram of the fusion process of multi-scale feature data and graph convolution feature data in an embodiment of the present invention;

[0048] Figure 3 Schematic diagram of a multi-scale feature data extraction process in an embodiment of the present invention;

[0049] Figure 4 Schematic diagram of the data processing process of the down-sampling module in an embodiment of the present invention;

[0050] Figure 5 Schematic diagram of the training process of the initial classification model in an embodiment of the present invention;

[0051] Figure 6A structural diagram of a mining area remote sensing scene classification model building device in an embodiment of the present invention;

[0052] Figure 7 Flow chart of a mining area remote sensing scene classification method in an embodiment of the present invention;

[0053] Figure 8 1 is a diagram showing the internal structure of a computer device in an embodiment of the present invention. DETAILED DESCRIPTION

[0054] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0055] like Figure 1 As shown, in one embodiment, a method for constructing a mining area remote sensing scene classification model is provided, comprising the following steps:

[0056] Step S1, obtaining remote sensing data of the mining area, the remote sensing data including optical image, radar image and elevation image data.

[0057] Specifically, the remote sensing data of the mining area can be obtained by aerial photography, satellite remote sensing and other means to obtain optical images, radar images and elevation image data of the mining area. For example, the optical image contains 4-band images with a resolution of 2m, the radar image contains a single band with a resolution of 5m, and the elevation image data (DEM data) contains a single band with a resolution of 10m.

[0058] Step S2, extracting surface data based on the elevation image data, and resampling the radar image, elevation image data and surface data to the same resolution as the optical image to obtain a temporary data set;

[0059] Specifically, the surface data of the frame area, such as slope data, aspect data and hill shadow data, are extracted from the DEM data. The surface data are all single-band and the resolution is consistent with the DEM data. Then the radar image, DEM data, slope data, aspect data and hill shadow data are resampled to the same resolution as the optical image (such as a resolution of 2m). The resampling ensures that all data have the same spatial resolution.

[0060] Step S3, fusing the temporary data set and the optical image to obtain an initial multimodal scene data set.

[0061] Specifically, all data with the same resolution are fused to obtain a fused data set; for example, the fused data set includes 4 optical image bands, 1 radar image band, 1 DEM data band, and single bands of slope, aspect, and hill shadow data, totaling 9 bands. Finally, through data clipping, such as clipping the obtained 9-band image data according to the set sample size from left to right and from top to bottom, the initial multimodal scene data set is obtained. This means that all samples in the data set are clipped into images containing 9-band information to ensure the data consistency and uniformity of each sample, which is convenient for subsequent data analysis and model training. The role of the fusion process is to achieve the fusion between different bands to obtain more comprehensive and high-quality image data, and provide more information for geographic information analysis and remote sensing applications.

[0062] Step S4, training and tuning the initial classification model according to the initial multimodal scene data set to obtain a remote sensing scene classification model, wherein the initial classification model is based on a CNN model and a GCN model, and the remote sensing scene classification model is used to classify remote sensing scenes in mining areas.

[0063] Specifically, the initial classification model based on the CNN model and the GCN model is trained and tuned using the initial multimodal scene dataset. Among them, the CNN model is good at processing image data, while the GCN model can effectively process graph structure data. The CNN and GCN models are combined to take advantage of their respective advantages to improve the performance of remote sensing scene classification. The powerful ability of CNN in processing image data and the advantage of GCN in processing graph structure data are combined to make the model more accurate and efficient in classifying mining remote sensing scenes.

[0064] The method for constructing a mining area remote sensing scene classification model in this embodiment first obtains more comprehensive and diverse surface information by integrating optical images, radar images and elevation image data, and can obtain more comprehensive scene information of the mining area. Compared with the method of using only a single data source, it can provide more accurate classification results, and extract surface data based on elevation images to better understand terrain features, which is helpful to consider the influence of terrain in remote sensing image classification, improve the model's understanding and recognition of landform features, and help improve the accuracy and generalization ability of the classification model. All remote sensing data are adjusted to the same resolution through resampling technology to ensure that the data is consistent in spatial scale. The consistency is conducive to improving the accuracy and reliability of classification. Finally, the initial classification model is constructed based on the CNN model and the GCN model. Among them, the CNN model can effectively extract the spatial features of the image, that is, the CNN model can be used to capture the spatial distribution, texture features and edge information of the objects in the mining area image, and the GCN model can process the relationship information of the graph data, and can capture the spatial relationship and terrain information between the objects. That is, the connection relationship between the objects can be obtained through the GCN network to help understand the geographic spatial relationship. The combination of the two models can better comprehensively consider the spectral, spatial and terrain characteristics, and improve the classification effect of the mining area remote sensing scene. Compared with the traditional remote sensing classification method, this comprehensive model has stronger feature expression ability and classification accuracy.

[0065] In some embodiments, in step S4, the initial multimodal scene dataset includes a plurality of original images and label image data corresponding to the original images; the initial classification model is trained and tuned according to the initial multimodal scene dataset to obtain a remote sensing scene classification model, including:

[0066] Step S41, classify the original image using the initial classification model to obtain a temporary classification result.

[0067] Step S42, optimizing the initial classification model according to the temporary classification result and the corresponding label image data, and using the optimized initial classification model as the remote sensing scene classification model.

[0068] Specifically, feature extraction and model training: The initial multimodal scene dataset includes the original image and the corresponding labeled image data. The labeled image data corresponds to the real label of the object category of each pixel. The initial classification model is trained using the initial multimodal scene dataset. By training the model, it can learn the features and identification rules of the object category from the input multimodal data. The initial classification model is tuned based on the temporary classification results and label data. The model parameters are fine-tuned and optimized through the feedback results to improve the accuracy and effect of the model. The tuned initial classification model is determined as the remote sensing scene classification model, which can be used to identify and classify remote sensing scenes in mining areas. By training and tuning on actual scene data, different object categories can be more accurately identified, and the accuracy and generalization ability of the classification model can be improved. Through the continuous tuning process, the classification model is closer to the real data distribution, and the adaptability and robustness of the model to new data are improved. By continuously optimizing the model parameters, the error rate in the classification process can be reduced, and the precision and accuracy of the classification of remote sensing scenes in mining areas can be improved.

[0069] In some embodiments, in step S41, the original image is classified by the initial classification model to obtain a temporary classification result, including:

[0070] Step S411, performing a multi-scale feature extraction operation on the original image through a CNN model to obtain multi-scale feature data.

[0071] Step S412: Process the original image through the GCN model to obtain graph convolution feature data.

[0072] Step S413: fuse the multi-scale feature data and the graph convolution feature data to obtain fused feature data, and classify the fused feature data to obtain a temporary classification result.

[0073] Specifically, the CNN model is used to perform multi-scale extraction on the original image in the initial multimodal scene dataset to obtain multi-scale feature data. At the same time, the GCN model is used to perform graph convolution on the original image to obtain graph convolution feature data. The above two data are fused, and the fused data are classified to obtain a temporary classification result.

[0074] CNN can effectively extract features of different scales and different abstract levels in the original image, helping to identify the information of the objects in the image. GCN is suitable for graph structure data, which can capture the connection relationship and spatial information between the objects in the image, and help understand the relationship between the objects. Finally, by fusing the different feature information extracted by CNN and GCN, the local features and global context information can be comprehensively utilized to improve the accuracy of object classification. That is, by extracting multi-scale features through CNN, the features of objects at different scales in the image can be captured, which helps to improve the robustness and generalization ability of classification. By extracting graph convolution features through GCN, the connection and spatial layout between objects can be better understood, and the ability to model the relationship between objects can be improved. Fusion of multi-scale features and graph convolution features can comprehensively utilize the advantages of different features to improve classification precision and accuracy. The fused feature data can better describe the complex features of the objects, which helps to improve the performance of the classification model. Therefore, by combining the advantages of CNN and GCN, that is, through the steps of multi-scale feature extraction, object relationship modeling and feature fusion, the accuracy and efficiency of remote sensing scene classification in mining areas can be effectively improved, and the challenges brought by the complexity and irregularity of remote sensing image scenes can be solved.

[0075] In some embodiments, Figure 2 As shown in the figure, in the process of fusion of scale feature data and graph convolution feature data, the multi-scale feature data (feature map of C'*H'*W', where C' represents the number of channels, H' represents the height, and W' represents the width) is converted into a two-dimensional matrix H'W'*C' to facilitate the subsequent matrix multiplication operation. The graph convolution feature data (matrix C'*N) is transposed into a matrix N*C', and then through the nonlinear activation function σ, matrix multiplication is performed with the two-dimensional matrix H'W'*C'. The result is a matrix of H'W'*N, where N is the number of new feature maps. The obtained H'W'*N matrix is ​​converted into a feature map of N'*H'*W', that is, it is converted back into an image form. Then, the number of channels is adjusted through a 1x1 convolution operation, and features are further extracted by changing the number of channels. It should be noted that the feature map C'*H'*W' is equivalent to C'×H'×W' in the figure, that is, * in the feature map or matrix is ​​equivalent to ×.

[0076] The adjusted feature map is then added to the graph convolution feature data (image), and the label information is injected to keep the shape of the feature map unchanged while integrating the label information.

[0077] Such fusion operations can make full use of the multi-scale features and label information of the image, further optimize the performance and feature expression capabilities of the network, and the entire process can improve the accuracy and efficiency of image processing tasks.

[0078] In some embodiments, in step S411, a multi-scale feature extraction operation is performed on the original image through a CNN model to obtain multi-scale feature data, including:

[0079] Step S4111, extract features from the original image to obtain a preliminary convolution feature map, and perform multi-scale information extraction on the preliminary convolution feature map to obtain multiple temporary feature data.

[0080] Step S4112, splicing the temporary feature data, and preprocessing the spliced ​​data to obtain temporary data.

[0081] Step S4113, obtaining corresponding temporary multi-scale feature data according to the temporary data and each temporary feature data.

[0082] Step S4113, extract edge information from the preliminary convolution feature map to obtain edge information features.

[0083] Step S4115: All temporary multi-scale feature data and edge information features are merged to obtain multi-scale feature data.

[0084] Specifically, first, the original image is feature extracted to obtain a preliminary convolution feature map. Various feature information in the image, such as texture, shape, etc., can be extracted through the convolution operation.

[0085] Through the preliminary convolution feature map, multi-scale information extraction is performed on the preliminary convolution feature map to obtain temporary feature data corresponding to different scales, that is, feature maps of different scales, such as Figure 3 As shown in the figure, for the multi-scale information extraction process, the dilated convolution operation with different dilation rates can be used to extract the multi-scale information, and then the obtained feature maps are convolutionally distributed along the channel ( Figure 3 As shown in the figure, 4 channels are spliced ​​together with corresponding expansion rates of rate = 1, 3, 5, 7 respectively; then, after global average pooling and channel compression, temporary data matching the number of channels of temporary feature data is obtained, and the temporary data is limited to the range of [0, 1] by sigmoid (activation function), and then multiplied with each temporary feature data, that is, element-wise multiplied with the feature maps (temporary feature data) of different scales mentioned above (this process is an enhancement of temporary feature data, that is, an enhancement process of multi-scale features), to obtain enhanced multi-scale features, that is, temporary multi-scale feature data, and finally, the enhanced multi-scale features (temporary multi-scale feature data) are spliced ​​along the channel direction, and then fused with the edge information features extracted from the original image to obtain multi-scale feature data, that is, edge and attention enhanced multi-scale features. Among them, the edge information contains the contour and boundary information of the object in the image, which helps to improve the image segmentation and recognition capabilities of the classification model.

[0086] Because the sizes of objects in the original image may vary, and convolution extracts local information at a fixed size, using only one type of convolution is not suitable for extracting features of objects with large size differences. Therefore, convolutions with different kernel sizes are used, not only to expand the information flow, but also because features of different scales have different effects on classification. Therefore, a scale attention mechanism is used to give different weights to features of different scales.

[0087] The advantages of this process are: (1) through multi-scale feature extraction and edge information extraction, a richer and more comprehensive representation of ground object features can be obtained, helping to improve the performance of the classification model; (2) the fusion of multi-scale information and edge information can help improve the classification model's grasp of the ground object's morphology and features, making the classification more accurate and reliable. (3) through edge information extraction and fusion, the shape edge features of the ground object can be better captured, improving the classification model's ability to express the ground object's appearance. Therefore, this process can use multi-scale feature extraction, edge information extraction and feature fusion to enhance the expression of ground object features, thereby improving the accuracy and effect of the mining area remote sensing scene classification model.

[0088] In some embodiments, the GCN model includes a multi-layer graph convolutional network; in step S412, the original image is processed by the GCN model to obtain graph convolution feature data, including:

[0089] Step S4121, topology is performed on the original image based on the block method and the nearest neighbor algorithm to obtain a first graph node feature matrix and a first adjacency matrix.

[0090] Step S4122: input the first graph node feature matrix and the first adjacency matrix into the first layer graph convolution network to perform graph convolution operation to obtain a first graph convolution feature image.

[0091] Step S4123, topology is performed on the first graph convolution feature image based on the block method and the nearest neighbor algorithm to obtain the second graph node feature matrix and the second adjacency matrix, and then the second graph node feature matrix and the second adjacency matrix are input into the second layer of graph convolution network for graph convolution operation to obtain the second graph convolution feature image, and the iterative process is repeated until all graph convolution networks complete data processing to obtain graph convolution feature data.

[0092] Specifically, the original image is topologically analyzed based on the block method and the nearest neighbor algorithm to obtain the first graph node feature matrix and the first adjacency matrix: by performing data blocking operations on the image, the node feature matrix and the adjacency matrix in the image are established to construct the graph structure.

[0093] Input the first layer of graph convolution network to perform graph convolution operation to obtain the first graph convolution feature image: perform graph convolution operation on the first graph node feature matrix and adjacency matrix to obtain the feature image after the first layer of convolution.

[0094] Based on the block method and the nearest neighbor algorithm, the first graph convolution feature image is topologically analyzed to obtain the second graph node feature matrix and the second adjacency matrix: the first layer of graph convolution feature image is subjected to data blocking operation to obtain the second-level node features and adjacency matrix.

[0095] Input the second layer of graph convolution network to perform graph convolution operation to obtain the second graph convolution feature image: perform graph convolution operation on the second graph node feature matrix and adjacency matrix to obtain the feature image after the second layer of convolution.

[0096] The above process is repeated and iterated until all graph convolutional network layers complete data processing and obtain the final graph convolutional feature data.

[0097] The original image is topologically reconstructed using the block method and the nearest neighbor algorithm, and the topological structure of the graph is constructed. The nearest neighbor algorithm (k nearest neighbor algorithm) is used to update the topological structure of the graph after each graph convolution layer, so that the model can dynamically reorganize the connection relationship of the graph according to the current feature situation. This can more flexibly adapt to the needs of expressing different image features, thereby improving the classification accuracy of the model.

[0098] Using the k-nearest neighbor algorithm to update the topology of the graph can help the model better capture the neighboring nodes related to each node, thereby obtaining a wider range of image information. Determining the connection relationship between nodes through the nearest neighbor algorithm can make the model focus more on learning the local relationship between nodes. This helps the model better capture the correlation between adjacent pixels in the image, improves the expressiveness of features, and helps improve the model's ability to understand the surrounding environment and contextual information, thereby improving the accuracy of classification tasks.

[0099] Applying a feedforward network to each node enables each node to learn more complex and abstract feature representations. The feedforward network can extract higher-level features through a multi-layer neural network, thereby improving the model's abstraction and representation capabilities for image features. At the same time, applying a feedforward network to each node enables each node to learn informative feature representations. This can reduce the loss of information when it is transmitted in the network and help the model better capture key information and features in the image.

[0100] In some embodiments, in step S4121, the original image is topologically processed based on a block-by-block method and a nearest neighbor algorithm to obtain a first graph node feature matrix and a first adjacency matrix, including:

[0101] Step SA1, segment the original image to obtain multiple image blocks, and linearly embed each image block to obtain a first graph node feature matrix.

[0102] Step SA2: construct the topological structure of the graph according to the first graph node feature matrix and based on the nearest neighbor algorithm to obtain a first adjacency matrix.

[0103] Specifically, the original image is divided into multiple small image blocks to facilitate processing and analysis and improve computational efficiency. Feature extraction and embedding operations are performed on each image block, and the image block is converted into a feature vector representation to form a first graph node feature matrix. For the first graph node feature matrix, the topological relationship of the graph is established according to the nearest neighbor algorithm to determine the connection between the object nodes.

[0104] That is, by processing the original image in a block-based manner, the local features of the image can be better captured, which helps to improve the precision and accuracy of object classification. The topological structure of the graph constructed based on the nearest neighbor algorithm can help model the adjacency and connection between objects, thereby better understanding the spatial relationship between objects. Therefore, by dividing the original image into blocks and constructing a topological structure, the characteristics and connection relationships of the objects can be better characterized, which helps to improve the performance and accuracy of the remote sensing scene classification model in the mining area.

[0105] In some embodiments, in step SA2, the topological structure of the graph is constructed based on the first graph node feature matrix and the nearest neighbor algorithm to obtain the first adjacency matrix, including:

[0106] Step SA21, determine the adjacency relationship graph of each node in the first graph node feature matrix, wherein the distance between each node in the first graph node feature matrix is ​​calculated to find the nearest neighbor node whose distance to the target node is less than a preset threshold, connect the target node with the nearest neighbor node to establish an edge, and construct the adjacency relationship graph of the target node, wherein the target node is any node in the first graph node feature.

[0107] Step SA22, constructing the topological structure of the graph according to all adjacency relationship graphs to obtain an adjacency matrix.

[0108] Specifically, by calculating the node features in the first graph node feature matrix, a distance matrix between different nodes is obtained to find the most similar neighboring nodes. For example, the distance between each node in the first graph node feature matrix is ​​calculated to find the k nearest neighbor nodes (the selection of k can be determined according to the specific actual situation) closest to the target node v (the target node v is any node in the first graph node feature matrix). The target node v is established with the k nearest neighbor nodes respectively, and the adjacency relationship graph of the target node v is constructed. The adjacency relationship graphs of all nodes are integrated to establish the topological structure of the entire graph and obtain the final adjacency matrix.

[0109] That is, by calculating the distance between nodes and establishing adjacency relationships, the similarity and connection relationship between objects can be more accurately represented, which helps to improve the accuracy of the modeling of spatial relationships. Constructing an accurate adjacency relationship graph and adjacency matrix can help the model better understand the connection and spatial distribution between objects, thereby improving the accuracy and precision of mining remote sensing scene classification. By constructing an accurate adjacency relationship structure, the model's feature extraction and classification capabilities can be improved, which helps to optimize the performance and performance of the classification model.

[0110] In some embodiments, the initial classification model further includes a downsampling module. In step S4113, all temporary feature multi-scale data and edge information features are fused to obtain multi-scale feature data, including:

[0111] Step ST1: fuse all temporary multi-scale feature data and edge information features to obtain a temporary feature image.

[0112] Step ST2, downsampling the temporary feature image through a downsampling module to obtain multi-scale feature data, wherein a first down-sampled feature map is obtained by pooling the temporary feature image, the temporary feature image is segmented to obtain multiple temporary images, all temporary images are spliced ​​to obtain a second down-sampled feature map, and the first down-sampled feature map and the second down-sampled feature map are fused to obtain multi-scale feature data.

[0113] Specifically, all temporary multi-scale feature data and edge information features are fused to obtain a temporary feature image. The temporary feature image is processed by the downsampling module to obtain multi-scale feature data. Downsampling operations are often used in image processing to reduce image size while retaining important feature information. The purpose of this step is to further extract features through downsampling operations and ensure the complementarity of features.

[0114] In some embodiments, Figure 4As shown in the figure, the data processing process of the downsampling module is as follows: one branch: the temporary multi-scale feature map (C'*H'*W') is cut into four pieces and spliced ​​along the channel direction to obtain 4C'*H' / 2*W' / 2, and then the channel is compressed by 1*1 convolution to obtain C'*H' / 2*W' / 2 to match the number of feature map channels obtained by maximum pooling; the other branch: the temporary multi-scale feature map (C'*H'*W') is subjected to maximum pooling to obtain C'*H' / 2*W' / 2, and then the two are added together, that is, spliced ​​in the channel direction, and finally 1x1 convolution is used to compress the channel to obtain multi-scale feature data (C' / 2*H' / 2*W' / 2), and two down-sampled feature maps (the first down-sampled feature map and the second down-sampled feature map) are obtained by different down-sampling methods, and the two down-sampled feature maps are fused to reduce the loss of small target feature information.

[0115] In one implementation, if Figure 5 As shown, remote sensing data of the mining area, remote sensing data optical image, radar image and elevation image data are obtained; surface data is extracted according to the elevation image data, and the radar image, elevation image data and surface data are resampled to the same resolution as the optical image to obtain a temporary data set; the temporary data set is fused with the optical image to obtain an initial multimodal scene data set;

[0116] Among them, the CNN model branch extracts features from the original image in the initial multimodal scene data set to obtain a preliminary convolution feature map, and extracts multi-scale information from the preliminary convolution feature map to obtain multiple temporary feature data; splices each temporary feature data, and preprocesses the spliced ​​data to obtain temporary data; obtains corresponding temporary multi-scale feature data based on the temporary data and each temporary feature data; extracts edge information from the preliminary convolution feature map to obtain edge information features; fuses the edge information features and the temporary multi-scale feature data to obtain a temporary feature image, and processes the temporary feature image through the downsampling module to obtain multi-scale feature data.

[0117] The GCN model branch divides the original image into multiple image blocks, each block is used as a node in the graph structure, and each image block is linearly embedded to obtain the graph node feature matrix. The topological structure of the graph is constructed based on the k-nearest neighbor algorithm to obtain the adjacency matrix for graph convolution operations. The graph node feature matrix and the adjacency matrix are input into the multi-layer graph convolution network. The network has four graph convolution layers. After each layer of graph convolution, the k-nearest neighbor algorithm is used to update the topological structure (adjacency matrix) of the graph, and a feedforward network is applied to each node. Finally, the output of the GCN model, namely the graph convolution feature data, is obtained.

[0118] Finally, the multi-scale feature data and the graph convolution feature data are fused to obtain fused feature data, and the fused feature data is classified to obtain a temporary classification result.

[0119] The mining area remote sensing scene classification model construction method described in this embodiment can obtain more comprehensive and diverse surface information by integrating optical image, radar image and elevation image data, and can obtain more comprehensive scene information of the mining area. Compared with the method of using only a single data source, it can provide more accurate classification results, and extract surface data based on elevation image to better understand the terrain characteristics, which is helpful to consider the influence of terrain in remote sensing image classification, improve the model's understanding and recognition of landform features, and help improve the accuracy and generalization ability of the classification model. All remote sensing data are adjusted to the same resolution through resampling technology to ensure the data at the spatial scale. The consistency in the above is conducive to improving the accuracy and reliability of classification. Finally, a classification model is constructed based on the CNN model and the GCN model. Among them, the CNN model can effectively extract the spatial features of the image, that is, the CNN model can be used to capture the spatial distribution, texture features and edge information of the objects in the mining area image, and the GCN model can process the relationship information of the graph data, and can capture the spatial relationship and terrain information between the objects. That is, the connection relationship between the objects can be obtained through the GCN network, which helps to understand the geographic spatial relationship. The combination of the two models can better comprehensively consider the spectral, spatial and terrain characteristics, and improve the classification effect of the mining area remote sensing scene. Compared with the traditional remote sensing classification method, this comprehensive model has stronger feature expression ability and classification accuracy.

[0120] like Figure 6 As shown, another embodiment of the present invention provides a mining area remote sensing scene classification model construction device, comprising:

[0121] The acquisition unit is used to acquire remote sensing data of the mining area, wherein the remote sensing data includes optical images, radar images and elevation image data.

[0122] The processing unit is used to extract surface data based on the elevation image data, and resample the radar image, the elevation image data and the surface data to the same resolution as the optical image to obtain a temporary data set.

[0123] The temporary data set and the optical image are fused to obtain an initial multimodal scene data set.

[0124] A construction unit is used to train and tune the initial classification model according to the initial multimodal scene data set to obtain a remote sensing scene classification model, wherein the initial classification model is based on a CNN model and a GCN model, and the remote sensing scene classification model is used to classify remote sensing scenes in mining areas.

[0125] Another embodiment of the present invention provides a mining area remote sensing scene classification model construction device, including a memory and a processor: the memory is used to store a computer program; the processor is used to implement the mining area remote sensing scene classification model construction method when executing the computer program.

[0126] It should be noted that the device may be a computer device such as a server or a mobile terminal.

[0127] like Figure 7 As shown, another embodiment of the present invention also provides a mining area remote sensing scene classification method, comprising the following steps:

[0128] Step A1, acquiring target remote sensing data of the mining area to be tested, and obtaining a target multimodal scene data set according to the target remote sensing data.

[0129] Step A2, inputting the target multimodal scene data set into the remote sensing scene classification model obtained by the mining area remote sensing scene classification model construction method, and obtaining the remote sensing scene classification result of the mining area to be tested.

[0130] Specifically, first, the remote sensing data (target remote sensing data) of the mining area to be tested can be obtained through aerial photography, satellite remote sensing and other means, including image data obtained by multiple sensors, lidar data, etc. These data contain information on different characteristics in the mining area, such as land use type, vegetation coverage, etc. The data obtained by different sensors and data with different characteristics are integrated together to form a target multimodal scene data set. This data set contains data of various types and characteristics, providing richer information for the subsequent construction of classification models. The target multimodal scene data set is input into the mining area remote sensing scene classification model to obtain the classification result of the target mining area remote sensing scene. By using the constructed remote sensing scene classification model, the remote sensing data of the mining area to be tested is classified and identified to obtain the classification result of the target mining area remote sensing scene, that is, different areas or objects in the mining area are classified and divided, such as mining area type, land use, etc.

[0131] Another embodiment of the present invention further provides a mining area remote sensing scene classification device, comprising:

[0132] The acquisition module is used to acquire the target remote sensing data of the mining area to be tested, and obtain the target multimodal scene data set according to the target remote sensing data.

[0133] The processing module is used to input the target multimodal scene data set into the remote sensing scene classification model obtained by the mining area remote sensing scene classification model construction method to obtain the remote sensing scene classification result of the mining area to be tested.

[0134] It should be noted that the device may be a computer device such as a server or a mobile terminal.

[0135] Figure 8 The internal structure diagram of a computer device in one embodiment is shown. The computer device includes a processor, a memory, a network interface, an input device and a display screen connected through a system bus. Among them, the memory includes a non-volatile storage medium and an internal memory. The non-volatile storage medium of the computer device stores an operating system and may also store a computer program. When the computer program is executed by the processor, the processor can implement a mining area remote sensing scene classification method. The internal memory may also store a computer program. When the computer program is executed by the processor, the processor can implement a mining area remote sensing scene classification method. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covered on the display screen, or a button, trackball or touchpad provided on the housing of the computer device, or an external keyboard, touchpad or mouse, etc.

[0136] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned steps of mining area remote sensing scene classification are implemented.

[0137] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0138] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0139] The above description is only a specific embodiment of the present invention, so that those skilled in the art can understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but should conform to the widest scope consistent with the principles and novel features invented herein.

[0140] Although the present invention is disclosed as above, the protection scope of the present invention is not limited thereto. Those skilled in the art may make various changes and modifications without departing from the spirit and scope of the present invention, and these changes and modifications will fall within the protection scope of the present invention.

Claims

1. A method for constructing a mining area remote sensing scene classification model, characterized in that: include: Acquiring remote sensing data of the mining area, wherein the remote sensing data includes optical images, radar images and elevation image data; Extracting surface data based on the elevation image data, and resampling the radar image, the elevation image data, and the surface data to the same resolution as the optical image to obtain a temporary data set; fusing the temporary data set with the optical image to obtain an initial multimodal scene data set, wherein the initial multimodal scene data set includes a plurality of original images and label image data corresponding to the original images; The initial classification model is trained and tuned according to the initial multimodal scene data set to obtain a remote sensing scene classification model, wherein the initial classification model is based on a CNN model and a GCN model, and the remote sensing scene classification model is used to classify remote sensing scenes in mining areas, including: The original image is classified by the initial classification model to obtain a temporary classification result, including: Performing a multi-scale feature extraction operation on the original image through the CNN model to obtain multi-scale feature data includes: Performing feature extraction on the original image to obtain a preliminary convolution feature map, and performing multi-scale information extraction on the preliminary convolution feature map to obtain multiple temporary feature data; Splicing the temporary feature data, and preprocessing the spliced ​​data to obtain temporary data; Obtaining corresponding temporary multi-scale feature data according to the temporary data and each of the temporary feature data; Extracting edge information from the preliminary convolution feature map to obtain edge information features; The step of fusing all the temporary multi-scale feature data and the edge information features to obtain the multi-scale feature data includes: Fusing all the temporary multi-scale feature data and the edge information features to obtain a temporary feature image, wherein the initial classification model further includes a downsampling module; The temporary feature image is downsampled by the downsampling module to obtain the multi-scale feature data, wherein the temporary feature image is pooled to obtain a first downsampled feature map, the temporary feature image is segmented to obtain a plurality of temporary images, all the temporary images are spliced ​​to obtain a second downsampled feature map, and the first downsampled feature map and the second downsampled feature map are fused to obtain the multi-scale feature data; Performing data processing on the original image through the GCN model to obtain graph convolution feature data; Fusing the multi-scale feature data and the graph convolution feature data to obtain fused feature data, and classifying the fused feature data to obtain the temporary classification result; The initial classification model is optimized according to the temporary classification result and the corresponding label image data, and the optimized initial classification model is used as the remote sensing scene classification model.

2. The method for constructing a mining area remote sensing scene classification model according to claim 1, characterized in that: The GCN model includes a multi-layer graph convolution network; the data processing of the original image by the GCN model to obtain graph convolution feature data includes: The original image is topologically processed based on a block-by-block method and a nearest neighbor algorithm to obtain a first graph node feature matrix and a first adjacency matrix; Inputting the first graph node feature matrix and the first adjacency matrix into a first-layer graph convolutional network to perform a graph convolution operation to obtain a first graph convolution feature image; The first graph convolution feature image is topologically processed based on the block method and the nearest neighbor algorithm to obtain a second graph node feature matrix and a second adjacency matrix. The second graph node feature matrix and the second adjacency matrix are then input into a second-layer graph convolution network for graph convolution operation to obtain a second graph convolution feature image. The iterative process is repeated until all the graph convolution networks complete data processing to obtain the graph convolution feature data.

3. The method for constructing a mining area remote sensing scene classification model according to claim 2, characterized in that: The topology of the original image is performed based on the block method and the nearest neighbor algorithm to obtain a first graph node feature matrix and a first adjacency matrix, including: Segmenting the original image to obtain a plurality of image blocks, and linearly embedding each of the image blocks to obtain the first graph node feature matrix; According to the first graph node feature matrix and based on a nearest neighbor algorithm, a topological structure of the graph is constructed to obtain the first adjacency matrix.

4. The method for constructing a mining area remote sensing scene classification model according to claim 2, characterized in that: The step of constructing the topological structure of the graph based on the first graph node feature matrix and the nearest neighbor algorithm to obtain the first adjacency matrix includes: Determine an adjacency relationship graph of each node in the first graph node feature matrix, wherein the distance between each node in the first graph node feature matrix is ​​calculated to find a nearest neighbor node whose distance to a target node is less than a preset threshold, connect the target node to the nearest neighbor node to establish an edge, and construct an adjacency relationship graph of the target node, wherein the target node is any node in the first graph node feature; The topological structure of the graph is constructed according to all the adjacency relationship graphs to obtain the adjacency matrix.

5. A remote sensing scene classification model construction device, characterized in that: include: An acquisition unit, used to acquire remote sensing data of a mining area, wherein the remote sensing data includes optical images, radar images and elevation image data; A processing unit is used to extract surface data based on the elevation image data, and resample the radar image, the elevation image data and the surface data to the same resolution as the optical image to obtain a temporary data set; fuse the temporary data set with the optical image to obtain an initial multimodal scene data set, wherein the initial multimodal scene data set includes a plurality of original images and label image data corresponding to the original images; A construction unit is used to train and tune the initial classification model according to the initial multimodal scene data set to obtain a remote sensing scene classification model, wherein the initial classification model is based on a CNN model and a GCN model, and the remote sensing scene classification model is used to classify remote sensing scenes in mining areas, including: classifying the original image through the initial classification model to obtain a temporary classification result, including: performing a multi-scale feature extraction operation on the original image through the CNN model to obtain multi-scale feature data, including: extracting features from the original image to obtain a preliminary convolution feature map, and extracting multi-scale information from the preliminary convolution feature map to obtain multiple temporary feature data; splicing each of the temporary feature data, and preprocessing the spliced ​​data to obtain temporary data; obtaining corresponding temporary multi-scale feature data according to the temporary data and each of the temporary feature data; extracting edge information from the preliminary convolution feature map to obtain edge information features; fusing all of the temporary multi-scale feature data and the edge information features to obtain the multi-scale feature data, including The method comprises: fusing all the temporary multi-scale feature data and the edge information features to obtain a temporary feature image, wherein the initial classification model further includes a downsampling module; performing a downsampling operation on the temporary feature image through the downsampling module to obtain the multi-scale feature data, wherein the temporary feature image is pooled to obtain a first down-sampled feature map, the temporary feature image is segmented to obtain a plurality of temporary images, all the temporary images are spliced ​​to obtain a second down-sampled feature map, the first down-sampled feature map and the second down-sampled feature map are fused to obtain the multi-scale feature data; performing data processing on the original image through the GCN model to obtain graph convolution feature data; fusing the multi-scale feature data and the graph convolution feature data to obtain fused feature data, and classifying the fused feature data to obtain the temporary classification result; optimizing the initial classification model according to the temporary classification result and the corresponding label image data, and using the optimized initial classification model as the remote sensing scene classification model.

6. A mining area remote sensing scene classification method, characterized in that: include: Acquire target remote sensing data of the mining area to be tested, and obtain a target multimodal scene data set based on the target remote sensing data; The target multimodal scene data set is input into the remote sensing scene classification model obtained by the mining area remote sensing scene classification model construction method according to any one of claims 1 to 4 to obtain the remote sensing scene classification result of the mining area to be tested.

Images