A lithology scene classification method, electronic device and storage medium

Abstract

The present invention provides a lithology scene classification method, electronic device and storage medium, and relates to the field of image processing technology. The method comprises: inputting the acquired remote sensing image and knowledge data of the lithology scene into a trained lithology scene classification model to obtain the lithology scene classification result, wherein the lithology scene classification model comprises a transformer module, a graph attention network module, a cross-modal cross-attention module, an adaptive feature dimension reduction module and an output module; the graph attention network module is used to mine the knowledge features of the knowledge data; the cross-modal cross-attention module is used to fuse image features and knowledge features to generate lithology sensitive feature data; the adaptive feature dimension reduction module is used to reduce the dimension of the lithology sensitive feature data to generate reduced dimension features; the output module is used to generate the lithology scene classification result according to the reduced dimension features. The present invention can improve the accuracy of lithology scene classification.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a lithology scene classification method, electronic equipment and storage medium. Background Art

[0002] Lithology scene classification refers to the process of automatically classifying rock types using image recognition technology. This process usually involves the analysis of geological images in order to identify different types of rocks and their distribution in specific scenarios. Lithology scene classification is of great significance for geological research, mineral exploration, environmental protection, and infrastructure construction.

[0003] In the prior art, classification algorithms are generally used to perform image recognition on remote sensing images and lithological knowledge of lithological scenes to obtain lithological scene classification results. However, lithological knowledge includes geological characteristics such as co-occurrence and mutual occurrence relationships of lithologies. Due to the different representation methods of various geological characteristics, the data sources of remote sensing images and lithological knowledge are different, resulting in low accuracy of classification algorithms in classifying lithological scenes using remote sensing images and lithological knowledge of lithological scenes. Summary of the invention

[0004] The problem solved by the present invention is how to improve the accuracy of lithology scene classification.

[0005] In a first aspect, the present invention provides a lithology scene classification method, comprising:

[0006] Input the acquired remote sensing image and knowledge data of the lithological scene into the trained lithological scene classification model to obtain the lithological scene classification result, wherein the lithological scene classification model includes a transformer module, a graph attention network module, a cross-modal cross-attention module, an adaptive feature dimension reduction module and an output module; the output end of the graph attention network and the output end of the transformer model are respectively connected to the input end of the cross-modal cross-attention module, and the cross-modal cross-attention module, the adaptive feature dimension reduction module and the output module are connected in sequence;

[0007] The transformer module is used to mine the image features of the remote sensing image;

[0008] The graph attention network module is used to mine the knowledge features of the knowledge data;

[0009] The cross-modal cross-attention module is used to fuse the image features and the knowledge features to generate lithology sensitive feature data;

[0010] The adaptive feature dimension reduction module is used to reduce the dimension of the lithology sensitive feature data to generate reduced dimension features;

[0011] The output module is used to generate the lithology scene classification result according to the dimensionality reduction feature.

[0012] Optionally, the remote sensing image includes spectral data and topographic data, the image features include spectral features and topographic features, and the transformer module includes a first TransXNet network and a second TransXNet network, the first TransXNet network is used to process the spectral data and generate the spectral features, and the second TransXNet network is used to process the topographic data and generate the topographic features.

[0013] Optionally, the first TransXNet network includes a first processing layer and at least one second processing layer, the first processing layer and at least one second processing layer are connected in sequence, the first processing layer includes a first patch embedding layer and a transformer block, and the second processing layer includes a second patch embedding layer and the transformer block;

[0014] The first patch embedding layer is used to process the spectral data to generate a first spectral vector;

[0015] The transformer block of the first processing layer is used to process the first spectral vector to generate a first spectral vector feature;

[0016] The second patch embedding layer is used to process the first spectrum vector feature or the second spectrum vector feature output by the previous second processing layer to generate a second spectrum vector;

[0017] The transformer block of the second processing layer is used to process the second spectral vector to generate the second spectral vector feature, and the second spectral vector feature generated by the last transformer block is the spectral feature.

[0018] Optionally, the transformer block includes a dynamic position encoding layer, a first normalization layer, a dynamic token mixer, a first integration layer, a second normalization layer, a multi-scale shuffle feedforward network and a second integration layer, wherein the dynamic position encoding layer, the first normalization layer, the dynamic token mixer, the first integration layer, the second normalization layer, the multi-scale shuffle feedforward network and the second integration layer are connected in sequence, and the output end of the dynamic position encoding layer is also connected to the input end of the first integration layer, and the output end of the second normalization layer is also connected to the input end of the second integration layer;

[0019] The dynamic position encoding layer is used to perform position encoding on the first spectral vector or the second spectral vector to generate spectral encoding data;

[0020] The first normalization layer is used to normalize the spectral encoding data to generate encoding normalized data;

[0021] The dynamic token mixer is used to perform mixing conversion on the encoded normalized data to generate mixing conversion data;

[0022] The first integration layer is used to integrate the spectrum encoding data and the mixing conversion data to generate first integrated data;

[0023] The second normalization layer is used to normalize the first integrated data to generate integrated normalized data;

[0024] The multi-scale shuffled feedforward network is used to perform channel shuffling operations and forward propagation operations on the integrated normalized data to generate multi-scale shuffled data;

[0025] The second integration layer is used to integrate the integrated normalized data and the multi-scale shuffled data to generate the first spectral vector feature or the second spectral vector feature.

[0026] Optionally, the dynamic token mixer includes a first convolution layer, a second convolution layer, an overlapping spatial dimension reduction attention layer, an input-dependent deep convolution layer, and a compressed token enhancer, wherein the first convolution layer is connected to the overlapping spatial dimension reduction attention layer, the second convolution layer is connected to the input-dependent deep convolution layer, and the output ends of the overlapping spatial dimension reduction attention layer and the input-dependent deep convolution layer are both connected to the input end of the compressed token enhancer;

[0027] The first convolutional layer is used to map the encoded normalized data to generate a first feature map;

[0028] The second convolutional layer maps the encoded normalized data to generate a second feature map;

[0029] The overlapping space dimension reduction attention layer is used to reduce the overlapping space of the first feature map to extract global information;

[0030] The input-dependent deep convolution layer is used to adjust convolution kernel parameters according to the second feature map, and extract local information according to the convolution kernel parameters;

[0031] The compressed token enhancer is used to perform token aggregation on the global information and the local information to generate the mixed frequency conversion data.

[0032] Optionally, before inputting the acquired remote sensing image and knowledge data of the lithology scene into a trained lithology scene classification model to obtain the lithology scene classification result, the method further includes:

[0033] Constructing an initial training model and first-scale training data, and training the initial training model according to the first-scale training data to obtain a first training model;

[0034] Constructing second-scale training data according to the first-scale training data, and training the first training model according to the second-scale training data to obtain a second training model;

[0035] According to the second-scale training data, third-scale training data are constructed, and the second training model is trained according to the third-scale training data to obtain a third training model, and the third training model is used as the trained lithology scene classification model.

[0036] Optionally, the adaptive feature dimensionality reduction module includes a third convolutional layer, a BN layer and an activation function layer, and the third convolutional layer, the BN layer and the activation function layer are connected in sequence.

[0037] Optionally, the lithology scene classification model further includes a loss module, and the loss module is respectively connected to the graph attention network, the transformer model, the cross-modal cross attention module, the adaptive feature dimension reduction module and the output module;

[0038] The loss module is used to perform feature comparison between the dimensionality reduction features, the image features and the knowledge features to generate a contrast loss, and is also used to generate a classification loss based on the lithology scene classification results, and to update the parameters of the graph attention network, the transformer model and the cross-modal cross attention module respectively based on the contrast loss and the classification loss.

[0039] In a second aspect, the present invention provides an electronic device, including a memory and a processor;

[0040] The memory is used to store computer programs;

[0041] The processor is used to implement the lithology scene classification method as described in the first aspect when executing the computer program.

[0042] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the lithology scene classification method as described in the first aspect is implemented.

[0043] The beneficial effects of the lithology scene classification method, electronic device and storage medium of the present invention are:

[0044] The remote sensing images and knowledge data with different data sources are input into the trained lithology scene classification model. The remote sensing images and knowledge data are processed respectively by the Transformer module and the graph attention network module in the trained lithology scene classification model. Since the Transformer module is good at processing sequence data and can capture long-distance dependencies, it can effectively process remote sensing images. The graph attention network module is suitable for processing structured knowledge data and can better understand the inherent structure of knowledge data. The corresponding image features and knowledge features are extracted through the Transformer module and the graph attention network module, which have better representation capabilities, and then through cross-modal cross The attention module effectively integrates these two different types of features based on the cross-modal cross-attention mechanism, balances the differences between remote sensing images and knowledge data sources, and utilizes the association between image features and knowledge features to generate more comprehensive and accurate lithology-sensitive feature data, thereby improving the accuracy of lithology scene classification. The lithology-sensitive feature data is then reduced in dimension through the adaptive feature dimensionality reduction module to generate reduced-dimensional features, which not only reduces the computational cost, but also removes redundant information and retains the most relevant and discriminative features, which helps to improve the accuracy of lithology scene classification and reduce the risk of overfitting. Finally, the output module generates high-precision lithology scene classification results to complete the classification task. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 A schematic diagram of the structure of a lithology scene classification model provided by an embodiment of the present invention;

[0046] Figure 2 A schematic diagram of the structure of a first TransXNet network and a second TransXNet network provided in an embodiment of the present invention;

[0047] Figure 3 A schematic diagram of the structure of a transformer block provided in an embodiment of the present invention;

[0048] Figure 4 A schematic diagram of the structure of a dynamic token mixer provided in an embodiment of the present invention;

[0049] Figure 5 A schematic diagram of the structure of a multi-scale shuffled feedforward network provided in an embodiment of the present invention;

[0050] Figure 6 A schematic diagram of the structure of a lithology scene classification model provided by another embodiment of the present invention;

[0051] Figure 7 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0052] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. Although certain embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be interpreted as being limited to the embodiments described herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present invention. It should be understood that the drawings and embodiments of the present invention are only for exemplary purposes and are not intended to limit the scope of protection of the present invention.

[0053] It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0054] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to"; the term "based on" means "based at least in part on"; the term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments"; the term "optionally" means "optional embodiments". The relevant definitions of other terms will be given in the following description. It should be noted that the concepts of "first", "second", etc. mentioned in the present invention are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0055] It should be noted that the modifications of "one" and "plurality" mentioned in the present invention are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0056] The names of the messages or information exchanged between multiple devices in the embodiments of the present invention are only used for illustrative purposes, and are not used to limit the scope of these messages or information.

[0057] In view of the problems existing in the above-mentioned related technologies, this embodiment provides a lithology scene classification method, an electronic device and a storage medium.

[0058] An embodiment of the present invention provides a lithology scene classification method, including:

[0059] The acquired remote sensing images and knowledge data of the lithological scene are input into the trained lithological scene classification model to obtain the lithological scene classification result, wherein the lithological scene classification model includes a transformer module, a graph attention network module, a cross-modal cross-attention module, an adaptive feature dimensionality reduction module and an output module; the output end of the graph attention network and the output end of the transformer model are respectively connected to the input end of the cross-modal cross-attention module, and the cross-modal cross-attention module, the adaptive feature dimensionality reduction module and the output module are connected in sequence.

[0060] Specifically, remote sensing images of lithology scenes are generated by data collected by sensors on satellites or aircraft, which can reveal the characteristics of rocks and geological structures on the earth's surface. The lithology knowledge of lithology scenes contains geological features such as the co-occurrence and mutual occurrence relationship of lithology, such as geological maps. By inputting the acquired remote sensing images and knowledge data of lithology scenes into the trained lithology scene classification model, accurate lithology scene classification results can be obtained. Figure 1 As shown, the lithology scene classification model includes a transformer module, a graph attention network module, a cross-modal cross-attention module, an adaptive feature dimensionality reduction module and an output module; the output end of the graph attention network and the output end of the transformer model are respectively connected to the input end of the cross-modal cross-attention module, and the cross-modal cross-attention module, the adaptive feature dimensionality reduction module and the output module are connected in sequence.

[0061] The transformer module is used to mine the image features of the remote sensing image.

[0062] Specifically, a transformer module is used to mine the image features of the remote sensing image. The transformer module can perform efficient parallel calculations through its own self-attention mechanism, consider the information of all positions in the sequence and the global contextual relationships, and effectively capture long-distance dependencies. By observing the self-attention weights, it is possible to understand the parts that the model focuses on when making decisions, thereby improving the model's interpretability to obtain image features with more representation capabilities.

[0063] The graph attention network module is used to mine the knowledge features of the knowledge data.

[0064] Specifically, the graph attention network module encodes the knowledge data to obtain the encoded features and then mines them.

[0065] Exemplarily, the graph attention network module includes multiple single-layer graph attention layers to encode the knowledge data, and the encoded features obtained are a set of lithology node features Where N is the number of nodes, R is a real number set, i is the i-th node, and F is the number of features of each lithology node. is the lithology node feature of the i-th node. The output of the single-layer graph attention layer is a set of new node features Among them, F′ is the number of features of the output node, is the new lithology node feature of the i-th node. In order to obtain sufficient expression ability to transform the input features into higher-level features, a linear transformation W∈R is used for each lithology node. F′×F Then the attention coefficient formula is used to perform self-attention on the lithology node to calculate the attention coefficient. The attention coefficient formula includes:

[0066]

[0067] Among them, e ij is the attention coefficient, which indicates the importance of the feature of node j to node i, a is the attention network, W is the projection matrix, is the lithology node characteristic of the jth node.

[0068] Each lithology node participates in the calculation of other lithology nodes. By using the masked attention mechanism to inject the graph structure, the lithology nodes are calculated. ij , j∈N i , N i are the neighbors of node i in the graph, normalized using the softmax function:

[0069]

[0070] Among them, the attention mechanism α is a single-layer feedforward neural network ′ Network, α ij is the attention weight, i.e. the normalized result, k is the kth node, and the weight vector Parameterization, and use LeakyReLU for nonlinear operation, e ik is the attention coefficient, which indicates the importance of the feature of node k to node i. After full expansion, the coefficient calculated by the attention mechanism is:

[0071]

[0072] in, is the weight vector The normalized attention weights are used to calculate the linear combination of the corresponding features as the output features of each lithology node. The output features are:

[0073]

[0074] in, is the output feature, and σ represents the activation function, such as ReLU, sigmoid, etc.

[0075] In order to stabilize the learning process of self-attention, the graph attention network module uses a multi-head attention mechanism, that is, it uses K independent attention mechanisms and then connects their features to obtain the following output feature representation.

[0076]

[0077] Among them, ∥ represents the splicing operation, represents the normalized attention coefficient calculated by the kth attention mechanism, W k is the weight matrix for the corresponding linear transformation of the input.

[0078] The multi-head self-attention mechanism calculates different attention heads multiple times to obtain multiple attention scores and feature combinations. Finally, the average operation is used in the nonlinear layer at the end of the graph attention network module to average the results of these attention heads to obtain a stable and comprehensive feature representation. The image features of the remote sensing image are finally obtained as follows:

[0079]

[0080] The cross-modal cross-attention module is used to fuse the image features and the knowledge features to generate lithology-sensitive feature data.

[0081] Specifically, the cross-modal cross-attention module adopts a cross-modal cross-attention mechanism to calculate the weights between the image features and the knowledge features to determine their importance in the attention mechanism, thereby fusing them and outputting lithology sensitive feature data. First, the cross-modal cross-attention module uses a cross-attention weight calculation formula to confirm the correlation between vectors. The cross-attention weight calculation formula includes:

[0082]

[0083] Among them, E ij Represents vector q i and k j The correlation between i represents the query vector, k j represents the key vector and T is the transpose operation.

[0084] Then, according to the correlation between the vectors, the attention weight vector is determined using the cross-attention output calculation formula. The cross-attention output calculation formula includes:

[0085]

[0086] Among them, a irepresents the attention weight vector, v j represents the value vector, E iI is the i-th query vector q i and the I-th key vector k I The relevance score between them is calculated by the dot product and reflects the similarity between the query and the key.

[0087] After obtaining the weight vectors of image features and knowledge features respectively through the above formula, the weight vectors are used to determine which features have higher importance in the fusion process, and the image features and knowledge features of different modalities are weightedly combined to obtain lithology sensitive feature data to achieve the fusion of image features and knowledge features.

[0088] The adaptive feature dimension reduction module is used to reduce the dimension of the lithology sensitive feature data to generate reduced dimension features.

[0089] Specifically, the adaptive feature dimensionality reduction module reduces the dimensionality of high-dimensional lithology-sensitive feature data to obtain a feature map with a channel number of 1, namely, a dimensionality reduction feature, so as to reduce the feature complexity and at the same time reduce the computational cost, remove redundant information, and retain the most relevant and discriminative features, thereby improving the classification accuracy and reducing the risk of overfitting.

[0090] The output module is used to generate the lithology scene classification result according to the dimensionality reduction feature.

[0091] Specifically, the output module performs classification according to the dimension reduction features through the classifier, i.e., the fully connected layer, to obtain the lithology scene classification results.

[0092] In this embodiment, remote sensing images and knowledge data with differential data sources are input into a trained lithology scene classification model, and the remote sensing images and knowledge data are processed respectively by the Transformer module and the graph attention network module in the trained lithology scene classification model. Since the Transformer module is good at processing sequence data and can capture long-distance dependencies, it can effectively process remote sensing images. The graph attention network module is suitable for processing structured knowledge data and can better understand the inherent structure of knowledge data. The corresponding image features and knowledge features are extracted through the Transformer module and the graph attention network module, which have better representation capabilities. Then, the cross-modal cross-attention module can be used according to the cross-modal cross-attention mechanism to obtain the image features and knowledge features respectively using the above formulas. After the weight vector is obtained, the weight vector is used to determine which features have higher importance in the fusion process, and the image features and knowledge features of different modalities are weighted and combined to obtain lithology sensitive feature data, which effectively fuses these two different types of features, balances the differences between remote sensing images and knowledge data sources, and utilizes the association between image features and knowledge features to generate more comprehensive and accurate lithology sensitive feature data, thereby improving the accuracy of lithology scene classification. Then, the lithology sensitive feature data is reduced in dimension through the adaptive feature dimensionality reduction module to generate reduced dimensionality features, which not only reduces the computational cost, but also may remove redundant information and retain the most relevant and discriminative features, which helps to improve the accuracy of lithology scene classification and reduce the risk of overfitting. Finally, the output module generates high-precision lithology scene classification results to complete the classification task.

[0093] Optionally, the remote sensing image includes spectral data and topographic data, the image features include spectral features and topographic features, and the transformer module includes a first TransXNet network and a second TransXNet network, the first TransXNet network is used to process the spectral data and generate the spectral features, and the second TransXNet network is used to process the topographic data and generate the topographic features.

[0094] Specifically, the remote sensing image includes spectral data and terrain data, the spectral data includes multispectral data and hyperspectral data, the transformer module includes a first TransXNet network and a second TransXNet network, the first TransXNet network and the second TransXNet network have the same structure, but process different data, the first TransXNet network is used to process the spectral data and generate spectral features, and the second TransXNet network is used to process the terrain data and generate terrain features. In this embodiment, the cross-modal cross-attention module is also used to fuse spectral features, terrain features, and knowledge features to generate lithology-sensitive feature data.

[0095] Optionally, the first TransXNet network includes a first processing layer and at least one second processing layer, the first processing layer and at least one second processing layer are connected in sequence, the first processing layer includes a first patch embedding layer and a transformer block, and the second processing layer includes a second patch embedding layer and the transformer block;

[0096] The first patch embedding layer is used to process the spectral data to generate a first spectral vector;

[0097] The transformer block of the first processing layer is used to process the first spectral vector to generate a first spectral vector feature;

[0098] The second patch embedding layer is used to process the first spectrum vector feature or the second spectrum vector feature output by the previous second processing layer to generate a second spectrum vector;

[0099] The transformer block of the second processing layer is used to process the second spectral vector to generate the second spectral vector feature, and the second spectral vector feature generated by the last transformer block is the spectral feature.

[0100] Specifically, Figure 2As shown, the first TransXNet network includes a first processing layer and at least one second processing layer, the first processing layer and the at least one second processing layer are connected sequentially, the first processing layer includes a first patch embedding layer and a transformer block, the second processing layer includes a second patch embedding layer and a transformer block, the first patch embedding layer is implemented by a 7×7 convolution layer with a stride of 4 and a batch normalization layer BN, and the second patch embedding layer is implemented by a 3×3 convolution layer with a stride of 2 and a batch normalization layer BN. The first patch embedding layer processes the spectral data to generate a first spectral vector, the transformer block of the first processing layer processes the first spectral vector to generate a first spectral vector feature, and passes the first spectral vector feature to the second processing layer, the second patch embedding layer of the first second processing layer processes the first spectral vector feature to generate a second spectral vector, the transformer block of the first second processing layer processes the second spectral vector to generate a second spectral vector feature, and then passes the second spectral vector feature to the next second processing layer, the second patch embedding layer of the next second processing layer processes the second spectral vector feature output by the previous second processing layer to generate a second spectral vector, and so on, until the last second processing layer, the second spectral vector feature generated by the transformer block of the last second processing layer is used as the spectral feature. It should be understood that the number of second processing layers depends on the actual situation. This embodiment provides three second processing layers to achieve the best effect and input-output ratio.

[0101] Optionally, the transformer block includes a dynamic position encoding layer, a first normalization layer, a dynamic token mixer, a first integration layer, a second normalization layer, a multi-scale shuffle feedforward network and a second integration layer, wherein the dynamic position encoding layer, the first normalization layer, the dynamic token mixer, the first integration layer, the second normalization layer, the multi-scale shuffle feedforward network and the second integration layer are connected in sequence, and the output end of the dynamic position encoding layer is also connected to the input end of the first integration layer, and the output end of the second normalization layer is also connected to the input end of the second integration layer;

[0102] The dynamic position encoding layer is used to perform position encoding on the first spectral vector or the second spectral vector to generate spectral encoding data;

[0103] The first normalization layer is used to normalize the spectral encoding data to generate encoding normalized data;

[0104] The dynamic token mixer is used to perform mixing conversion on the encoded normalized data to generate mixing conversion data;

[0105] The first integration layer is used to integrate the spectrum encoding data and the mixing conversion data to generate first integrated data;

[0106] The second normalization layer is used to normalize the first integrated data to generate integrated normalized data;

[0107] The multi-scale shuffled feedforward network is used to perform channel shuffling operations and forward propagation operations on the integrated normalized data to generate multi-scale shuffled data;

[0108] The second integration layer is used to integrate the integrated normalized data and the multi-scale shuffled data to generate the first spectral vector feature or the second spectral vector feature.

[0109] Specifically, Figure 3 As shown, the transformer block includes a dynamic position coding layer, a first normalization layer, a dynamic token mixer, a first integration layer, a second normalization layer, a multi-scale shuffle feedforward network and a second integration layer. The dynamic position coding layer, the first normalization layer, the dynamic token mixer, the first integration layer, the second normalization layer, the multi-scale shuffle feedforward network and the second integration layer are connected in sequence, and the output end of the dynamic position coding layer is also connected to the input end of the first integration layer, and the output end of the second normalization layer is also connected to the input end of the second integration layer; the dynamic position coding layer of the transformer block of the first processing layer is used to perform position coding on the first spectral vector to generate spectral coding data; the dynamic position coding layer of the transformer block of the second processing layer is used to perform position coding on the second spectral vector, Generate spectral coded data; the first normalization layer is used to normalize the spectral coded data to generate coded normalized data; the dynamic token mixer aggregates global information and local details in an input-dependent manner, and applies efficient global attention modules and input-dependent deep convolutions to evenly divided feature segments, thereby giving the network a powerful inductive bias and an expanded effective receptive field, and then performs mixing conversion on the coded normalized data to generate mixing conversion data; the first integration layer is used to integrate the spectral coded data and the mixing conversion data, that is, the concat operation, to generate the first integrated data; the second normalization layer is used to normalize the first integrated data to generate integrated normalized data; the multi-scale shuffled feedforward network is used to perform channel shuffling operations and forward propagation operations on the integrated normalized data to generate multi-scale shuffled data.

[0110] Optionally, the dynamic token mixer includes a first convolution layer, a second convolution layer, an overlapping spatial dimension reduction attention layer, an input-dependent deep convolution layer, and a compressed token enhancer, wherein the first convolution layer is connected to the overlapping spatial dimension reduction attention layer, the second convolution layer is connected to the input-dependent deep convolution layer, and the output ends of the overlapping spatial dimension reduction attention layer and the input-dependent deep convolution layer are both connected to the input end of the compressed token enhancer;

[0111] The first convolutional layer is used to map the encoded normalized data to generate a first feature map;

[0112] The second convolutional layer maps the encoded normalized data to generate a second feature map;

[0113] The overlapping space dimension reduction attention layer is used to reduce the overlapping space of the first feature map to extract global information;

[0114] The input-dependent deep convolution layer is used to adjust convolution kernel parameters according to the second feature map, and extract local information according to the convolution kernel parameters;

[0115] The compressed token enhancer is used to perform token aggregation on the global information and the local information to generate the mixed frequency conversion data.

[0116] Specifically, Figure 4 As shown, the dynamic token mixer includes a first convolution layer, a second convolution layer, an overlapping spatial dimension reduction attention layer, an input-dependent deep convolution layer, and a compressed token enhancer. The first convolution layer is connected to the overlapping spatial dimension reduction attention layer, the second convolution layer is connected to the input-dependent deep convolution layer, and the output ends of the overlapping spatial dimension reduction attention layer and the input-dependent deep convolution layer are both connected to the input end of the compressed token enhancer; the first convolution layer is used to map the encoded normalized data to generate a first feature map; the second convolution layer maps the encoded normalized data to generate a second feature map; the first convolution layer and the second convolution layer are both 1×1 convolutions, which perform linear transformations between channels on the input feature map to achieve feature separation, which helps to capture different feature information separately. And the 1×1 convolution maps the original features to a new feature space through linear transformation, and then a nonlinear activation function (such as ReLU) can be applied to enhance the nonlinear representation ability of the model. The overlapping space dimension reduction attention layer is used to reduce the overlapping space of the first feature map and extract global information; the input-dependent deep convolution layer is used to adjust the convolution kernel parameters according to the second feature map, and extract local information according to the convolution kernel parameters; the compressed token enhancer is used to tokenize the global information and the local information to generate mixed frequency conversion data.

[0117] In one embodiment, Figure 5 As shown, the multi-scale shuffled feedforward network includes a first 1x1 convolution, a second 1x1 convolution, a 3x3 convolution, a 5x5 convolution, a 7x7 convolution, a third integration layer, a channel shuffle layer and a third 1x1 convolution, the input ends of the second 1x1 convolution, the 3x3 convolution, the 5x5 convolution and the 7x7 convolution are respectively connected to the output end of the first 1x1 convolution, the output ends of the second 1x1 convolution, the 3x3 convolution, the 5x5 convolution and the 7x7 convolution are respectively connected to the input end of the third integration layer, and the third integration layer, the channel shuffle layer and the third 1x1 convolution are connected in sequence; After the integrated normalized data is processed by the first 1x1 convolution, four parallel convolutions of different scales are used. Each convolution processes a quarter of the channel. The four parallel convolutions of different scales can effectively capture multi-scale information. Then, the outputs of the four parallel convolutions of different scales are integrated through the third integration layer, that is, the concat operation, and then enter the channel shuffling layer to perform channel shuffling operations to improve the feature representation capability and promote information exchange in the channel dimension. Finally, the multi-scale shuffled data is obtained through the third 1x1 convolution processing; wherein, the channel shuffling operation includes:

[0118] If the shape of the input feature map X is (H, W, C), it is divided into g groups, each with channels, and then reshape:

[0119] X reshaped =Reshape(X,(H,W,g,C g ));

[0120] Among them, Reshape is the reshaping operation, X reshaped is the reshaped result. The reshaped shape is (H, W, g, C g ).

[0121] Then transpose:

[0122] X transposed =Transpose(X reshaped , (0, 1, 3, 2));

[0123] Among them, Transpose is the transposition operation, X transposed is the transposed result. The shape after transposition is (H, W, C g , g).

[0124] Finally reshape it back to its original shape:

[0125] X shuffled =Reshape(X transposed , (H, W, C));

[0126] Among them, X shuffledThe final shape is restored to (H, W, C) as a shuffled result, indicating that the channel order has been shuffled.

[0127] Optionally, before inputting the acquired remote sensing image and knowledge data of the lithology scene into a trained lithology scene classification model to obtain the lithology scene classification result, the method further includes:

[0128] Constructing an initial training model and first-scale training data, and training the initial training model according to the first-scale training data to obtain a first training model;

[0129] Constructing second-scale training data according to the first-scale training data, and training the first training model according to the second-scale training data to obtain a second training model;

[0130] According to the second-scale training data, third-scale training data are constructed, and the second training model is trained according to the third-scale training data to obtain a third training model, and the third training model is used as the trained lithology scene classification model.

[0131] Specifically, before the acquired remote sensing images and knowledge data of the lithological scene are input into the trained lithological scene classification model to obtain the lithological scene classification result, it is necessary to construct the lithological scene classification model and train it. In order to enable the lithological scene classification model to process knowledge data of different scales and remote sensing images of different resolutions, an initial training model and first-scale training data are first constructed, and the initial training model is trained with the first-scale training data to obtain a first training model. Then, based on the first-scale training data, second-scale training data with a different scale and resolution from the first-scale training data are constructed. Through model migration, the first training model is trained according to the second-scale training data to obtain a second training model. Finally, based on the second-scale training data, third-scale training data with a different scale and resolution from the second-scale training data are constructed. Through model migration, the second training model is trained according to the third-scale training data to obtain a third training model. The third training model is used as the trained lithological scene classification model, so that the trained lithological scene classification model has the ability to process knowledge data of different scales and remote sensing images of different resolutions. For example, first-scale training data, second-scale training data, and third-scale training data can be constructed in sequence from small scale to large scale and from low resolution to high resolution, so as to gradually train through a progressive enhancement strategy to deepen and capture the model's understanding of lithological characteristics at different levels and improve the model's classification accuracy for lithological scenes.

[0132] Optionally, the adaptive feature dimensionality reduction module includes a third convolutional layer, a BN layer and an activation function layer, and the third convolutional layer, the BN layer and the activation function layer are connected in sequence.

[0133] Specifically, the adaptive feature dimensionality reduction module includes a third convolutional layer, a BN layer and an activation function layer, and the third convolutional layer, the BN layer and the activation function layer are superimposed in sequence to complete the dimensionality reduction operation.

[0134] Optionally, the lithology scene classification model further includes a loss module, and the loss module is respectively connected to the graph attention network, the transformer model, the cross-modal cross attention module, the adaptive feature dimension reduction module and the output module;

[0135] The loss module is used to perform feature comparison between the dimensionality reduction features, the image features and the knowledge features to generate a contrast loss, and is also used to generate a classification loss based on the lithology scene classification results, and to update the parameters of the graph attention network, the transformer model and the cross-modal cross attention module respectively based on the contrast loss and the classification loss.

[0136] Specifically, Figure 6 As shown, the lithology scene classification model also includes a loss module, which is respectively connected to the graph attention network, the transformer model, the cross-modal cross-attention module, the adaptive feature dimension reduction module and the output module; the loss module is used to compare the dimension reduction features with the image features and the knowledge features to generate a contrast loss, that is, to obtain the knowledge loss and the image loss, and when the remote sensing image includes spectral data and terrain data, the corresponding image loss includes spectral loss and terrain loss, and the knowledge loss, spectral loss and terrain loss are weighted according to the weighted formula to obtain the contrast loss; then, according to the lithology scene classification results and lithology labels, the classification loss is generated, and according to the contrast loss and classification loss, the parameters of the graph attention network, the transformer model and the cross-modal cross-attention module are respectively updated to continuously optimize the lithology scene classification model and improve the fusion performance and accuracy of the lithology scene classification model. Among them, the weighted formula includes:

[0137]

[0138] in, is the contrast loss, For knowledge loss, is the spectral loss, is the terrain loss, and α, β, and γ are weight coefficients obtained based on experience.

[0139] Exemplarily, the loss module may choose whether to remain in the lithology scenario classification model according to actual conditions.

[0140] like Figure 7As shown, an electronic device 700 provided by an embodiment of the present invention includes a memory 710 and a processor 720; the memory 710 is used to store a computer program; the processor 720 is used to implement the lithology scene classification method as described above when executing the computer program.

[0141] In other words, an electronic device 700 includes a memory 710 and a processor 720 coupled to the memory 710; the memory 710 is configured to store a computer program; and the processor 720 is configured to perform the following operations when executing the computer program:

[0142] Input the acquired remote sensing image and knowledge data of the lithological scene into the trained lithological scene classification model to obtain the lithological scene classification result, wherein the lithological scene classification model includes a transformer module, a graph attention network module, a cross-modal cross-attention module, an adaptive feature dimension reduction module and an output module; the output end of the graph attention network and the output end of the transformer model are respectively connected to the input end of the cross-modal cross-attention module, and the cross-modal cross-attention module, the adaptive feature dimension reduction module and the output module are connected in sequence;

[0143] The transformer module is used to mine the image features of the remote sensing image;

[0144] The graph attention network module is used to mine the knowledge features of the knowledge data;

[0145] The cross-modal cross-attention module is used to fuse the image features and the knowledge features to generate lithology sensitive feature data;

[0146] The adaptive feature dimension reduction module is used to reduce the dimension of the lithology sensitive feature data to generate reduced dimension features;

[0147] The output module is used to generate the lithology scene classification result according to the dimensionality reduction feature.

[0148] An embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the lithology scene classification method as described above is implemented.

[0149] In other words, a non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor performs the following operations:

[0150] Input the acquired remote sensing image and knowledge data of the lithological scene into the trained lithological scene classification model to obtain the lithological scene classification result, wherein the lithological scene classification model includes a transformer module, a graph attention network module, a cross-modal cross-attention module, an adaptive feature dimension reduction module and an output module; the output end of the graph attention network and the output end of the transformer model are respectively connected to the input end of the cross-modal cross-attention module, and the cross-modal cross-attention module, the adaptive feature dimension reduction module and the output module are connected in sequence;

[0151] The transformer module is used to mine the image features of the remote sensing image;

[0152] The graph attention network module is used to mine the knowledge features of the knowledge data;

[0153] The cross-modal cross-attention module is used to fuse the image features and the knowledge features to generate lithology sensitive feature data;

[0154] The adaptive feature dimension reduction module is used to reduce the dimension of the lithology sensitive feature data to generate reduced dimension features;

[0155] The output module is used to generate the lithology scene classification result according to the dimensionality reduction feature.

[0156] An electronic device 700 that can be used as a server or client of the present invention will now be described, which is an example of a hardware device that can be applied to various aspects of the present invention. The electronic device 700 is intended to represent various forms of digital electronic computer equipment, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device 700 can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples, and are not intended to limit the implementation of the present invention described herein and / or required.

[0157] The electronic device 700 includes a computing unit, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) or a computer program loaded from a storage unit into a random access memory (RAM). In the RAM, various programs and data required for the operation of the device can also be stored. The computing unit, ROM, and RAM are connected to each other via a bus. An input / output (I / O) interface is also connected to the bus.

[0158] A person of ordinary skill in the art can understand that the implementation of all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc. In the present application, the unit described as a separate component may or may not be physically separated, and the component displayed as a unit may or may not be a physical unit, that is, it may be located in one place, or it may be distributed on multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment of the present invention. In addition, each functional unit in each embodiment of the present invention can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0159] Although the present invention is disclosed as above, the protection scope of the present invention is not limited thereto. Those skilled in the art may make various changes and modifications without departing from the spirit and scope of the present invention, and these changes and modifications will fall within the protection scope of the present invention.

Claims

1. A lithology scene classification method, characterized in that: include: The acquired remote sensing image and knowledge data of the lithology scene are input into the trained lithology scene classification model to obtain the lithology scene classification result, wherein the lithology scene classification model includes a transformer module, a graph attention network module, a cross-modal cross-attention module, an adaptive feature dimension reduction module and an output module; the output end of the graph attention network and the output end of the transformer module are respectively connected to the input end of the cross-modal cross-attention module, and the cross-modal cross-attention module, the adaptive feature dimension reduction module and the output module are connected in sequence; The transformer module is used to mine the image features of the remote sensing image; The graph attention network module is used to mine the knowledge features of the knowledge data; The cross-modal cross-attention module is used to fuse the image features and the knowledge features to generate lithology sensitive feature data; The adaptive feature dimension reduction module is used to reduce the dimension of the lithology sensitive feature data to generate reduced dimension features; The output module is used to generate the lithology scene classification result according to the dimensionality reduction feature.

2. The lithology scene classification method according to claim 1, characterized in that: The remote sensing image includes spectral data and terrain data, the image features include spectral features and terrain features, the transformer module includes a first TransXNet network and a second TransXNet network, the first TransXNet network is used to process the spectral data and generate the spectral features, and the second TransXNet network is used to process the terrain data and generate the terrain features.

3. The lithology scene classification method according to claim 2, characterized in that: The first TransXNet network includes a first processing layer and at least one second processing layer, the first processing layer and at least one second processing layer are connected in sequence, the first processing layer includes a first patch embedding layer and a transformer block, and the second processing layer includes a second patch embedding layer and the transformer block; The first patch embedding layer is used to process the spectral data to generate a first spectral vector; The transformer block of the first processing layer is used to process the first spectral vector to generate a first spectral vector feature; The second patch embedding layer is used to process the first spectrum vector feature or the second spectrum vector feature output by the previous second processing layer to generate a second spectrum vector; The transformer block of the second processing layer is used to process the second spectral vector to generate the second spectral vector feature, and the second spectral vector feature generated by the last transformer block is the spectral feature.

4. The lithology scene classification method according to claim 3, characterized in that: The transformer block includes a dynamic position encoding layer, a first normalization layer, a dynamic token mixer, a first integration layer, a second normalization layer, a multi-scale shuffle feedforward network, and a second integration layer, wherein the dynamic position encoding layer, the first normalization layer, the dynamic token mixer, the first integration layer, the second normalization layer, the multi-scale shuffle feedforward network, and the second integration layer are connected in sequence, and an output end of the dynamic position encoding layer is also connected to an input end of the first integration layer, and an output end of the second normalization layer is also connected to an input end of the second integration layer; The dynamic position encoding layer is used to perform position encoding on the first spectral vector or the second spectral vector to generate spectral encoding data; The first normalization layer is used to normalize the spectral encoding data to generate encoding normalized data; The dynamic token mixer is used to perform mixing conversion on the encoded normalized data to generate mixing conversion data; The first integration layer is used to integrate the spectrum encoding data and the mixing conversion data to generate first integrated data; The second normalization layer is used to normalize the first integrated data to generate integrated normalized data; The multi-scale shuffled feedforward network is used to perform channel shuffling operations and forward propagation operations on the integrated normalized data to generate multi-scale shuffled data; The second integration layer is used to integrate the integrated normalized data and the multi-scale shuffled data to generate the first spectral vector feature or the second spectral vector feature.

5. The lithology scene classification method according to claim 4, characterized in that: The dynamic token mixer includes a first convolution layer, a second convolution layer, an overlapping spatial dimension reduction attention layer, an input-dependent deep convolution layer, and a compressed token enhancer, wherein the first convolution layer is connected to the overlapping spatial dimension reduction attention layer, the second convolution layer is connected to the input-dependent deep convolution layer, and the output ends of the overlapping spatial dimension reduction attention layer and the input-dependent deep convolution layer are both connected to the input end of the compressed token enhancer; The first convolutional layer is used to map the encoded normalized data to generate a first feature map; The second convolutional layer maps the encoded normalized data to generate a second feature map; The overlapping space dimension reduction attention layer is used to reduce the overlapping space of the first feature map to extract global information; The input-dependent deep convolution layer is used to adjust convolution kernel parameters according to the second feature map, and extract local information according to the convolution kernel parameters; The compressed token enhancer is used to perform token aggregation on the global information and the local information to generate the mixed frequency conversion data.

6. The lithology scene classification method according to claim 1, characterized in that: Before inputting the acquired remote sensing image and knowledge data of the lithology scene into the trained lithology scene classification model to obtain the lithology scene classification result, the method further includes: Constructing an initial training model and first-scale training data, and training the initial training model according to the first-scale training data to obtain a first training model; Constructing second-scale training data according to the first-scale training data, and training the first training model according to the second-scale training data to obtain a second training model; According to the second-scale training data, third-scale training data are constructed, and the second training model is trained according to the third-scale training data to obtain a third training model, and the third training model is used as the trained lithology scene classification model.

7. The lithology scene classification method according to claim 1, characterized in that: The adaptive feature dimension reduction module includes a third convolution layer, a BN layer and an activation function layer, and the third convolution layer, the BN layer and the activation function layer are connected in sequence.

8. The lithology scene classification method according to any one of claims 1 to 7, characterized in that: The lithology scene classification model also includes a loss module, which is respectively connected to the graph attention network, the transformer model, the cross-modal cross attention module, the adaptive feature dimension reduction module and the output module; The loss module is used to perform feature comparison between the dimensionality reduction features, the image features and the knowledge features to generate a contrast loss, and is also used to generate a classification loss based on the lithology scene classification results, and to update the parameters of the graph attention network, the transformer model and the cross-modal cross attention module respectively based on the contrast loss and the classification loss.

9. An electronic device, characterized in that: including memory and processor; The memory is used to store computer programs; The processor is used to implement the lithology scene classification method according to any one of claims 1 to 8 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the lithology scene classification method according to any one of claims 1 to 8 is implemented.

Images