Geochemical data prediction method and device based on multimodal data fusion

Abstract

The present application relates to the technical field of multimodal data fusion, and discloses a geochemical data prediction method and device based on multimodal data fusion; the method includes: inputting multimodal geoscience data into a geochemical data prediction model to obtain geochemical prediction data; the multimodal geoscience data includes vegetation information, digital elevation model, aeromagnetic data and remote sensing image; the spatial multimodal extractor of the geochemical data prediction model includes multiple feature extraction modules, each feature extraction module is used to extract the spatial features of geoscience data of a modality; the multi-head convolution attention layer is used to fuse the spatial features extracted by each feature extraction module; the spectral convolution module is used to extract the spectral features of the remote sensing image; the feature fusion layer is used to fuse multiple spatial features and spectral features. The present application realizes the precise fusion of multimodal geoscience data and improves the accuracy of data prediction.

Description

Technical Field

[0001] The present disclosure relates to the technical field of multimodal data fusion, and in particular to a geochemical data prediction method and device based on multimodal data fusion. Background Art

[0002] The detection of geochemical data is an important means in the fields of geological exploration and environmental monitoring. At present, in the field of geochemical data prediction, traditional point-based machine learning regression methods, such as linear regression, support vector machine and random forest, have been widely used. However, the prediction accuracy of these models is often low and cannot achieve the ideal prediction effect. Summary of the invention

[0003] In view of the above situation, the embodiments of the present application provide a geochemical data prediction method and device based on multimodal data fusion, which aims to solve the above problems or at least partially solve the above problems.

[0004] In a first aspect, an embodiment of the present application provides a geochemical data prediction method based on multimodal data fusion, the method comprising: obtaining multimodal geoscience data, the multimodal geoscience data comprising vegetation information, digital elevation model, aeromagnetic data and remote sensing image; inputting the multimodal geoscience data into a geochemical data prediction model to obtain geochemical prediction data; wherein the geochemical data prediction model comprises a spatial multimodal extractor, a multi-head convolutional attention layer, a spectral convolution module and a feature fusion module; the spatial multimodal extractor comprises a plurality of feature extraction modules, each feature extraction module being used to extract spatial features of geoscience data of a certain modality; the multi-head convolutional attention layer being used to fuse the spatial features extracted by each feature extraction module; the spectral convolution module being used to extract the spectral features of the remote sensing image; the feature fusion layer being used to fuse the spatial features fused by the multi-head convolutional attention layer with the spectral features extracted by the spectral convolution module.

[0005] In a second aspect, an embodiment of the present application also provides a geochemical data prediction device based on multimodal data fusion, the device comprising: an acquisition module for acquiring multimodal geoscience data, the multimodal geoscience data comprising vegetation information, digital elevation model, aeromagnetic data and remote sensing image; a prediction module for inputting the multimodal geoscience data into a geochemical data prediction model to acquire geochemical prediction data; wherein the geochemical data prediction comprises a spatial multimodal extractor, a multi-head convolutional attention layer, a spectral convolution module and a feature fusion module; the spatial multimodal extractor comprises a plurality of feature extraction modules, each feature extraction module being used to extract spatial features of geoscience data of a modality; the multi-head convolutional attention layer being used to fuse the spatial features extracted by each feature extraction module; the spectral convolution module being used to extract the spectral features of the remote sensing image; the feature fusion layer being used to fuse the spatial features fused by the multi-head convolutional attention layer with the spectral features extracted by the spectral convolution module.

[0006] In a third aspect, an embodiment of the present application further provides an electronic device, comprising: a processor; and a memory arranged to store computer executable instructions, wherein the executable instructions, when executed, cause the processor to perform the steps of the first aspect described above.

[0007] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, which stores one or more programs. When the one or more programs are executed by an electronic device including multiple application programs, the electronic device executes the steps of the first aspect above.

[0008] At least one of the above technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects:

[0009] Geochemical data is predicted by fusing multimodal spatial data and spectral data. Among them, each modal data in the multimodal earth science data extracts features through an independent feature extraction module, ensuring the independent processing logic of each modal data, so as to better explore the respective data characteristics; the spatial features of different modal data are fused through the multi-head convolution attention mechanism, which improves the interoperability between different data sources, better understands the spatial change trend, and solves the semantic gap problem between different modal data; further, by introducing a spectral convolution module for spectral feature extraction of remote sensing images, the introduction of the convolution module greatly enhances the model's understanding and processing capabilities of spectral data, and the convolution network can extract local features more effectively; finally, geochemical data is predicted by fusing multimodal spatial data and spectral data. This application makes full use of the global feature extraction capability of the spatial multimodal extractor and the local feature processing capability of the convolution network to achieve the precise fusion of multimodal earth science data and improve the accuracy of data prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0011] Figure 1 A schematic diagram of a process of a geochemical data prediction method based on multimodal data fusion provided in an embodiment of the present application is shown;

[0012] Figure 2 The following is a diagram showing the architecture of the geochemical data prediction model provided by the embodiment of the present application;

[0013] Figure 3 A schematic diagram of a geochemical data prediction model provided by another embodiment of the present application is shown;

[0014] Figure 4 The following is an architecture diagram of a multi-head attention layer provided by an embodiment of the present application;

[0015] Figure 5 A structural diagram of a geochemical data prediction device based on multimodal data fusion provided in an embodiment of the present application is shown;

[0016] Figure 6 A schematic structural diagram of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0018] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that such use is interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the term "including" and its variants are to be interpreted as open-ended terms meaning "including but not limited to".

[0019] As described in the background technology, traditional point-based machine learning regression methods, such as linear regression, support vector machine and random forest, have been widely used. For example, in the related art, a remote sensing geochemical nonlinear inversion model is constructed based on a stack encoder and an extreme learning machine, and the band information of the corresponding position pixel element in the remote sensing image is selected by sampling points. The band information is used as an independent variable and the geochemical data is used as a dependent variable to train the SAE-ELM model and establish a generalized prediction model. However, the model can only input the pixel information of one position, without considering the spatial information of the sampling point field and the positioning error caused by sampling through GPS. Although satellite remote sensing images can reflect the surface features of most areas, the response of surface features is often not accurate enough in areas where large areas are prepared for coverage or blocked by other objects.

[0020] Another related technology proposed a prediction method based on random forests, which simulates the concentration of elements in the soil through high-resolution and geophysical data. Although geophysical data was used, the fusion method was too simple to fully utilize the advantages of the data, and lacked consideration of spatial information, resulting in the model's inadequate performance in capturing geographic distribution characteristics and spatial correlation. In addition, there is a semantic gap between geoscience data of different modalities, and different geochemical elements also have different responses to these data.

[0021] Based on this, this application proposes a geochemical data prediction method based on multimodal data fusion, which predicts geochemical data by fusing multimodal spatial data and spectral data. Among them, each modal data in the multimodal earth science data extracts features through a separate feature extraction module, ensuring the independent processing logic of each modal data, so as to better explore the respective data characteristics; the spatial features of different modal data are fused through a multi-head convolution attention mechanism, which improves the interoperability between different data sources, better understands the spatial change trend, and solves the semantic gap problem between different modal data; further, by introducing a spectral convolution module for spectral feature extraction of remote sensing images, the introduction of the convolution module greatly enhances the model's understanding and processing capabilities of spectral data, and the convolution network can extract local features more effectively; finally, geochemical data is predicted by fusing multimodal spatial data and spectral data. This application makes full use of the global feature extraction capability of Swin Transformer and the local feature processing capability of the convolution network to achieve the precise fusion of multimodal earth science data and improve the accuracy of data prediction.

[0022] The present application is described in detail below through specific embodiments.

[0023] Figure 1 The schematic diagram of the process of the geochemical data prediction method based on multimodal data fusion provided in the embodiment of the present application is shown. Figure 1 It can be seen that the present application at least includes step S101-step S102:

[0024] Step S101: Acquire multimodal earth science data.

[0025] Among them, multimodal earth science data includes vegetation information, digital elevation model, aeromagnetic data and remote sensing images. Vegetation information refers to the type, distribution, structure, vegetation index and other information prepared on the earth. Digital Elevation Model (DEM) is a digital model used to represent the undulating terrain on the earth's surface, and realizes the digital simulation of the terrain surface through terrain elevation data. Aeromagnetic data, that is, aeromagnetic measurement data, is earth data obtained by measuring the strength or gradient of the geomagnetic field during the flight of an aeromagnetic magnetometer. Remote sensing images usually rely on platforms such as remote sensing satellites, aircraft or drones. The sensors carried can capture electromagnetic waves of different bands to form images, that is, remote sensing images contain electromagnetic wave information of objects on the earth's surface. It is worth noting that the multimodal earth science data in the embodiments of the present application are not limited to the above four types of data, and earth science data of any modality are applicable to the scheme of the present application.

[0026] Although remote sensing data can reflect the surface characteristics of most areas, the reflection of surface characteristics is often not accurate enough in areas that are covered by large areas or blocked by other objects. Therefore, multimodal earth science data is introduced in the embodiments of the present application to obtain accurate chemical data of the earth through earth data in various modes.

[0027] In some embodiments, the geoscience data of each modality is interpolated to generate image data with the same resolution as the remote sensing image to achieve pixel-level alignment of the data, and the image data of each modality is further input into a geochemical data prediction model.

[0028] In some embodiments, for each modality of geoscience data, the sample is X , is the number of blocks, is the number of channels for each geoscience data, 50 indicates the block size.

[0029] Step S102: input the multimodal geoscience data into a geochemical data prediction model to obtain geochemical prediction data.

[0030] Among them, geochemical prediction data include the concentration of metal elements such as copper, lead, zinc, and tungsten in the soil. By obtaining geochemical prediction data, it is possible to map the concentration of geochemical data, which can be used in the fields of geological rocks, mineral exploration, sedimentology, etc., such as predicting and warning the development trend of elements, compounds and other geochemical parameters; predicting various geological events; detecting environmental changes, etc.

[0031] The geochemical data prediction method based on multimodal data fusion of the present application is implemented based on the geochemical data prediction model. Figure 2 A schematic diagram of the structure of a geochemical data prediction model according to an embodiment of the present application is shown. Figure 2 It can be seen that the geochemical data prediction model 200 includes a spatial multimodal extractor 201, a multi-head convolutional attention layer 202, a spectral convolution module 203 and a feature fusion module 204, wherein the spatial multimodal extractor 201 is connected to the multi-head convolutional attention layer 202, the multi-head convolutional attention layer 202 and the spectral convolution module 203 are respectively connected to the feature fusion module 204, and the output of the feature fusion module 204 is used as the output of the geochemical data prediction model 200.

[0032] Specifically, the spatial multimodal extractor includes multiple feature extraction modules, each of which is used to extract the spatial features of geoscience data of a certain mode. Among them, the feature extraction module is based on the Swin Transformer architecture. That is, in this application, each modality of geoscience data is extracted through a separate Swin Transformer module to extract features, which ensures the independent processing logic of each data, can better mine the data features of each modality data, and can increase or delete the amount of input data according to actual conditions, that is, the model architecture of the entire geochemical data prediction model can be flexibly changed, showing higher flexibility and adaptability. Optionally, Swin Transformer can also be replaced with deep learning models such as residual network (ResNet) and convolutional neural network (CNN) to achieve similar results.

[0033] The multi-head convolutional attention layer is used to fuse the spatial features extracted by each feature extraction module, improving the interoperability between geoscience data of different modalities.

[0034] The spectral convolution module is used to extract the spectral features of remote sensing images. Compared with the model that only relies on Swin Transformer, the introduction of the convolution module greatly enhances the model's understanding and processing capabilities of spectral data. Especially in the spectral data dimension, the convolutional network can extract local features more effectively.

[0035] The feature fusion layer is used to fuse the spatial features fused by the multi-head convolutional attention layer and the spectral features extracted by the spectral convolution module.

[0036] In some embodiments, the geochemical data prediction model further includes a two-dimensional convolutional regression layer, which is used to predict geochemical data based on the fused features.

[0037] from Figure 1 It can be seen from the method shown that each modal data in the multimodal earth science data in this application extracts features through a separate feature extraction module, ensuring the independent processing logic of each modal data, so as to better explore the data characteristics of each modality; the spatial features of different modal data are fused through the multi-head convolution attention mechanism, which improves the interoperability between different data sources, better understands the spatial change trend, and solves the semantic gap problem between different modal data; further, by introducing a spectral convolution module for spectral feature extraction of remote sensing images, the introduction of the convolution module greatly enhances the model's understanding and processing capabilities of spectral data, and the convolution network can extract local features more effectively; finally, geochemical data is predicted by fusing multimodal spatial data and spectral data. This application makes full use of the global feature extraction capability of SwinTransformer and the local feature processing capability of the convolutional network, and combines Swin Transformer and the convolutional network to achieve the precise fusion of multimodal earth science data and improve the accuracy of data prediction.

[0038] Combine the following Figure 3 The geochemical data prediction model architecture shown explains the geochemical data prediction model in detail.

[0039] In some embodiments of the present application, each feature extraction module in the spatial multimodal extractor includes an image segmentation layer (Patch partition), a linear embedding layer (Linear embedding), and a multi-head attention layer (SwinTransformer block). Among them, the image segmentation layer is used to divide the earth science data into non-overlapping blocks, and the linear embedding layer is used to project each segmented block into a low-dimensional space through a convolution operation to obtain an embedded feature x: x=Projection(patch)#(1.1), where Projection is a projection method, usually using two-dimensional convolution for projection. The multi-head attention layer is used to extract local features and global features.

[0040] Among them, the Swin Transformer block includes the Window Multi-Head Self-Attention (W-MSA) mechanism and the Shifted Window Multi-Head Self-Attention (SW-MSA) mechanism. The Window Multi-Head Self-Attention mechanism is used to capture the local features within a fixed window through the self-attention mechanism, and the Shifted Window Multi-Head Self-Attention mechanism is used to capture the interactive features between different windows through the self-attention mechanism.

[0041] Specifically, the Swin Transformer block contains two key parts, W-MSA and SW-MSA, and both are stacked with N layers. Among them, W-MSA is a multi-head attention mechanism based on windows. This attention mechanism can extract features within a local window, effectively reducing computational complexity and improving computational efficiency. SW-MSA is a multi-head attention mechanism based on a moving window. By moving the position of the window, the information between different windows can interact, thereby capturing global features. This feature extraction method is very suitable for analyzing and extracting data features with spatial variation trends.

[0042] like Figure 4 As shown in the figure, the Swin Transformer block includes layer normalization LN, multi-layer perceptron MLP, W-MSA and SW-MSA. After the embedded features output by Linear embedding are input into the Swin Transformer block, the embedded features are divided into multiple small windows. After the self-attention mechanism is calculated in the window, the window of the next layer is moved by half a window or a preset window size in each dimension, and then the self-attention mechanism is calculated to capture the interactive features of different windows.

[0043] In some embodiments of the present application, in the above method, the spectral convolution module (Spectrum Block) is implemented by a one-dimensional convolutional neural network, which consists of three repeated blocks, each of which contains a one-dimensional convolution layer (Conv1D), a batch normalization layer (BatchNorm) and a ReLU activation function. Specifically, the remote sensing image is subjected to feature extraction through a one-dimensional convolution layer, and then the data distribution is standardized through a batch normalization layer, and finally the nonlinear expression ability of the model is enhanced through the ReLU activation function. The embodiment of the present application extracts the deep-level features of the input data layer by layer, and finally reduces the number of parameters and the amount of calculation of the subsequent layers through the maximum pooling layer (MaxPooling), thereby improving the calculation efficiency and reducing the risk of overfitting.

[0044] In some embodiments of the present application, in the above method, the Multi-Head CrossAttention (MHCA) layer captures the characteristics of each modality of geoscience data through convolution and deconvolution operations of different scales, and fuses the characteristics of geoscience data of different modalities.

[0045] Specifically, the MHCA mechanism processes feature tensors of different sizes through multiple independent branches, and the output feature tensors of each branch are spliced ​​along the channel dimension. Each branch includes multiple attention heads, each of which performs convolution and deconvolution operations respectively, and uses different convolution kernel sizes to capture spatial relationships at various scales. In the attention head, the convolution layer adjusts the spatial and channel dimensions of the input tensor, and the subsequent deconvolution layer restores the tensor to its original size. Finally, the outputs of all heads are merged and processed through the sigmoid function to generate an attention map, which is used to adjust the original spliced ​​tensor.

[0046] In the embodiments of the present application, by extracting the features of multimodal geoscience data in the spatial and spectral domains and establishing a nonlinear relationship with geochemical data, the concept of machine translation is innovatively introduced into geochemical data prediction, significantly improving the accuracy of prediction. Furthermore, by designing a regression predictor based on Swin Transformer and convolution, the global feature extraction capability of Swin Transformer and the local feature processing advantages of convolutional networks are fully utilized to achieve accurate modeling and fusion of multi-source geoscience data, greatly improving the accuracy of data prediction.

[0047] In some embodiments of the present application, a method for training a geochemical data prediction model is also provided. When training the geochemical data prediction model, a model architecture based on a convolutional network is first built, and the spectral features of the remote sensing image are extracted through the convolutional network, and the surface features are identified and classified through the spectral features. However, although remote sensing images can reflect the surface features of most areas, the reflection of surface features is often not accurate enough in areas covered by large areas of vegetation or blocked by other objects. Therefore, the surface features cannot be fully identified only by extracting the spectral features of the remote sensing image. Therefore, the embodiment of the present application combines the Swin Transformer architecture with the convolutional network, and uses the global feature extraction capability of the Swin Transformer and the local feature processing capability of the convolutional architecture to build the model architecture of the model to be trained. Furthermore, considering that geoscience data has multiple modalities and each modal geoscience data has its own characteristics, in order to avoid the weakening of the characteristics of a certain modality of geoscience data, a SwinTransformer architecture is designed for each modality of geoscience data. The spatial characteristics of each modality of geoscience data are extracted through the Swin Transformer architecture, and then the spatial characteristics of multiple modal geoscience data are processed and fused through multiple independent branches in the multi-head multimodal attention mechanism (MHCA). Finally, the spectral features extracted by the convolutional network are fused, and the corresponding geochemical data concentration is predicted through convolutional regression.

[0048] The model architecture of the model to be trained is explained above. During actual training, multiple groups of training data are first obtained, each group of training data includes multimodal geoscience data and its corresponding geochemical data, wherein the multimodal chemical data includes but is not limited to vegetation information, digital elevation model data, aeromagnetic data and remote sensing images. Since the data of multimodal chemical data other than remote sensing images is data information, the multimodal data is first interpolated into image data. In one embodiment, in order to achieve pixel-level alignment of data, the multimodal data is interpolated into image data with the same resolution as the remote sensing image. The image data and its corresponding geochemical data are input into the model to be trained, and through a pre-set loss function, when the loss value of the loss function is minimized, the model training is completed, which is the geochemical data prediction model described in the above embodiment of the present application.

[0049] In some embodiments, when building a model architecture, the global feature extraction capability can be achieved by replacing the Swin Transformer model with a deep learning model such as ResNet or CNN. Implementation methods that are the same as the concept of this solution are included in the scope of protection of this application.

[0050] In an embodiment of the present application, the global feature extraction capability of Swin Transformer and the local feature processing capability of convolutional network are utilized to combine Swin Transformer and convolutional network to realize the precise fusion of multimodal earth science data and improve the accuracy of data prediction.

[0051] In some embodiments of the present application, a geochemical data prediction device based on multimodal data fusion is provided, and the geochemical data prediction device based on multimodal data fusion corresponds one-to-one to the geochemical data prediction method based on multimodal data fusion in the above embodiments. Figure 5 As shown, the geochemical data prediction device based on multimodal data fusion includes: an acquisition module 301 and a prediction module 302. The detailed description of each functional module is as follows:

[0052] The acquisition module is used to acquire multimodal earth science data, including vegetation information, digital elevation model, aeromagnetic data and remote sensing images;

[0053] A prediction module is used to input multimodal geoscience data into a geochemical data prediction model to obtain geochemical prediction data;

[0054] Among them, the geochemical data prediction model includes a spatial multimodal extractor, a multi-head convolutional attention layer, a spectral convolution module and a feature fusion module; the spatial multimodal extractor includes multiple feature extraction modules, each feature extraction module is used to extract the spatial features of a modality of earth science data; the multi-head convolutional attention layer is used to fuse the spatial features extracted by each feature extraction module; the spectral convolution module is used to extract the spectral features of remote sensing images; the feature fusion layer is used to fuse the spatial features fused by the multi-head convolutional attention layer and the spectral features extracted by the spectral convolution module.

[0055] In some embodiments of the present application, in the above-mentioned device, each feature extraction module in the spatial multimodal extractor includes an image segmentation layer, a linear embedding layer, and a multi-head attention layer;

[0056] The image segmentation layer is used to segment geoscience data into non-overlapping blocks;

[0057] The linear embedding layer is used to convert each segmented block into an embedded feature;

[0058] The multi-head attention layer is used to extract local features and interaction features.

[0059] In some embodiments of the present application, in the above-mentioned device, the multi-head attention layer includes a fixed window multi-head self-attention mechanism and a moving window multi-head self-attention mechanism;

[0060] Among them, the window multi-head self-attention mechanism is used to capture the local features within a fixed window through the multi-head self-attention mechanism; the moving window multi-head attention mechanism is used to capture the interactive features between different windows through the multi-head self-attention mechanism.

[0061] In some embodiments of the present application, in the above-mentioned device, the spectral feature extractor is generated based on a one-dimensional convolutional network, including multiple blocks, each block including a one-dimensional convolutional layer, a batch normalization layer and an activation function.

[0062] In some embodiments of the present application, in the above-mentioned device, the multi-head convolutional attention layer captures the characteristics of each modality of earth science data through convolution and deconvolution operations of different scales, and fuses the characteristics of earth science data of different modalities.

[0063] In some embodiments of the present application, in the above-mentioned device, the geochemical data prediction model also includes a two-dimensional convolutional regression layer; the two-dimensional convolutional regression layer is used to predict geochemical data based on fusion features.

[0064] In some embodiments of the present application, in the above-mentioned device, the geoscience data of each modality is interpolated to generate image data with the same resolution as the remote sensing image; and each image data is input into the geochemical data prediction model.

[0065] It should be noted that any of the above-mentioned geochemical data prediction devices based on multimodal data fusion can implement the above-mentioned geochemical data prediction method based on multimodal data fusion in a one-to-one correspondence, which will not be repeated here.

[0066] Figure 6 FIG. 1 is a schematic diagram showing the structure of an electronic device provided in an embodiment of the present application. Figure 6 As shown, at the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and a memory. The memory may include a memory, such as a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage, etc. Of course, the electronic device may also include hardware required for other services.

[0067] The processor, network interface and memory can be interconnected through an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0068] The memory is used to store programs. Specifically, the program may include program code, and the program code includes computer operation instructions. The memory may include internal memory and non-volatile memory, and provides instructions and data to the processor.

[0069] The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming a geochemical data prediction device based on multimodal data fusion at the logical level. The processor executes the program stored in the memory and is specifically used to execute the above method.

[0070] The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module may be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0071] The electronic device can execute the geochemical data prediction method based on multimodal data fusion provided in multiple embodiments of the present application, and realize a geochemical data prediction device based on multimodal data fusion. Figure 5 The functions of the illustrated embodiment will not be described in detail in the embodiments of the present application.

[0072] An embodiment of the present application also proposes a computer-readable storage medium, which stores one or more programs, and the one or more programs include instructions. When the instructions are executed by an electronic device including multiple application programs, the electronic device can execute the geochemical data prediction method based on multimodal data fusion provided by multiple embodiments of the present application.

[0073] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0074] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0075] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0076] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0077] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0078] Memory may include non-permanent storage in a computer-readable medium, in the form of random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0079] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0080] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity or device including the element.

[0081] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0082] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A geochemical data prediction method based on multimodal data fusion, characterized in that: The method comprises: Acquiring multimodal geoscience data, wherein the multimodal geoscience data includes vegetation information, digital elevation models, aeromagnetic data, and remote sensing images; Inputting the multimodal geoscience data into a geochemical data prediction model to obtain geochemical prediction data; Among them, the geochemical data prediction model includes a spatial multimodal extractor, a multi-head convolutional attention layer, a spectral convolution module and a feature fusion module; the spatial multimodal extractor includes multiple feature extraction modules, each feature extraction module is used to extract the spatial features of geoscience data of a modality; the multi-head convolutional attention layer is used to fuse the spatial features extracted by each feature extraction module; the spectral convolution module is used to extract the spectral features of the remote sensing image; the feature fusion layer is used to fuse the spatial features fused by the multi-head convolutional attention layer and the spectral features extracted by the spectral convolution module.

2. The method according to claim 1, characterized in that Each feature extraction module in the spatial multimodal extractor includes an image segmentation layer, a linear embedding layer, and a multi-head attention layer; The image segmentation layer is used to segment the geoscience data into non-overlapping blocks; The linear embedding layer is used to convert each segmented block into an embedded feature; The multi-head attention layer is used to extract local features and interaction features.

3. The method according to claim 2, characterized in that The multi-head attention layer includes a fixed window multi-head self-attention mechanism and a moving window multi-head self-attention mechanism; Among them, the fixed window multi-head self-attention mechanism is used to capture local features within the fixed window through the multi-head self-attention mechanism; the moving window multi-head attention mechanism is used to capture the interactive features between different windows through the multi-head self-attention mechanism.

4. The method according to claim 1, characterized in that: The spectral convolution module is generated based on a one-dimensional convolutional network and includes multiple blocks, each of which includes a one-dimensional convolutional layer, a batch normalization layer and an activation function.

5. The method according to claim 1, characterized in that The multi-head convolutional attention layer captures the characteristics of each modality of geoscience data through convolution and deconvolution operations of different scales, and fuses the characteristics of geoscience data of different modalities.

6. The method according to claim 1, characterized in that The geochemical data prediction model also includes a two-dimensional convolutional regression layer; the two-dimensional convolutional regression layer is used to predict geochemical data based on fusion features.

7. The method according to any one of claims 1 to 6, characterized in that The inputting the multimodal geoscience data into a geochemical data prediction model comprises: Interpolating the geoscience data of each modality to generate image data having the same resolution as the remote sensing image; Each image data is input into the geochemical data prediction model.

8. A geochemical data prediction device based on multimodal data fusion, characterized in that: The device comprises: An acquisition module, used for acquiring multimodal earth science data, wherein the multimodal earth science data includes vegetation information, digital elevation model, aeromagnetic data and remote sensing image; A prediction module, used for inputting the multimodal geoscience data into a geochemical data prediction model to obtain geochemical prediction data; Among them, the geochemical data prediction model includes a spatial multimodal extractor, a multi-head convolutional attention layer, a spectral convolution module and a feature fusion module; the spatial multimodal extractor includes multiple feature extraction modules, each feature extraction module is used to extract the spatial features of geoscience data of a modality; the multi-head convolutional attention layer is used to fuse the spatial features extracted by each feature extraction module; the spectral convolution module is used to extract the spectral features of the remote sensing image; the feature fusion layer is used to fuse the spatial features fused by the multi-head convolutional attention layer and the spectral features extracted by the spectral convolution module.

9. An electronic device, comprising: processor; as well as A memory arranged to store computer executable instructions, which, when executed, cause the processor to perform the steps of the geochemical data prediction method based on multimodal data fusion as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing one or more programs, which, when executed by an electronic device including multiple application programs, enables the electronic device to perform the steps of the geochemical data prediction method based on multimodal data fusion as described in any one of claims 1 to 7.

Images