Multimodal spatio-temporal feature fusion and prediction method, system, device and medium

Abstract

The application provides a multi-modal spatio-temporal feature fusion and prediction method, system, device and medium, comprising: acquiring multi-source heterogeneous data of a target area and respectively performing feature coding to obtain multiple modal features; dynamically calculating attention weights based on context information of the modal features, and weighting and fusing the multiple modal features based on the attention weights to obtain fused spatio-temporal features; respectively extracting time sequence evolution features of the fused spatio-temporal features in a time dimension and spatial topology dependent features of the fused spatio-temporal features in a space dimension, and performing joint prediction based on the two to obtain a prediction result of the target area. The application realizes adaptive deep fusion of multi-source heterogeneous data and spatio-temporal joint accurate prediction, and improves the robustness and accuracy of agricultural remote sensing monitoring in a complex environment.

Description

Technical Field

[0001] This invention relates to the field of interdisciplinary technology of geographic information science and artificial intelligence, specifically to a method, system, device and medium for multimodal spatiotemporal feature fusion and prediction. Background Technology

[0002] Currently, in the field of agricultural remote sensing monitoring and geographic intelligent perception, crop growth analysis and yield prediction usually adopt vegetation index calculation based on single-modality satellite imagery (such as optical or radar) or traditional machine learning classification methods.

[0003] In practical applications, optical images are easily obscured by clouds and fog, resulting in missing time-series data. While synthetic aperture radar (SAR) images have penetrating power, they have limited ability to distinguish crop types. At the same time, meteorological and ground observation data often have poor spatial continuity, making it difficult to effectively integrate them with remote sensing image data.

[0004] Existing multi-source data processing methods mostly employ simple feature splicing or fixed-weight fusion, which cannot dynamically adjust the contribution of each modality according to data quality fluctuations. Furthermore, traditional models struggle to simultaneously capture the continuous evolution of geographical phenomena in the temporal dimension and the neighborhood diffusion dependence features in the spatial dimension. This results in low prediction accuracy and poor robustness in complex environments, failing to meet the urgent need for deep fusion and dynamic perception of multimodal spatiotemporal information in precision agriculture decision-making. Summary of the Invention

[0005] To address the shortcomings of existing technologies, the purpose of this invention is to provide a method, system, device, and medium for multimodal spatiotemporal feature fusion and prediction.

[0006] A multimodal spatiotemporal feature fusion and prediction method according to the present invention includes: Acquire multi-source heterogeneous data of the target region, and perform feature encoding on the multi-source heterogeneous data to obtain multiple modal features; Based on the contextual information of each modal feature, the attention weight of each modal feature is dynamically calculated, and multiple modal features are weighted and fused based on the attention weight to obtain fused spatiotemporal features; The temporal evolution features of the fused spatiotemporal features in the time dimension and the spatial topological dependence features in the spatial dimension are extracted respectively. Based on the temporal evolution features and the spatial topological dependence features, a joint prediction is performed to obtain the prediction result of the target region.

[0007] Preferably, the step of dynamically calculating the attention weight of each modal feature based on the context information of each modal feature, and then weighting and fusing multiple modal features based on the attention weights to obtain fused spatiotemporal features includes: An adaptive attention mechanism is used to evaluate the reliability of each modal feature in the context information and to calculate the attention weight coefficient corresponding to each modal feature. The fused spatiotemporal features are obtained by weighting and summing multiple modal features based on the attention weight coefficients.

[0008] Preferably, the multi-source heterogeneous data includes optical images, synthetic aperture radar images, and meteorological data; The calculation of the attention weight coefficients corresponding to each modality feature includes: Through formula Calculate the first Attention weight coefficients for each modality ;in, For the first Modal features, For the first The learnable weight matrix corresponding to each mode It is a normalized exponential function; The step of weighting and summing multiple modal features based on the attention weight coefficients to obtain the fused spatiotemporal features includes: Through formula Calculate the fused spatiotemporal features ;in, This indicates a weighted summation of all modalities participating in the fusion. This indicates a weighted operation.

[0009] Preferably, the step of extracting the temporal evolution features of the fused spatiotemporal features in the time dimension and the spatial topological dependency features in the spatial dimension, and performing joint prediction based on the temporal evolution features and the spatial topological dependency features to obtain the prediction result of the target region includes: In the time dimension, the temporal evolution features are extracted through sequence modeling; In the spatial dimension, the spatial topological dependency features are extracted based on the region graph structure; The prediction result is obtained by fusing the temporal evolution features and the spatial topological dependency features.

[0010] Preferably, the regional map structure is defined based on a regional adjacency matrix constructed from geographic spatial distances; The extraction of temporal evolution features through sequence modeling in the time dimension includes: The temporal evolution features are obtained by performing temporal-dimensional sequence evolution modeling on the fused spatiotemporal features using a Transformer network. The extraction of spatial topological dependency features based on the region graph structure in the spatial dimension includes: By using a graph convolutional network and the region adjacency matrix, the spatial topological dependency model of the fused spatiotemporal features is performed to obtain the spatial topological dependency features.

[0011] Preferably, the step of performing feature encoding on the multi-source heterogeneous data to obtain multiple modal features includes: For the optical images in the multi-source heterogeneous data, spatial features are extracted using a convolutional neural network to obtain the corresponding modal features; For the synthetic aperture radar image in the multi-source heterogeneous data, texture and structural information are extracted through complex convolution structure to obtain the corresponding modal features; For the meteorological and soil sensor data in the multi-source heterogeneous data, the temporal change trend is captured by a long short-term memory network to obtain the corresponding modal features.

[0012] Preferably, acquiring multi-source heterogeneous data of the target area includes: A unified geographic coordinate reference system is used to sequentially perform radiometric correction, atmospheric correction, resampling, and registration on the collected multi-source raw data to achieve data spatial alignment and obtain the multi-source heterogeneous data.

[0013] Preferably, the method includes: Multi-source heterogeneous data of the target area is acquired through multi-type sensing hardware, and multiple modal features are obtained by using computing hardware to encode the features of the multi-source heterogeneous data respectively. Based on the contextual information of each modal feature, the attention weight of each modal feature is dynamically calculated, and multiple modal features are weighted and fused based on the attention weight to obtain fused spatiotemporal features; The temporal evolution features of the fused spatiotemporal features in the time dimension and the spatial topological dependence features in the spatial dimension are extracted using dedicated computing hardware for spatiotemporal interaction networks. Based on the temporal evolution features and the spatial topological dependence features, joint prediction is performed to obtain the prediction results for the target region. The agricultural intelligent decision-making results are then output by visualization and gateway hardware.

[0014] A multimodal spatiotemporal feature fusion and prediction system according to the present invention includes: The feature encoding module is used to acquire multi-source heterogeneous data of the target region and perform feature encoding on the multi-source heterogeneous data to obtain multiple modal features; The cross-modal fusion module is used to dynamically calculate the attention weight of each modal feature based on the context information of each modal feature, and to perform weighted fusion of multiple modal features based on the attention weight to obtain fused spatiotemporal features; The spatiotemporal prediction module is used to extract the temporal evolution features of the fused spatiotemporal features in the time dimension and the spatial topological dependency features in the spatial dimension, and to perform joint prediction based on the temporal evolution features and the spatial topological dependency features to obtain the prediction result of the target region.

[0015] A computer device according to the present invention includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the multimodal spatiotemporal feature fusion and prediction method.

[0016] According to the present invention, a computer-readable storage medium is provided thereon storing a computer program, which, when executed by a processor, implements the multimodal spatiotemporal feature fusion and prediction method.

[0017] Compared with the prior art, the present invention has the following beneficial effects: 1. This application employs an adaptive attention mechanism to dynamically allocate fusion weights based on the reliability of each modality's features within the current context. When the quality of a particular modality's data deteriorates due to cloud cover or sensor malfunction, the system automatically reduces its weight and increases the contribution of other reliable modalities. This allows the multi-source data to complement each other at the feature level, improving the stability of the fusion results in complex environments.

[0018] 2. This application constructs a dual-branch prediction architecture by using sequential modeling in the time dimension and graph structure modeling in the spatial dimension. It simultaneously captures the evolution of crop growth over time and spatial correlations such as the spread of pests and diseases and the convergence of growth between adjacent plots. This solves the problem that a single model cannot take into account both temporal and spatial characteristics, making the prediction results more consistent with the physical processes in actual agricultural production.

[0019] 3. This application selects convolutional neural networks, complex convolutional structures, and long short-term memory networks for feature encoding based on the physical characteristics of optical images, SAR images, and meteorological sensor data, respectively. It also constructs regional adjacency matrices based on real geographical distances to match the model structure with the data characteristics, avoiding the simple stacking of general networks and making it more applicable to agricultural remote sensing monitoring scenarios.

[0020] 4. The technical principle by which this invention achieves the "hardware" technical effect is: to solve the acquisition defects of single imaging hardware and the shortcomings of weak hardware differentiation of ground objects and insufficient sensor distribution through the multi-terminal hardware collaborative acquisition principle; and to solve the hardware underlying data conflict caused by the mismatch of resolution, imaging scale and data units of different acquisition hardware through the hardware computing power layered processing and alignment principle.

[0021] Therefore, the quantifiable hardware technology effects of this invention are: complementary and collaborative multi-type sensing hardware enables complete data collection of agricultural areas across multiple time phases, scales, and all weather conditions, eliminating time-series data gaps and ensuring the integrity and continuity of original sensing data; relying on hardware-based full-domain sensing data input, the model prediction accuracy is improved by 12% compared to the traditional single-hardware data source LSTM solution, spatial analysis consistency is improved by 15%, and the hardware data linkage response lag rate is reduced by 25% in sudden disease and drought scenarios. Attached Figure Description

[0022] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a flowchart of the multi-source data agricultural intelligent sensing and prediction method according to an embodiment of this application; Figure 2 This is a schematic diagram of the multimodal feature coding network structure according to an embodiment of this application; Figure 3 This is a schematic diagram of the cross-modal attention fusion mechanism in an embodiment of this application; Figure 4 This is a temporal prediction structure diagram of the spatiotemporal interactive attention network according to an embodiment of this application. Detailed Implementation

[0023] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0024] like Figures 1 to 4 As shown in the figure, this embodiment provides a multimodal spatiotemporal feature fusion and prediction method. This method is applied to agricultural remote sensing monitoring scenarios and aims to solve the problems of difficulty in fusing multi-source heterogeneous data and insufficient capture of spatiotemporal features of crop growth. This method achieves high-precision perception and prediction of agricultural conditions in the target area by constructing a complete processing link from data alignment, differential coding, adaptive fusion to spatiotemporal joint prediction. The invention will be described in detail below.

[0025] Step S1: Acquire multi-source heterogeneous data of the target area, and sequentially perform radiometric correction, atmospheric correction, resampling and registration on the acquired multi-source raw data to achieve data spatial alignment.

[0026] Because satellite remote sensing, UAV imagery, and ground sensor data have different acquisition mechanisms and spatial resolutions, direct fusion can lead to severe spatial misalignment and radiometric bias. This embodiment uses a unified WGS84 geographic coordinate reference system as the spatial benchmark.

[0027] During processing, the optical and SAR images are first radiometrically calibrated and atmospherically corrected to eliminate sensor noise and aerosol scattering effects. Then, using high-resolution orthorectified images as a reference, multi-source data are registered and resampled at the sub-pixel level using mutual information or phase correlation algorithms to ensure that all data are strictly aligned at the pixel scale.

[0028] After the above processing, the modal data are strictly aligned in space, providing a consistent basis for subsequent feature-level fusion and avoiding feature mismatch caused by geometric deviations.

[0029] Step S2 involves encoding features from the multi-source heterogeneous data to obtain multiple modal features. To address the differences in physical characteristics among the different modalities, this embodiment employs a differentiated encoding strategy rather than a general network stacking approach to ensure the effectiveness of feature extraction.

[0030] like Figure 2 As shown, for optical images, which are rich in surface reflectance spectrum and texture information, a convolutional neural network (CNN) is used to extract spatial features. The encoding process can be represented as follows:

[0031] in, This represents the input optical image data. This represents a convolutional neural network used for feature extraction from optical images. This represents the extracted optical spatial features.

[0032] For synthetic aperture radar (SAR) imagery, since its data is essentially a complex signal and contains phase information reflecting the structure and dielectric properties of ground features, a complex convolution structure is used for processing. This approach simultaneously preserves amplitude and phase information and extracts texture and structural features, avoiding the loss of crucial physical information due to model extraction. The encoding process can be represented as follows:

[0033] in, This represents the input SAR image data. This indicates a SAR image feature extraction network employing a complex convolutional structure. This represents the extracted SAR texture and structural features.

[0034] The spatial features from optical and SAR sources are merged through a cross-modal stitching operation to form:

[0035] in This represents the spatial characteristics after stitching together optical and SAR data. This indicates a cross-modal feature splicing operation.

[0036] For meteorological and soil sensor data, which exhibit typical time-series signals and long-range dependencies, Long Short-Term Memory (LSTM) networks are used to capture temporal trends. The hidden state of the LSTM at time t is shown below. The calculation formula is as follows:

[0037] in, This represents the meteorological or soil observation input vector at time t. and These are the weight matrix and bias term inside the LSTM unit, respectively. This represents a nonlinear transformation operation that includes a forget gate, an input gate, and an output gate.

[0038] Finally, the spatial features S and temporal features are combined. Concatenate them into a unified modal feature representation:

[0039] in, This represents the spatial characteristics after stitching together optical and SAR data. The time-series characteristics of the LSTM output. This indicates a feature concatenation operation. This represents the unified modal feature representation after splicing.

[0040] Through the aforementioned differential encoding, each modality data is converted into a semantically aligned high-dimensional feature representation, laying the foundation for subsequent fusion.

[0041] Step S3: Based on the contextual information of each modal feature, dynamically calculate the attention weight of each modal feature, and perform weighted fusion of multiple modal features based on the attention weight to obtain fused spatiotemporal features.

[0042] In agricultural monitoring, the reliability of different modal data fluctuates drastically with environmental conditions. For example, cloud cover can render optical images ineffective, while heavy rainfall can interfere with SAR backscatter signals. To address this, this embodiment introduces an Adaptive Multi-modal Attention (AMA) mechanism to achieve dynamic fusion based on the reliability of each modal feature in the current context.

[0043] like Figure 3 As shown, this mechanism does not use fixed weights, but automatically adjusts the contribution of each modality based on the real-time quality of the input features. The attention weight coefficient of the i-th modality... Calculated using the following formula:

[0044] in, The feature vector obtained by encoding the i-th mode in step S2 is... Let be the learnable weight matrix corresponding to the i-th modality, used to map features to the reliability assessment space. The function is a normalized exponential function, ensuring that the sum of all modal weights is 1.

[0045] Based on this, spatiotemporal features are integrated. Obtained through the following formula:

[0046] in, This indicates a weighted summation of all modalities involved in the fusion (such as optical, SAR, and meteorological). This indicates a weighted operation.

[0047] Through this mechanism, when optical features are detected... When the confidence level decreases due to cloud and fog interference, the network will automatically reduce its corresponding confidence level. It also increases the weight of SAR or meteorological features, thereby achieving dynamic denoising and complementary enhancement at the feature level and improving the robustness of data fusion under complex weather conditions.

[0048] Step S4: Extract the temporal evolution features in the time dimension and the spatial topological dependence features in the spatial dimension of the fused spatiotemporal features, and perform joint prediction based on the temporal evolution features and spatial topological dependence features to obtain the prediction results of the target region.

[0049] Crop growth follows the temporal evolution of phenological stages and is also influenced by the microclimate of adjacent plots and the spatial diffusion of pests and diseases. For example... Figure 4 As shown, this embodiment uses a dual-branch parallel structure of the Spatiotemporal Interactive Attention Network (STIA) for joint modeling.

[0050] In the time dimension, the Transformer network is used to model the sequence evolution of the fused spatiotemporal features, and the self-attention mechanism is used to capture the global dependencies in the long time series to obtain the temporal evolution features.

[0051] In the spatial dimension, spatial topological dependency features are extracted based on a regional graph structure and a graph convolutional network (GCN). The regional graph structure here is not built based on abstract feature similarity, but rather on a region adjacency matrix strictly defined by geospatial distance. The Euclidean geographical distance between the center points of each field within the target region is calculated. If the distance between two points is less than a preset threshold, an edge is established and weighted; otherwise, no edge is established.

[0052] This graph structure construction method based on real geographical proximity can accurately depict the spatial continuity of physical processes such as water and fertilizer transport and disease spread in the agricultural environment, avoiding the spurious correlations that may be generated by purely data-driven mapping.

[0053] The joint prediction process can be represented as:

[0054] in, This indicates the final prediction result. To integrate spatiotemporal features, This represents a time series modeling network based on a self-attention mechanism. This represents a graph convolutional network based on the region adjacency matrix. This indicates a spatiotemporal interactive attention network that fuses the outputs of the temporal and spatial branches to generate prediction results.

[0055] Finally, the temporal evolution features and spatial topological dependency features are spliced ​​or added together and fused, and the results such as crop growth index and yield prediction map are output through a fully connected layer.

[0056] This embodiment also provides a multimodal spatiotemporal feature fusion and prediction system, which includes a feature encoding module, a cross-modal fusion module, and a spatiotemporal prediction module.

[0057] The feature encoding module is responsible for acquiring multi-source heterogeneous data of the target area and performing feature encoding on each modality of the data. Specifically, spatial features are extracted from optical images using a convolutional neural network, texture and structural information are extracted from SAR images using a complex convolutional structure, and temporal trends are captured from meteorological and soil sensor data using a long short-term memory network. Finally, multiple semantically aligned modal features are output.

[0058] The cross-modal fusion module receives the modal features output by the feature encoding module, evaluates the reliability of each modality in the current context based on the adaptive attention mechanism, dynamically calculates the attention weight coefficients, and performs a weighted summation of each modal feature according to the weights to output the fused spatiotemporal features.

[0059] After receiving the fused spatiotemporal features, the spatiotemporal prediction module performs sequence evolution modeling in the time dimension through a Transformer network, and topological dependency modeling in the spatial dimension through a graph convolutional network and a regional adjacency matrix constructed based on geographical distance. Finally, the outputs of the two branches are fused and passed through a fully connected layer to obtain the prediction result.

[0060] This embodiment also provides a computer device, including a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it completes the steps of the multimodal spatiotemporal feature fusion and prediction method. The memory can be a medium such as RAM, ROM, flash memory, disk, or optical disk, and the processor can be a general-purpose processor, a special-purpose processor, or a programmable logic device such as an FPGA.

[0061] This embodiment provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, it triggers the processor to complete the steps of the multimodal spatiotemporal feature fusion and prediction method. The storage medium can be a tangible, non-volatile storage medium, such as a USB flash drive, portable hard drive, optical disc, magnetic disk, or semiconductor memory.

[0062] Those skilled in the art will understand that, besides implementing the system and its various devices, modules, and units provided by this invention in the form of purely computer-readable program code, the same functions can be achieved entirely through logical programming of the method steps, making the system and its various devices, modules, and units of this invention function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the system and its various devices, modules, and units provided by this invention can be considered as a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered as structures within the hardware component; alternatively, the devices, modules, and units for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0063] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A multimodal spatiotemporal feature fusion and prediction method, characterized in that, include: Acquire multi-source heterogeneous data of the target region, and perform feature encoding on the multi-source heterogeneous data to obtain multiple modal features; Based on the contextual information of each modal feature, the attention weight of each modal feature is dynamically calculated, and multiple modal features are weighted and fused based on the attention weight to obtain fused spatiotemporal features; The temporal evolution features of the fused spatiotemporal features in the time dimension and the spatial topological dependence features in the spatial dimension are extracted respectively. Based on the temporal evolution features and the spatial topological dependence features, a joint prediction is performed to obtain the prediction result of the target region.

2. The multimodal spatiotemporal feature fusion and prediction method according to claim 1, characterized in that, The process involves dynamically calculating the attention weights of each modal feature based on its contextual information, and then weighted and fused multiple modal features based on these attention weights to obtain a fused spatiotemporal feature, including: An adaptive attention mechanism is used to evaluate the reliability of each modal feature in the context information and to calculate the attention weight coefficient corresponding to each modal feature. The fused spatiotemporal features are obtained by weighting and summing multiple modal features based on the attention weight coefficients.

3. The multimodal spatiotemporal feature fusion and prediction method according to claim 2, characterized in that, The multi-source heterogeneous data includes optical images, synthetic aperture radar images, and meteorological data; The calculation of the attention weight coefficients corresponding to each modality feature includes: Through formula Calculate the first Attention weight coefficients for each modality ;in, For the first Modal features, For the first The learnable weight matrix corresponding to each mode It is a normalized exponential function; The step of weighting and summing multiple modal features based on the attention weight coefficients to obtain the fused spatiotemporal features includes: Through formula Calculate the fused spatiotemporal features ;in, This indicates a weighted summation of all modalities participating in the fusion. This indicates a weighted operation.

4. The multimodal spatiotemporal feature fusion and prediction method according to claim 1, characterized in that, The process involves extracting the temporal evolution features of the fused spatiotemporal features in the time dimension and the spatial topological dependency features in the spatial dimension, and then performing joint prediction based on the temporal evolution features and the spatial topological dependency features to obtain the prediction result of the target region, including: In the time dimension, the temporal evolution features are extracted through sequence modeling; In the spatial dimension, the spatial topological dependency features are extracted based on the region graph structure; The prediction result is obtained by fusing the temporal evolution features and the spatial topological dependency features.

5. The multimodal spatiotemporal feature fusion and prediction method according to claim 4, characterized in that, The region map structure is defined based on a region adjacency matrix constructed from geographic spatial distances; The extraction of temporal evolution features through sequence modeling in the time dimension includes: The temporal evolution features are obtained by performing temporal-dimensional sequence evolution modeling on the fused spatiotemporal features using a Transformer network. The extraction of spatial topological dependency features based on the region graph structure in the spatial dimension includes: By using a graph convolutional network and the region adjacency matrix, the spatial topological dependency model of the fused spatiotemporal features is performed to obtain the spatial topological dependency features.

6. The multimodal spatiotemporal feature fusion and prediction method according to claim 1, characterized in that, The feature encoding of the multi-source heterogeneous data yields multiple modal features, including: For the optical images in the multi-source heterogeneous data, spatial features are extracted using a convolutional neural network to obtain the corresponding modal features; For the synthetic aperture radar image in the multi-source heterogeneous data, texture and structural information are extracted through complex convolution structure to obtain the corresponding modal features; For the meteorological and soil sensor data in the multi-source heterogeneous data, the temporal change trend is captured by a long short-term memory network to obtain the corresponding modal features.

7. The multimodal spatiotemporal feature fusion and prediction method according to claim 1, characterized in that, include: Multi-source heterogeneous data of the target area is acquired through multi-type sensing hardware, and multiple modal features are obtained by using computing hardware to encode the features of the multi-source heterogeneous data respectively. Based on the contextual information of each modal feature, the attention weight of each modal feature is dynamically calculated, and multiple modal features are weighted and fused based on the attention weight to obtain fused spatiotemporal features; The temporal evolution features of the fused spatiotemporal features in the time dimension and the spatial topological dependence features in the spatial dimension are extracted using dedicated computing hardware for spatiotemporal interaction networks. Based on the temporal evolution features and the spatial topological dependence features, joint prediction is performed to obtain the prediction results for the target region. The agricultural intelligent decision-making results are then output by visualization and gateway hardware.

8. A multimodal spatiotemporal feature fusion and prediction system, characterized in that, include: The feature encoding module is used to acquire multi-source heterogeneous data of the target region and perform feature encoding on the multi-source heterogeneous data to obtain multiple modal features; The cross-modal fusion module is used to dynamically calculate the attention weight of each modal feature based on the context information of each modal feature, and to perform weighted fusion of multiple modal features based on the attention weight to obtain fused spatiotemporal features; The spatiotemporal prediction module is used to extract the temporal evolution features of the fused spatiotemporal features in the time dimension and the spatial topological dependency features in the spatial dimension, and to perform joint prediction based on the temporal evolution features and the spatial topological dependency features to obtain the prediction result of the target region.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the multimodal spatiotemporal feature fusion and prediction method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the multimodal spatiotemporal feature fusion and prediction method according to any one of claims 1 to 7.

Images