Dual-mode multi-task general analysis method and system for spatio-temporal data

Abstract

The invention discloses a spatio-temporal data dual-mode multi-task general analysis method and a spatio-temporal data dual-mode multi-task general analysis system. The method comprises the following steps: acquiring spatio-temporal data of different modes; converting the spatio-temporal data of different modalities into data sequences of the same format, and generating road network representation vectors according to the spatio-temporal data of different modalities; performing up-sampling on the data sequence and the road network representation vector to obtain a spatio-temporal data feature sequence; and determining a text instruction and a task placeholder, and analyzing the spatio-temporal data by using the spatio-temporal data analysis model in combination with the spatio-temporal data feature sequence. According to the method, the problem of incompatibility between different data modes is effectively solved, and the requirements of different tasks can be flexibly met.

Description

Technical Field

[0001] The present invention relates to the technical field of data analysis, and more specifically to a dual-modal multi-task general analysis method and system for spatiotemporal data. Background Art

[0002] Spatiotemporal data analysis has wide applications in intelligent transportation systems (ITS), smart cities, and location-based services (LBS). Existing spatiotemporal data analysis methods mainly focus on specific tasks, which results in limited generalization ability of the models across tasks.

[0003] At present, multi-task general deep learning models have achieved remarkable success in fields such as natural language processing (NLP), computer vision (CV), and multimedia (MM), but they still face many challenges in the field of spatiotemporal data analysis. Including:

[0004] 1) Challenges in constructing a unified representation of spatiotemporal data: Two important types of data in spatiotemporal data analysis—individual trajectories and group traffic status—are usually viewed as two incompatible data modalities. Trajectory data is usually represented as a sequence of geographic units, such as road segments or points of interest (POIs); while traffic status data is represented as a graph structure with dynamic signals (such as traffic speed). How to unify the data of these two different modalities is a challenge.

[0005] Existing methods generally build general spatiotemporal data representations to support various downstream tasks, but most of these methods focus on individual static geographic elements, such as road networks and points of interest (POIs), or only focus on trajectory data.

[0006] In recent years, trajectory representation models such as TremBR and START have proposed to introduce the periodicity of temporal information, and traffic state models such as T-wave, TrajNet and TrGNN have captured multi-hop spatial dependencies by propagating information along trajectories. Although current representation models overlap in the fields of trajectory and traffic state, developing a unified representation perspective for these two types of data remains an unresolved research area;

[0007] 2) Challenges in unifying heterogeneous spatiotemporal data analysis tasks: Spatiotemporal data analysis tasks are highly heterogeneous, including:

[0008] The data format varies for different tasks; for example, the output of the travel time estimation task is a continuous time, while the output of the path planning task is another trajectory;

[0009] Heterogeneous tasks vary greatly in complexity; that is, supervisory signals require annotations of different granularities, resulting in a variety of training formats.

[0010] Although general models have been built, these general models use special models as tools and call these tools during the training process to achieve cross-task learning. This paradigm builds a general spatiotemporal data analysis system rather than a single model, and its ability to reveal the correlation between different tasks is limited.

[0011] Therefore, how to develop a general single model that can simultaneously process trajectory and traffic status data and perform spatiotemporal data analysis across multiple heterogeneous tasks remains a major challenge. Summary of the invention

[0012] In view of this, in order to at least partially solve the above-mentioned technical problems, the present invention provides a bimodal multi-task general analysis method and system for spatiotemporal data, aiming to achieve generalization across data sets based on traffic spatiotemporal data, and to process a variety of tasks based on spatiotemporal data, thereby realizing the analysis and prediction of traffic conditions.

[0013] In order to achieve the above object, the present invention adopts the following technical solution:

[0014] In a first aspect, the present application discloses a general analysis method for spatiotemporal data in dual-mode and multi-task mode, comprising:

[0015] Acquire spatiotemporal data of different modalities;

[0016] Convert spatiotemporal data of different modes into data sequences of the same format, and generate road network representation vectors based on spatiotemporal data of different modes;

[0017] Using the data sequence as an index, extract the corresponding spatiotemporal data feature sequence from the road network representation vector;

[0018] Determine text instructions and task placeholders, combine the spatiotemporal data feature sequence, and use the spatiotemporal data analysis model to analyze the spatiotemporal data.

[0019] Furthermore, the spatiotemporal data comes from a road network of the area to be analyzed, and each node in the road network contains static road trajectory information and dynamic traffic status.

[0020] Furthermore, the sequence format is: [geographic location, instantaneous time index, state interval index];

[0021] The geographical location is the specific road section where you are currently located, the instantaneous time is the current time, and the status interval is the traffic status information within the statistical preset time interval.

[0022] Furthermore, a road network representation vector is obtained according to the spatiotemporal data of different modes; the steps include:

[0023] The static road network features and dynamic road network features in different modal spatiotemporal data are extracted through the characterizer and then spliced;

[0024] Determine the K and V values ​​based on the concatenated features, combine the learnable Q values, and output the attention-weighted features through the cross-attention mechanism;

[0025] The attention weighted features are further extracted through the MLP network to obtain the road network representation vector.

[0026] Furthermore, the characterizer comprises a first FFN network, a GAT network and a second FFN network which are connected in sequence.

[0027] Further, task placeholders include classification and regression.

[0028] Furthermore, the spatiotemporal data analysis model adopts a hierarchical training strategy, and the training steps include:

[0029] Pre-train the spatiotemporal data analysis model based on the spatiotemporal data feature sequence and task placeholders;

[0030] Incorporate text instructions to fine-tune spatiotemporal data analysis models;

[0031] Introduce reinforcement learning to improve the performance of spatiotemporal data analysis models.

[0032] Furthermore, the spatiotemporal data analysis model is composed of multiple stacked Blocks, each of which includes a parallel Value network, a Query network, and a Key network. The output end of each network is commonly connected to a multi-head attention network, and the output end of the multi-head attention network is sequentially connected to a normalization layer and a feedforward neural network.

[0033] In a second aspect, the present application discloses a spatiotemporal data bimodal multi-task general analysis system, which applies the spatiotemporal data bimodal multi-task general analysis method as described above, wherein the system comprises:

[0034] A data acquisition module is used to acquire spatiotemporal data of different modalities;

[0035] A format conversion module is used to convert spatiotemporal data of different modalities into data sequences of the same format;

[0036] A road network representation vector generation module, used to generate a road network representation vector according to spatiotemporal data of different modes;

[0037] The spatiotemporal data feature sequence extraction module is used to extract the road network representation vector using the data sequence as an index;

[0038] The spatiotemporal data analysis module is used to analyze the spatiotemporal data using the spatiotemporal data analysis model based on text instructions and task placeholders combined with spatiotemporal data feature sequences.

[0039] Furthermore, it also includes a spatiotemporal data analysis model training module, which is used to train the spatiotemporal data analysis model using a hierarchical training strategy, and the training steps include:

[0040] Pre-train the spatiotemporal data analysis model based on the spatiotemporal data feature sequence and task placeholders;

[0041] Incorporate text instructions to fine-tune spatiotemporal data analysis models;

[0042] Introduce reinforcement learning to improve the performance of spatiotemporal data analysis models.

[0043] It can be seen from the above technical solution that the present invention discloses a general bimodal multi-task analysis method and system for spatiotemporal data, which mainly realizes the unified representation of spatiotemporal data of different modalities through spatiotemporal units (ST Units) and spatiotemporal tokenizers, and introduces interactive prompts and hierarchical training strategies to cope with the challenges of task heterogeneity.

[0044] Compared with the existing technology,

[0045] 1. The present invention successfully constructs a unified spatiotemporal data representation framework by converting individual trajectory and group traffic status data into data sequences of the same format and generating road network representation vectors. This framework effectively solves the incompatibility problem between different data modalities, allowing the two types of data to be analyzed on a common basis.

[0046] 2. Different from the paradigm of general models as tool calls in the prior art, the spatiotemporal data analysis model proposed in this invention can directly process multiple heterogeneous tasks, such as classification and regression, through a single model, without the need to design a dedicated model for each task, which helps to reveal the correlation between different spatiotemporal data analysis tasks. The combination of task placeholders and text instructions enables the model to flexibly adapt to the needs of different tasks, greatly improving the efficiency of task processing.

[0047] 3. The hierarchical training strategy, including pre-training, fine-tuning, and reinforcement learning, significantly improves the generalization ability of the model, enabling the model to not only perform well on specific datasets, but also maintain high performance across datasets. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0049] Figure 1 This is a flowchart of a general dual-modal multi-task analysis method for spatiotemporal data;

[0050] Figure 2 This is a schematic diagram of the structure of ST Tokenizer;

[0051] Figure 3 Sample graph of input data for trajectory generation task;

[0052] Figure 4 Schematic diagram of the three-stage training method;

[0053] Figure 5 The data, model, and task correspondence diagram for spatiotemporal analysis using the current benchmark model;

[0054] Figure 6 This is a relationship diagram between data, models and tasks obtained by using the solution of the present invention. DETAILED DESCRIPTION

[0055] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0056] In order to overcome the above problems in the prior art, the embodiment of the present invention first discloses a dual-modal multi-task general analysis method for spatiotemporal data; Figure 1 , Figure 1 Schematic diagram of the flow of the general bimodal multi-task analysis method for spatiotemporal data.

[0057] The analysis method steps include:

[0058] S1. Acquire spatiotemporal data of different modalities; in one embodiment, it is used to process common spatiotemporal applications based on road networks, including trajectory and traffic status tasks; that is, in the data set of this application, all heterogeneous spatiotemporal data come from the road network of the area to be analyzed, and the network is constructed as a graph structure, and each node contains static road trajectory information and dynamic traffic status.

[0059] S2, converting the spatiotemporal data of different modes into data sequences of the same format, and generating a road network representation vector according to the spatiotemporal data of different modes;

[0060] In this application, both trajectory data and traffic status data are essentially sequence data sampled from a dynamic road network. From this perspective, the main difference between trajectory data and traffic status data lies in the sampling method.

[0061] Therefore, in one embodiment, in response to the challenge of unified data representation, the present application proposes spatiotemporal units (STUnits) for converting spatiotemporal data of two different modalities, trajectory and traffic state sequence, into a unified format; that is, spatiotemporal data of both modalities can be represented as ST unit sequences.

[0062] ST unit sequence is a data format, specifically,

[0063] The format structure is: [geographic location, instantaneous time index, status interval index];

[0064] The geographic location is the specific road section you are currently on, and the instantaneous time refers to the current time accurate to seconds. The status interval is used to measure the traffic status, with half an hour as the interval, and the traffic status information in the time interval is counted.

[0065] In another embodiment, the present application designs a spatiotemporal tokenizer (ST Tokenizer) for generating a road network representation vector according to spatiotemporal data of different modalities, wherein the road network representation vector is a graph structure for representing a dynamic road network;

[0066] The generation steps include:

[0067] The static road network features and dynamic road network features in different modal spatiotemporal data are extracted through the characterizer and then spliced;

[0068] Determine the K and V values ​​based on the concatenated features, combine the learnable Q values, and output the attention-weighted features through the cross-attention mechanism;

[0069] The attention weighted features are further extracted through the MLP network to obtain the road network representation vector.

[0070] In one embodiment, the structure of ST Tokenizer refers to Figure 2 ; Including static characterizer, dynamic characterizer and fusion characterizer;

[0071] The static characterizer and the dynamic characterizer have the same structure, including a first FFN network, a GAT network and a second FFN network connected in sequence. The present invention can achieve deep fusion of spatiotemporal data through the characterizer. This fusion not only enhances the comprehensiveness of data analysis, but also improves the explanatory power and prediction accuracy of the model.

[0072] The first FFN network (fully connected feedforward network) is used to preliminarily process static road network information and extract features;

[0073] GAT network (Graph Attention Network), used to capture spatial dependencies in the road network and emphasize important road segment features;

[0074] The second FFN network (fully connected feedforward network) is used to further process and abstract the features output by the GAT network.

[0075] The features processed by the static characterizer and the dynamic characterizer are further concatenated and merged into a richer feature representation, which helps the model better understand the overall situation of the road network.

[0076] The K and V values ​​are further determined based on the concatenated features, and combined with the learnable Q values, the attention-weighted features are output through the cross-attention mechanism; the K (Key) and V (Value) values ​​represent the important parts of the road network features, and the importance of different features can be determined by interactive attention calculation with the learnable Q values.

[0077] The fusion representation includes cross attention and MLP network;

[0078] The role of cross attention is to obtain the importance weight of each feature by calculating the similarity between Q and K, and then weighted sum V value to obtain the final attention weighted feature;

[0079] The MLP network is used to process the features output by the cross-attention mechanism, further extract and abstract information, and finally output a road network representation vector. This vector can be a comprehensive representation of the entire road network or an intermediate representation for a specific prediction task.

[0080] In this embodiment, ST Tokenizer combines static and dynamic information, utilizes graph neural network and attention mechanism, can effectively capture the complex relationship of road network, and provide strong feature support for subsequent road network analysis or prediction tasks.

[0081] In one embodiment, the learnable Q is a trainable variable, that is, the loss is calculated by paying attention to the output and the label, thereby calculating the gradient of the loss relative to Q, and using an optimization algorithm to update the parameters of Q. The learnable Q enables the model to extract useful information from the input features more effectively.

[0082] S3, using the data sequence as an index, obtaining the corresponding data in the road network representation vector, and obtaining a spatiotemporal data feature sequence;

[0083] In this embodiment, it can be considered that different modal spatiotemporal data are sampled in different ways. Specifically, the data sequence formed by the trajectory and traffic data provides a data index, and the spatiotemporal feature information is extracted from the dynamic road network representation according to the index, thereby obtaining a unified representation of the trajectory and traffic status (spatiotemporal data feature sequence).

[0084] S4. Determine text instructions and task placeholders, combine the spatiotemporal data feature sequence, and use the spatiotemporal data analysis model to analyze the spatiotemporal data.

[0085] To address the challenge of task heterogeneity, this embodiment introduces interactive prompts, that is, uniformly marking specific task data from different heterogeneous tasks, including input data and task-related specifications.

[0086] For input data, based on the sequence of spatiotemporal data features, this application adds text instructions to guide the model to perform task types;

[0087] For output tasks, since multiple tasks may share the same spatiotemporal input, it is difficult for the model to determine the specific task type based on spatiotemporal data alone. Therefore, to solve this problem, this application introduces a task instruction mechanism as a task identifier to indicate the output type and quantity of each task. Based on this, the data of multiple spatiotemporal tasks can be integrated into one dataset for joint training.

[0088] In one embodiment, the spatiotemporal tasks are divided into four categories, and their output forms are summarized into two categories: classification of static road segment IDs and regression of dynamic features.

[0089] As a preferred solution, the task placeholders are defined as: [CLS] for classification, and [REG] for regression.

[0090] In an exemplary embodiment, a personalized text instruction template is provided for each task to clarify the task type. Therefore, in this way, the format of all input data can be unified. Specifically, the input of the model is divided into three parts: text instructions, spatiotemporal data feature sequence, and task placeholder.

[0091] To further demonstrate the connection between text instructions, spatiotemporal data, and task placeholders, this application takes the trajectory generation task as an example and provides a specific input data sample. Figure 3 .

[0092] In one embodiment, heterogeneous tasks differ in complexity and training paradigms. For example, generation tasks output sequences and use sequence labels for supervision, while classification and regression tasks rely on single labels for supervision. To address these challenges, the present application designs a three-stage training process to meet the needs of tasks of different complexity, including trajectory reconstruction in model pre-training; task-oriented prompt fine-tuning to adapt the model to the task; and generative reinforcement learning to enhance the performance of trajectory generation tasks. Figure 4 , the specific training steps include:

[0093] 1) Pre-train the spatiotemporal data analysis model based on the spatiotemporal data feature sequence and task placeholders;

[0094] In this stage, only spatiotemporal data (ST data) and task placeholders are used for training. Specifically, the input of the model is divided into three parts: text instructions, spatiotemporal data, and task placeholders. For the output of the model, only the task placeholders are taken as the final result. The number of task placeholders corresponds to the number of results that the model needs to output. Regression tasks are obtained through regression placeholders ([REG]), and classification tasks are obtained through classification placeholders ([CLS]);

[0095] 2) Incorporate text instructions to fine-tune the spatiotemporal data analysis model; with the help of text instructions, the model is fine-tuned jointly on multiple tasks. After this stage, the model is able to handle classification and regression tasks.

[0096] Since the input data of all tasks is unified into three parts: text instructions + spatiotemporal data + task placeholders, all tasks can be trained uniformly through a set of model architectures. Each type of task has its own text instruction template as a specific task identifier.

[0097] 3) Introduce reinforcement learning to improve the performance of spatiotemporal data analysis models.

[0098] The last stage introduces reinforcement learning to specifically improve the performance of the model on sequence labeling tasks (such as generation tasks).

[0099] This embodiment adopts the PPO algorithm in reinforcement learning; including:

[0100] First, the distance between the generated trajectory and the true trajectory is calculated by DTW. This distance calculation method can assign a distance value to each token in the trajectory. These distance values ​​are then accumulated to generate the reward value in reinforcement learning, and then the loss value in reinforcement learning is obtained.

[0101] In this application, all tasks can be trained by sequence modeling. Given GPT-2's powerful capabilities in sequence modeling, it can be used as the model's infrastructure to build a spatiotemporal data analysis model.

[0102] In one embodiment, the spatiotemporal data analysis model is a bimodal interactive general model (BIGST) for spatiotemporal data analysis. The model architecture of BIGST is as follows: Figure 1 shown.

[0103] BIGST includes multiple stacked blocks to learn complex sequence-to-sequence mappings. The blocks include parallel Value networks, Query networks, and Key networks, and the outputs of each network are connected to a multi-head attention network, and the outputs of the multi-head attention network are connected to a normalization layer and a feedforward neural network in sequence. This architecture design of the present invention is efficient, can capture long-distance spatiotemporal dependencies, and improves the computational efficiency and prediction accuracy of the model.

[0104] in,

[0105] Value network: used to extract "value" information from input data;

[0106] Query network: used to extract "query" information from input data;

[0107] Key network: used to extract "key" information.

[0108] These three networks are usually fully connected layers that process the input data in parallel and produce corresponding value, query, and key vectors.

[0109] In a multi-head attention network, the outputs from the Q, K, and V networks are split into multiple "heads", each of which computes a portion of the attention output. The final output of the multi-head attention network is the concatenation of the outputs of all heads, which is able to capture the relationship between different parts of the input sequence.

[0110] After the multi-head attention network, the output data passes through a normalization layer to normalize each feature of each sample to stabilize the training process and accelerate convergence.

[0111] The features after multi-head attention and layer normalization are then further processed through a feedforward neural network (FFN network) to increase the nonlinear capability of the model.

[0112] In this embodiment, there are residual connections (jump connections) between the multi-head attention network and the FFN network, as well as after the FFN network. These connections add the input data directly to the output of each sub-module, which helps prevent the gradient vanishing problem and allows the model to train deeper networks.

[0113] In an practicable embodiment, the present application discloses a spatiotemporal data dual-modal multi-task general analysis system, which applies the spatiotemporal data dual-modal multi-task general analysis method as described above, wherein the system includes:

[0114] A data acquisition module is used to acquire spatiotemporal data of different modalities;

[0115] A format conversion module is used to convert spatiotemporal data of different modalities into data sequences of the same format;

[0116] A road network representation vector generation module, used to generate a road network representation vector according to spatiotemporal data of different modes;

[0117] The spatiotemporal data feature sequence extraction module is used to extract the road network representation vector using the data sequence as an index;

[0118] The spatiotemporal data analysis module is used to analyze the spatiotemporal data using the spatiotemporal data analysis model based on text instructions and task placeholders combined with spatiotemporal data feature sequences.

[0119] Furthermore, it also includes a spatiotemporal data analysis model training module, which is used to train the spatiotemporal data analysis model using a hierarchical training strategy, and the training steps include:

[0120] Pre-train the spatiotemporal data analysis model based on the spatiotemporal data feature sequence and task placeholders;

[0121] Incorporate text instructions to fine-tune spatiotemporal data analysis models;

[0122] Introduce reinforcement learning to improve the performance of spatiotemporal data analysis models.

[0123] Since the detailed execution steps of this system are consistent with the above-mentioned general analysis method of spatiotemporal data bimodal multi-task, they will not be repeated here.

[0124] The bimodal multi-task general analysis method of spatiotemporal data in this application can integrate multiple data types and process multiple tasks simultaneously. In order to clarify the effects that this application can achieve, it is further compared with three different benchmark models. The data sets, models and tasks corresponding to each benchmark model are detailed in Figure 5 ;

[0125] Depend on Figure 5 It can be seen that in the existing methods, the three baselines have differences in model architecture and training paradigm. Further, the method or system of the present application is used to try to process these different tasks at the same time. The relationship between the obtained data, model and task is as follows: Figure 6 As shown;

[0126] Depend on Figure 6It can be seen that the advantage of this application lies in its powerful multi-task capability. BIGST can process three different baselines using only one model, and all of them can achieve SOTA performance.

[0127] The present invention has achieved an important technological breakthrough in the field of spatiotemporal data analysis, and has provided a powerful analytical tool for intelligent transportation systems, urban planning and other related fields, and has broad application prospects and significant technical advantages.

[0128] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0129] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A dual-modal multi-task general analysis method for spatiotemporal data, characterized in that: Acquire spatiotemporal data of different modalities; Convert spatiotemporal data of different modes into data sequences of the same format, and generate road network representation vectors based on spatiotemporal data of different modes; Using the data sequence as an index, extract the corresponding spatiotemporal data feature sequence from the road network representation vector; Determine text instructions and task placeholders, combine the spatiotemporal data feature sequence, and use the spatiotemporal data analysis model to analyze the spatiotemporal data.

2. The dual-modal multi-task general analysis method for spatiotemporal data according to claim 1 is characterized in that: The spatiotemporal data comes from the road network of the area to be analyzed, and each node in the road network contains static road trajectory information and dynamic traffic status.

3. The dual-modal multi-task general analysis method for spatiotemporal data according to claim 1 is characterized in that: The sequence format is: [geographic location, instantaneous time index, state interval index]; The geographical location is the specific road section where you are currently located, the instantaneous time is the current time, and the status interval is the traffic status information within the statistical preset time interval.

4. The dual-modal multi-task general analysis method for spatiotemporal data according to claim 1 is characterized in that: Obtaining a road network representation vector based on spatiotemporal data of different modes; the steps include: The static road network features and dynamic road network features in different modal spatiotemporal data are extracted through the characterizer and then spliced; Determine the K and V values ​​based on the concatenated features, combine the learnable Q values, and output the attention-weighted features through the cross-attention mechanism; The attention weighted features are further extracted through the MLP network to obtain the road network representation vector.

5. The dual-modal multi-task general analysis method for spatiotemporal data according to claim 4 is characterized in that: The characterizer consists of a first FFN network, a GAT network and a second FFN network connected in sequence.

6. The dual-modal multi-task general analysis method for spatiotemporal data according to claim 1 is characterized in that: Task placeholders include classification and regression.

7. The dual-modal multi-task general analysis method for spatiotemporal data according to claim 1 is characterized in that: The spatiotemporal data analysis model adopts a hierarchical training strategy. The training steps include: Pre-train the spatiotemporal data analysis model based on the spatiotemporal data feature sequence and task placeholders; Incorporate text instructions to fine-tune spatiotemporal data analysis models; Introduce reinforcement learning to improve the performance of spatiotemporal data analysis models.

8. The dual-modal multi-task general analysis method for spatiotemporal data according to claim 1 is characterized in that: The spatiotemporal data analysis model is composed of multiple stacked blocks, each of which includes a parallel Value network, a Query network, and a Key network. The output end of each network is commonly connected to a multi-head attention network, and the output end of the multi-head attention network is sequentially connected to a normalization layer and a feedforward neural network.

9. A dual-modal multi-task general analysis system for spatiotemporal data, characterized in that: The system using the dual-modal multi-task general analysis method for spatiotemporal data described in any one of claims 1 to 8 comprises: A data acquisition module is used to acquire spatiotemporal data of different modalities; A format conversion module is used to convert spatiotemporal data of different modalities into data sequences of the same format; A road network representation vector generation module, used to generate a road network representation vector according to spatiotemporal data of different modes; The spatiotemporal data feature sequence extraction module is used to extract the road network representation vector using the data sequence as an index; The spatiotemporal data analysis module is used to analyze the spatiotemporal data using the spatiotemporal data analysis model based on text instructions and task placeholders combined with spatiotemporal data feature sequences.

10. The spatiotemporal data dual-modal multi-task general analysis system according to claim 9, characterized in that: It also includes a spatiotemporal data analysis model training module, which is used to train the spatiotemporal data analysis model using a hierarchical training strategy. The training steps include: Pre-train the spatiotemporal data analysis model based on the spatiotemporal data feature sequence and task placeholders; Incorporate text instructions to fine-tune spatiotemporal data analysis models; Introduce reinforcement learning to improve the performance of spatiotemporal data analysis models.

Images