Space-time big data processing method based on pre-training-fine tuning architecture

Abstract

The invention discloses a space-time big data processing method based on a pre-training-fine tuning architecture, and solves the problems of complex relation and high calculation cost when a traditional method processes space-time multi-attribute data by combining a pre-trained Transform model and a prompt fine tuning mechanism. Specifically, the model framework comprises a time encoder, a space encoder and a head network, and common features among a plurality of time-space attributes are efficiently captured in a parameter sharing mode. In the pre-training stage, the model learns a general spatio-temporal pattern in a large-scale spatio-temporal data set, and then fine tuning is performed on specific spatio-temporal attributes by using a trainable spatio-temporal prompt token through a prompt fine tuning stage, so that efficient task adaptation and low calculation cost are realized. The method not only can improve the space-time prediction precision, but also has good mobility, adapts to unseen space-time attributes, and has wide application prospects.

Description

Technical Field

[0001] This invention belongs to the field of big data technology, specifically relating to a spatiotemporal big data processing method based on a pre-training-fine-tuning architecture. Background Technology

[0002] With the acceleration of urbanization, spatiotemporal big data is playing an increasingly important role in the construction of smart cities. Multiple fields, including urban management, traffic monitoring, human activity analysis, and social event prediction, rely on the mining and analysis of spatiotemporal data. However, due to the complexity of spatiotemporal data—its multidimensionality and multiple attributes—existing technologies face significant challenges in handling multi-attribute spatiotemporal prediction problems. Traditional spatiotemporal data processing methods often only predict a single attribute, neglecting the interrelationships and common characteristics between different attributes.

[0003] Although some spatiotemporal multi-task learning methods attempt to handle multiple spatiotemporal tasks, their task-specificity and model complexity make them difficult to extend to more tasks and attributes. These methods lack effective modeling of relationships between multiple attributes and have high training costs. Therefore, how to effectively handle spatiotemporal multi-attribute prediction, capture common features in spatiotemporal data, and improve the adaptability and transferability of models has become a pressing technical challenge. Summary of the Invention

[0004] To overcome the shortcomings of existing technologies, this invention provides a spatiotemporal big data processing method based on a pre-training-fine-tuning architecture. By combining a pre-trained Transformer model and a cue-based fine-tuning mechanism, it solves the problems of complex relationships and high computational costs faced by traditional methods when processing spatiotemporal multi-attribute data. Specifically, the model framework includes a temporal encoder, a spatial encoder, and a head network, which efficiently captures common features among multiple spatiotemporal attributes by sharing parameters. In the pre-training stage, the model learns common spatiotemporal patterns in large-scale spatiotemporal datasets. Subsequently, in the cue-based fine-tuning stage, trainable spatiotemporal cue tokens are used to fine-tune the model for specific spatiotemporal attributes, thereby achieving efficient task adaptation and low computational cost. This method not only improves the accuracy of spatiotemporal prediction but also has good transferability, adapting to unseen spatiotemporal attributes, and has broad application prospects.

[0005] The technical solution adopted by this invention to solve its technical problem is as follows: Step 1: Build the model framework; It employs a Transformer-based architecture to capture knowledge of multiple spatiotemporal attributes; the Transformer architecture consists of a temporal encoder, a spatial encoder, and a header; Step 2: Construct a spatiotemporal Transformer network; Step 3: Pre-training phase; During the pre-training phase, the spatiotemporal Transformer network is trained using all spatiotemporal attributes; Step 4: Fine-tuning phase.

[0006] Preferably, step 1 specifically comprises: Step 1-1: In the pre-training phase, the architecture is trained in a parameter-sharing manner, enabling the model to learn the common spatiotemporal features among all attributes.

[0007] Steps 1-2: Propose a fine-tuning phase to adapt to each attribute; Steps 1-3: By fixing the main body of the pre-trained framework, namely the temporal encoder and the spatial encoder, and updating only the cue symbols and the head, the spatiotemporal multi-attribute prediction scheme PromptST captures specific spatial and temporal patterns of each individual attribute through trainable parameters.

[0008] Preferably, step 2 specifically comprises: Step 2-1: Time encoder; The time encoder consists of stacked multi-head attention blocks and a position-by-position feedforward layer; Step 2-2: Spatial encoder; After capturing temporal dependencies using a temporal encoder, a spatial encoder is introduced to learn spatial information; spatial location information is added and processed using stacked multi-head attention and feedforward layers.

[0009] Preferably, step 2-1 specifically comprises: Step 2-1-1: Location embedding; Based on formula Study time information, among which, and These are learnable weights and biases. It is the learned representation. It is the embedding size; This represents the transpose function. Represents the sigmoid function; Based on formula Embed trainable locations into labeled time series. This represents the trainable position embedding vector corresponding to the time series; Step 2-1-2: Multi-head attention; Based on formula Calculate self-attention, where, These are the query, key, and value matrices, respectively; multi-head attention divides the input features into several parts and performs self-attention calculation on each part separately; Represents the Attention function. Represents the normalization function; Based on formula , After features are processed in different embedding subspaces, they are concatenated to integrate diverse information, among which... , , , It is a learnable parameter matrix; Indicates the output matrix. This indicates the output of the attention head. This represents the collection of output results; , They represent Matrix feature dimensions; Indicates the number of attention heads; Based on formula Perform multi-head attention operations; This represents the overall computation function. This indicates that the representation layer is normalized. Indicates input data; Step 2-1-3: Feedforward layer; Based on formula , Perform residual connections and layer normalization, where, , It is a parameter matrix. , It is a bias; Indicates the output function. This represents the activation function. Indicates the output of the feedforward sublayer; Based on formula Calculate the first Mid-time after the next iteration; based on the formula conduct Multi-head attention in the layer-time encoder layer and capture of temporal dependencies in the feedforward layer; , They represent the first Layer, First The output of the layer time encoder, This represents the final output of the time encoder. This represents the sum of the outputs of each time encoder layer.

[0010] Preferably, step S2-2 specifically includes: Step 2-2-1: Location embedding; Based on formula Learn the positional information between grids; Indicates spatial location embedding, The initial input to the spatial encoder is represented; Step 2-2-2: Spatial representation learning; Based on formula Calculate the first The space of layer iteration output; , They represent the first Layer, First The output of the layer space encoder; Based on formula The space representing the output. This represents the final output of the spatial encoder; Indicates the total number of layers in the spatial encoder; Step 2-2-3: Header; Based on formula Represents the spatiotemporal properties of the prediction The future state, in which, and These represent the weight matrix and the bias, respectively. This indicates the prediction result.

[0011] Preferably, step 3 specifically comprises: Step 3-1: Given the spatiotemporal attribute matrix Simultaneously using a single spatiotemporal attribute matrix For output By concatenating the C attributes together, we get ; Step 3-2: Based on the formula Denotes the optimization objective of pre-training, where, For the model parameters of the time encoder and the spatial encoder, The parameters for the head are used for model output. ; This represents the root mean square error function. This represents the mean absolute error function. This represents the overall loss function.

[0012] Preferably, step 4 specifically comprises: Step 4-1: Spacetime Hints; A spatiotemporal cue token is used and inserted into the feature sequence of each temporal encoder layer to trigger [the event / event]. The general spatiotemporal change patterns learned in the course; Will A learnable spacetime cue token Input time representation integrated into each layer In the time dimension; therefore, the time representation dimension of the connection is... After multi-head attention and feedforward processing, the added data is truncated. To maintain the original sequence length, the truncated sequence is... Same as before adding the token; based on the formula: Indicates the first Midway through the next iteration; Step 4-2: Prompt for fine-tuning; load ,freeze and initialize each spatiotemporal attribute. and notification token Based on formula This represents the optimization process, where the model output... .

[0013] An electronic device includes a processor and a memory; the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the electronic device performs the above-described spatiotemporal big data processing method.

[0014] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described spatiotemporal big data processing method.

[0015] A chip includes a processor for retrieving and running a computer program from a memory, causing a device equipped with the chip to perform the aforementioned spatiotemporal big data processing method.

[0016] A computer program product includes a computer storage medium storing a computer program, the computer program including instructions executable by at least one processor, which, when executed by the at least one processor, implement the above-described spatiotemporal big data processing method.

[0017] The beneficial effects of this invention are as follows: This invention proposes a novel spatiotemporal Transformer backbone structure and, combined with corresponding pre-training strategies, effectively captures common spatiotemporal variation patterns across multiple attributes. Through this structure, the model can efficiently process complex spatiotemporal data, extract similar patterns from multi-dimensional data, and thus improve the processing efficiency of spatiotemporal tasks.

[0018] Furthermore, this invention designs an innovative spatiotemporal cue fine-tuning strategy that can efficiently indicate the spatial and temporal characteristics of a single target attribute. During the fine-tuning process, only a small number of trainable parameters are adjusted, thereby significantly reducing computational and storage costs while ensuring prediction accuracy. Attached Figure Description

[0019] Figure 1 This is a spatial and temporal distribution diagram of multiple spatiotemporal attributes in Embodiment 1 of the present invention.

[0020] Figure 2 This is a diagram of the PromptST framework of the present invention; Figure 3 This is a diagram illustrating the spatiotemporal prompting fine-tuning process of the present invention. Detailed Implementation

[0021] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0022] To address the challenges of spatiotemporal multi-attribute prediction, this invention combines a pre-trained Transformer model with a fine-tuning mechanism. This approach efficiently captures common features in spatiotemporal data and adapts to specific task requirements. By employing this method, complex relationships in spatiotemporal multi-attribute prediction can be resolved, improving the model's computational efficiency and prediction accuracy.

[0023] To achieve the above-mentioned objectives, this invention proposes a spatiotemporal big data processing method based on a pre-training-fine-tuning architecture, the method comprising the following steps: S1: Model Framework Construction. An efficient Transformer-based architecture is proposed to capture general knowledge across multiple spatiotemporal attributes. The Transformer architecture consists of a temporal encoder, a spatial encoder, and a head. To balance model training across multiple attributes, a single spatiotemporal attribute is input to the Transformer in parallel. In this way, the common spatiotemporal features shared by all spatiotemporal attributes can be addressed in a parametrically efficient manner, while adapting to specific attributes.

[0024] S2: Constructing a Spatiotemporal Transformer Network. For spatiotemporal multi-attribute prediction involving complex relationships among numerous attributes, a high-efficiency backbone network based on Transformer is proposed. This method features a compact architecture and significant performance in modeling the spatiotemporal dependencies of multiple attributes.

[0025] S3: Pre-training phase. To address the common features among multiple spatiotemporal attributes, a parameter-sharing training strategy is designed. In this pre-training phase, the spatiotemporal Transformer is trained using all spatiotemporal attributes.

[0026] S4: Hint-based fine-tuning phase. A fast training strategy is needed to minimize training and deployment costs for diverse attributes. A pre-training-hint-based fine-tuning model is proposed to reduce the heavy training costs while achieving better performance with only a minimal number of trainable parameters.

[0027] The specific steps for constructing the model framework described in step S1 are as follows: S1-1: In the pre-training phase, the architecture is trained in a parameter-sharing manner, enabling the model to learn the common spatiotemporal features among all attributes.

[0028] S1-2: To improve the performance of specific spatiotemporal attributes, a cue fine-tuning stage is proposed to adapt to each attribute, so as to benefit from the learned general spatiotemporal variation patterns, which is beneficial for specific attribute modeling.

[0029] S1-3: By fixing the main body of the pre-trained framework, namely the temporal encoder and the spatial encoder, and updating only the cue symbols and the head, PromptST can capture specific spatial and temporal patterns of each individual attribute with tiny trainable parameters.

[0030] The specific steps for constructing the spatiotemporal Transformer network described in step S2 are as follows: S2-1: Time encoder. The time encoder consists of stacked multi-head attention blocks and position-by-position feedforward layers.

[0031] S2-2: Spatial Encoder. After capturing temporal dependencies using a temporal encoder, a simple yet effective spatial encoder is introduced to learn spatial information. Similar to the temporal encoder, spatial location information is added and processed using stacked multi-head attention and feedforward layers.

[0032] The specific steps of the time encoder described in step S2-1 are as follows: S2-1-1: Location Embedding. Based on the formula. Study time information, among which, and These are learnable weights and biases. It is the learned representation. It refers to the embedding size.

[0033] Based on formula Trainable location embeddings are used to label time series. S2-1-2: Multi-head attention. Based on the formula... Calculate self-attention, where, These are the query, key, and value matrices, respectively. Multi-head attention divides the input features into several parts and performs self-attention calculations on each part separately.

[0034] Based on formula , After features are processed in different embedding subspaces, they are concatenated to integrate diverse information, among which... , , , It is a learnable parameter matrix.

[0035] Based on formula Perform multi-head attention operations. S2-1-3: Feedforward layer. Based on the formula... , Perform residual connections and layer normalization, where, , It is a parameter matrix. , It is a bias.

[0036] Based on formula Calculate the first Mid-time after the next iteration; based on the formula conduct The spatial encoder uses a multi-head attention layer and a feedforward layer to capture temporal dependencies. The specific steps of the spatial encoder described in step S2-2 are as follows: S2-2-1: Location Embedding. Based on the formula. Learning positional information between grid cells. S2-2-2: Spatial representation learning. Based on formula. Calculate the first The middle of the space output by the layer iteration; Based on formula Indicates the output space. S2-2-3: Header. Based on the formula. Represents the spatiotemporal properties of the prediction The future state, in which, and These represent the weight matrix and the bias, respectively.

[0037] The specific steps of the pre-training phase described in step S3 are as follows: S3-1: Given a spatiotemporal attribute matrix Simultaneously using a single spatiotemporal attribute matrix (in, To train the spatiotemporal Transformer. For the output... By concatenating the C attributes together, we get .

[0038] S3-2: Based on formula Denotes the optimization objective of pre-training, where, For the model parameters of the time encoder and the spatial encoder, The parameters for the head are used for model output. .

[0039] The specific steps of the fine-tuning phase described in step S4 are as follows: S4-1: Spatiotemporal Cue. A novel spatiotemporal cue token is proposed, which is inserted into the feature sequence of each temporal encoder layer to trigger [something]. The general spatiotemporal change patterns learned in the course.

[0040] Will A learnable spacetime cue token Input time representation integrated into each layer In the time dimension. Therefore, the time representation dimension of the connection is... After multi-head attention and feedforward layer processing, the added... To maintain the original sequence length, the truncated sequence is... Same as before adding the token. Based on the formula. Indicates the first The time interval after the next iteration.

[0041] S4-2: Hint for fine-tuning. Loading ,freeze and initialize each spatiotemporal attribute. and notification token Based on the formula This represents the optimization process, where the model output... .

[0042] This embodiment provides an effective spatiotemporal multi-attribute prediction solution—PromptST. A spatiotemporal Transformer and a parameter-sharing training scheme are designed to handle common sense between different spatiotemporal attributes. A spatiotemporal cue fine-tuning strategy is then detailed to adapt to specific attributes in a lightweight manner. Through pre-training and cue fine-tuning stages, PromptST can adapt to specific target attributes by cueing the base model while maintaining learned common sense, thereby enhancing its ability to capture specific spatiotemporal features. Extensive experiments on real-world datasets validate that PromptST achieves state-of-the-art performance. Furthermore, it demonstrates good transferability to unseen spatiotemporal attributes, bringing promising application potential to urban computing. The following is a detailed description of the specific embodiment: Example 1: See attached document Figure 1 PromptST, a pre-training and hint-based fine-tuning method, is built upon the following background: For example, the 3112 area complaint records in New York City are attached. Figure 1 Images (a), (b), and (c) respectively display complaints regarding "general buildings," "illegal parking," and "noise." (Attached) Figure 1 (a) and (b) exhibit similar spatial patterns due to the prevalence of building and parking issues in cities. In contrast, (c) shows a prominent tall volume within grid (4, 3), which can be attributed to the high pedestrian flow in the city center. Furthermore, the temporal trends of different properties are similar or differ. (Appendix) Figure 1(d) shows the changes in the three complaint attributes over three days. All three types of complaints occurred in the morning, but noise complaints were more frequent in the evening, while the other two peaked at midday. These multiple spatiotemporal attributes are interconnected, and analyzing and utilizing their correlations is crucial for spatiotemporal multi-attribute prediction and unified smart city management.

[0043] However, existing spatiotemporal data mining efforts have failed to adequately handle spatiotemporal multi-attribute prediction. Spatiotemporal prediction methods typically focus on high performance for a single attribute, which requires integrating inductive biases based on expert knowledge and designing highly task-specific architectures. Most existing models neglect the relationships between multiple spatiotemporal attributes, thus failing to address the multi-attribute prediction problem.

[0044] On the other hand, spatiotemporal multi-task learning methods focus on solving two specific tasks with similar spatiotemporal variation patterns, but due to their highly specific architecture, they are difficult to extend to numerous tasks. To solve the problem of spatiotemporal multi-attribute prediction, three challenges need to be overcome: Capturing common features among multiple attributes requires high-capacity and effective spatiotemporal modeling.

[0045] The model should be able to handle specific attributes well while preserving shared knowledge.

[0046] The model should have higher time and space efficiency to facilitate widespread deployment.

[0047] Large pre-trained models offer a solution by leveraging shared knowledge learned from large datasets to improve the performance of target attributes. However, this pre-training and fine-tuning approach suffers from significant retraining costs and catastrophic forgetting. To address this issue, we propose a pre-training and prompting fine-tuning method—PromptST—to effectively solve the spatiotemporal multi-attribute prediction problem.

[0048] Example 2: A pre-training and hint-based fine-tuning method—PromptST—has the following specific steps: S1: Construct the model framework; First, a Transformer-based architecture is constructed to capture general knowledge of multiple spatiotemporal properties. This architecture includes the following core modules: Time encoder: Used to capture time dependencies in spatiotemporal data.

[0049] Spatial encoder: Used to capture spatial dependencies in spatiotemporal data.

[0050] Head network: Used to map spatiotemporal features and finally output prediction results.

[0051] The temporal encoder models the temporal dependencies of spatiotemporal data using stacked multi-head attention layers and a feedforward neural network. Its task is to capture latent patterns in time-series data, such as seasonal variations in traffic flow and peak traffic congestion periods. (See appendix...) Figure 2 (b) illustrates the specific structure of the time encoder: it first maps the input time series data to a new representation space through position embedding, then learns the temporal dependencies through a multi-head attention mechanism, and finally uses a feedforward neural network to process the temporal dependencies.

[0052] The spatial encoder is responsible for capturing spatial dependencies in spatiotemporal data. It primarily models spatial data by learning the location information of spatial grids and the relationships between them. The spatial encoder and temporal encoder share the same spatiotemporal Transformer framework, handling spatial dependencies through a stacked multi-head attention mechanism and feedforward layers. (See appendix...) Figure 2 (c) illustrates the structure of a spatial encoder: it first encodes the input spatial data through position embedding, then applies a multi-head attention mechanism to capture spatial dependencies between different grids, and finally processes the spatial features further through a feedforward neural network.

[0053] The role of the head network is to map the spatiotemporal features extracted by the spatiotemporal encoder to the output space of the target task. In spatiotemporal multi-attribute prediction tasks, the head network maps spatiotemporal features to future spatiotemporal states (e.g., predicting the probability of traffic flow or crime in a certain area in the future) through an MLP structure.

[0054] S2: Construct a spatiotemporal Transformer network; S2-1: Preparation of Input Data. The input data is a spatiotemporal dataset containing multiple spatiotemporal attributes, such as traffic flow, vehicle speed, and weather data. The input data for each spatiotemporal attribute includes time series and spatial grid information, such as the time series of traffic flow data and its corresponding spatial grid.

[0055] S2-2: Parallel Input Data. Data for all spatiotemporal attributes is input into the spatiotemporal Transformer model in parallel. Data for each spatiotemporal attribute is processed separately by a temporal encoder and a spatial encoder. In this stage, the temporal encoder and spatial encoder of the spatiotemporal Transformer capture the temporal and spatial dependencies of each spatiotemporal attribute, respectively.

[0056] S2-3: Shared Parameters. To balance model training across multiple spatiotemporal attributes, all spatiotemporal attributes share the same set of model parameters. This allows the spatiotemporal Transformer to learn common features across multiple spatiotemporal attributes, improving the model's versatility. Parameter sharing not only reduces computational overhead but also helps the model maintain consistent learning performance when facing different spatiotemporal attributes.

[0057] S2-4: Multi-head Attention and Feedforward Neural Networks. The temporal encoder learns the temporal dependencies of each spatiotemporal attribute through a multi-head attention mechanism, capturing the correlations between different time steps. The spatial encoder uses a similar method, capturing the dependencies between different spatial grids through a multi-head attention mechanism. After multi-head attention computation at each layer, the output is further processed by a feedforward neural network to enhance the model's feature learning ability.

[0058] S2-5: Spatiotemporal Feature Fusion. The outputs of the temporal encoder and the spatial encoder are fused together to form a feature representation containing spatiotemporal information. These spatiotemporal features will be used in the subsequent fine-tuning stage to predict future spatiotemporal states.

[0059] S2-6: Pre-training Phase. During the pre-training phase, the Spatiotemporal Transformer is trained using a large-scale spatiotemporal dataset to learn the common features of the spatiotemporal data. The goal of the model is to capture the general patterns of the spatiotemporal data without excessive task-specific fine-tuning.

[0060] S3: Indicates the fine-tuning stage; After the pre-training phase, the process moves to the cue fine-tuning phase, where spatiotemporal cue tokens are inserted into the input sequence to adapt it to the specific spatiotemporal attribute prediction task. The goal of the fine-tuning process is to improve the prediction performance for a particular task by adjusting a small number of parameters (such as the cue tokens and the parameters of the head network).

[0061] Specifically, to enable the model to adjust its output based on different spatiotemporal attributes, spatiotemporal cue tokens are inserted into the input sequence of each temporal encoder layer. These tokens are trainable, allowing the model to better adapt to the target task. The cue tokens have the same shape as the input data, and each token provides additional contextual information for each spatiotemporal attribute, guiding the model to focus on spatiotemporal features relevant to the target task.

[0062] Specifically, during the fine-tuning process, the main body of the spatiotemporal Transformer (time encoder and spatial encoder) remains frozen, as shown in the attached diagram. Figure 3Only the spatiotemporal cue token and the head network are fine-tuned. This avoids catastrophic forgetting while efficiently adjusting the model's spatiotemporal features. The goal of fine-tuning is to minimize the prediction error of the target task by adjusting the parameters of the cue token and the head network to improve the prediction accuracy of the target spatiotemporal attributes.

[0063] S4: Performance Evaluation and Application; After pre-training and fine-tuning, the model will be evaluated using various real-world datasets (such as traffic flow prediction and urban safety incident prediction) to verify its spatiotemporal prediction capabilities. During the evaluation process, the model's prediction accuracy, computational efficiency, and transferability will be the main evaluation metrics.

[0064] The pre-training and prompting fine-tuning method provided in this invention—the PromptST method—effectively solves the problem of spatiotemporal multi-attribute prediction by using a Transformer-based spatiotemporal encoder combined with a prompting fine-tuning strategy. The model can efficiently learn the common features among multiple spatiotemporal attributes and adapt to specific tasks with minimal fine-tuning, thus demonstrating strong application potential in spatiotemporal data processing and prediction for smart cities.

Claims

1. A spatiotemporal big data processing method based on a pre-training-fine-tuning architecture, characterized in that, Includes the following steps: Step 1: Build the model framework; It employs a Transformer-based architecture to capture knowledge of multiple spatiotemporal attributes; the Transformer architecture consists of a temporal encoder, a spatial encoder, and a header; Step 2: Construct a spatiotemporal Transformer network; Step 3: Pre-training phase; During the pre-training phase, the spatiotemporal Transformer network is trained using all spatiotemporal attributes; Step 4: Fine-tuning phase.

2. The spatiotemporal big data processing method based on a pre-training-fine-tuning architecture according to claim 1, characterized in that, Step 1 specifically involves: Step 1-1: In the pre-training phase, train the architecture in a parameter-sharing manner to enable the model to learn the common spatiotemporal features among all attributes; Steps 1-2: Propose a fine-tuning phase to adapt to each attribute; Steps 1-3: By fixing the main body of the pre-trained framework, namely the temporal encoder and the spatial encoder, and updating only the cue symbols and the head, the spatiotemporal multi-attribute prediction scheme PromptST captures specific spatial and temporal patterns of each individual attribute through trainable parameters.

3. The spatiotemporal big data processing method based on a pre-training-fine-tuning architecture according to claim 2, characterized in that, Step 2 specifically involves: Step 2-1: Time encoder; The time encoder consists of stacked multi-head attention blocks and a position-by-position feedforward layer; Step 2-2: Spatial encoder; After capturing temporal dependencies using a time encoder, a spatial encoder is introduced to learn spatial information. Add spatial location information and process it using stacked multi-head attention and feedforward layers.

4. The spatiotemporal big data processing method based on a pre-training-fine-tuning architecture according to claim 3, characterized in that, Step 2-1 specifically involves: Step 2-1-1: Location embedding; Based on formula Study time information, among which, and These are learnable weights and biases. It is the learned representation. It is the embedding size; This represents the transpose function. Represents the sigmoid function; Based on formula Embed trainable locations into labeled time series. This represents the trainable position embedding vector corresponding to the time series; Step 2-1-2: Multi-head attention; Based on formula Calculate self-attention, where, These are the query, key, and value matrices, respectively; multi-head attention divides the input features into several parts and performs self-attention calculation on each part separately; Represents the Attention function. Represents the normalization function; Based on formula , After features are processed in different embedding subspaces, they are concatenated to integrate diverse information, among which... , , , It is a learnable parameter matrix; Indicates the output matrix. This indicates the output of the attention head. This represents the collection of output results; , They represent Matrix feature dimensions; Indicates the number of attention heads; Based on formula Perform multi-head attention operations; This represents the overall computation function. This indicates that the representation layer is normalized. Indicates input data; Step 2-1-3: Feedforward layer; Based on formula , Perform residual connections and layer normalization, where, , It is a parameter matrix. , It is a bias; Indicates the output function. This represents the activation function. Indicates the output of the feedforward sublayer; Based on formula Calculate the first Mid-time after the next iteration; based on the formula conduct Multi-head attention in the layer-time encoder layer and capture of temporal dependencies in the feedforward layer; , They represent the first Layer, First The output of the layer time encoder, This represents the final output of the time encoder. This represents the sum of the outputs of each time encoder layer.

5. The spatiotemporal big data processing method based on a pre-training-fine-tuning architecture according to claim 4, characterized in that, Step S2-2 specifically involves: Step 2-2-1: Location embedding; Based on formula Learn the positional information between grids; Indicates spatial location embedding, This represents the initial input to the spatial encoder; Step 2-2-2: Spatial representation learning; Based on formula Calculate the first The space of layer iteration output; , They represent the first Layer, First The output of the layer space encoder; Based on formula The space representing the output. This represents the final output of the spatial encoder; Indicates the total number of layers in the spatial encoder; Step 2-2-3: Head; Based on formula Represents the spatiotemporal properties of the prediction The future state, in which, and These represent the weight matrix and the bias, respectively. This indicates the prediction result.

6. The spatiotemporal big data processing method based on a pre-training-fine-tuning architecture according to claim 5, characterized in that, Step 3 specifically involves: Step 3-1: Given the spatiotemporal attribute matrix Simultaneously using a single spatiotemporal attribute matrix For output By concatenating the C attributes together, we get ; Step 3-2: Based on the formula Denotes the optimization objective of pre-training, where, For the model parameters of the time encoder and the spatial encoder, The parameters for the head are used for model output. ; This represents the root mean square error function. This represents the mean absolute error function. This represents the overall loss function.

7. The spatiotemporal big data processing method based on a pre-training-fine-tuning architecture according to claim 6, characterized in that, Step 4 specifically involves: Step 4-1: Spacetime Hints; A spatiotemporal cue token is used and inserted into the feature sequence of each temporal encoder layer to trigger [the event / event]. The general spatiotemporal change patterns learned in the course; Will A learnable spacetime cue token Input time representation integrated into each layer In the time dimension; therefore, the time representation dimension of the connection is... After multi-head attention and feedforward processing, the added data is truncated. To maintain the original sequence length, the truncated sequence is... Same as before the token was added; Based on the formula: Indicates the first Midway through the next iteration; Step 4-2: Prompt for fine-tuning; load ,freeze and initialize each spatiotemporal attribute. and notification token ; Based on formula This represents the optimization process, where the model output... .

8. An electronic device, characterized in that, include: Processor and memory; The memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory to cause the electronic device to perform the method as described in any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.

10. A chip, characterized in that, include: A processor for retrieving and running a computer program from memory, causing a device on which the chip is mounted to perform the method as described in any one of claims 1 to 7.

Images