Space-time prediction method and device of space-time general model based on prompt fine tuning

Abstract

The invention relates to the technical field of space-time prediction, in particular to a space-time prediction method and device of a space-time general model based on prompt fine tuning, and the method employs the space-time general model based on prompt fine tuning to process a space-time prediction task, and the processing comprises the steps: obtaining the space-time prediction task for a target scene, determining spatio-temporal data according to the spatio-temporal prediction task, processing the spatio-temporal data to obtain a plurality of spatio-temporal tensors, determining a spatial memory pool and a time memory pool based on the plurality of spatio-temporal tensors, and performing feature extraction on the plurality of spatio-temporal tensors to obtain a plurality of spatio-temporal features; the spatial memory pool and the time memory pool are used for processing the multiple spatial-temporal characteristics to obtain prompt information, and a prediction result is determined according to the target mask and the prompt information. According to the spatio-temporal prediction method and device provided by the embodiment of the invention, spatio-temporal prediction tasks of various scenes in an urban environment can be realized.

Description

Technical Field

[0001] The present disclosure relates to the field of spatiotemporal prediction technology, and in particular to a spatiotemporal prediction method and device based on a prompt-fine-tuned spatiotemporal universal model. Background Art

[0002] Currently, pre-trained basic models have achieved remarkable success in the fields of natural language processing (NLP) and computer vision (CV), especially in few-shot and zero-shot scenarios. However, similar breakthroughs have not yet been achieved in the field of urban spatiotemporal modeling.

[0003] With the continuous development of deep learning technology, related technologies have proposed using deep learning to perform urban spatiotemporal forecasting. However, these approaches still have many limitations. The effectiveness of spatiotemporal models is usually limited to a single city or a specific dataset, which is highly restrictive. Therefore, current solutions are still far from being universal models. Summary of the Invention

[0004] In view of this, the present disclosure proposes a spatiotemporal prediction method and device based on a general spatiotemporal model with prompt fine-tuning, which can make predictions based on multi-level spatiotemporal features and realize spatiotemporal prediction tasks for various scenarios in urban environments.

[0005] According to one aspect of the present disclosure, a spatiotemporal prediction method based on a spatiotemporal general model with prompt fine-tuning is provided, wherein the method processes a spatiotemporal prediction task using a spatiotemporal general model with prompt fine-tuning, and the processing comprises the following steps: obtaining a spatiotemporal prediction task for a target scene, determining spatiotemporal data according to the spatiotemporal prediction task, wherein the spatiotemporal data comprises data of multiple dimensions, wherein the multiple dimensions comprise a spatial dimension, a temporal dimension, and a target dimension, wherein the target dimension is determined according to the target scene, and the target scene is any one of a transportation scene, a crowd flow scene, and a life service scene in an urban environment; processing the spatiotemporal data to obtain multiple spatiotemporal tensors, and determining a spatial memory pool and a temporal memory pool based on the multiple spatiotemporal tensors, wherein the spatial memory pool and the temporal memory pool are determined based on the spatial memory pool. The memory pool includes multiple spatial key-value pairs, and the temporal memory pool includes multiple temporal key-value pairs; feature extraction is performed on the multiple spatiotemporal tensors to obtain spatial proximity features representing spatial proximity attributes, spatial hierarchical features representing spatial hierarchical attributes, temporal proximity features representing temporal proximity attributes, and temporal period features representing temporal period attributes; the spatial proximity features, the spatial hierarchical features, the temporal proximity features, and the temporal period features are processed using the spatial memory pool and the temporal memory pool to obtain multiple prompt information; a target mask is determined based on a preset mask rule and the multiple spatiotemporal tensors, and a prediction result corresponding to the spatiotemporal prediction task is determined according to the target mask and the multiple prompt information, wherein the mask rule is determined based on multiple mask strategies.

[0006] In one possible implementation, the spatiotemporal data is processed to obtain a plurality of spatiotemporal tensors, including: smoothing the spatiotemporal data to obtain smoothed data; segmenting the smoothed data based on the multiple dimensions to obtain a plurality of three-dimensional tensors; and converting the multiple three-dimensional tensors of a first shape into a plurality of spatiotemporal tensors of a second shape based on a preset convolutional layer, wherein the size of the convolution kernel of the convolutional layer is the same as the size of the spatiotemporal tensor, and the first shape is larger than the second shape.

[0007] In one possible implementation, the general spatiotemporal model includes an encoder; wherein, determining a spatial memory pool and a temporal memory pool based on the multiple spatiotemporal tensors includes: encoding the multiple spatiotemporal tensors using the encoder to obtain encoding results; determining multiple spatial key-value pairs and multiple temporal key-value pairs based on the encoding results; constructing the spatial memory pool based on the multiple spatial key-value pairs, and constructing the temporal memory pool based on the multiple temporal key-value pairs.

[0008] In one possible implementation, feature extraction is performed on the multiple spatiotemporal tensors to obtain spatial proximity features representing spatial proximity attributes, spatial hierarchical features representing spatial hierarchical attributes, temporal proximity features representing temporal proximity attributes, and temporal period features representing temporal period attributes, including: using a cosine function to position encode data at odd positions in a sequence corresponding to each of the spatiotemporal tensors, and using a sine function to position encode data at even positions in a sequence corresponding to each of the spatiotemporal tensors to obtain the spatial proximity features, the spatial hierarchical features, the temporal proximity features, and the temporal period features.

[0009] In a possible implementation, the number of the spatial key-value pairs is the same as the number of the time key-value pairs, each of the spatial key-value pairs includes a spatial key and a spatial value, and each of the time key-value pairs includes a time key and a time value; wherein, the spatial proximity feature, the spatial hierarchical feature, the time proximity feature, and the time period feature are processed using the spatial memory pool and the time memory pool to obtain a plurality of prompt information, including: determining a first weight for the spatial proximity attribute based on all the spatial keys and the spatial proximity features in the spatial memory pool, and determining spatial proximity prompt information based on the first weight and all the spatial values ​​in the spatial memory pool; determining a spatial proximity prompt information based on all the spatial keys and the spatial hierarchical features in the spatial memory pool. For the second weight of the spatial layered attribute, the spatial layered prompt information is determined based on the second weight and all the spatial values ​​in the spatial memory pool; the third weight for the temporal proximity attribute is determined based on all the time keys in the temporal memory pool and the temporal proximity feature, and the temporal proximity prompt information is determined based on the third weight and all the time values ​​in the temporal memory pool; the fourth weight for the temporal period attribute is determined based on all the time keys in the temporal memory pool and the temporal period feature, and the temporal period prompt information is determined based on the fourth weight and all the time values ​​in the temporal memory pool; the spatial proximity prompt information, the spatial layered prompt information, the temporal proximity prompt information, and the temporal period prompt information are used as the multiple prompt information.

[0010] In one possible implementation, the method also includes a training process for the spatiotemporal universal model, which training process includes: obtaining sample data under multiple sample scenarios, the multiple sample scenarios including the transportation scenario, the crowd flow scenario, and the life service scenario; constructing a training sample set based on the sample data, and performing training based on the training sample set to obtain a trained spatiotemporal universal model.

[0011] In one possible implementation, the multiple masking strategies include a random masking strategy for capturing fine-grained spatiotemporal relationships, a pipeline masking strategy for improving the spatial extrapolation capability of the spatiotemporal general model, a block masking strategy for enhancing spatial transitivity, and a temporal masking strategy for improving the ability of the spatiotemporal general model to capture temporal dependencies.

[0012] According to another aspect of the present disclosure, a spatiotemporal prediction device based on a spatiotemporal general model with prompt fine-tuning is provided, comprising: an acquisition module for acquiring a spatiotemporal prediction task for a target scene, and determining spatiotemporal data according to the spatiotemporal prediction task, wherein the spatiotemporal data includes data of multiple dimensions, wherein the multiple dimensions include a spatial dimension, a temporal dimension, and a target dimension, and the target dimension is determined according to the target scene, and the target scene is any one of a transportation scene, a crowd flow scene, and a life service scene in an urban environment; a determination module for processing the spatiotemporal data to obtain multiple spatiotemporal tensors, and determining a spatial memory pool and a temporal memory pool based on the multiple spatiotemporal tensors, wherein the spatial memory pool includes multiple spatial key-value pairs, and the temporal memory pool includes multiple time key-value pairs; a feature extraction module for extracting features from the multiple spatiotemporal tensors to obtain spatial proximity features representing spatial proximity attributes, spatial hierarchical features representing spatial hierarchical attributes, temporal proximity features representing temporal proximity attributes, and temporal period features representing temporal period attributes; a processing module for processing the spatial proximity features, the spatial hierarchical features, the temporal proximity features, and the temporal period features using the spatial memory pool and the temporal memory pool to obtain multiple prompt information; a prediction module for determining a target mask based on a preset mask rule and the multiple spatiotemporal tensors, and determining a prediction result corresponding to the spatiotemporal prediction task according to the target mask and the multiple prompt information, wherein the mask rule is determined based on multiple mask strategies.

[0013] In one possible implementation, the spatiotemporal data is processed to obtain a plurality of spatiotemporal tensors, including: smoothing the spatiotemporal data to obtain smoothed data; segmenting the smoothed data based on the multiple dimensions to obtain a plurality of three-dimensional tensors; and converting the multiple three-dimensional tensors of a first shape into a plurality of spatiotemporal tensors of a second shape based on a preset convolutional layer, wherein the size of the convolution kernel of the convolutional layer is the same as the size of the spatiotemporal tensor, and the first shape is larger than the second shape.

[0014] In one possible implementation, the general spatiotemporal model includes an encoder; wherein, determining a spatial memory pool and a temporal memory pool based on the multiple spatiotemporal tensors includes: encoding the multiple spatiotemporal tensors using the encoder to obtain encoding results; determining multiple spatial key-value pairs and multiple temporal key-value pairs based on the encoding results; constructing the spatial memory pool based on the multiple spatial key-value pairs, and constructing the temporal memory pool based on the multiple temporal key-value pairs.

[0015] In one possible implementation, feature extraction is performed on the multiple spatiotemporal tensors to obtain spatial proximity features representing spatial proximity attributes, spatial hierarchical features representing spatial hierarchical attributes, temporal proximity features representing temporal proximity attributes, and temporal period features representing temporal period attributes, including: using a cosine function to position encode data at odd positions in a sequence corresponding to each of the spatiotemporal tensors, and using a sine function to position encode data at even positions in a sequence corresponding to each of the spatiotemporal tensors to obtain the spatial proximity features, the spatial hierarchical features, the temporal proximity features, and the temporal period features.

[0016] In a possible implementation, the number of the spatial key-value pairs is the same as the number of the time key-value pairs, each of the spatial key-value pairs includes a spatial key and a spatial value, and each of the time key-value pairs includes a time key and a time value; wherein, the spatial proximity feature, the spatial hierarchical feature, the time proximity feature, and the time period feature are processed using the spatial memory pool and the time memory pool to obtain a plurality of prompt information, including: determining a first weight for the spatial proximity attribute based on all the spatial keys and the spatial proximity features in the spatial memory pool, and determining spatial proximity prompt information based on the first weight and all the spatial values ​​in the spatial memory pool; determining a spatial proximity prompt information based on all the spatial keys and the spatial hierarchical features in the spatial memory pool. For the second weight of the spatial layered attribute, the spatial layered prompt information is determined based on the second weight and all the spatial values ​​in the spatial memory pool; the third weight for the temporal proximity attribute is determined based on all the time keys in the temporal memory pool and the temporal proximity feature, and the temporal proximity prompt information is determined based on the third weight and all the time values ​​in the temporal memory pool; the fourth weight for the temporal period attribute is determined based on all the time keys in the temporal memory pool and the temporal period feature, and the temporal period prompt information is determined based on the fourth weight and all the time values ​​in the temporal memory pool; the spatial proximity prompt information, the spatial layered prompt information, the temporal proximity prompt information, and the temporal period prompt information are used as the multiple prompt information.

[0017] In one possible implementation, the device also includes a training module, which is used to perform a training process for the spatiotemporal universal model. The training process includes: obtaining sample data under multiple sample scenarios, the multiple sample scenarios including the transportation scenario, the crowd flow scenario, and the life service scenario; constructing a training sample set based on the sample data, and performing training based on the training sample set to obtain a trained spatiotemporal universal model.

[0018] In one possible implementation, the multiple masking strategies include a random masking strategy for capturing fine-grained spatiotemporal relationships, a pipeline masking strategy for improving the spatial extrapolation capability of the spatiotemporal general model, a block masking strategy for enhancing spatial transitivity, and a temporal masking strategy for improving the ability of the spatiotemporal general model to capture temporal dependencies.

[0019] According to another aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0020] According to another aspect of the present disclosure, a non-volatile computer-readable storage medium is provided, on which computer program instructions are stored, wherein the computer program instructions implement the above method when executed by a processor.

[0021] According to another aspect of the present disclosure, a computer program product is provided, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0022] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the disclosure and, together with the description, serve to explain the principles of the disclosure.

[0024] Figures 1 to 3 A schematic diagram of a spatiotemporal prediction method based on a spatiotemporal general model with prompt fine-tuning according to an embodiment of the present disclosure is shown.

[0025] Figures 4 and 5 A block diagram of a spatiotemporal prediction apparatus based on a spatiotemporal general model fine-tuned by hints according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0026] Various exemplary embodiments, features, and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise indicated.

[0027] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0028] In addition, numerous specific details are provided in the following detailed description to better illustrate the present disclosure. Those skilled in the art will appreciate that the present disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art are not described in detail in order to highlight the main points of the present disclosure.

[0029] In order to facilitate those skilled in the art to understand the technical solution provided by the embodiments of the present disclosure, the technical environment in which the technical solution is implemented is described below.

[0030] Currently, pre-trained basic models have achieved remarkable success in the fields of natural language processing (NLP) and computer vision (CV), especially in few-shot and zero-shot scenarios. However, similar breakthroughs have not yet been achieved in the field of urban spatiotemporal modeling. Although existing spatiotemporal modeling methods have made remarkable progress, their effectiveness is usually limited to one city, that is, one specific dataset, and they are highly limited. In other words, existing methods require a specific dataset for each specific city or different application scenarios in an urban environment to train a model capable of handling the corresponding spatiotemporal prediction task. The reliance on large amounts of training data further hinders the model's generalization potential. Therefore, current solutions are still a long way from being a "universal model."

[0031] The importance of a universal model lies in its ability to address the data scarcity problem prevalent in urban areas, namely the imbalance in data collection across different cities and sectors. A universal spatiotemporal model should possess the following two capabilities. First, it must be able to leverage the rich data in urban environments for training, particularly data with diverse spatiotemporal characteristics. Training of the universal model should ensure the acquisition of sufficient and rich information. Second, it should demonstrate strong generalization across a wide range of spatiotemporal scenarios. In particular, when training data is limited or missing, the model should still be able to make accurate predictions without significant performance degradation. This capability comes not only from diverse data training but also from shared and transferable knowledge across various spatiotemporal scenarios.

[0032] However, it is not easy for spatiotemporal models to possess the above two capabilities. Spatiotemporal data presents several unique challenges that make it impossible to directly use general models in the current natural language or vision fields. The first challenge comes from the diverse data formats of spatiotemporal datasets. Unlike languages ​​with a unified one-dimensional structure or images and videos that follow a standard format, spatiotemporal data from different sources exhibit completely different characteristics, and the dimensions, temporal duration, and spatial range of variables may vary greatly. The second challenge comes from the significant differences in data distributions of different spatiotemporal datasets. Faced with the different levels of spatiotemporal patterns brought about by different application scenarios, it may be difficult for the model to adapt to and fit these differences simultaneously.

[0033] Based on this, the embodiment of the present disclosure utilizes deep learning technology to construct a spatiotemporal general model for urban spatiotemporal prediction tasks. The spatiotemporal general model has the processing capability and strong generalization capability to cope with spatiotemporal prediction tasks of various spatiotemporal scenarios in urban environments, thereby solving the problem that the spatiotemporal model can only handle specific urban scenarios. The spatiotemporal general model provided by the embodiment of the present disclosure is formed based on the encoder-decoder architecture of the transformer. In other words, the spatiotemporal general model may include an encoder and a decoder, wherein the specific processing process of the Transformer model can be determined according to the actual structure, which is not limited in this article. The spatiotemporal scenarios cover various spatiotemporal fields in urban environments, such as transportation, cellular networks, and human mobility.

[0034] In order to solve the above technical problems, an embodiment of the present disclosure provides a spatiotemporal prediction method based on a spatiotemporal general model with prompt fine-tuning, which uses the spatiotemporal general model with prompt fine-tuning to process spatiotemporal prediction tasks, and the processing includes obtaining a spatiotemporal prediction task for a target scene, determining spatiotemporal data according to the spatiotemporal prediction task, the spatiotemporal data including data of multiple dimensions, the multiple dimensions including spatial dimension, temporal dimension, and target dimension, the target dimension is determined according to the target scene, and the target scene is any one of a transportation scene, a crowd flow scene, and a life service scene in an urban environment; processing the spatiotemporal data to obtain multiple spatiotemporal tensors, and determining a spatial memory pool and a temporal memory pool based on the multiple spatiotemporal tensors, the spatial memory pool including multiple spatial key-value pairs, and the temporal memory pool including multiple temporal key-value pairs; processing the multiple temporal tensors The spatial tensor is used for feature extraction to obtain spatial proximity features representing spatial proximity attributes, spatial layered features representing spatial layered attributes, temporal proximity features representing temporal proximity attributes, and temporal period features representing temporal period attributes; the spatial proximity features, spatial layered features, temporal proximity features, and temporal period features are processed using the spatial memory pool and the temporal memory pool to obtain multiple prompt information; a target mask is determined based on a preset mask rule and multiple spatiotemporal tensors, and a prediction result of the corresponding spatiotemporal prediction task is determined according to the target mask and the multiple prompt information, wherein the mask rule is determined based on multiple mask strategies. In this way, prediction can be made based on multi-level spatiotemporal features such as spatial proximity features, spatial layered features, temporal proximity features, and temporal period features, thereby realizing spatiotemporal prediction tasks for various scenarios in urban environments.

[0035] Figures 1 to 3 A schematic diagram of a spatiotemporal prediction method based on a spatiotemporal general model with prompt fine-tuning according to an embodiment of the present disclosure is shown. Figures 1 to 3 The spatiotemporal prediction method provided by the embodiment of the present disclosure is schematically illustrated.

[0036] The spatiotemporal prediction method provided by the disclosed embodiments utilizes a general spatiotemporal model based on cue fine-tuning to process spatiotemporal prediction tasks. During this process, the multi-layered spatiotemporal features of the spatiotemporal data corresponding to the target scene are captured from multiple perspectives, thereby generating spatiotemporal cue information closely related to the target scene and determining the spatiotemporal prediction results for the target scene.

[0037] like Figure 1 As shown, the spatiotemporal prediction method provided by the embodiment of the present disclosure may include the following steps S101 to S105.

[0038] Step S101: Acquire a spatiotemporal prediction task for a target scene, and determine spatiotemporal data according to the spatiotemporal prediction task.

[0039] The target scenario is any one of the various scenarios in the urban environment. For example, the target scenario can be any one of the transportation scenarios, crowd flow scenarios, and life service scenarios. Correspondingly, the spatiotemporal prediction task can be a traffic speed prediction task, vehicle flow prediction task, pedestrian flow prediction task, circuit network congestion prediction task, etc. in a certain city.

[0040] Spatiotemporal data has a four-dimensional (4D) structure. In other words, it consists of data in multiple dimensions. These dimensions include two spatial dimensions, a temporal dimension, and a target dimension. The target dimension is determined based on the target scenario. For example, for a crowd flow prediction task, the target dimension might represent inflow and outflow.

[0041] The traditional Transformer architecture is used to process one-dimensional (1D) sequence data. To this end, this spatiotemporal prediction method first preprocesses the spatiotemporal data to obtain 1D sequence data, which is later referred to as the spatiotemporal tensor. Based on this spatiotemporal tensor, the model can then determine multiple levels of spatiotemporal features, namely spatial proximity features, spatial layer features, temporal proximity features, and temporal period features.

[0042] Step S102: Process the spatiotemporal data to obtain multiple spatiotemporal tensors, and determine a spatial memory pool and a temporal memory pool based on the multiple spatiotemporal tensors.

[0043] In one possible implementation, processing the spatiotemporal data in step S102 to obtain a plurality of spatiotemporal tensors may include: smoothing the spatiotemporal data to obtain smoothed data, for example, compressing the original spatiotemporal data into a smoothed popular representation, i.e., smoothed data, through a compression encoder; segmenting the smoothed data based on the above-mentioned spatial dimension, time dimension, and target dimension to obtain a plurality of three-dimensional tensors, for example, segmenting multiple levels of instances based on each dimension of the spatiotemporal data corresponding to the target scene, and these instances are three-dimensional (3D) tensors, which can be represented as And it is in a first shape L×H×W, where L represents the length dimension of the three-dimensional tensor, H represents the height dimension of the three-dimensional tensor, and W represents the width dimension of the three-dimensional tensor, Represents the domain, the three dimensions L, H, and W together define the first shape of the three-dimensional tensor, which can include Figure 2 The spatial proximity tensor X in sc , spatial layered tensor X sh , temporal proximity tensor X tc , time period tensor X tp, so that multi-level spatiotemporal features can be determined based on these three-dimensional tensors in the future; based on the preset convolution layer, multiple three-dimensional tensors of the first shape are converted into multiple spatiotemporal tensors of the second shape, and the size of the convolution kernel of the convolution layer is the same as the size of the spatiotemporal tensor. For example, the converted spatiotemporal tensor is a three-dimensional tensor with a smaller shape, and the spatiotemporal tensor can be expressed as E x And it has a second shape l×h×w, l represents the length dimension of the space-time tensor, h represents the height dimension of the space-time tensor, and w represents the width dimension of the space-time tensor. The three dimensions l, h, and w together define the second shape of the space-time tensor. The first shape is larger than the second shape. E x That is, it is 1D sequence data and its sequence length is L'×H'×W', L'=L / l, H'=H / h, W'=W / w, where the conversion can be implemented using a 3D convolution layer and can be expressed as E x =Conv3d(X), where X is a three-dimensional tensor. The kernel size and stride of this 3D convolutional layer can be set to (l, h, w). The remaining parameters are described above and will not be repeated here. In this way, the spatiotemporal marker can be used to implement the above preprocessing to obtain multiple spatiotemporal tensors, allowing the spatiotemporal universal model to process various spatiotemporal data in a unified sequence format.

[0044] The spatiotemporal general model provided by the embodiment of the present disclosure is formed by using the encoder-decoder architecture of Transformer. In one possible implementation, determining the spatial memory pool and the temporal memory pool based on multiple spatiotemporal tensors in step S102 may include: encoding the multiple spatiotemporal tensors using an encoder to obtain an encoding result. For example, the processing process of the encoder can be expressed as E enc =Encoder(E x ); Based on the encoding result, multiple spatial key-value pairs and multiple time key-value pairs are determined, the number of spatial key-value pairs and time key-value pairs is the same, each spatial key-value pair includes a spatial key and a spatial value, and each time key-value pair includes a time key and a time value. For example, multiple spatial key-value pairs can be represented as KM s ={(k s,0 ,m s,0 ),(k s,1 ,m s,1 ),…,(k s,N-1 ,m s,N-1 )}, where (k s,0 ,m s,0 ) represents the first spatial key-value pair, (k s,1 ,m s,1 ) represents the second space key-value pair, (k s,N-1 ,m s,N-1 ) represents the Nth spatial key-value pair. Taking the Nth spatial key-value pair as an example, k s,N-1Represents the spatial key in the Nth spatial key-value pair, m s,N-1 Represents the spatial value in the Nth spatial key-value pair. The spatial keys and spatial values ​​in the remaining spatial key-value pairs are similar to the Nth spatial key-value pair and will not be repeated here. Similarly, multiple time key-value pairs can be represented as KM t ={(k t,0 ,m t,0 ),(k t,1 ,m t,1 ),…,(k t,N-1 ,m t,N-1 )}, where (k t,0 ,m t,0 ) represents the first time key-value pair, (k t,1 ,m t,1 ) represents the second time key-value pair, (k t,N-1 ,m t,N-1 ) represents the Nth time key-value pair, k t,N-1 Indicates the time key in the Nth time key-value pair, m t,N-1 It represents the time value in the Nth time key-value pair. The time keys and time values ​​in the remaining time key-value pairs are similar to those in the Nth time key-value pair, which will not be repeated here. N represents the number of spatial key-value pairs or time key-value pairs, and N can be set in advance. A spatial memory pool is constructed based on multiple spatial key-value pairs, and a time memory pool is constructed based on multiple time key-value pairs. The specific construction method can be determined according to actual conditions and combined with the actual structure of the model, and the embodiments of the present disclosure do not limit this.

[0045] Step S103: extract features from multiple spatiotemporal tensors to obtain spatial proximity features representing spatial proximity attributes, spatial hierarchical features representing spatial hierarchical attributes, temporal proximity features representing temporal proximity attributes, and temporal period features representing temporal period attributes.

[0046] In one possible implementation, step S103 may include: using a cosine function to positionally encode data at odd positions in the sequence corresponding to each spatiotemporal tensor, and using a sine function to positionally encode data at even positions in the sequence corresponding to each spatiotemporal tensor, to obtain spatial proximity features, spatial hierarchical features, temporal proximity features, and temporal period features. For example, the position encoding process of each spatiotemporal tensor can be implemented in the spatial dimension and the temporal dimension using the following formula (1):

[0047]

[0048] In formula (1), e t Represents the space-time tensor E x The elements in [e t ] j Represents the element e tThe position encoding result, j represents the element e in the sequence corresponding to the spatiotemporal tensor t where cos represents the cosine function, sin represents the sine function, t represents the time corresponding to the space-time tensor, and M represents the maximum sequence length, which can be considered a hyperparameter and is generally preset to 5000. In this way, by using sine and cosine functions to positionally encode multiple space-time tensors, the generalization ability of the space-time general model can be enhanced.

[0049] By performing the above-mentioned preprocessing and position encoding on the spatiotemporal data, the feature extraction process of the spatiotemporal data can be realized. Figure 2 The corresponding spatial neighbor tensor X is shown sc The spatial proximity feature E sc , corresponding to the spatial layered tensor X sh The spatial hierarchical features E sh , corresponding to the time proximity tensor X tc The temporal proximity feature E tc , corresponding to the time period tensor X tp Time period characteristics E tp These multiple levels of spatiotemporal features. Among them, spatial proximity features can represent the characteristics of spatial proximity attributes in spatiotemporal data, spatial layering features can represent the characteristics of spatial layering attributes in spatiotemporal data, temporal proximity features can represent the characteristics of temporal proximity attributes in spatiotemporal data, and temporal period features can represent the characteristics of temporal period attributes in spatiotemporal data. Based on these spatiotemporal features, we can obtain interesting information in the spatiotemporal data corresponding to the target scene.

[0050] Although the spatiotemporal patterns of different spatiotemporal scenarios, i.e., different application scenarios in urban environments, vary greatly, they still share some basic principles. For example, the location of commercial centers in different cities may be different, but the way they are related to other functional areas may be a shared feature. Similarly, although the time period patterns vary in different scenarios, they share the basic concept of "repeating patterns." Therefore, in the process of processing the spatiotemporal prediction task using the spatiotemporal general model, this spatiotemporal prediction method performs multi-level spatiotemporal feature extraction on the spatiotemporal data of the target scene. It can align these spatiotemporal features that share a shared association and have a potential relationship, so that subsequent steps can use these spatiotemporal features to generate prompt information for fine-tuning, thereby generating more accurate spatiotemporal prediction results.

[0051] In order to further improve the capabilities of the spatiotemporal general model, the present spatiotemporal prediction method designs spatiotemporal knowledge-guided prompt fine-tuning in the process of the spatiotemporal general model processing spatiotemporal prediction tasks. Since some spatiotemporal patterns have high-dimensional and complex characteristics, it is unrealistic to train fixed prompts for each situation. For example, for the dynamic position data of groups concerned in the task of predicting pedestrian flow in urban environments, since the position data represents the aggregated result of human dynamic movement trajectory data, it is inappropriate to set fixed prompts for the dynamic change process to perform spatiotemporal prediction tasks. In order to solve this problem, the present spatiotemporal prediction method designs a prompt storage mechanism based on a memory network, namely a spatial memory pool and a temporal memory pool. The spatial memory pool includes multiple spatial key-value pairs, and the temporal memory pool includes multiple temporal key-value pairs. The method for determining the spatial memory pool and the temporal memory pool is detailed in step S102 and will not be repeated here. In the prompt fine-tuning process provided by the present spatiotemporal prediction method, such as Figure 3 As shown, the spatial memory pool and the temporal memory pool can be learned and optimized based on multi-level spatiotemporal features to store valuable information about spatiotemporal domain knowledge, thereby generating prompt information that helps to accurately achieve spatiotemporal prediction tasks.

[0052] Step S104: using the spatial memory pool and the temporal memory pool to process the spatial proximity feature, the spatial layering feature, the temporal proximity feature, and the temporal period feature to obtain a plurality of prompt information.

[0053] The plurality of prompt information may include spatial proximity prompt information, spatial layered prompt information, temporal proximity prompt information, and temporal period prompt information. In a possible implementation, step S104 may include: determining a first weight for the spatial proximity attribute based on all spatial keys and spatial proximity features in the spatial memory pool, and determining the spatial proximity prompt information based on the first weight and all spatial values ​​in the spatial memory pool; determining a second weight for the spatial layered attribute based on all spatial keys and spatial layered features in the spatial memory pool, and determining the spatial layered prompt information based on the second weight and all spatial values ​​in the spatial memory pool; determining a third weight for the temporal proximity attribute based on all time keys and temporal proximity features in the temporal memory pool, and determining the temporal proximity prompt information based on the third weight and all time values ​​in the temporal memory pool; determining a fourth weight for the temporal period attribute based on all time keys and temporal period features in the temporal memory pool, and determining the temporal period prompt information based on the fourth weight and all time values ​​in the temporal memory pool; using the spatial proximity prompt information, spatial layered prompt information, temporal proximity prompt information, and temporal period prompt information as the plurality of prompt information.

[0054] For example, multiple prompt information can be obtained through the following formula (2):

[0055]

[0056] In formula (2), [k s,0 ;k s,1 ;…;k s,N-1 ] represents N spatial keys in N spatial key-value pairs, m s,i From the above multiple space key-value pairs, m s,i Represents the i-th spatial value, i∈N, E sc represents the spatial proximity feature, α sc Represents the first weight, which includes the weight corresponding to each space key, α sc,i represents the weight corresponding to the i-th spatial key in the first weight, P sc Indicates spatial proximity prompt information, E sh represents the spatial hierarchical features, α sh Represents the second weight, which includes the weight corresponding to each space key, α sh,i represents the weight corresponding to the i-th spatial key in the second weight, P sh Indicates spatial hierarchical prompt information, [k t,0 ;k t,1 ;…;k t,N-1 ] represents the N time keys in the N time key-value pairs, m t,i From the above multiple time key-value pairs, m t,i represents the i-th time value, E tc represents the temporal proximity feature, α tc Represents the third weight, which includes the weight corresponding to each time key, α tc,i Indicates the weight corresponding to the i-th time key in the third weight, P tc Indicates time proximity prompt information, E tp represents the time period characteristics, α tp Represents the fourth weight, which includes the weight corresponding to each time key, α tp,i Indicates the weight corresponding to the i-th time key in the fourth weight, P tp Represents the time period prompt information, and T represents transposition. These prompt information are integrated into the input space of the Transformer architecture, such as Figure 2 As shown in , these prompt information and spatiotemporal tensors can be used together as input to the attention block in Transformer for attention processing, where the attention block can be determined based on the multi-head attention mechanism, normalization layer, and feedforward network, but Figure 2 This is just an example. The specific structure can be flexibly set according to actual needs. In addition, the fixed or fine-tuned settings of the parameters of each module / network in the spatiotemporal general model can be found in Figure 2 .

[0057] That is to say, in the process of generating useful prompt information by this spatiotemporal prediction method, valuable information can be extracted from the above-mentioned memory pool based on the multi-level spatiotemporal characteristics, that is, the representation of spatiotemporal attributes can be used as the query basis, that is, the relevant embedded representation vectors such as spatial proximity prompt information, spatial layered prompt information, temporal proximity prompt information, and temporal period prompt information can be obtained. This enables the general spatiotemporal model to adaptively select the most useful information for prediction.

[0058] Step S105 : determining a target mask based on a preset mask rule and a plurality of spatiotemporal tensors, and determining a prediction result corresponding to the spatiotemporal prediction task according to the target mask and the plurality of prompt information.

[0059] The masking rules are determined based on multiple masking strategies. In one possible implementation, these strategies include a random masking strategy for capturing fine-grained spatiotemporal relationships, a pipeline masking strategy for improving the spatial extrapolation capability of the spatiotemporal general model, a block masking strategy for enhancing spatial transitivity, and a temporal masking strategy for improving the spatiotemporal general model's ability to capture temporal dependencies. These masking strategies can be applied in the decoder of the spatiotemporal general model.

[0060] The random masking strategy is similar to the method used in the masked autoencoder (MAE), which randomly masks the input spatiotemporal data. Its purpose is to capture fine-grained spatiotemporal relationships. In practice, the random masking strategy randomly selects a certain proportion of data for masking. The spatiotemporal general model needs to predict the masked value based on its surrounding context information, thereby learning more complex spatiotemporal relationships. For example, the random masking strategy can be expressed by the following formula (3):

[0061]

[0062] In formula (3), M represents the mask determined by the random masking strategy, M~U[0,1] means that M obeys the uniform distribution on [0,1], r represents the mask ratio, and the remaining parameters are detailed in the previous article and will not be repeated here.

[0063] The pipeline masking strategy simulates the situation where some spatial units are completely missing data in all time dimensions, which reflects the situation in the real world where some sensors that monitor the dynamic position of groups may not work properly. The purpose of the pipeline masking strategy is to improve the spatial extrapolation capability. In practical applications, the pipeline masking strategy randomly selects a certain proportion of spatial units and completely masks the data of all time periods. The spatiotemporal universal model needs to predict the value of the masked spatial unit based on the information of other spatial units, thereby improving the spatiotemporal universal model's ability to understand and infer spatial information. For example, the pipeline masking strategy can be expressed by the following formula (4):

[0064]

[0065] In formula (4), M represents the mask determined by the pipeline mask strategy. The remaining parameters are detailed in the previous text and will not be repeated here.

[0066] The block masking strategy involves the complete omission of the entire spatial unit block in all time periods. Due to the limited context information, the reconstruction task becomes more complicated. The goal of the block masking strategy is to enhance spatial transitivity. In practice, the block masking strategy randomly selects a certain proportion of spatial blocks and completely masks their data in all time periods. The spatiotemporal universal model needs to predict the value of the masked spatial block based on the information of other spatial blocks, thereby improving the spatiotemporal universal model's modeling ability for spatial transitivity. For example, the block masking strategy can be expressed by the following formula (5):

[0067] M~Uniform(1,2),E x =Ex[:,H′(M-1) / 2:H′M / 2] Formula (5)

[0068] In formula (5), M represents the mask determined by the block mask strategy, M~Uniform(1,2) means that M obeys the uniform distribution on (1,2), and the remaining parameters are detailed in the previous article and will not be repeated here.

[0069] The time masking strategy involves masking future data, forcing the model to reconstruct the future based only on historical information. The purpose of the time masking strategy is to improve the ability of the spatiotemporal general model to capture time dependencies from the past to the future. In practical applications, the time masking strategy masks data within a certain period of time in the future. The spatiotemporal general model needs to predict future data based on past information, thereby learning the modeling ability of time dependencies. For example, the time masking strategy can be expressed by the following formula (6):

[0070] M=Concat([1 (1-r)L′×H′×W′ , 0 rL′×H′W′ ]), E x =E x [M=1] Formula (6)

[0071] In formula (6), M represents the mask determined by the time mask strategy, Concat represents the concatenation process, and 1 (1-r)L'×H'×W' represents a matrix of shape (1-r)L'×H'×W' with all elements being 1, 0 rL'×H'×W' Represents a matrix of shape rL'×H'×W' with all elements set to 0. The remaining parameters are described above and are not repeated here.

[0072] In this way, by adopting these diverse masking strategies, the spatiotemporal general model can systematically enhance its modeling capabilities and simultaneously handle spatiotemporal, spatial, and temporal relationships. The introduction of these strategies will help the spatiotemporal general model better understand and adapt to the spatiotemporal data characteristics of different cities, and simultaneously capture spatiotemporal correlations containing coarse-grained and fine-grained spatiotemporal correlations, thereby improving the performance and generalization ability of the spatiotemporal general model on spatiotemporal tasks.

[0073] The prediction result of the corresponding spatiotemporal prediction task is determined according to the target mask and multiple prompt information. This process can be implemented by the decoder in the spatiotemporal general model. For example, the process can be expressed as Y = Decoder (P i ,M i ), where Y represents the prediction result of the spatiotemporal prediction task, such as Figure 3 As shown, P i Indicates the total prompt information, P i It is determined based on spatial proximity prompt information, spatial layer prompt information, temporal proximity prompt information, and time period prompt information. i Embedding vector representing the target mask, M i It may include embedding vectors M1, M2, ... M corresponding to the original shape of the spatiotemporal data L .

[0074] This spatiotemporal prediction method also includes a training process for the spatiotemporal universal model. In one possible implementation, the training process may include: obtaining sample data under multiple sample scenarios, including transportation scenarios, crowd flow scenarios, and life service scenarios; constructing a training sample set based on the sample data, and performing training based on the training sample set to obtain a trained spatiotemporal universal model. For example, a data set of more than 20 spatiotemporal scenarios is obtained, which covers multiple cities and involves different scenarios in multiple urban environments such as crowd flow, dynamic population, traffic speed, cellular network usage, taxi trips, and bicycle demand. The spatiotemporal universal model is trained using this data including multiple scenarios. The specific training method can be flexibly set according to actual needs, and the embodiments of the present disclosure do not limit this.

[0075] In the process of training the spatiotemporal universal model, the task of self-supervised learning is usually completed through mask reconstruction or autoregressive prediction. For example, in a visual model, a visual patch, i.e., a part of the image, is usually randomly masked, and then the masked pixels are reconstructed by the model. The present spatiotemporal prediction method designs the above-mentioned multi-angle masking strategy, performs masking and reconstruction in the pre-training stage, and improves the spatiotemporal universal model's comprehensive modeling capabilities for the complex spatiotemporal relationships and interwoven dynamics of group dynamic positions. In some embodiments, the encoder-decoder structure in the spatiotemporal universal model follows the method in the masked autoencoder, that is, the spatiotemporal tensor is masked at a certain ratio, and then the encoder receives the unmasked patch, and the decoder receives the encoder's output and the masked patch, and reconstructs them. Among them, the present spatiotemporal prediction method uses a decoder with a size comparable to the encoder network parameters to play a key role in both the training and prompt fine-tuning stages.

[0076] In the process of testing the spatiotemporal universal model, the mean absolute error (MAE) and root mean square error (RMSE) are used to compare the spatiotemporal universal model proposed in this spatiotemporal prediction method with the existing baseline model. These two performance metrics can be calculated using the following formula (7):

[0077]

[0078] In formula (7), represents the root mean square error, represents the mean absolute error, Y=Y1,…,Y N is the true value of the actual spatiotemporal data, Y i represents the true value of the i-th actual spatiotemporal data, is the value predicted by the spatiotemporal general model, represents the value predicted by the i-th spatiotemporal general model, and N is the total number of test samples.

[0079] Table 1 below shows the performance metrics of existing baseline models and general spatiotemporal models in different application scenarios. The baseline models use heuristic methods, deep urban spatiotemporal prediction methods, video prediction methods, multivariate time series prediction methods, and urban transfer learning methods. The baseline models include the adaptive Hamiltonian model (HA), the autoregressive integrated moving average model (ARIMA), the spatial-temporal residual network (STResNet), the adaptive recurrent feature map model (ACFM), the spatial-temporal identity model (STID), the spatial-temporal normalization model (STNorm), the spatial-temporal graph convolutional network model (STGSP), the spatiotemporal prediction architecture based on mask and contrastive learning (MC-STL), the multi-scale attention unit (MAU), the prediction recurrent neural network (PredRNN), the multimodal integration model (MIM), the similarity vector prediction model (SimVP), the temporal attention unit (TAU), the clustered multivariate time series model (PatchTST), the transformer variant model (iTransformer), and the unified clustered multivariate time series model (PatchTST (one-for-all)). As shown in Table 1, the spatiotemporal universal model proposed in this spatiotemporal prediction method performs well in different scenarios.

[0080] Table 1 Results of performance metrics of existing baseline models and spatiotemporal general models in different application scenarios

[0081]

[0082] The implementation of this spatiotemporal prediction method is schematically illustrated using two typical scenarios: crowd flow prediction and traffic speed prediction.

[0083] Crowd flow prediction refers to the technology of predicting the flow of people at a specific time and place in the future based on historical data and current conditions. It is of great significance in urban planning, traffic management, commercial operations and other fields. Crowd flow data is an important spatiotemporal data. In this case, spatial gridded time series crowd flow data is used and includes timestamps and corresponding crowd flow data. Timestamps are usually recorded at fixed time intervals (such as every hour or every half hour). Crowd flow data can be the actual number of people, the flow of people entering and leaving the area, etc. The given shape of the spatiotemporal data in this case is T×H×W, where T h ×H×W is the spatiotemporal data of h steps in history as the input of the spatiotemporal universal model. The model will predict the future spatiotemporal dynamics, and the output shape and size are (TT h )×H×W prediction results.

[0084] Traffic speed prediction refers to the technology of predicting traffic speeds at specific times and locations in the future based on historical data and current conditions. Similar to crowd flow prediction, traffic speed prediction is also of great significance in urban planning, traffic management and other fields. In this example, spatially gridded time series traffic speed data is used, including timestamps and corresponding traffic speed data. Timestamps are usually recorded at fixed time intervals (such as every hour or every half hour), and traffic speed data can be actual speeds, driving speeds of road sections, etc. Similarly to the crowd flow prediction scenario, traffic speed data can be used as input to the spatiotemporal general model to achieve the corresponding prediction task, and the details will not be repeated here.

[0085] This spatiotemporal prediction method forms a spatiotemporal universal model based on self-supervised pre-training technology and spatiotemporal cue fine-tuning technology. On the one hand, the spatiotemporal pre-training technology improves the general performance of the spatiotemporal universal model in capturing multi-level and multi-angle spatiotemporal correlations. On the other hand, fine-tuning is performed by cue information determined based on spatiotemporal features, thereby improving the generalization ability of the spatiotemporal universal model.

[0086] The embodiment of the present disclosure provides a spatiotemporal prediction device based on a prompt-fine-tuned spatiotemporal universal model. Figure 4 FIG. 1 is a block diagram of a spatiotemporal prediction device based on a spatiotemporal general model fine-tuned by hints according to an embodiment of the present disclosure. Figure 4 As shown, the spatiotemporal prediction device 100 includes the following acquisition module 101, determination module 102, feature extraction module 103, processing module 104, and prediction module 105.

[0087] Acquisition module 101 is used to acquire a spatiotemporal prediction task for a target scene, and determine spatiotemporal data based on the spatiotemporal prediction task. The spatiotemporal data includes data of multiple dimensions, and the multiple dimensions include spatial dimension, time dimension, and target dimension. The target dimension is determined based on the target scene, and the target scene is any one of a transportation scene, a crowd flow scene, and a life service scene in an urban environment.

[0088] The determination module 102 is used to process the spatiotemporal data to obtain multiple spatiotemporal tensors, and determine a spatial memory pool and a temporal memory pool based on the multiple spatiotemporal tensors, wherein the spatial memory pool includes multiple spatial key-value pairs, and the temporal memory pool includes multiple temporal key-value pairs.

[0089] The feature extraction module 103 is used to extract features from the multiple spatiotemporal tensors to obtain spatial proximity features representing spatial proximity attributes, spatial hierarchical features representing spatial hierarchical attributes, temporal proximity features representing temporal proximity attributes, and temporal period features representing temporal period attributes.

[0090] The processing module 104 is configured to process the spatial proximity feature, the spatial layering feature, the temporal proximity feature, and the temporal period feature using the spatial memory pool and the temporal memory pool to obtain a plurality of prompt information.

[0091] The prediction module 105 is used to determine a target mask based on a preset mask rule and the multiple spatiotemporal tensors, and determine a prediction result corresponding to the spatiotemporal prediction task according to the target mask and the multiple prompt information, wherein the mask rule is determined based on multiple mask strategies.

[0092] In one possible implementation, the spatiotemporal data is processed to obtain a plurality of spatiotemporal tensors, including: smoothing the spatiotemporal data to obtain smoothed data; segmenting the smoothed data based on the multiple dimensions to obtain a plurality of three-dimensional tensors; and converting the multiple three-dimensional tensors of a first shape into a plurality of spatiotemporal tensors of a second shape based on a preset convolutional layer, wherein the size of the convolution kernel of the convolutional layer is the same as the size of the spatiotemporal tensor, and the first shape is larger than the second shape.

[0093] In one possible implementation, the general spatiotemporal model includes an encoder; wherein, determining a spatial memory pool and a temporal memory pool based on the multiple spatiotemporal tensors includes: encoding the multiple spatiotemporal tensors using the encoder to obtain encoding results; determining multiple spatial key-value pairs and multiple temporal key-value pairs based on the encoding results; constructing the spatial memory pool based on the multiple spatial key-value pairs, and constructing the temporal memory pool based on the multiple temporal key-value pairs.

[0094] In one possible implementation, feature extraction is performed on the multiple spatiotemporal tensors to obtain spatial proximity features representing spatial proximity attributes, spatial hierarchical features representing spatial hierarchical attributes, temporal proximity features representing temporal proximity attributes, and temporal period features representing temporal period attributes, including: using a cosine function to position encode data at odd positions in a sequence corresponding to each of the spatiotemporal tensors, and using a sine function to position encode data at even positions in a sequence corresponding to each of the spatiotemporal tensors to obtain the spatial proximity features, the spatial hierarchical features, the temporal proximity features, and the temporal period features.

[0095] In a possible implementation, the number of the spatial key-value pairs is the same as the number of the time key-value pairs, each of the spatial key-value pairs includes a spatial key and a spatial value, and each of the time key-value pairs includes a time key and a time value; wherein, the spatial proximity feature, the spatial hierarchical feature, the time proximity feature, and the time period feature are processed using the spatial memory pool and the time memory pool to obtain a plurality of prompt information, including: determining a first weight for the spatial proximity attribute based on all the spatial keys and the spatial proximity features in the spatial memory pool, and determining spatial proximity prompt information based on the first weight and all the spatial values ​​in the spatial memory pool; determining a spatial proximity prompt information based on all the spatial keys and the spatial hierarchical features in the spatial memory pool. For the second weight of the spatial layered attribute, the spatial layered prompt information is determined based on the second weight and all the spatial values ​​in the spatial memory pool; the third weight for the temporal proximity attribute is determined based on all the time keys in the temporal memory pool and the temporal proximity feature, and the temporal proximity prompt information is determined based on the third weight and all the time values ​​in the temporal memory pool; the fourth weight for the temporal period attribute is determined based on all the time keys in the temporal memory pool and the temporal period feature, and the temporal period prompt information is determined based on the fourth weight and all the time values ​​in the temporal memory pool; the spatial proximity prompt information, the spatial layered prompt information, the temporal proximity prompt information, and the temporal period prompt information are used as the multiple prompt information.

[0096] In one possible implementation, the device also includes a training module, which is used to perform a training process for the spatiotemporal universal model. The training process includes: obtaining sample data under multiple sample scenarios, the multiple sample scenarios including the transportation scenario, the crowd flow scenario, and the life service scenario; constructing a training sample set based on the sample data, and performing training based on the training sample set to obtain a trained spatiotemporal universal model.

[0097] In one possible implementation, the multiple masking strategies include a random masking strategy for capturing fine-grained spatiotemporal relationships, a pipeline masking strategy for improving the spatial extrapolation capability of the spatiotemporal general model, a block masking strategy for enhancing spatial transitivity, and a temporal masking strategy for improving the ability of the spatiotemporal general model to capture temporal dependencies.

[0098] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments. For the sake of brevity, it will not be repeated here.

[0099] The present disclosure also provides a computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions implement the above method when executed by a processor. The computer-readable storage medium may be a volatile or non-volatile computer-readable storage medium.

[0100] An embodiment of the present disclosure further proposes an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0101] An embodiment of the present disclosure also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0102] Figure 5 A block diagram of a spatiotemporal prediction apparatus based on a spatiotemporal general model fine-tuned by hints according to an embodiment of the present disclosure is shown. For example, apparatus 1900 may be provided as a server or terminal device. Figure 5 The apparatus 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions, such as an application, that can be executed by the processing component 1922. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute the instructions to perform the above-described method.

[0103] The device 1900 may also include a power supply component 1926 configured to perform power management of the device 1900, a wired or wireless network interface 1950 configured to connect the device 1900 to a network, and an input / output interface 1958 (I / O interface). The device 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server 2003. TM , MacOS X TM , Unix TM ,Linux TM , FreeBSD TM or similar.

[0104] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by the processing component 1922 of the apparatus 1900 to perform the above-described method.

[0105] The present disclosure may be a system, method and / or computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0106] A computer-readable storage medium can be a tangible device that can hold and store instructions for use by an instruction execution device. A computer-readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination thereof. As used herein, a computer-readable storage medium is not to be construed as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through an electrical wire.

[0107] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in the computer-readable storage medium in each computing / processing device.

[0108] The computer program instructions for performing the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, and conventional procedural programming languages ​​such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., utilizing an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions. The electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0109] Various aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0110] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0111] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0112] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and the part of the module, program segment or instruction contains one or more executable instructions for realizing the prescribed logical function. In some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the prescribed function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0113] While various embodiments of the present disclosure have been described above, the foregoing description is intended to be illustrative, non-exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or technological improvements in the marketplace, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A spatiotemporal prediction method based on a spatiotemporal general model fine-tuned by prompts, characterized in that: The method uses a spatiotemporal general model based on cue fine-tuning to process a spatiotemporal prediction task, and the processing includes the following steps: Acquire a spatiotemporal prediction task for a target scene, and determine spatiotemporal data according to the spatiotemporal prediction task, wherein the spatiotemporal data includes data of multiple dimensions, wherein the multiple dimensions include a spatial dimension, a temporal dimension, and a target dimension, and the target dimension is determined according to the target scene, and the target scene is any one of a transportation scene, a crowd flow scene, and a life service scene in an urban environment; Processing the spatiotemporal data to obtain a plurality of spatiotemporal tensors, and determining a spatial memory pool and a temporal memory pool based on the plurality of spatiotemporal tensors, wherein the spatial memory pool includes a plurality of spatial key-value pairs, and the temporal memory pool includes a plurality of temporal key-value pairs; Extracting features from the multiple spatiotemporal tensors to obtain spatial proximity features representing spatial proximity attributes, spatial hierarchical features representing spatial hierarchical attributes, temporal proximity features representing temporal proximity attributes, and temporal periodic features representing temporal periodic attributes; The spatial proximity feature, the spatial layering feature, the temporal proximity feature, and the temporal period feature are processed by using the spatial memory pool and the temporal memory pool to obtain a plurality of prompt information; A target mask is determined based on a preset mask rule and the multiple spatiotemporal tensors, and a prediction result corresponding to the spatiotemporal prediction task is determined according to the target mask and the multiple prompt information, wherein the mask rule is determined based on multiple mask strategies.

2. The method according to claim 1, characterized in that The spatiotemporal data is processed to obtain a plurality of spatiotemporal tensors, including: Smoothing the spatiotemporal data to obtain smoothed data; Segmenting the smoothed data based on the multiple dimensions to obtain multiple three-dimensional tensors; Based on a preset convolutional layer, the multiple three-dimensional tensors in a first shape are converted into multiple spatiotemporal tensors in a second shape, wherein the size of the convolution kernel of the convolutional layer is the same as the size of the spatiotemporal tensor, and the first shape is larger than the second shape.

3. The method according to claim 1, characterized in that The spatiotemporal general model includes an encoder; wherein a spatial memory pool and a temporal memory pool are determined based on the multiple spatiotemporal tensors, including: Encoding the multiple spatiotemporal tensors using the encoder to obtain an encoding result; Determine a plurality of spatial key-value pairs and a plurality of temporal key-value pairs based on the encoding result; The spatial memory pool is constructed based on the multiple spatial key-value pairs, and the temporal memory pool is constructed based on the multiple temporal key-value pairs.

4. The method according to any one of claims 1 to 3, characterized in that Feature extraction is performed on the multiple spatiotemporal tensors to obtain spatial proximity features representing spatial proximity attributes, spatial hierarchical features representing spatial hierarchical attributes, temporal proximity features representing temporal proximity attributes, and temporal periodic features representing temporal periodic attributes, including: The data at odd positions in the sequence corresponding to each of the space-time tensors are position-encoded using a cosine function, and the data at even positions in the sequence corresponding to each of the space-time tensors are position-encoded using a sine function to obtain the spatial proximity feature, the spatial stratification feature, the temporal proximity feature, and the temporal period feature.

5. The method according to any one of claims 1 to 3, characterized in that: The number of the spatial key-value pairs is the same as the number of the temporal key-value pairs, each of the spatial key-value pairs includes a spatial key and a spatial value, and each of the temporal key-value pairs includes a temporal key and a temporal value; wherein the spatial proximity feature, the spatial hierarchical feature, the temporal proximity feature, and the temporal period feature are processed by the spatial memory pool and the temporal memory pool to obtain a plurality of prompt information, including: Determine a first weight for a spatial proximity attribute according to all spatial keys in the spatial memory pool and the spatial proximity feature, and determine spatial proximity prompt information according to the first weight and all spatial values ​​in the spatial memory pool; Determine a second weight for a spatial hierarchical attribute according to all spatial keys in the spatial memory pool and the spatial hierarchical feature, and determine spatial hierarchical prompt information according to the second weight and all spatial values ​​in the spatial memory pool; Determine a third weight for a time proximity attribute according to all time keys in the time memory pool and the time proximity feature, and determine time proximity prompt information according to the third weight and all time values ​​in the time memory pool; Determine a fourth weight for a time period attribute according to all time keys in the time memory pool and the time period feature, and determine time period prompt information according to the fourth weight and all time values ​​in the time memory pool; The spatial proximity prompt information, the spatial layered prompt information, the temporal proximity prompt information, and the temporal period prompt information are used as the plurality of prompt information.

6. The method according to any one of claims 1 to 3, characterized in that: The method also includes a training process of the spatiotemporal general model, the training process comprising: Acquire sample data under multiple sample scenarios, wherein the multiple sample scenarios include the transportation scenario, the crowd flow scenario, and the life service scenario; A training sample set is constructed according to the sample data, and training is performed based on the training sample set to obtain a trained spatiotemporal universal model.

7. The method according to any one of claims 1 to 3, characterized in that The multiple masking strategies include a random masking strategy for capturing fine-grained spatiotemporal relationships, a pipeline masking strategy for improving the spatial extrapolation capability of the spatiotemporal general model, a block masking strategy for enhancing spatial transitivity, and a temporal masking strategy for improving the ability of the spatiotemporal general model to capture temporal dependencies.

8. A spatiotemporal prediction device based on a spatiotemporal general model fine-tuned by prompts, characterized in that: include: An acquisition module, used to acquire a spatiotemporal prediction task for a target scene, and determine spatiotemporal data according to the spatiotemporal prediction task, wherein the spatiotemporal data includes data of multiple dimensions, wherein the multiple dimensions include a spatial dimension, a temporal dimension, and a target dimension, and the target dimension is determined according to the target scene, and the target scene is any one of a transportation scene, a crowd flow scene, and a life service scene in an urban environment; A determination module, configured to process the spatiotemporal data to obtain a plurality of spatiotemporal tensors, and determine a spatial memory pool and a temporal memory pool based on the plurality of spatiotemporal tensors, wherein the spatial memory pool includes a plurality of spatial key-value pairs, and the temporal memory pool includes a plurality of temporal key-value pairs; A feature extraction module is used to extract features from the multiple spatiotemporal tensors to obtain spatial proximity features representing spatial proximity attributes, spatial hierarchical features representing spatial hierarchical attributes, temporal proximity features representing temporal proximity attributes, and temporal periodic features representing temporal periodic attributes; A processing module, used to process the spatial proximity feature, the spatial layering feature, the temporal proximity feature, and the temporal period feature using the spatial memory pool and the temporal memory pool to obtain a plurality of prompt information; A prediction module is used to determine a target mask based on a preset mask rule and the multiple spatiotemporal tensors, and determine a prediction result corresponding to the spatiotemporal prediction task according to the target mask and the multiple prompt information, wherein the mask rule is determined based on multiple mask strategies.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to implement the method described in any one of claims 1 to 7 when executing the instructions stored in the memory.

10. A non-volatile computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images