Pre-training method and device of space-time model, equipment and storage medium

Abstract

The invention provides a spatio-temporal model pre-training method, device and equipment and a storage medium, relates to the technical field of artificial intelligence, in particular to the technical fields of neural networks, large models and the like, and can be applied to scenes related to spatio-temporal data, such as large-scale urban data analysis, traffic flow prediction, weather forecast, environment monitoring and the like. According to the specific implementation scheme, spatial features are extracted through a spatial feature extraction network of a to-be-trained spatial-temporal model based on first spatial-temporal information in a spatial-temporal diagram; extracting time sequence signal features based on the first spatio-temporal information through a time feature extraction network of the spatio-temporal model to be trained; obtaining a space guide time feature based on the space feature and the time sequence signal feature through a space-time fusion conversion network of the to-be-trained space-time model; and pre-training the to-be-trained spatiotemporal model based on second spatiotemporal information corresponding to the spatial guidance time feature and the first spatiotemporal information to obtain a pre-trained spatiotemporal model.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to technical fields such as neural networks and large models, and can be applied to scenarios involving spatiotemporal data such as large-scale urban data analysis, traffic flow prediction, weather forecasting, and environmental monitoring. Background Art

[0002] Multivariate Time Series (MTS) is widely used in many fields such as medicine, finance, multimedia, transportation, environmental science, and biological science. Traditional methods mostly use statistical methods, such as K-nearest neighbor, expectation maximization algorithm, and matrix decomposition. With the rise of deep learning, technologies such as recurrent neural networks and generative adversarial networks have begun to be applied to missing value filling tasks. Summary of the invention

[0003] The present invention provides a method, apparatus, device and storage medium for pre-training a spatiotemporal model.

[0004] According to one aspect of the present disclosure, a method for pre-training a spatiotemporal model is provided, comprising:

[0005] Extracting spatial features based on the first spatiotemporal information in the spatiotemporal graph through a spatial feature extraction network of the spatiotemporal model to be trained;

[0006] Extracting time series signal features based on the first time-space information through a time feature extraction network of the time-space model to be trained;

[0007] Obtaining a spatially guided temporal feature based on the spatial feature and the time series signal feature through a spatiotemporal fusion conversion network of the spatiotemporal model to be trained;

[0008] Based on the second spatiotemporal information corresponding to the spatially guided temporal feature and the first spatiotemporal information, the spatiotemporal model to be trained is pre-trained to obtain a pre-trained spatiotemporal model.

[0009] According to another aspect of the present disclosure, a pre-training device for a spatiotemporal model is provided, comprising:

[0010] A first extraction module, configured to extract spatial features based on first spatiotemporal information in the spatiotemporal graph through a spatial feature extraction network of a spatiotemporal model to be trained;

[0011] A second extraction module is used to extract time series signal features based on the first spatiotemporal information through a time feature extraction network of the spatiotemporal model to be trained;

[0012] A fusion module, used for obtaining a spatial guided time feature based on the spatial feature and the time series signal feature through a spatiotemporal fusion conversion network of the spatiotemporal model to be trained;

[0013] The pre-training module is used to pre-train the spatiotemporal model to be trained based on the second spatiotemporal information corresponding to the spatial guided time feature and the first spatiotemporal information to obtain a pre-trained spatiotemporal model.

[0014] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0015] at least one processor; and

[0016] a memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute any method in the embodiments of the present disclosure.

[0018] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any method according to the embodiments of the present disclosure.

[0019] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which implements any method according to the embodiments of the present disclosure when executed by a processor.

[0020] According to the present disclosure, the spatial features in the space-time graph can be fused with the time series signal features through the space-time model to be trained, which includes a spatial feature extraction network, a temporal feature extraction network and a space-time fusion conversion network, and the space can be used to reasonably guide time, so that the pre-trained space-time model can achieve better prediction accuracy.

[0021] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0023] Figure 1 is a flowchart of a method for pre-training a spatiotemporal model according to an embodiment of the present disclosure;

[0024] Figure 2 is a flowchart of a method for pre-training a spatiotemporal model according to another embodiment of the present disclosure;

[0025] Figure 3a and Figure 3b is a schematic diagram of a process of spatial feature extraction according to an embodiment of the present disclosure;

[0026] Figure 4 is a schematic diagram of a representation learning algorithm architecture based on spatiotemporal decoding provided according to an embodiment of the present disclosure;

[0027] Figure 5 yes Figure 4 Schematic diagram of the structure of the convolution encoder for time signals;

[0028] Figure 6 yes Figure 4 Schematic diagram of the structure of SGTT;

[0029] Figure 7 It is a flow chart of an application method for regional traffic flow prediction according to an embodiment of the present disclosure;

[0030] Figure 8 is a schematic structural diagram of a pre-training device for a spatiotemporal model according to an embodiment of the present disclosure;

[0031] Fig. 9 is a schematic structural diagram of a pre-training device for a spatiotemporal model according to another embodiment of the present disclosure;

[0032] Fig.10 is a block diagram of an electronic device for implementing an embodiment of the present disclosure. DETAILED DESCRIPTION

[0033] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0034] Figure 1 FIG. 1 is a flow chart of a method 100 for pre-training a spatiotemporal model according to an embodiment of the present disclosure. In one embodiment, the method includes:

[0035] S101, extracting spatial features based on first spatiotemporal information in the spatiotemporal graph through a spatial feature extraction network of a spatiotemporal model to be trained;

[0036] S102, extracting time series signal features based on the first spatiotemporal information through a time feature extraction network of the spatiotemporal model to be trained;

[0037] S103, obtaining a spatial guided time feature based on the spatial feature and the time series signal feature through the space-time fusion conversion network of the space-time model to be trained;

[0038] S104: Based on the second spatiotemporal information corresponding to the spatially guided temporal feature and the first spatiotemporal information, pre-train the spatiotemporal model to be trained to obtain a pre-trained spatiotemporal model.

[0039] In an embodiment of the present disclosure, the spatiotemporal model to be trained can be constructed based on a neural network. The network structure of the spatiotemporal model to be trained may include a spatial feature extraction network, a temporal feature extraction network, and a spatiotemporal fusion conversion network. One or more of the spatial feature extraction network, the temporal feature extraction network, and the spatiotemporal fusion conversion network may include a neural network architecture. The training samples of the spatiotemporal model to be trained may include one, one batch, or multiple batches of spatiotemporal graphs. A spatiotemporal graph may include spatiotemporal data of multiple regions. The spatiotemporal data of each region may include multiple nodes. The first spatiotemporal information may include the timing signals and spatial relationships of one or more nodes. The timing signals in the first spatiotemporal information may include historical timing signals and / or current timing signals. The spatial relationship of nodes is a static topological attribute of nodes.

[0040] In some examples, the first spatiotemporal information is divided into a static part and a dynamic part. The static part may include the historical time series signal and spatial relationship of the node, and the dynamic part may include the current time series signal of the node. The spatiotemporal model can extract the spatial features of the node offline based on the static part of the first spatiotemporal information, and can extract the time series signal features of the node online based on the dynamic part of the first spatiotemporal information. For a spatiotemporal graph in a training sample, the spatial features in the first spatiotemporal information of the spatiotemporal graph can be extracted through a spatial feature extraction network. For example, the spatial feature extraction network can extract the spatial features of a node based on the historical time series signal of a node and the spatial relationship between the node and other nodes. For the spatiotemporal graph, the time series signal features in the first spatiotemporal information of the spatiotemporal graph can be extracted through a temporal feature extraction network. For example, the temporal feature extraction network can extract the time series signal features of a node based on the current time series signal of a node. The steps of extracting spatial features and time series signal features may have no timing requirements, and the spatial features may be extracted first, or the time series signal features may be extracted first, or they may be extracted in parallel. Through the pre-trained space-time model, the spatial features and time series signal features of a node in a certain area in the space-time graph can be extracted, and the spatial features and time series signal features of multiple nodes in multiple areas in the space-time graph can also be extracted.

[0041] Furthermore, the spatial features and time series signal features extracted from the space-time graph can be fused through the space-time fusion conversion network to obtain space-guided time features. The space-guided time features can also be converted into corresponding second space-time information through the space-time model to be trained. Then, based on the first space-time information input in the space-time graph and the predicted second space-time information, the space-time model to be trained is pre-trained to obtain a pre-trained space-time model. For example, a loss function can be constructed based on the first space-time information and the second space-time information. The loss function value calculated based on the first space-time information and the second space-time information. Then, based on the loss function value, it is determined whether it is necessary to adjust the parameters of the space-time model to be trained. If adjustment is required, after the adjustment, the space-time graph in the training sample can be used to continue the above-mentioned pre-training process until the pre-training end condition is met. After the pre-training is completed, the pre-trained space-time model can be obtained.

[0042] Through the space-time model to be trained, which includes a spatial feature extraction network, a temporal feature extraction network and a space-time fusion conversion network, the spatial features in the space-time graph can be fused with the time series signal features, and the space can be used to reasonably guide the time, so that the pre-trained space-time model can achieve better prediction accuracy.

[0043] Figure 2 2 is a flow chart of a method 200 for pre-training a spatiotemporal model according to another embodiment of the present disclosure. The method 200 can be used to implement step S101 in the method 100 for pre-training a spatiotemporal model. In one embodiment, the method 200 includes: the spatial feature convolution network includes a spatial graph convolution encoder, and the spatial feature extraction network of the spatiotemporal model to be trained is used to extract the spatial feature based on the first spatiotemporal information in the spatiotemporal graph, and further includes:

[0044] S201. Input the first time series signals and spatial relations of multiple nodes in the first spatiotemporal information into the spatial graph convolution encoder to obtain spatial features of one or more first nodes among the multiple nodes.

[0045] In the disclosed embodiment, the spatiotemporal graph generally includes a plurality of nodes. The first spatiotemporal information may include time series signals of a plurality of variables (wherein a plurality of variables may be understood as a plurality of nodes, and a time series may be referred to as a time series). For example, the first spatiotemporal information may include a first time series signal of a plurality of nodes, a second time series signal, and a spatial relationship between a plurality of nodes. The first time series signal of a certain node may include a historical time series signal of the node, that is, a time series signal of the node at a historical moment before the current moment. For example, a node in a traffic spatiotemporal graph such as a smart surveillance camera has a traffic flow at time t-1. The spatial relationship of a certain node may include a spatial position relationship between the node and other nodes. For example, a node A1 in a traffic spatiotemporal graph is connected to other nodes A2, A3, and A4 around it on a road network, and the spatial relationship of a plurality of nodes may include information such as the distance between node A1 and nodes A2, A3, and A4. The first time series signal of a certain node may include a current time series signal. For example, the traffic flow of a monitoring node at time t in a traffic spatiotemporal graph.

[0046] In the embodiment of the present disclosure, the spatial graph convolution encoder in the spatiotemporal model to be trained can extract the spatial features of one or more first nodes based on the first time series signals and spatial relationships of the multiple nodes. Figure 3a and Figure 3b As shown, the spatial relationship between node A1 and nodes A2, A3 and A4, such as the distance, and the historical time series signals of nodes A1, A2, A3 and A4, such as the historical traffic, are input into the spatial graph convolution encoder 301, and the spatial features of node A1 can be obtained. For another example, if node A1 is connected to nodes A2, A3 and A4, and node A5 is connected to nodes A2 and A6, the spatial relationship between node A1 and nodes A2, A3 and A4, such as the distance, and the spatial relationship between node A5 and nodes A2 and A6, such as the distance, and the historical time series signals of nodes A1, A2, A3, A4, A5 and A6 can also be input into the spatial graph convolution encoder 301, and the spatial features of nodes A1 and A5 can be obtained.

[0047] Through the spatial graph convolution encoder, the spatial features of one or more first nodes can be extracted from the first time series signals and spatial relations of multiple nodes in the first spatiotemporal information, which is conducive to unified modeling using the spatial correlation and temporal dynamics of the nodes in the spatiotemporal graph, and improving the prediction accuracy of the pre-trained spatiotemporal model. The use of historical first time series signals to provide spatial features can complete the joint modeling of spatiotemporal features without the need for online information transmission between nodes, improve computing efficiency, and improve the prediction accuracy and reasoning efficiency of the pre-trained spatiotemporal model. In addition, the first spatiotemporal information is not limited to a specific single data set, but can include data sets in multiple regions. The scheme of the embodiment of the present disclosure can effectively handle spatiotemporal data integration across regions and data sets, and improve performance in large-scale application scenarios. In a multi-region scenario, the scheme based on the embodiment of the present disclosure can be unified modeled based on the spatial correlation and temporal dynamics of the data in each region, which is conducive to reducing the overhead of model training and improving the generalization ability of the model, thereby achieving better results in different regions or applications.

[0048] In one implementation, S201 inputs the first time series signals and spatial relationships of multiple nodes in the first spatiotemporal information into the spatial graph convolution encoder to obtain spatial features of one or more first nodes among the multiple nodes, including:

[0049] Inputting first time series signals and spatial relations of multiple nodes in the first spatiotemporal information into the spatial graph convolution encoder to obtain initial features of the multiple nodes; the multiple nodes include one or more first nodes and one or more second nodes related to each first node;

[0050] Based on the features of one or more second nodes related to the first node in the current layer of the spatial graph convolutional encoder, updating the features of the first node in the next layer of the spatial graph convolutional encoder by using an intermediate variable updating method;

[0051] The spatial features of the one or more first nodes are outputted through the spatial graph convolutional encoder.

[0052] In an embodiment of the present disclosure, the spatial graph convolutional encoder may include multiple layers such as an input layer, an intermediate layer, and an output layer. After the first time series signals and spatial relationships of multiple nodes are input from the input layer, the intermediate variables of each layer can be updated by an intermediate variable update method until the spatial features of one or more first nodes are output from the output layer. For example, the input layer can convert the first time series signal and spatial relationship of a first node and its related second node into initial features. According to the distance or connectivity relationship between the nodes, it can be determined which second nodes the first node is related to. The second node can also be called an adjacent node of the first node. Then, the features of the first node in the l+1th layer of the spatial graph convolutional encoder are updated based on the features of the second node in the lth layer of the spatial graph convolutional encoder using an intermediate variable update method.

[0053] For example, the first time series signals and spatial relationships of nodes A1, A2, A3, and A4 are input to the spatial graph convolution encoder. If A4 is regarded as the first node and the others are regarded as the second node, the features of node A4 at the l+1 layer can be updated according to the features of nodes A1, A2, and A3 at the lth layer; if A2 is regarded as the first node and the others are regarded as the second node, the features of node A2 at the l+1 layer can be updated according to the features of nodes A1, A3, and A4 at the lth layer; and so on, the features of each node at the l+1 layer can be obtained. Furthermore, the features of these nodes at the l+2 layer can be updated according to the features of these nodes at the l+1 layer. Until finally, the spatial features of each node are output.

[0054] The spatial graph convolution encoder can use an intermediate variable update method based on the features of a second node related to a first node in the current layer to update the features of the first node in the next layer, and then output the spatial features of the first node. This can quickly and accurately extract the spatial features of one or more nodes in the spatial graph, thereby improving computational efficiency.

[0055] In one embodiment, the intermediate variable updating method is determined based on the spatial distance information between the first node and each second node related to it, the characteristics of each second node in the current layer, and the learnable parameters.

[0056] An example of a formula for updating intermediate variables is as follows:

[0057]

[0058] in, Represents the variable of the i-th node (the first node) in the l+1th layer (the next layer) of the graph convolutional network; represents the intermediate variable of the jth node (the second node) in the lth layer (the current layer) of the graph convolutional network; e jiRepresents the spatial distance information from the jth node to the ith node; represents the element corresponding to the jth row and the ith column of the normalized adjacency matrix (for example, the adjacency matrix may include the spatial distance information between a first node i and all second nodes j related to it). In addition, is a learnable parameter and σ represents the activation function.

[0059] The spatial distance information in the disclosed embodiment can be an actual distance or a relative distance, such as a weighted physical spatial distance. The characteristics of the nodes in the next layer are updated by the spatial distance information between the nodes, the characteristics of the nodes in the current layer, etc., and the spatiotemporal information can be multi-layered encoded to obtain spatial features. Spatial features can also be called spatial representations or spatial representations, etc. The spatial correlation modeling between different nodes mainly depends on the topological relationship (such as spatial relationship) between the nodes. The disclosed embodiment can adopt an offline calculation method, such as obtaining the spatial discrimination information (such as spatial features) of each node in advance based on historical time series signals and spatial relationships, and then the information can be transmitted in real time during the sequence modeling process, which effectively improves the efficiency of online model reasoning.

[0060] In one embodiment, if Figure 2 As shown, the method 200 can also be used to implement step S102 in the pre-training method 100 of the spatiotemporal model. The method 200 includes: the spatial feature convolution network includes a time signal convolution encoder, and the time feature extraction network of the spatiotemporal model to be trained extracts the time series signal feature based on the first spatiotemporal information, and further includes:

[0061] S202. Input the second time series signals of one or more first nodes in the first time-space information into the time signal convolution encoder to obtain time series signal features of the one or more first nodes.

[0062] In the disclosed embodiment, the first spatiotemporal information of a first node may include the current time series signal of the node. The current time series signal may be time series dynamic information in a real scene. For example, the time signal convolution encoder performs convolution encoding processing on the current time series signals of nodes A1, A2, A3 and A4, and the time series signal features of nodes A1, A2, A3 and A4 may be obtained respectively. The time signal convolution encoder may extract dynamic time series signal features from the second time series signal in a real scene, which is beneficial to fuse the dynamic time series signal features with the static space features and improve the prediction accuracy of the spatiotemporal model.

[0063] In the embodiments of the present disclosure, although the first timing signal is a historical timing signal and the second timing signal is a current timing signal for illustration, this is not a limitation. In actual application scenarios, the first timing signal and the second timing signal can be the same timing signal or different timing signals. For example, the first timing signal and the second timing signal can both be current timing signals.

[0064] In one implementation, S202 inputs the second time series signal of the first node in the first spatiotemporal information into the time signal convolution encoder to obtain the time series signal feature of the first node, including:

[0065] In the first convolution layer of the time signal convolution encoder, the local timing signal within the first sliding convolution window and its corresponding binary mask are used to calculate the timing signal convolution result. The first sliding convolution window corresponds to the first convolution kernel and the first sliding step. The timing signal convolution result is used to determine the timing signal characteristics of the first node.

[0066] In an embodiment of the present disclosure, the time signal convolution encoder may include one or more first convolution layers. The first convolution layer may use a sliding convolution window to perform a convolution operation on the input time series signal according to a specific time step. For example, the first sliding convolution window corresponds to the first convolution kernel and the first sliding step. Based on the first sliding step, a local time series signal in the sliding convolution window can be obtained. The local time series signal may include a time series positions within the first sliding step, and the signal value at each time series position may be valid, invalid or missing. A binary mask may be used to indicate whether the signal value at each time series position of the local time series signal is valid. For example, the local time series signal in the first sliding convolution window includes 5 time series positions, wherein the signal values ​​at positions 1, 3, and 4 are valid, and the signal values ​​at other positions are invalid or missing. The binary mask of the local time series signal may be {1, 0, 1, 1, 0}. Multiple groups of local time series signals in a second time series signal and their corresponding binary masks can be obtained through the first sliding convolution window. Based on the local timing signal of a first node in the current layer and its corresponding binary mask, the local timing signal of the first node in the next layer can be calculated. Multiple layers of masked causal convolution can be stacked in the time signal convolution encoder (or called time signal convolution module, time convolution module, etc.). The local timing signal of the output of the current first convolution layer can be used as the local timing signal of the input of the next first convolution layer, and the binary mask of the next layer can be calculated based on the binary mask of the current layer. Until all first convolution layers are processed. The final processing results of all first convolution layers can be used as the timing signal features of the first node, or the timing signal features of the first node can be obtained after other processing.

[0067] In the embodiment of the present disclosure, the first convolution layer of the time signal convolution encoder can process the irregular missing and timing information redundancy problems reflected in the time series signal in the real scene through the local timing signal in the first sliding convolution window and its corresponding binary mask, and extract higher-dimensional timing dynamic information.

[0068] In one implementation, S202 inputs the second time series signal of the first node in the first spatiotemporal information into the time signal convolution encoder to obtain the time series signal feature of the first node, and further includes:

[0069] After passing through one or more of the first convolutional layers, the convolution result of the timing signal is downsampled using the local timing signal in the second sliding convolution window of the second convolutional layer and its corresponding binary mask to obtain the timing signal characteristics of the first node; wherein the second sliding convolution window corresponds to the second convolution kernel and the second sliding step; the second convolution kernel is larger than the first convolution kernel, and / or the second sliding step is larger than the first sliding step.

[0070] At the end of the first convolutional layer, a convolution scheme with a large convolution kernel and a large step size (or called stride) can be introduced, that is, the second convolutional layer. The second sliding convolution window of the second convolutional layer corresponds to the second convolution kernel and the second sliding step size. The second convolution kernel is larger than the first convolution kernel corresponding to the first sliding convolution window, and the second sliding step size is larger than the first sliding step size corresponding to the first sliding convolution window for efficient feature downsampling. If the sequence length of the second time series signal is L, the values ​​of the second convolution kernel and the second sliding step size of the second convolutional layer connected after the first convolutional layer are both p, and after downsampling, k = L / p timestamp discrete representations can be obtained, such as the time series signal feature of the first node. If the complexity of the time signal convolution encoder is O(L 2 ), the complexity of the subsequent time signal convolution encoder can be reduced to O(L 2 / k 2 ), which can improve the efficiency of model pre-training and inference.

[0071] In one embodiment, if Figure 2 As shown, the method 200 can also be used to implement step S103 in the pre-training method 100 of the spatiotemporal model. The spatiotemporal fusion transformation network includes a spatiotemporal fusion transformer (Transformer), which can also be called a spatial-guided temporal transformer. The method 200 includes: obtaining a spatial-guided temporal feature based on the spatial feature and the time series signal feature through the spatiotemporal fusion transformation network of the spatiotemporal model, and further includes:

[0072] S203, through the first attention layer on the first path of the spatiotemporal fusion converter, the time position code and the time series signal feature of the first node are fused to obtain the time fusion feature of the first node; wherein the time position code is obtained by convolution based on the time embedding feature according to the set step size;

[0073] S204, sending the time fusion feature of the first node from the first attention layer to the second attention layer on the second path of the spatiotemporal fusion converter;

[0074] S205, fusing the time fusion feature of the first node with the space feature of the first node through the second attention layer to obtain the time-space compatibility feature of the first node;

[0075] S206: Obtain a spatially guided time feature of the first node based on the spatiotemporal compatibility feature of the first node and the time fusion feature of the first node.

[0076] In the disclosed embodiment, the spatiotemporal fusion converter may include two paths, each of which has an attention layer. The attention layers on the two paths may be crossed during forward propagation and may be truncated during reverse propagation. After the temporal position code is obtained by convolution according to the set step size based on the temporal embedding feature, the temporal position code H t The time series signal features of the first node are input into the first path of the spatiotemporal fusion converter. The time position encoding H t and the timing signal characteristics of the first node Fusion, get the time fusion feature of the first node

[0077] Assume that the representation of each timestamp consists of an aperiodic component and several periodic dimensions. If the input calendar feature x of timestamp t is t ∈R C , the corresponding time embedding (or called time embedding feature, time embedding representation, etc.) can be represented as follows:

[0078]

[0079] Here, ⊕ represents the concatenation operation and F(·) represents a periodic activation function, such as sine and cosine functions. and is a learnable parameter, and b np and b p is a bias term associated with these parameters. In this way, the periodicity factor can be captured in a learnable way, thus avoiding the bias of relying on experience.

[0080] Then, a 1×p convolution with stride p is applied to the temporal embedding (the second convolutional layer) to generate the final temporal position encoding Further, this specific location information H t Signal representation added to discretization , as the input of the spatiotemporal fusion transformer to capture the associations between the fragments. The following is an example:

[0081]

[0082] The spatiotemporal fusion transformer can also support cross-attention mechanism. The temporal fusion features of the first node are sent from the first attention layer to the second attention layer on the second path of the spatiotemporal fusion transformer. The temporal fusion features of each layer can be expressed as

[0083] The temporal fusion features of each layer of the first node are fused through the second attention layer of the spatiotemporal fusion transformer The spatial feature (also called spatial representation) of the first node Fusion, to obtain the spatiotemporal compatibility features of the first node

[0084] The spatiotemporal compatibility characteristics of the first node output by the spatiotemporal fusion converter Fusion with the time fusion feature of the first node It can be spliced, fused or otherwise processed to obtain the spatial guided temporal feature z of the first node i (It can also be called spatial guided temporal representation).

[0085] Through the attention layers on multiple paths, not only can different temporal features be fused to obtain temporal fusion features, but also temporal and spatial features can be fused to obtain spatiotemporal compatible features. This is conducive to taking into account both dynamic and static features and improving the prediction accuracy of the spatiotemporal model.

[0086] In one implementation, based on the second spatiotemporal information corresponding to the spatial guided temporal feature and the first spatiotemporal information, the spatiotemporal model to be trained is pre-trained to obtain a pre-trained spatiotemporal model, and S104 further includes:

[0087] Inputting the spatial guided time feature of the first node into a decoder to reconstruct the second spatial and temporal information, wherein the second spatial and temporal information may include the reconstructed time series signal and spatial relationship of the first node;

[0088] Obtaining a loss function value based on the second spatiotemporal information and the first spatiotemporal information;

[0089] Based on the loss function value, the spatiotemporal model to be trained is pre-trained to obtain the pre-trained spatiotemporal model.

[0090] The spatiotemporal model may include a decoder, and the input of the decoder may include a spatially guided temporal feature z of a first node i , the output of the decoder may include the reconstructed timing signal and spatial relationship of the first node. The loss function used for pre-training of the spatiotemporal model can be constructed based on the first timing information and the second timing information. The timing signal and spatial relationship of one or more first nodes in the first timing information and the second timing information are substituted into the loss function to calculate the loss function value. If the loss function value does not converge, the parameters of the spatiotemporal model to be trained can be adjusted, and then the spatiotemporal graph can be used to continue training based on the adjusted model.

[0091] In one embodiment, the time series signal and / or spatial relationship in the first spatiotemporal information input into the spatial feature extraction network is partially masked or fully masked, and the second spatiotemporal information is predicted by the spatiotemporal model to be trained based on the masked first spatiotemporal information; and the loss function value is obtained based on the second spatiotemporal information and the first spatiotemporal information, including:

[0092] Based on the second spatiotemporal information, the unmasked first spatiotemporal information and the mask information, a first loss function value and a second loss function value are obtained, wherein the first loss function value represents the reconstruction loss of the masked sampling effective value, and the second loss function value represents the reconstruction loss of the unmasked sampling effective value;

[0093] Based on the first loss function value and the second loss function value, a third loss function value is obtained.

[0094] In the embodiments of the present disclosure, mask processing of a node in a space-time graph is taken as an example for explanation, and the mask processing method of other nodes in the space-time graph, or one or more nodes in other space-time graphs is similar.

[0095] The time series signals of one or more nodes in the first spatiotemporal information may be partially or fully masked first. The spatial relationship between different nodes in the first spatiotemporal information may also be partially or fully masked. For example, the masking strategy may include: randomly masking one or more of the partial time series signals in the historical time series signals of a first node in the first spatiotemporal information, the partial time series signals in the current time series signals, and the spatial relationship. Then, the masked first spatiotemporal information is input into the spatiotemporal model to be trained. For example, the spatiotemporal model to be trained extracts the spatial features of the first node based on the historical time series signals and spatial relationships of the second node related to the masked first node, and extracts the time series signal features of the first node based on the current time series signal (e.g., partial mask) of the masked first node. Then, the spatiotemporal model to be trained fuses the spatial features and time series signal features of the first node, and predicts the second spatiotemporal information such as the time series signal and spatial relationship of the first node. Furthermore, based on the spatiotemporal information of the masked first node and the output unmasked spatiotemporal information of the first node, the loss function value may be calculated. The loss function value can reflect the difference in information before and after the first node mask. If the difference is large, pre-training can be continued until the pre-training goal is met.

[0096] Through mask processing, the input data, such as the first spatiotemporal information, can be randomly masked in the time dimension and / or space dimension. Since the input data of the spatiotemporal model itself may have certain random missing, this masking strategy can enhance the robustness of the pre-trained spatiotemporal model.

[0097] In one implementation, based on the loss function value, pre-training the spatiotemporal model to be trained to obtain the pre-trained spatiotemporal model includes:

[0098] When the loss function value does not converge, back-propagation is used to adjust the parameters of one or more network structures of the spatial graph convolution encoder, the temporal signal convolution encoder, the spatiotemporal fusion converter and the decoder in the spatiotemporal model to be trained; wherein, in the spatiotemporal fusion converter, the gradient on the cross-attention layer from the feedback path of the second path to the feedback path of the first path is blocked;

[0099] When the loss function value converges, the pre-trained spatiotemporal model is obtained.

[0100] In the disclosed embodiment, after forward propagation, the parameters of each network module in the spatiotemporal model to be trained can be adjusted according to a certain gradient by back propagation. For example, the adjustable model parameters include one or more of the learnable parameters of the spatial graph convolution encoder, the convolution weights and bias terms of the temporal signal convolution encoder, and the learnable parameters of the spatiotemporal fusion converter. After adjustment, the above pre-training steps can be repeated until the loss function value converges. During the back propagation process, the gradient from the second attention layer to the first attention layer is blocked, that is, the gradient is not back propagated from the second attention layer to the first attention layer. By adopting a gradient blocking mechanism on the connection of the cross-attention layer, the static spatial influence is isolated from the dynamic time label, the misleading gradient flow from static space to dynamic time modeling is reduced, and the prediction accuracy and reasoning efficiency of the model are improved.

[0101] In one embodiment, the first spatiotemporal information in the spatiotemporal graph includes spatiotemporal data of multiple regions of the spatiotemporal graph, and the first node is any node in the multiple regions; the spatiotemporal data of the multiple regions are used to pre-train the spatiotemporal model. In the embodiment of the present disclosure, the first spatiotemporal information includes the timing signals and spatial relationships of the nodes of multiple regions in the spatiotemporal graph. Unified analysis and modeling of data from different regions can improve the accuracy and adaptability of the impact model. Pre-training the spatiotemporal model with spatiotemporal data from multiple regions can improve the generalization ability of the pre-trained spatiotemporal model when processing large-scale data across regions.

[0102] In one embodiment, the first spatiotemporal information in the spatiotemporal graph also includes the spatiotemporal data of a single area of ​​the spatiotemporal graph, and the first node is an arbitrary node in the single area; the spatiotemporal data of the single area is used to fine-tune the pre-trained spatiotemporal model. In the embodiment of the present disclosure, the pre-trained encoder is spliced ​​with a prediction head of random parameters, and a time series signal such as a traffic prediction is used as a supervisory signal to fine-tune the pre-trained spatiotemporal model in a single area. For example, in a single area, a 24-step signal is used as input to predict the traffic value of the next 12 steps. The loss function used for fine-tuning can take the form of mean square error, etc. The object of fine-tuning can include one or more of the spatial graph convolution encoder, the time signal convolution encoder, the spatiotemporal fusion converter and the decoder in the pre-trained spatiotemporal model. For example, only the parameters of the spatiotemporal fusion converter are fine-tuned. For example, the parameters of the time signal convolution encoder and the spatiotemporal fusion converter are fine-tuned. By fine-tuning the pre-trained spatiotemporal model with the spatiotemporal data of a single area, the prediction accuracy of the spatiotemporal model in the area can be improved.

[0103] The embodiments of the present disclosure involve the following related technologies:

[0104] 1. Multivariate Time Series Forecasting Methods

[0105] Multivariate time series (MTS) forecasting is a critical task in many fields such as transportation and environmental science. Deep learning methods can use Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs) to improve the accuracy of MTS forecasting. For example, Temporal Convolutional Network (TCN) and its variants treat time series as a one-dimensional structure and extract high-level features through convolution kernels. For another example, Long Short-Term Memory (LSTM) network and Gated Recurrent Unit (GRU) use autoregressive framework for forecasting. With the rise of Transformer architecture, methods such as Informer (a deep learning model architecture for long sequence time series forecasting) and Autoformer (Transformer model with autocorrelation decomposition) can be used in MTS forecasting by taking advantage of its ability to handle long-term dependencies. These techniques are widely used in different fields, especially in scenarios where long-term dependencies need to be handled.

[0106] With the development of graph network learning, spatio-temporal graph neural networks (STGNNs) have emerged. This type of method not only recognizes the temporal pattern of each node through a sequence model, but also propagates information between nodes through graph neural networks (GNNs), thereby effectively capturing spatiotemporal dependencies. According to different spatial learning strategies, these methods can be divided into two categories: one is a strategy based on static graphs, such as using topological distance and dynamic time warping (DTW) distance to construct static graphs; the other method dynamically constructs a latent graph, and generates a graph structure in an adaptive manner based on the similarity between the current time embeddings, thereby reflecting more flexible spatial correlations.

[0107] 2. Multivariate Time Series Completion Method

[0108] In the task of filling missing values ​​in multivariate time series, statistical methods such as K-nearest neighbors, expectation maximization algorithm and matrix decomposition can be used. Techniques such as recurrent neural network (RNN) and generative adversarial network (GAN) can also be applied to the task of filling missing values. For example, Bidirectional Recurrent Imputation for Time–Series (BRITS) uses a bidirectional recurrent dynamic system to handle missing data. Generative Adversarial Imputation Network (GAIN) combines RNN and GAN to achieve generative filling. Transformer architecture has also been applied to sequence filling tasks, such as Self-Attentive Imputation for Time Series (SAITS) which uses the self-attention mechanism of diagonal masks to capture fine-grained feature associations between different time steps, thereby enhancing the accuracy of missing value filling.

[0109] 3. Time Series Representation Learning Algorithm

[0110] In the field of time series representation learning, contrastive learning and masked autoencoders can be included. Contrastive learning optimizes by defining positive and negative sample pairs, making the representations of similar samples closer and the representations of dissimilar samples more separated. For example, Semantic Role Labeling (SRL) uses a ternary loss function to optimize the similarity between random subsequences; Temporal Neighborhood Coding (TNC) introduces a debiased contrast target to ensure the distinguishability of signal distribution inside and outside the neighborhood. Masked autoencoders use data masking as a data augmentation strategy to guide the encoder to reconstruct the masked part, thereby fully capturing the data distribution characteristics. For example, the Transformer-based Sequence-to-Sequence (TST) autoencoder uses the Transformer to extract dense vectors and trains with a denoising objective; PatchTST (TST for patching) further explores the ability of the Transformer in time series pre-training and uses its characteristics to improve the quality of representation learning. These methods are more suitable for training single-region datasets.

[0111] The disclosed embodiments can improve the multivariate time series (MTS) prediction and filling technology in the following aspects:

[0112] 1. Comprehensive modeling of spatial and temporal dependencies

[0113] Methods such as convolutional neural networks and recurrent neural networks mainly focus on the temporal dependency of time series, but do not adequately consider the complex spatial correlations between variables. The solution of the disclosed embodiment focuses on the spatial correlations between variables. In the case of complex data environments and changeable spatiotemporal relationships, the modeling from a global perspective can improve the prediction effect when faced with highly correlated multivariate data.

[0114] 2. Static and dynamic graph construction

[0115] When modeling spatial dependencies, spatiotemporal graph neural network methods can rely on static graphs based on distance functions or dynamically generated potential graphs. However, static graphs lack flexibility and are difficult to adapt to changing spatiotemporal relationships; although dynamic graphs can provide more sophisticated spatial modeling, they are prone to instability when data quality (such as noise and missing values) is poor. In addition, the high computational cost of dynamic graphs limits their application in large-scale real-time prediction tasks. The solution of the disclosed embodiment can combine the advantages of static graphs and dynamic graphs.

[0116] 3. Multi-region data expansion and adaptability

[0117] Related time series representation learning techniques, such as contrastive learning and masked autoencoders, are usually optimized for a single data set and lack sufficient generalization capabilities when processing large-scale cross-regional data. The solution of the disclosed embodiment can perform unified analysis and modeling for data from different regions, thereby improving the accuracy and adaptability of the spatiotemporal model.

[0118] The solution of the embodiment of the present disclosure can improve the data expansion capability in the multivariate time series pre-training process and can process multi-region spatiotemporal data sets.

[0119] If the multivariate time series pre-training method is limited to a specific single data set, it may not be able to effectively handle the integration of spatiotemporal data across regions and data sets, resulting in poor performance in large-scale application scenarios. Especially in multi-region scenarios, since the spatial correlation and temporal dynamics of data in each region are difficult to model uniformly, it is usually necessary to develop a pre-training model for each region separately. This approach not only increases the cost of model training, but also limits the generalization ability of the model, making it difficult to achieve the best results in different regions or applications.

[0120] To this end, the embodiments of the present disclosure propose a learning framework for decoupled spatiotemporal representation (which may be referred to as DeSTR for short). This framework can be used to improve the scalability of multivariate time series pre-training models in multi-dataset and multi-region scenarios. The framework adopts a dual encoder structure to capture the dynamic characteristics within the time series and the spatial correlation between multiple variables, and then combines these two types of information through the Transformer that guides the time series in space, giving the time features more discriminative spatial information. Compared with other spatiotemporal joint models, the embodiments of the present disclosure can complete the joint modeling of spatiotemporal features without the need for online information transfer between nodes, improve computing efficiency, and thus enable large-scale, cross-regional spatiotemporal data integration and analysis.

[0121] The application scenarios of the embodiments of the present disclosure include, but are not limited to, large-scale urban data analysis, traffic flow prediction, weather forecast, environmental monitoring and other application fields involving spatiotemporal data. Through pre-training of multi-regional data, the spatiotemporal model can better adapt to downstream tasks in different regions and has a wide range of industry application prospects, especially in real-time applications that require efficient spatiotemporal data processing, which can significantly reduce the computational cost of model reasoning.

[0122] Figure 4 It is a schematic diagram of a representation learning algorithm architecture based on spatiotemporal decoding according to an embodiment of the present disclosure. The architecture mainly includes the following parts: a spatial graph convolution encoder 401, a temporal signal convolution encoder 402, a spatial-guided temporal transformer 403 and a decoder 404. The above-mentioned parts specifically implement decoupled spatiotemporal representation learning through the following steps:

[0123] 1. Spatial Graph Convolutional Coding

[0124] First, the spatial relationship is extracted based on the input multivariate time series through the graph convolutional network. For example, one-hot encoding can be used to initialize the initial feature value of each node in the multivariate time series. For the intermediate variable of the jth node in the lth layer of the graph convolutional network, The update can be performed according to the following exemplary formula:

[0125]

[0126] in, represents the variable of the i-th node in the l+1th layer of the graph convolutional network; e jirepresents the weighted geospatial distance from the jth node to the ith node, corresponds to the element corresponding to row j and column i of the normalized adjacency matrix. In addition, is a learnable parameter and σ represents the activation function.

[0127] Since the spatial correlation modeling between different nodes only depends on the topological relationship between the nodes, this process can adopt offline calculation to obtain the spatial discrimination information of each node, thereby avoiding the need for real-time information transmission in the sequence modeling process and effectively improving the efficiency of online model reasoning.

[0128] 2. Convolutional Coding of Time Signals

[0129] Figure 5 yes Figure 4 Schematic diagram of the structure of the time signal convolution encoder in . The time signal convolution encoder can be called a time signal convolution module or a time convolution module. The time signal convolution encoder may include a masked causal convolution (MaskedCausal Convolution) module 501. The masked causal convolution module 501 may include modules such as causal convolution 5011 (CausalConv.) and renormalization 5012 (Re-Norm). The time convolution module can be used to process irregular missing and time series information redundancy problems reflected in time series signals (which can be referred to as time series, time series, time series signals, etc.) in real scenarios. The time signal convolution encoder can handle irregular missing problems in signals. For example, a sliding convolution window with a kernel size of 1×S is set to operate on the length L of the input time series signal. Let W and b represent the convolution weight and the corresponding bias term, respectively. For any specific time step (i.e., sliding step), let represents the local signal in the current window, and As the corresponding binary mask, it indicates the missing signal value at each position. The convolution operation can be expressed as:

[0130]

[0131] Among them, ⊙ represents element-wise multiplication, ||m l || 1 Calculate the sum of the valid signal values. In addition, the mask m can be updated according to a simple principle l If all signal values ​​covered by the current kernel are missing, the corresponding output masks will be set to zero. An example of the update formula can be expressed as:

[0132]

[0133] The subscript t represents the output time point corresponding to the current local window. The time series representation obtained by training is affected by the effective signal value, and the scaling factor Adaptive adjustment can be provided for different numbers of these valid values.

[0134] In order to extract higher-dimensional temporal dynamic information, multiple layers of masked causal convolution can be stacked in the temporal signal convolution module. Since the information density of the output features is low, a convolution scheme with a large convolution kernel and a large stride can be introduced at the end of the above convolution layer. For example Figure 5 The convolution layer 502 in , thus performing efficient feature downsampling. For example, the values ​​of the large convolution kernel and the sliding step size are both p, and after downsampling, k = L / p timestamp discrete representations can be obtained. Since the complexity of Transformer is O(L 2 ), the above downsampling operation reduces the complexity of the subsequent Transformer to O(L 2 / k 2 ), achieving significant efficiency improvements.

[0135] The time signal convolution encoder can output a discrete signal representation

[0136] 3. Spatial-Guided Temporal Transformer module, also known as the spatially guided temporal transformer module, can also be abbreviated as SGTT or Transformer encoder.

[0137] Figure 6 yes Figure 4 After extracting the decoupled time signal features and the corresponding node static topology attributes through steps 1 and 2 above, the discretized continuous signal is modeled according to the designed SGTT, and the spatial features are introduced as static discriminant information.

[0138] A new temporal position encoding is introduced to utilize the timestamp information in an adaptive way. It can be assumed that the representation of each timestamp consists of a non-periodic component and several periodic dimensions. The input calendar feature x of timestamp t is t ∈R C , the corresponding time embedding can be expressed as:

[0139]

[0140] in, represents the concatenation operation, and F(·) represents a periodic activation function, such as sine and cosine functions. and is a learnable parameter, and b np and b p is a bias term associated with these parameters. In this way, the periodicity factor can be captured in a learnable way, thus avoiding the bias of relying on experience.

[0141] Subsequently, a 1×p convolution with stride p can be applied to the temporal embedding to generate the final temporal position encoding This specific position information is further added to the discretized signal representation , as the input of the Transformer encoder, for example, through a multi-head self-attention channel of the Transformer encoder to capture the association between fragments, the formula example is as follows:

[0142]

[0143] Among them, the multi-head self-attention channel includes a normalization module 6021, a multi-head self-attention module 6022, a normalization module 6023 and a feedforward module 6024.

[0144] In order to provide static spatial properties for the above temporal modeling, a cross-attention mechanism can also be used. For example, through another multi-head cross-attention channel of the Transformer encoder, the spatial representation of the i-th node As a query, each term embedded with the signal representation Combined with the calculation compatibility function, the formula example is as follows:

[0145]

[0146] Among them, the multi-head cross-attention channel includes a normalization module 6011, a multi-head cross-attention module 6012, a normalization module 6013 and a feedforward module 6014.

[0147] Since the static node representation is independent of the current input time series, which is constantly changing during training, it may introduce misleading gradient flow from static space to dynamic time modeling. The disclosed embodiment can adopt a gradient blocking mechanism on each cross-attention connection to isolate the static space influence from the dynamic time labeling to reduce or avoid misleading gradient flow. Finally, the dual channels of SGTT generate updated temporal and spatial embeddings, which are fused together after the stacked SGTT blocks to obtain the final spatial guided temporal representation z i , the formula example is as follows:

[0148]

[0149] Among them, f Φ is a fully connected feed-forward network with parameters Φ; express The elements in ;[·;·] represent concatenation operations.

[0150] 4. Spatiotemporal Masked Autoencoder

[0151] In an exemplary masking strategy, the input multivariate time series can be randomly masked in both time and space dimensions. For example, the time series signal and / or spatial relationship of the i-th node in the multivariate time series is partially or fully masked. In real scenarios, since there are certain random missing data themselves, this masking strategy can enhance the robustness of the pre-trained encoder.

[0152] 5. Reconstruction and loss calculation

[0153] Using a lightweight decoder, such as a multi-layer perceptron, we can guide the temporal representation z based on space i To reconstruct the input signal S out . S out It can be understood as the output value of the pre-trained spatiotemporal model. In the pre-training stage, the optimization goal is to make the masked signal and the unmasked signal consistent with the true value S of the input signal. in Specifically, the following objective function can be optimized:

[0154]

[0155] Among them, λ is a hyperparameter. L mask and L valid They represent the reconstruction loss of the effective values ​​of masked and unmasked samples, respectively. Includes a sliding window with a binary mask m at multiple steps l .

[0156] Figure 7 FIG. 7 is a flow chart of an application method 700 for regional traffic flow prediction according to an embodiment of the present disclosure. Taking the traffic flow prediction in a city as an example, Figure 7 As shown, one or more of the following steps may be included:

[0157] S701. Collect traffic flow data sets in multiple areas:

[0158] In order to give full play to the advantages brought by data expansion, as much traffic data as possible can be collected in specific applications, such as road network traffic in multiple cities. In addition, since the detectors of the road network in real scenarios may have equipment failures, resulting in long-term abnormal traffic at a single node, anomaly detection methods can be used to discover abnormal data and perform mask processing during data processing.

[0159] S702, sampling DeSTR framework for joint pre-training:

[0160] The cleaned traffic data of multiple areas are used as input data, and joint pre-training is performed under the DeSTR framework to optimize the model parameters in the encoder and obtain a set of static spatial representations of each node.

[0161] S703, fine-tuning the pre-trained encoder on a single region using the traffic prediction as a supervisory signal:

[0162] The pre-trained encoder is spliced ​​with a prediction head with random parameters, and the traffic prediction is used as the supervision signal to fine-tune the pre-trained encoder on a single region. For example, with a 24-step signal as input, the traffic value for the next 12 steps is predicted. The mean square error loss is used to fine-tune the model downstream.

[0163] S704, deploying the back-end algorithm to predict traffic flow:

[0164] The fine-tuned model is deployed on the backend algorithm to predict possible future traffic values ​​in real time based on the historical traffic data returned by the monitoring probes.

[0165] If multivariate time series analysis is conducted independently on a single data set, the effect gain brought by pre-training at the data expansion level cannot be fully utilized. The spatiotemporal pre-training method of the disclosed embodiment includes a method for joint pre-training of multi-region spatiotemporal data, which can realize joint pre-training on multiple regional data sets, thereby obtaining a universal encoder, and can also be flexibly applied to downstream applications of a single region on multiple tasks, achieving better prediction accuracy and reasoning efficiency.

[0166] Figure 8 8 is a schematic diagram of a pre-training device 800 for a spatiotemporal model according to an embodiment of the present disclosure. In one embodiment, the device may include:

[0167] A first extraction module 801 is used to extract spatial features based on first spatiotemporal information in the spatiotemporal graph through a spatial feature extraction network of a spatiotemporal model to be trained;

[0168] A second extraction module 802 is used to extract time series signal features based on the first spatiotemporal information through a time feature extraction network of the spatiotemporal model to be trained;

[0169] A fusion module 803 is used to obtain a spatial guided time feature based on the spatial feature and the time series signal feature through a spatiotemporal fusion conversion network of the spatiotemporal model to be trained;

[0170] The pre-training module 804 is used to pre-train the spatiotemporal model to be trained based on the second spatiotemporal information corresponding to the spatial guided temporal feature and the first spatiotemporal information to obtain a pre-trained spatiotemporal model.

[0171] Fig. 9 It is a structural schematic diagram of a pre-training device 900 for a spatiotemporal model according to another embodiment of the present disclosure. The device 900 includes: a first extraction module 901, a second extraction module 902, a fusion module 903 and a pre-training module 904. The functions of the above modules can refer to the functions of each module of the pre-training device 800 for the spatiotemporal model of the above embodiment. In one embodiment, the spatial feature convolution network includes a spatial graph convolution encoder, and the first extraction module 901 is used to input the first time series signals and spatial relations of multiple nodes in the first spatiotemporal information into the spatial graph convolution encoder to obtain the spatial features of one or more first nodes among the multiple nodes.

[0172] In one implementation, the first extraction module 901 includes:

[0173] An input submodule 9011 is used to input the first time series signals and spatial relations of multiple nodes in the first spatiotemporal information into the spatial graph convolution encoder to obtain initial features of the multiple nodes; the multiple nodes include one or more first nodes and one or more second nodes related to each first node;

[0174] A variable updating submodule 9012 is used to update the feature of the first node in the next layer of the spatial graph convolutional encoder by using an intermediate variable updating method based on the feature of one or more second nodes related to the first node in the current layer of the spatial graph convolutional encoder;

[0175] The output submodule 9013 is used to output the spatial features of the one or more first nodes through the spatial graph convolution encoder.

[0176] In one embodiment, the intermediate variable updating method is determined based on the spatial distance information between the first node and each second node related to it, the characteristics of each second node in the current layer, and the learnable parameters.

[0177] In one embodiment, Fig. 9 As shown, the spatial feature convolution network includes a time signal convolution encoder, and the second extraction module 902 is used to input the second time series signal of one or more first nodes in the first space-time information into the time signal convolution encoder to obtain the time series signal characteristics of the one or more first nodes.

[0178] In one embodiment, the second extraction module 902 includes:

[0179] The first convolution submodule 9021 is used to calculate the convolution result of the timing signal in the first convolution layer of the time signal convolution encoder, using the local timing signal in the first sliding convolution window and its corresponding binary mask, the first sliding convolution window corresponds to the first convolution kernel and the first sliding step, and the timing signal convolution result is used to determine the timing signal characteristics of the first node.

[0180] In one implementation, the second extraction module 902 further includes:

[0181] The second convolution submodule 9022 is used to downsample the convolution result of the timing signal using the local timing signal in the second sliding convolution window of the second convolution layer and its corresponding binary mask after passing through one or more of the first convolution layers, so as to obtain the timing signal characteristics of the first node; wherein the second sliding convolution window corresponds to the second convolution kernel and the second sliding step; the second convolution kernel is larger than the first convolution kernel, and / or the second sliding step is larger than the first sliding step.

[0182] In one embodiment, Fig. 9 As shown, the time-space fusion conversion network includes a time-space fusion converter, and the fusion module 903 includes:

[0183] The first fusion submodule 9031 is used to fuse the time position code and the time series signal feature of the first node through the first attention layer on the first path of the spatiotemporal fusion converter to obtain the time fusion feature of the first node; wherein the time position code is obtained by convolution based on the time embedding feature according to the set step size;

[0184] A sending submodule 9032, configured to send the time fusion feature of the first node from the first attention layer to the second attention layer on the second path of the spatiotemporal fusion converter;

[0185] A second fusion submodule 9033 is used to fuse the time fusion feature of the first node with the spatial feature of the first node through the second attention layer to obtain the time-space compatibility feature of the first node;

[0186] The spatiotemporal feature acquisition submodule 9034 is used to obtain the spatial guidance time feature of the first node based on the spatiotemporal compatibility feature of the first node and the time fusion feature of the first node.

[0187] In one embodiment, Fig. 9 As shown, the pre-training module 904 includes:

[0188] A reconstruction submodule 9041, configured to input the spatial guided time feature of the first node into a decoder to reconstruct the second spatial and temporal information, where the second spatial and temporal information may include the reconstructed time series signal and spatial relationship of the first node;

[0189] A loss calculation submodule 9042, configured to obtain a loss function value based on the second spatiotemporal information and the first spatiotemporal information;

[0190] The pre-training submodule 9043 is used to pre-train the spatiotemporal model to be trained based on the loss function value to obtain the pre-trained spatiotemporal model.

[0191] In one embodiment, the timing signal and / or spatial relationship in the first spatiotemporal information input into the spatial feature extraction network is partially masked or fully masked, and the second spatiotemporal information is predicted by the spatiotemporal model to be trained based on the masked first spatiotemporal information; the loss calculation submodule 9042 is also used to obtain a first loss function value and a second loss function value based on the second spatiotemporal information, the unmasked first spatiotemporal information and the mask information, the first loss function value representing the reconstruction loss of the masked sampling effective value, and the second loss function value representing the reconstruction loss of the unmasked sampling effective value; based on the first loss function value and the second loss function value, a third loss function value is obtained.

[0192] In one embodiment, the pre-training submodule 9043 is also used to adjust the parameters of one or more network structures of the spatial graph convolution encoder, the temporal signal convolution encoder, the spatiotemporal fusion converter and the decoder in the spatiotemporal model to be trained by back propagation when the loss function value has not converged; wherein, in the spatiotemporal fusion converter, the gradient on the cross-attention layer from the feedback path of the second path to the feedback path of the first path is blocked; until the loss function value converges, the pre-trained spatiotemporal model is obtained.

[0193] In one embodiment, the first spatiotemporal information in the spatiotemporal graph includes spatiotemporal data of multiple regions of the spatiotemporal graph, and the first node is any node in the multiple regions; the spatiotemporal data of the multiple regions are used to pre-train the spatiotemporal model.

[0194] In one embodiment, the first spatiotemporal information in the spatiotemporal graph also includes spatiotemporal data of a single area of ​​the spatiotemporal graph, and the first node is any node in the single area; the spatiotemporal data of the single area is used to fine-tune the pre-trained spatiotemporal model.

[0195] For the description of specific functions and examples of each module and submodule of the device in the embodiment of the present disclosure, reference can be made to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0196] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0197] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0198] Fig.10 A schematic block diagram of an example electronic device 1000 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0199] like Fig.10 As shown, the device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from a storage unit 1008 into a random access memory (RAM) 1003. In the RAM 1003, various programs and data required for the operation of the device 1000 can also be stored. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0200] A number of components in the device 1000 are connected to the I / O interface 1005, including: an input unit 1006, such as a keyboard, a mouse, etc.; an output unit 1007, such as various types of displays, speakers, etc.; a storage unit 1008, such as a disk, an optical disk, etc.; and a communication unit 1009, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1009 allows the device 1000 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0201] The computing unit 1001 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1001 performs the various methods and processes described above, such as the pre-training method of the spatiotemporal model. For example, in some embodiments, the pre-training method of the spatiotemporal model may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 1008. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1000 via the ROM 1002 and / or the communication unit 1009. When the computer program is loaded into the RAM 1003 and executed by the computing unit 1001, one or more steps of the pre-training method of the spatiotemporal model described above may be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured to execute the pre-training method of the spatiotemporal model in any other appropriate manner (eg, by means of firmware).

[0202] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0203] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0204] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0205] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0206] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0207] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0208] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0209] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A pre-training method for a spatiotemporal model, comprising: Extracting spatial features based on the first spatiotemporal information in the spatiotemporal graph through a spatial feature extraction network of the spatiotemporal model to be trained; Extracting time series signal features based on the first spatiotemporal information through a time feature extraction network of the spatiotemporal model to be trained; Obtaining space-guided time features based on the spatial features and the time series signal features through the space-time fusion conversion network of the space-time model to be trained; Based on the second spatiotemporal information corresponding to the spatially guided temporal feature and the first spatiotemporal information, the spatiotemporal model to be trained is pre-trained to obtain a pre-trained spatiotemporal model.

2. The method according to claim 1, wherein: The spatial feature convolution network includes a spatial graph convolution encoder, and extracts spatial features based on first spatiotemporal information in the spatiotemporal graph through a spatial feature extraction network of a spatiotemporal model to be trained, including: The first time series signals and spatial relations of multiple nodes in the first spatiotemporal information are input into the spatial graph convolution encoder to obtain spatial features of one or more first nodes among the multiple nodes.

3. The method according to claim 2, wherein: Inputting first time series signals and spatial relations of multiple nodes in the first spatiotemporal information into the spatial graph convolution encoder to obtain spatial features of one or more first nodes among the multiple nodes, including: Inputting first time series signals and spatial relations of multiple nodes in the first spatiotemporal information into the spatial graph convolution encoder to obtain initial features of the multiple nodes; the multiple nodes include one or more first nodes and one or more second nodes related to each first node; Based on features of one or more second nodes related to the first node in the current layer of the spatial graph convolutional encoder, updating features of the first node in the next layer of the spatial graph convolutional encoder by using an intermediate variable updating method; The spatial features of the one or more first nodes are outputted through the spatial graph convolutional encoder.

4. The method according to claim 3, wherein: The intermediate variable updating method is determined based on the spatial distance information between the first node and each second node related to it, the characteristics of each second node in the current layer, and the learnable parameters.

5. The method according to any one of claims 1 to 4, wherein: The spatial feature convolution network includes a time signal convolution encoder, and extracts time series signal features based on the first spatiotemporal information through the time feature extraction network of the spatiotemporal model to be trained, including: The second time series signals of one or more first nodes in the first time-space information are input into the time signal convolution encoder to obtain the time series signal features of the one or more first nodes.

6. The method according to claim 5, wherein: Inputting the second time series signal of the first node in the first time-space information into the time signal convolution encoder to obtain the time series signal feature of the first node, including: In the first convolution layer of the time signal convolution encoder, the local timing signal within the first sliding convolution window and its corresponding binary mask are used to calculate the timing signal convolution result, the first sliding convolution window corresponds to the first convolution kernel and the first sliding step, and the timing signal convolution result is used to determine the timing signal characteristics of the first node.

7. The method according to claim 6, wherein: Inputting the second time series signal of the first node in the first time-space information into the time signal convolution encoder to obtain the time series signal feature of the first node, further comprising: After passing through one or more of the first convolution layers, the convolution result of the timing signal is downsampled using the local timing signal in the second sliding convolution window of the second convolution layer and its corresponding binary mask to obtain the timing signal features of the first node; wherein the second sliding convolution window corresponds to a second convolution kernel and a second sliding step; the second convolution kernel is larger than the first convolution kernel, and / or the second sliding step is larger than the first sliding step.

8. The method according to any one of claims 1 to 7, wherein the space-time fusion conversion network comprises a space-time fusion converter, and the space-time fusion conversion network of the space-time model obtains the space-guided time feature based on the spatial feature and the time series signal feature, comprising: The time position code and the time series signal feature of the first node are fused through the first attention layer on the first path of the spatiotemporal fusion converter to obtain the time fusion feature of the first node; wherein the time position code is obtained by convolution based on the time embedding feature according to the set step size; sending the time fused feature of the first node from the first attention layer to a second attention layer on a second path of the spatiotemporal fusion converter; fusing the time fusion feature of the first node with the space feature of the first node through the second attention layer to obtain the time-space compatible feature of the first node; Based on the spatiotemporal compatibility feature of the first node and the time fusion feature of the first node, a spatially guided time feature of the first node is obtained.

9. The method according to claim 8, wherein: Based on the second spatiotemporal information corresponding to the spatial guided time feature and the first spatiotemporal information, the spatiotemporal model to be trained is pre-trained to obtain a pre-trained spatiotemporal model, including: Inputting the spatial guided time feature of the first node into a decoder to reconstruct the second spatial and temporal information, wherein the second spatial and temporal information may include the reconstructed time series signal and spatial relationship of the first node; Obtaining a loss function value based on the second spatiotemporal information and the first spatiotemporal information; Based on the loss function value, the spatiotemporal model to be trained is pre-trained to obtain the pre-trained spatiotemporal model.

10. The method according to claim 9, wherein: The time series signal and / or spatial relationship in the first spatiotemporal information input into the spatial feature extraction network is partially masked or fully masked, and the second spatiotemporal information is predicted by the spatiotemporal model to be trained based on the masked first spatiotemporal information; Obtaining a loss function value based on the second spatiotemporal information and the first spatiotemporal information includes: Based on the second spatiotemporal information, the unmasked first spatiotemporal information and the mask information, a first loss function value and a second loss function value are obtained, wherein the first loss function value represents the reconstruction loss of the effective value of the masked sampling, and the second loss function value represents the reconstruction loss of the effective value of the unmasked sampling; Based on the first loss function value and the second loss function value, a third loss function value is obtained.

11. The method according to claim 10, wherein: Based on the loss function value, pre-training the spatiotemporal model to be trained to obtain the pre-trained spatiotemporal model includes: In the case where the loss function value has not converged, back propagation is used to adjust the parameters of one or more network structures of the spatial graph convolution encoder, the temporal signal convolution encoder, the spatiotemporal fusion converter and the decoder in the spatiotemporal model to be trained; wherein, in the spatiotemporal fusion converter, the gradient on the cross attention layer from the feedback path of the second path to the feedback path of the first path is blocked; When the loss function value converges, the pre-trained spatiotemporal model is obtained.

12. The method according to any one of claims 1 to 11, wherein: The first spatiotemporal information in the spatiotemporal graph includes spatiotemporal data of multiple regions of the spatiotemporal graph, and the first node is any node in the multiple regions; the spatiotemporal data of the multiple regions are used to pre-train the spatiotemporal model.

13. The method according to claim 12, wherein: The first spatiotemporal information in the spatiotemporal graph also includes spatiotemporal data of a single area of ​​the spatiotemporal graph, and the first node is any node in the single area; the spatiotemporal data of the single area is used to fine-tune the pre-trained spatiotemporal model.

14. A pre-training device for a spatiotemporal model, comprising: A first extraction module, configured to extract spatial features based on first spatiotemporal information in the spatiotemporal graph through a spatial feature extraction network of a spatiotemporal model to be trained; A second extraction module, configured to extract time series signal features based on the first spatiotemporal information through a time feature extraction network of the spatiotemporal model to be trained; A fusion module, configured to obtain a spatially guided temporal feature based on the spatial feature and the time series signal feature through a spatiotemporal fusion conversion network of the spatiotemporal model to be trained; A pre-training module is used to pre-train the spatiotemporal model to be trained based on the second spatiotemporal information corresponding to the spatial-guided temporal feature and the first spatiotemporal information to obtain a pre-trained spatiotemporal model.

15. The device according to claim 14, wherein: The spatial feature convolution network includes a spatial graph convolution encoder, and the first extraction module is used to input the first time series signals and spatial relationships of multiple nodes in the first spatiotemporal information into the spatial graph convolution encoder to obtain the spatial features of one or more first nodes among the multiple nodes.

16. The device according to claim 15, wherein: The first extraction module comprises: An input submodule, configured to input first time series signals and spatial relations of a plurality of nodes in the first spatiotemporal information into the spatial graph convolution encoder to obtain initial features of the plurality of nodes; the plurality of nodes comprising one or more first nodes and one or more second nodes associated with each first node; a variable updating submodule, configured to update the feature of the first node at the next layer of the spatial graph convolutional encoder by using an intermediate variable updating method based on the feature of one or more second nodes related to the first node at the current layer of the spatial graph convolutional encoder; An output submodule is used to output the spatial features of the one or more first nodes through the spatial graph convolution encoder.

17. The device according to claim 16, wherein: The intermediate variable updating method is determined based on the spatial distance information between the first node and each second node related to it, the characteristics of each second node in the current layer, and the learnable parameters.

18. The device according to any one of claims 14 to 17, wherein: The spatial feature convolution network includes a time signal convolution encoder, and the second extraction module is used to input the second time series signal of one or more first nodes in the first space-time information into the time signal convolution encoder to obtain the time series signal features of the one or more first nodes.

19. The device according to claim 18, wherein: The second extraction module comprises: The first convolution submodule is used to calculate the convolution result of the timing signal in the first convolution layer of the time signal convolution encoder, using the local timing signal within the first sliding convolution window and its corresponding binary mask, the first sliding convolution window corresponds to the first convolution kernel and the first sliding step, and the timing signal convolution result is used to determine the timing signal characteristics of the first node.

20. The device according to claim 19, wherein The second extraction module also includes: The second convolution submodule is used to downsample the convolution result of the timing signal using the local timing signal in the second sliding convolution window of the second convolution layer and its corresponding binary mask after passing through one or more of the first convolution layers, so as to obtain the timing signal characteristics of the first node; wherein the second sliding convolution window corresponds to a second convolution kernel and a second sliding step; the second convolution kernel is larger than the first convolution kernel, and / or the second sliding step is larger than the first sliding step.

21. The device according to any one of claims 14 to 20, wherein: The space-time fusion conversion network includes a space-time fusion converter, and the fusion module includes: A first fusion submodule, configured to fuse the time position code and the time series signal feature of the first node through a first attention layer on the first path of the spatiotemporal fusion converter to obtain a time fusion feature of the first node; wherein the time position code is obtained by convolution based on the time embedding feature according to a set step size; a sending submodule, configured to send the time fusion feature of the first node from the first attention layer to a second attention layer on a second path of the spatiotemporal fusion converter; A second fusion submodule, configured to fuse the time fusion feature of the first node with the spatial feature of the first node through the second attention layer to obtain a time-space compatible feature of the first node; The spatiotemporal feature acquisition submodule is used to obtain the spatially guided temporal feature of the first node based on the spatiotemporal compatibility feature of the first node and the temporal fusion feature of the first node.

22. The device according to claim 21, wherein The pre-training module comprises: A reconstruction submodule, configured to input the spatial guided time feature of the first node into a decoder, and reconstruct the second spatiotemporal information, wherein the second spatiotemporal information may include a time series signal and a spatial relationship after the reconstruction of the first node; A loss calculation submodule, used to obtain a loss function value based on the second spatiotemporal information and the first spatiotemporal information; The pre-training submodule is used to pre-train the spatiotemporal model to be trained based on the loss function value to obtain the pre-trained spatiotemporal model.

23. The device according to claim 22, wherein: The timing signal and / or spatial relationship in the first spatiotemporal information input into the spatial feature extraction network is partially masked or fully masked, and the second spatiotemporal information is predicted by the spatiotemporal model to be trained based on the masked first spatiotemporal information; the loss calculation submodule is also used to obtain a first loss function value and a second loss function value based on the second spatiotemporal information, the unmasked first spatiotemporal information and the mask information, the first loss function value representing the reconstruction loss of the effective value of the masked sampling, and the second loss function value representing the reconstruction loss of the effective value of the unmasked sampling; based on the first loss function value and the second loss function value, a third loss function value is obtained.

24. The device according to claim 23, wherein: The pre-training submodule is also used to adjust the parameters in one or more network structures of the spatial graph convolution encoder, the temporal signal convolution encoder, the spatiotemporal fusion converter and the decoder in the spatiotemporal model to be trained by back propagation when the loss function value has not converged; wherein, in the spatiotemporal fusion converter, the gradient on the cross-attention layer from the feedback path of the second path to the feedback path of the first path is blocked; until the loss function value converges, the pre-trained spatiotemporal model is obtained.

25. The device according to any one of claims 14 to 24, wherein: The first spatiotemporal information in the spatiotemporal graph includes spatiotemporal data of multiple regions of the spatiotemporal graph, and the first node is any node in the multiple regions; the spatiotemporal data of the multiple regions are used to pre-train the spatiotemporal model.

26. The device according to claim 25, wherein The first spatiotemporal information in the spatiotemporal graph also includes spatiotemporal data of a single area of ​​the spatiotemporal graph, and the first node is any node in the single area; the spatiotemporal data of the single area is used to fine-tune the pre-trained spatiotemporal model.

27. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 13.

28. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-13.

29. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 13.

Images