Space-time sequence prediction method based on separable convolution
By adopting a separable convolution method in spatiotemporal sequence prediction, using multi-scale decomposition and dual-flow structure, combining spatiotemporal selective scanning and deep separable convolution, the problems of low parallel computing efficiency, insufficient error accumulation and long-distance relationship learning ability in the prior art are solved, and efficient and accurate spatiotemporal sequence prediction is achieved.
Patent Information
- Application Number
- CN202411680773.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-22
- Publication Date
- 2025-02-28
AI Technical Summary
The prior art has problems such as poor parallel computing efficiency, cumulative errors, insufficient learning ability of long-distance relationships, excessive calculation amount, and ignoring the modeling of multi-scale and local spatiotemporal relationships in the prediction of space-time sequences.
The spatiotemporal sequence prediction method based on separable convolution is used to decompose and model the spatiotemporal sequence through multi-scale decomposition layer and dual-flow prediction and multi-scale merging layer, and the global and local spatiotemporal relationships are learned using spatiotemporal selective scanning and spatiotemporal depth separation convolution.
It improves the parallel computing efficiency and accuracy of spatiotemporal sequence prediction, can effectively learn global and local spatiotemporal relationships, prevents spatiotemporal relationship losses under different spatial scales, and significantly improves the detailed texture accuracy of the prediction results.
Smart Images
Figure CN119538733A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of spatiotemporal sequence prediction, and in particular relates to a spatiotemporal sequence prediction method based on separable convolution. Background Art
[0002] Spatiotemporal sequence prediction is the process of predicting future spatiotemporal sequences by analyzing the dependencies of existing spatiotemporal sequences in time and space. The task of spatiotemporal sequence prediction is to find another spatiotemporal sequence based on the given spatiotemporal sequence, so that the two spatiotemporal sequences before and after maintain continuity in time and space, so as to predict the future spatiotemporal change trend. Spatiotemporal sequence data can be a video without audio data, or it can be a sequence of images collected by different sensors or radars at different time points over a period of time arranged in chronological order. Spatiotemporal sequence prediction technology is used in fields such as weather forecasting, traffic flow prediction, and autonomous driving. For example, in weather forecasting, the future rainfall intensity is predicted based on the radar echo sequence collected by the meteorological radar over a period of time to predict the future high-resolution rainfall and the occurrence of extreme weather; in traffic flow prediction, the future traffic flow is predicted based on the traffic flow map sequence collected by the traffic detector in a certain area over a period of time to help people better choose traffic routes.
[0003] Traditional spatiotemporal series prediction methods mainly use statistical and regression methods to deal with dependencies in time and space dimensions. These methods often assume that the dependencies between spatiotemporal series are linear, so it is difficult to deal with the highly nonlinear characteristics of spatiotemporal series data in reality. At the same time, traditional methods often require a lot of manual feature engineering to extract useful features, which not only relies on expert experience, but also cannot guarantee that the extracted features can fully express the complexity of the data.
[0004] Deep learning-based spatiotemporal series prediction methods can effectively learn complex nonlinear spatiotemporal relationships without making linear assumptions about the data like traditional methods, so they excel in dealing with nonlinear spatiotemporal relationships. At the same time, deep learning methods can automatically extract high-level features from data, with low reliance on manual feature engineering and without requiring excessive expert experience.
[0005] Most of the existing deep learning methods are based on recurrent neural networks, convolutional neural networks and transformers to learn and predict the spatiotemporal relationship of spatiotemporal sequences. However, since the reasoning process of recurrent neural networks is performed serially step by time, the methods based on recurrent neural networks often have poor parallel computing efficiency and are prone to error accumulation during the reasoning process; since the convolutional neural network has insufficient learning ability for long-distance relationships, the methods based on convolutional neural networks are difficult to predict spatiotemporal sequences with a longer time range; since the time complexity of the transformer is quadratic, when the transformer is used to process spatiotemporal sequences with higher resolution, the problem of excessive computation will be encountered. Therefore, we propose a method based on selective scanning, which has high parallel computing efficiency and linear time complexity, and has good long-distance relationship learning ability.
[0006] In addition, existing deep learning-based spatiotemporal sequence prediction methods often only explicitly model single-scale and global spatiotemporal relationships, while ignoring the modeling of multi-scale and local spatiotemporal relationships. Therefore, the present invention proposes a dual-stream structure and spatiotemporal depth separable convolution, the former is used to model spatiotemporal relationships at multiple spatial scales, and the latter is used to model local spatiotemporal relationships.
[0007] Existing technologies mainly rely on recurrent neural networks or Transformers for learning global spatiotemporal relationships. Methods based on recurrent neural networks often have poor parallel computing efficiency and are prone to error accumulation during the reasoning process. In tasks with more prediction time steps, the reasoning time is long and the accuracy is low. Transformer-based methods encounter the problem of excessive computation when processing high-resolution spatiotemporal sequences. In order to alleviate this problem, these methods often perform lightweight processing on the Transformer's attention mechanism, such as limiting the scope of attention or replacing linear operations with convolution operations. These processing will undoubtedly damage the Transformer's own ability to learn global spatiotemporal relationships.
[0008] Existing technologies often only explicitly learn the structure of global spatiotemporal relationships, but not the structure of local spatiotemporal relationships, which results in the model being unable to capture local spatiotemporal relationships well during the reasoning process, and ultimately makes the prediction results lack detailed texture and become blurred.
[0009] The existing technology mainly adopts a single-stream network structure to mix the features of various scales of the spatiotemporal sequence together for learning. This structure can learn the same spatiotemporal relationship at different spatial scales, but it is easy to cause the loss of the difference between the spatiotemporal relationships at different spatial scales. Summary of the invention
[0010] In view of the above-mentioned deficiencies in the prior art, the spatiotemporal sequence prediction method based on separable convolution provided by the present invention solves the following problems existing in the prior art: 1) Existing technologies mainly rely on recurrent neural networks or Transformers for learning global spatiotemporal relationships. Methods based on recurrent neural networks often have poor parallel computing efficiency and are prone to error accumulation during the reasoning process. In tasks with more prediction time steps, the reasoning time is long and the accuracy is low. Transformer-based methods encounter the problem of excessive computation when processing high-resolution spatiotemporal sequences. In order to alleviate this problem, these methods often perform lightweight processing on the Transformer's attention mechanism, such as limiting the scope of attention or replacing linear operations with convolution operations. These processing will undoubtedly damage the Transformer's own ability to learn global spatiotemporal relationships.
[0011] 2) Existing technologies often only explicitly learn the structure of global spatiotemporal relationships, but not the structure of local spatiotemporal relationships, which results in the model being unable to capture local spatiotemporal relationships well during the reasoning process, and ultimately makes the prediction results lack detailed texture and become blurred.
[0012] 3) Existing technologies mainly use a single-stream network structure to mix the features of various scales of the spatiotemporal sequence together for learning. This structure can learn the same spatiotemporal relationship at different spatial scales, but it is easy to cause the loss of the spatial-temporal relationship with differences between different spatial scales.
[0013] In order to achieve the above-mentioned invention object, the technical solution adopted by the present invention is: a spatiotemporal sequence prediction method based on separable convolution, comprising the following steps: S1, input the original spatiotemporal sequence into the encoder and obtain the output of the encoder; S2, performing multi-scale decomposition on the output of the encoder through a multi-scale decomposition layer to obtain a first spatiotemporal sequence and a second spatiotemporal sequence of different spatial scales; S3, learning the spatiotemporal relationship between the first spatiotemporal sequence and the second spatiotemporal sequence through dual-stream prediction and multi-scale merging layer, merging the learning results in multi-scale to obtain a sixth spatiotemporal sequence; S4. Input the sixth spatiotemporal sequence into a decoder and decode it into a final predicted spatiotemporal sequence.
[0014] Further: In S1, the encoder includes two layers The convolutional layer of the encoder is used to obtain the output of The specific expression is:
[0015] In the formula, is the input of the encoder, , is a matrix, T is the time step of the original space-time sequence, C is the first channel number, H For long, W For width, , is the second channel number, is the convolution operation, for The first convolution kernel of for The second convolution kernel.
[0016] Further: in S2, the multi-scale decomposition layer includes a first PatchMerge layer, a first spatiotemporal relationship learning module, and a second PatchMerge layer connected in sequence, the first PatchMerge layer is also connected to the encoder and the first input end of the dual-stream prediction and multi-scale merging layer respectively, and the second PatchMerge layer is also connected to the second input end of the dual-stream prediction and multi-scale merging layer; S2 specifically includes: inputting the output of the encoder into the first PatchMerge layer to obtain the first spatiotemporal sequence ;
[0017] In the formula, The downsampling factor is The first PatchMerge layer, , is the third channel number; The output of the encoder is sequentially input into the first PatchMerge layer, the first spatiotemporal relationship learning module, and the second PatchMerge layer to obtain the second spatiotemporal sequence ;
[0018] In the formula, The downsampling factor is The second PatchMerge layer, This is the first spatiotemporal relationship learning module. , Is the fourth channel number.
[0019] Further: the first spatiotemporal relationship learning module includes a first normalization layer, a spatiotemporal gated linear unit, a second normalization layer and a convolutional gated linear unit connected in sequence; The input of the first normalization layer is used as the input of the first spatiotemporal relationship learning module, the output of the spatiotemporal gated linear unit is spliced with the input of the first normalization layer through a jump connection and used as the input of the second normalization layer, and the input of the convolutional gated linear unit is spliced with the input of the second normalization layer through a jump connection and used as the output of the first spatiotemporal relationship learning module; Output of the first spatiotemporal relationship learning module The specific expression is:
[0020] In the formula, is the output of the spatiotemporal gated linear unit, is a convolutional gated linear unit, is the second normalization layer;
[0021] In the formula, is the input of the first spatiotemporal relationship learning module, is a spatiotemporal gated linear unit, is the first normalization layer.
[0022] Further: the spatiotemporal gated linear unit includes a first fully connected layer, a second fully connected layer, a third fully connected layer, spatiotemporal selective scanning, spatiotemporal depth-separable convolution, and a SiLU activation function; Among them, the input end of the first fully connected layer and the input end of the second fully connected layer are both used as the input of the spatiotemporal gated linear unit, the output end of the first fully connected layer is connected to the input end of the SiLU activation function, the output end of the second fully connected layer is connected to the input end of the spatiotemporal selective scanning through the spatiotemporal depthwise separable convolution, the output of the SiLU activation function and the output of the spatiotemporal selective scanning are multiplied element by element and input into the third fully connected layer, and the output of the third fully connected layer is used as the output of the spatiotemporal gated linear unit;
[0023] In the formula, is the first fully connected layer, is the second fully connected layer, is the third fully connected layer, is a spatiotemporal depthwise separable convolution, For spatiotemporal selective scanning, is the SiLU activation function, is element-wise multiplication.
[0024] Further: the spatiotemporal depth separable convolution includes a first time point-by-point convolution, a space-by-depth convolution and a second time point-by-point convolution connected in sequence, and the input of the space-by-depth convolution and the output of the space-by-depth convolution are spliced through a jump connection as the input of the second time point-by-point convolution; The structure of the first time point-by-point convolution is the same as that of the second time point-by-point convolution. The method for processing the first input spatiotemporal sequence and generating the output of the time point-by-point convolution is as follows: The same channels of all frames of the first input spatiotemporal sequence are spliced, and the spliced results are spliced again in channel order to obtain the first feature map. The convolution kernel size is The grouped convolution operation is performed, where the number of groups is the number of channels, to learn the temporal relationship between the same channels of different frames of the input, and obtain the output of the temporal point-by-point convolution; the method of processing the second input spatiotemporal sequence by spatial convolution and generating the output of spatial depth-by-depth convolution is specifically as follows: All frames of the second input spatiotemporal sequence are spliced in time order to obtain a second feature map, and the second feature map is convolved with a kernel size of A grouped convolution operation is performed, where the group is the product of the time dimension size and the number of channels, to learn the spatial relationship within a single channel and obtain the output of the spatial depth-wise convolution.
[0025] Further: the method of processing the third input spatiotemporal sequence by spatiotemporal selective scanning and generating the output of spatiotemporal selective scanning is specifically: The patches of each frame of the third input spatiotemporal sequence are arranged in a preset direction, and the arrangements of all frames in the same direction are connected in series in time order to obtain four sequences, and the four sequences are respectively subjected to one-dimensional selective scanning, and the scanning results of the four sequences are combined to obtain the output of spatiotemporal selective scanning; Among them, the method of arranging according to a preset direction includes: arranging from left to right, from top to bottom; arranging from top to bottom, from left to right; arranging from right to left, from bottom to top; arranging from bottom to top, from right to left.
[0026] Further: in S3, the dual-stream prediction and multi-scale merging layer includes an M-layer spatiotemporal relationship learning module, an N-layer spatiotemporal relationship learning module, a nearest neighbor interpolation layer, a connection layer and a second spatiotemporal relationship learning module; Among them, one end of the N-layer spatiotemporal relationship learning module serves as the first input end of the dual-stream prediction and multi-scale merging layer, the other end of the N-layer spatiotemporal relationship learning module is connected to one end of the second spatiotemporal relationship learning module through the connection layer, the other end of the second spatiotemporal relationship learning module serves as the output end of the dual-stream prediction and multi-scale merging layer, one end of the M-layer spatiotemporal relationship learning module serves as the second input end of the dual-stream prediction and multi-scale merging layer, and the other end of the M-layer spatiotemporal relationship learning module is connected to the connection layer; The S3 comprises the following sub-steps: S31, inputting the first space-time sequence into an N-layer space-time relationship learning module to learn the space-time relationship, thereby obtaining a third space-time sequence; S32, inputting the second space-time sequence into the M-layer space-time relationship learning module to learn the space-time relationship, thereby obtaining a fourth space-time sequence; S33, input the fourth space-time sequence into the nearest neighbor interpolation layer to obtain the fifth space-time sequence ;
[0027] In the formula, is the third convolution kernel of 3×3, is the nearest neighbor interpolation layer, For the fourth space-time sequence, ; S34, multi-scale merging the fifth spatiotemporal sequence with the third spatiotemporal sequence through a connection layer, and inputting the merging result into the second spatiotemporal relationship learning module to obtain the sixth spatiotemporal sequence ;
[0028] In the formula, for The fourth convolution kernel, is the connection layer, is the third space-time sequence, .
[0029] Further: In S4, the decoder includes two layers The convolutional layer and the nearest neighbor interpolation layer are used to obtain the final predicted spatiotemporal sequence The specific expression is:
[0030] In the formula, for The sixth convolution kernel, for The fifth convolution kernel.
[0031] The beneficial effects of the present invention are: (1) The present invention provides a spatiotemporal sequence prediction method based on separable convolution. In order to verify the effect of the present invention, experiments were conducted on a spatiotemporal sequence prediction data set. From the prediction accuracy of each time step, the accuracy of the present invention decreases slowly with the increase of the time step, indicating that the spatiotemporal selective scanning of the present invention can well learn the global spatiotemporal relationship. In addition, compared with using only a single-stream structure, the accuracy of the present invention is improved by more than 1% after adding a dual-stream structure, which indicates that the dual-stream structure can indeed learn the spatiotemporal relationship with differences between different spatial scales. Compared with not using spatiotemporal depth separable convolution, the accuracy of the present invention is improved by more than 30% after adding a spatiotemporal depth separable convolution, which indicates that the spatiotemporal depth separable convolution can indeed enhance the model's ability to learn local spatiotemporal relationships. The above experimental data all show that the present invention can further improve the model's ability to learn spatiotemporal relationships and predict spatiotemporal sequences.
[0032] (2) The dual-stream spatiotemporal sequence prediction network proposed in the present invention includes an encoder, a multi-scale decomposition layer, a dual-stream prediction and multi-scale merging layer and a decoder connected in sequence, which can prevent the loss of spatiotemporal relationships with differences at different spatial scales. Compared with the single-stream network that mixes the features of various scales of the spatiotemporal sequence together for learning, the dual-stream spatiotemporal sequence prediction network explicitly decomposes the spatiotemporal sequence into two spatial scales, and models the spatiotemporal relationships at the two scales respectively, which can prevent the loss of spatiotemporal relationships with differences at different spatial scales.
[0033] (3) This paper proposes spatiotemporal selective scanning (STSS) as a new way to model global spatiotemporal relationships. It has linear time complexity and good parallel computing efficiency. It is not prone to error accumulation problems like recurrent neural networks when predicting more time steps. When processing high-resolution spatiotemporal sequences, it does not need to maintain computing efficiency by compromising long-distance relationship learning capabilities like Transformer. Spatiotemporal sequences contain one-dimensional temporal characteristics and two-dimensional spatial characteristics. Spatiotemporal selective scanning can explicitly take into account the characteristics of spatiotemporal sequences in these three dimensions and learn global spatiotemporal relationships.
[0034] (4) The present invention proposes a spatiotemporal depth separable convolution (STDSConv) to improve the accuracy of the prediction results on detailed textures. The spatiotemporal depth separable convolution consists of temporal point-by-point convolution and spatial depth-by-depth convolution. The temporal point-by-point convolution only learns the temporal relationship between the same channels of all frames in a single operation, and the spatial depth-by-depth convolution only learns the spatial relationship within a single channel of a single frame in a single operation. The spatiotemporal depth separable convolution explicitly models the local spatiotemporal relationship, which can prevent the loss of local spatiotemporal relationship during the reasoning process, so as to improve the accuracy of the prediction results on detailed textures. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] Figure 1 It is a flow chart of the spatiotemporal series prediction method based on separable convolution of the present invention; Figure 2 Schematic diagram of a dual-stream spatiotemporal sequence prediction network of the present invention; Figure 3 is a schematic diagram of a spatiotemporal relationship learning module of the present invention; Figure 4 Schematic diagram of the spatiotemporal depth-separable convolution of the present invention; Figure 5 Schematic diagram of spatiotemporal selective scanning of the present invention. DETAILED DESCRIPTION
[0036] The specific implementation modes of the present invention are described below so that those skilled in the art can understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.
[0037] like Figure 1 As shown, in one embodiment of the present invention, a spatiotemporal sequence prediction method based on separable convolution includes the following steps: S1, input the original spatiotemporal sequence into the encoder and obtain the output of the encoder; S2, performing multi-scale decomposition on the output of the encoder through a multi-scale decomposition layer to obtain a first spatiotemporal sequence and a second spatiotemporal sequence of different spatial scales; S3, learning the spatiotemporal relationship between the first spatiotemporal sequence and the second spatiotemporal sequence through dual-stream prediction and multi-scale merging layer, merging the learning results in multi-scale to obtain a sixth spatiotemporal sequence; S4. Input the sixth spatiotemporal sequence into a decoder and decode it into a final predicted spatiotemporal sequence.
[0038] In S1, the encoder consists of two layers The convolutional layer of the encoder is used to obtain the output of The specific expression is:
[0039] In the formula, is the input of the encoder, , is a matrix, T is the time step of the original space-time sequence, C is the first channel number, H For long, W For width, , is the second channel number, is the convolution operation, for The first convolution kernel of for The second convolution kernel.
[0040] In this embodiment, the original spatiotemporal sequence has time steps, each time step exists channels with length and width respectively and ,like Figure 2 As shown, Encoded by the encoder, we get Patch, that is .
[0041] In S2, the multi-scale decomposition layer includes a first PatchMerge layer, a first spatiotemporal relationship learning module, and a second PatchMerge layer connected in sequence, the first PatchMerge layer is also connected to the encoder and the first input end of the dual-stream prediction and multi-scale merging layer respectively, and the second PatchMerge layer is also connected to the second input end of the dual-stream prediction and multi-scale merging layer; In this embodiment, the dual-stream spatiotemporal sequence prediction network of the present invention is as follows Figure 2 As shown, it includes an encoder, a multi-scale decomposition layer, a dual-stream prediction and multi-scale merging layer and a decoder connected in sequence; In order to enable the network to model the spatiotemporal relationships of different spatial scales separately, the output of the encoder needs to be After decomposition, we obtain two space-time series with different spatial scales.
[0042] S2 specifically includes: inputting the output of the encoder into the first PatchMerge layer to obtain the first spatiotemporal sequence ;
[0043] In the formula, The downsampling factor is The first PatchMerge layer, , is the third channel number; The output of the encoder is sequentially input into the first PatchMerge layer, the first spatiotemporal relationship learning module, and the second PatchMerge layer to obtain the second spatiotemporal sequence ;
[0044] In the formula, The downsampling factor is The second PatchMerge layer, This is the first spatiotemporal relationship learning module. , Is the fourth channel number.
[0045] The first spatiotemporal relationship learning module includes a first normalization layer, a spatiotemporal gated linear unit, a second normalization layer, and a convolutional gated linear unit connected in sequence; The input of the first normalization layer is used as the input of the first spatiotemporal relationship learning module, the output of the spatiotemporal gated linear unit is spliced with the input of the first normalization layer through a jump connection and used as the input of the second normalization layer, and the input of the convolutional gated linear unit is spliced with the input of the second normalization layer through a jump connection and used as the output of the first spatiotemporal relationship learning module; like Figure 3 As shown, in this embodiment, in order to enable the network to explicitly learn local and global spatiotemporal relationships at the same time, the present invention constructs a spatiotemporal relationship learning module (STBlock). The spatiotemporal relationship learning module as a whole is composed of a spatiotemporal gated linear unit responsible for modeling the relationship between patches, and a convolutional gated linear unit (ConvGLU) responsible for modeling the internal relationship of patches. The spatiotemporal gated linear unit is mainly composed of a spatiotemporal deep separable convolution (STDSConv) responsible for modeling local spatiotemporal relationships and a spatiotemporal selective scanning (STSS) responsible for modeling global spatiotemporal relationships.
[0046] Output of the first spatiotemporal relationship learning module The specific expression is:
[0047] In the formula, is the output of the spatiotemporal gated linear unit, is a convolutional gated linear unit, is the second normalization layer;
[0048] In the formula, is the input of the first spatiotemporal relationship learning module, is a spatiotemporal gated linear unit, is the first normalization layer.
[0049] The spatiotemporal gated linear unit includes a first fully connected layer, a second fully connected layer, a third fully connected layer, spatiotemporal selective scanning, spatiotemporal depth separable convolution, and a SiLU activation function; Among them, the input end of the first fully connected layer and the input end of the second fully connected layer are both used as the input of the spatiotemporal gated linear unit, the output end of the first fully connected layer is connected to the input end of the SiLU activation function, the output end of the second fully connected layer is connected to the input end of the spatiotemporal selective scanning through the spatiotemporal depthwise separable convolution, the output of the SiLU activation function and the output of the spatiotemporal selective scanning are multiplied element by element and input into the third fully connected layer, and the output of the third fully connected layer is used as the output of the spatiotemporal gated linear unit;
[0050] In the formula, is the first fully connected layer, is the second fully connected layer, is the third fully connected layer, is a spatiotemporal depthwise separable convolution, For spatiotemporal selective scanning, is the SiLU activation function, is element-wise multiplication.
[0051] The spatiotemporal depth separable convolution includes a first time point-by-point convolution, a space-by-depth convolution, and a second time point-by-point convolution connected in sequence, and the input of the space-by-depth convolution and the output of the space-by-depth convolution are spliced through a jump connection as the input of the second time point-by-point convolution; The structure of the first time point-by-point convolution is the same as that of the second time point-by-point convolution. The method for processing the first input spatiotemporal sequence and generating the output of the time point-by-point convolution is as follows: The same channels of all frames of the first input spatiotemporal sequence are spliced, and the spliced results are spliced again in channel order to obtain the first feature map. The convolution kernel size is A grouped convolution operation is performed, where the number of groups is the number of channels, to learn the temporal relationship between the same channels of different input frames and obtain the output of temporal point-by-point convolution; The method of processing the second input spatiotemporal sequence by spatial depth-wise convolution and generating the output of spatial depth-wise convolution is specifically as follows: All frames of the second input spatiotemporal sequence are spliced in time order to obtain a second feature map, and the second feature map is convolved with a kernel size of A grouped convolution operation is performed, where the group is the product of the time dimension size and the number of channels, to learn the spatial relationship within a single channel and obtain the output of the spatial depth-wise convolution.
[0052] like Figure 4As shown, in this embodiment, the spatiotemporal depthwise separable convolution (STDSConv) is composed of time point-by-point convolution and space depth-by-depth convolution. The workflow of time point-by-point convolution is as follows: the same channels of different input frames are spliced together, and then the spliced results are spliced together in the order of channels to obtain a feature map, and then the feature map is grouped into groups equal to the number of channels. Convolution operation is performed to learn the temporal relationship between the same channels of each input frame, and finally the result of the convolution is restored to the shape of the input of the time point-by-point convolution.
[0053] The workflow of spatial depth-by-depth convolution is as follows: different input frames are spliced together in time order to obtain a feature map, and then the feature map is grouped into groups whose number is the product of the time dimension size and the number of channels. Convolution operations are performed to learn the spatial relationship within a single channel, and the final convolution result is restored to the shape of the input of the spatial depth-wise convolution.
[0054] The method of processing the third input spatiotemporal sequence by spatiotemporal selective scanning and generating the output of the spatiotemporal selective scanning is specifically: The patches of each frame of the third input spatiotemporal sequence are arranged in a preset direction, and the arrangements of all frames in the same direction are connected in series in time order to obtain four sequences, and the four sequences are respectively subjected to one-dimensional selective scanning, and the scanning results of the four sequences are combined to obtain the output of spatiotemporal selective scanning; Among them, the method of arranging according to a preset direction includes: arranging from left to right, from top to bottom; arranging from top to bottom, from left to right; arranging from right to left, from bottom to top; arranging from bottom to top, from right to left.
[0055] like Figure 5 As shown, in this embodiment, the spatiotemporal selective scanning explicitly takes into account the one-dimensional time and two-dimensional space characteristics of the spatiotemporal sequence. The specific workflow of the spatiotemporal selective scanning is: the patches of each input frame image are arranged in four directions, and then the arrangements of all frames in the same direction are connected in series in time order to obtain four sequences, such as Figure 5 Seq1, Seq2, Seq3 and Seq4 are shown, and then these four sequences are subjected to one-dimensional selective scanning respectively, and finally the four output sequences are merged into the same shape as the input of the spatiotemporal selective scanning module.
[0056] In S3, the dual-stream prediction and multi-scale merging layer includes an M-layer spatiotemporal relationship learning module, an N-layer spatiotemporal relationship learning module, a nearest neighbor interpolation layer, a connection layer, and a second spatiotemporal relationship learning module; Among them, one end of the N-layer spatiotemporal relationship learning module serves as the first input end of the dual-stream prediction and multi-scale merging layer, the other end of the N-layer spatiotemporal relationship learning module is connected to one end of the second spatiotemporal relationship learning module through the connection layer, the other end of the second spatiotemporal relationship learning module serves as the output end of the dual-stream prediction and multi-scale merging layer, one end of the M-layer spatiotemporal relationship learning module serves as the second input end of the dual-stream prediction and multi-scale merging layer, and the other end of the M-layer spatiotemporal relationship learning module is connected to the connection layer; The S3 comprises the following sub-steps: S31, inputting the first space-time sequence into an N-layer space-time relationship learning module to learn the space-time relationship, thereby obtaining a third space-time sequence; S32, inputting the second space-time sequence into the M-layer space-time relationship learning module to learn the space-time relationship, thereby obtaining a fourth space-time sequence; S33, input the fourth space-time sequence into the nearest neighbor interpolation layer to obtain the fifth space-time sequence ;
[0057] In the formula, is the third convolution kernel of 3×3, is the nearest neighbor interpolation layer, For the fourth space-time sequence, ; S34, multi-scale merging the fifth spatiotemporal sequence with the third spatiotemporal sequence through a connection layer, and inputting the merging result into the second spatiotemporal relationship learning module to obtain the sixth spatiotemporal sequence ;
[0058] In the formula, for The fourth convolution kernel, is the connection layer, is the third space-time sequence, .
[0059] In this embodiment, the first spatiotemporal relationship learning module and the second spatiotemporal relationship learning module have the same structure as the spatiotemporal relationship learning modules in the M-layer spatiotemporal relationship learning module and the N-layer spatiotemporal relationship learning module, and the method of processing input data is the same, so they are not repeated here.
[0060] In S4, the decoder includes two layers The convolutional layer and the nearest neighbor interpolation layer are used to obtain the final predicted spatiotemporal sequence The specific expression is:
[0061] In the formula, for The sixth convolution kernel, for The fifth convolution kernel.
[0062] The beneficial effects of the present invention are as follows: the present invention provides a spatiotemporal series prediction method based on separable convolution. In order to verify the effect of the present invention, experiments were conducted on a spatiotemporal series prediction data set. From the prediction accuracy of each time step, the accuracy of the present invention decreases slowly with the increase of the time step, indicating that the spatiotemporal selective scanning of the present invention can learn the global spatiotemporal relationship well. In addition, compared with using only a single-stream structure, the accuracy of the present invention is improved by more than 1% after adding a dual-stream structure, which indicates that the dual-stream structure can indeed learn the spatiotemporal relationship with differences between different spatial scales. Compared with not using spatiotemporal depth separable convolution, the accuracy of the present invention is improved by more than 30% after adding a spatiotemporal depth separable convolution, which indicates that the spatiotemporal depth separable convolution can indeed enhance the model's learning ability of local spatiotemporal relationships. The above experimental data all show that the present invention can further improve the model's spatiotemporal relationship learning ability and spatiotemporal series prediction ability.
[0063] The dual-stream spatiotemporal sequence prediction network proposed in the present invention includes an encoder, a multi-scale decomposition layer, a dual-stream prediction and multi-scale merging layer and a decoder connected in sequence, which can prevent the loss of spatial-temporal relationships with differences at different spatial scales. Compared with a single-stream network that mixes the features of various scales of the spatiotemporal sequence together for learning, the dual-stream spatiotemporal sequence prediction network explicitly decomposes the spatiotemporal sequence into two spatial scales, and models the spatiotemporal relationships at the two scales respectively, which can prevent the loss of spatial-temporal relationships with differences at different spatial scales.
[0064] This paper proposes spatiotemporal selective scanning (STSS) as a new global spatiotemporal relationship modeling method. Spatiotemporal selective scanning has linear time complexity and good parallel computing efficiency. When there are many prediction time steps, it will not easily have the problem of error accumulation like recurrent neural networks. When processing high-resolution spatiotemporal sequences, it does not need to maintain computing efficiency by damaging the long-distance relationship learning ability like Transformer. Spatiotemporal sequences contain one-dimensional time characteristics and two-dimensional spatial characteristics. Spatiotemporal selective scanning can explicitly take into account the characteristics of spatiotemporal sequences in these three dimensions and learn global spatiotemporal relationships.
[0065] The present invention proposes a spatiotemporal depth separable convolution (STDSConv) to improve the accuracy of prediction results on detailed textures. The spatiotemporal depth separable convolution consists of temporal point-by-point convolution and spatial depth-by-depth convolution. The temporal point-by-point convolution only learns the temporal relationship between the same channels of all frames in a single operation, and the spatial depth-by-depth convolution only learns the spatial relationship within a single channel of a single frame in a single operation. The spatiotemporal depth separable convolution explicitly models the local spatiotemporal relationship, which can prevent the loss of local spatiotemporal relationship during the reasoning process, so as to improve the accuracy of the prediction results on detailed textures.
[0066] In the description of the present invention, it is necessary to understand that the orientation or positional relationship indicated by the terms "center", "thickness", "upper", "lower", "horizontal", "top", "bottom", "inner", "outer", "radial", etc. is based on the orientation or positional relationship shown in the drawings, and is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, the terms "first", "second", and "third" are used only for descriptive purposes, and cannot be understood as indicating or implying the relative importance or the number of implicitly specified technical features. Therefore, the features defined by "first", "second", and "third" may explicitly or implicitly include one or more of the features.
Claims
1. A spatiotemporal series prediction method based on separable convolution, characterized in that: The following steps are involved: S1, input the original spatiotemporal sequence into the encoder to obtain the output of the encoder; S2, performing multi-scale decomposition on the output of the encoder through a multi-scale decomposition layer to obtain a first spatiotemporal sequence and a second spatiotemporal sequence of different spatial scales; S3, learning the spatiotemporal relationship between the first spatiotemporal sequence and the second spatiotemporal sequence through dual-stream prediction and multi-scale merging layer, merging the learning results in multi-scale to obtain a sixth spatiotemporal sequence; S4. Input the sixth spatiotemporal sequence into a decoder and decode it into a final predicted spatiotemporal sequence.
2. The method for spatiotemporal series prediction based on separable convolution according to claim 1, characterized in that: In S1, the encoder consists of two layers The convolutional layer of the encoder is used to obtain the output of The specific expression is: In the formula, is the input of the encoder, , is a matrix, T is the time step of the original space-time sequence, C is the first channel number, H For long, W For width, , is the second channel number, is the convolution operation, for The first convolution kernel of for The second convolution kernel.
3. The method for spatiotemporal series prediction based on separable convolution according to claim 2, characterized in that: In S2, the multi-scale decomposition layer includes a first PatchMerge layer, a first spatiotemporal relationship learning module, and a second PatchMerge layer connected in sequence, the first PatchMerge layer is also connected to the encoder and the first input end of the dual-stream prediction and multi-scale merging layer respectively, and the second PatchMerge layer is also connected to the second input end of the dual-stream prediction and multi-scale merging layer; S2 specifically includes: inputting the output of the encoder into the first PatchMerge layer to obtain the first spatiotemporal sequence ; In the formula, The downsampling factor is The first PatchMerge layer, , is the third channel number; The output of the encoder is sequentially input into the first PatchMerge layer, the first spatiotemporal relationship learning module, and the second PatchMerge layer to obtain the second spatiotemporal sequence ; In the formula, The downsampling factor is The second PatchMerge layer, This is the first spatiotemporal relationship learning module. , Is the fourth channel number.
4. The method for spatiotemporal series prediction based on separable convolution according to claim 3, characterized in that: The first spatiotemporal relationship learning module includes a first normalization layer, a spatiotemporal gated linear unit, a second normalization layer, and a convolutional gated linear unit connected in sequence; The input of the first normalization layer is used as the input of the first spatiotemporal relationship learning module, the output of the spatiotemporal gated linear unit is spliced with the input of the first normalization layer through a jump connection and used as the input of the second normalization layer, and the input of the convolutional gated linear unit is spliced with the input of the second normalization layer through a jump connection and used as the output of the first spatiotemporal relationship learning module; Output of the first spatiotemporal relationship learning module The specific expression is: In the formula, is the output of the spatiotemporal gated linear unit, is a convolutional gated linear unit, is the second normalization layer; In the formula, is the input of the first spatiotemporal relationship learning module, is a spatiotemporal gated linear unit, is the first normalization layer.
5. The method for spatiotemporal series prediction based on separable convolution according to claim 4, characterized in that: The spatiotemporal gated linear unit includes a first fully connected layer, a second fully connected layer, a third fully connected layer, spatiotemporal selective scanning, spatiotemporal depth separable convolution, and a SiLU activation function; Among them, the input end of the first fully connected layer and the input end of the second fully connected layer are both used as the input of the spatiotemporal gated linear unit, the output end of the first fully connected layer is connected to the input end of the SiLU activation function, the output end of the second fully connected layer is connected to the input end of the spatiotemporal selective scanning through the spatiotemporal depthwise separable convolution, the output of the SiLU activation function and the output of the spatiotemporal selective scanning are multiplied element by element and input into the third fully connected layer, and the output of the third fully connected layer is used as the output of the spatiotemporal gated linear unit; In the formula, is the first fully connected layer, is the second fully connected layer, is the third fully connected layer, is a spatiotemporal depthwise separable convolution, For spatiotemporal selective scanning, is the SiLU activation function, is element-wise multiplication.
6. The method for spatiotemporal series prediction based on separable convolution according to claim 5, characterized in that: The spatiotemporal depth separable convolution includes a first time point-by-point convolution, a space-by-depth convolution, and a second time point-by-point convolution connected in sequence, and the input of the space-by-depth convolution and the output of the space-by-depth convolution are spliced through a jump connection as the input of the second time point-by-point convolution; The structure of the first time point-by-point convolution is the same as that of the second time point-by-point convolution. The method for processing the first input spatiotemporal sequence and generating the output of the time point-by-point convolution is as follows: The same channels of all frames of the first input spatiotemporal sequence are spliced, and the spliced results are spliced again in channel order to obtain the first feature map. The convolution kernel size is A grouped convolution operation is performed, where the number of groups is the number of channels, to learn the temporal relationship between the same channels of different input frames and obtain the output of temporal point-by-point convolution; The method of processing the second input spatiotemporal sequence by spatial depth-wise convolution and generating the output of spatial depth-wise convolution is specifically as follows: All frames of the second input spatiotemporal sequence are spliced in time order to obtain a second feature map, and the second feature map is convolved with a kernel size of A grouped convolution operation is performed, where the group is the product of the time dimension size and the number of channels, to learn the spatial relationship within a single channel and obtain the output of the spatial depth-wise convolution.
7. The method for spatiotemporal series prediction based on separable convolution according to claim 5, characterized in that: The method of processing the third input spatiotemporal sequence by spatiotemporal selective scanning and generating the output of the spatiotemporal selective scanning is specifically: The patches of each frame of the third input spatiotemporal sequence are arranged in a preset direction, and the arrangements of all frames in the same direction are connected in series in time order to obtain four sequences, and the four sequences are respectively subjected to one-dimensional selective scanning, and the scanning results of the four sequences are combined to obtain the output of spatiotemporal selective scanning; Among them, the method of arranging according to a preset direction includes: arranging from left to right, from top to bottom; arranging from top to bottom, from left to right; arranging from right to left, from bottom to top; arranging from bottom to top, from right to left.
8. The method for spatiotemporal series prediction based on separable convolution according to claim 3, characterized in that: In S3, the dual-stream prediction and multi-scale merging layer includes an M-layer spatiotemporal relationship learning module, an N-layer spatiotemporal relationship learning module, a nearest neighbor interpolation layer, a connection layer, and a second spatiotemporal relationship learning module; Among them, one end of the N-layer spatiotemporal relationship learning module serves as the first input end of the dual-stream prediction and multi-scale merging layer, the other end of the N-layer spatiotemporal relationship learning module is connected to one end of the second spatiotemporal relationship learning module through the connection layer, the other end of the second spatiotemporal relationship learning module serves as the output end of the dual-stream prediction and multi-scale merging layer, one end of the M-layer spatiotemporal relationship learning module serves as the second input end of the dual-stream prediction and multi-scale merging layer, and the other end of the M-layer spatiotemporal relationship learning module is connected to the connection layer; The S3 comprises the following sub-steps: S31, inputting the first space-time sequence into an N-layer space-time relationship learning module to learn the space-time relationship, thereby obtaining a third space-time sequence; S32, inputting the second space-time sequence into the M-layer space-time relationship learning module to learn the space-time relationship, thereby obtaining a fourth space-time sequence; S33, input the fourth space-time sequence into the nearest neighbor interpolation layer to obtain the fifth space-time sequence ; In the formula, is the third convolution kernel of 3×3, is the nearest neighbor interpolation layer, For the fourth space-time sequence, ; S34, multi-scale merging the fifth spatiotemporal sequence with the third spatiotemporal sequence through a connection layer, and inputting the merging result into the second spatiotemporal relationship learning module to obtain the sixth spatiotemporal sequence ; In the formula, for The fourth convolution kernel, is the connection layer, is the third space-time sequence, .
9. The method for spatiotemporal series prediction based on separable convolution according to claim 8, characterized in that: In S4, the decoder consists of two layers The convolutional layer and the nearest neighbor interpolation layer are used to obtain the final predicted spatiotemporal sequence The specific expression is: In the formula, for The sixth convolution kernel, for The fifth convolution kernel.