A lightweight fine-grained spatiotemporal series prediction method and system
By constructing a space-time detail extraction module and an improved motion detail recurrent neural network, the fuzzy problem of existing space-time prediction models in image detail texture feature prediction is solved, and the model's ability to extract and predict fine-grained features is improved, especially in complex Performance in scenarios.
Patent Information
- Application Number
- CN202210101519.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-01-27
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2042-01-27
AI Technical Summary
The existing spatiotemporal prediction model based on long and short-term memory networks has a significant fuzzy trend in the prediction and retention of image detail texture features, and the excessive parameters introduced by the additional gating unit limit the use of the model in complex scenarios.
By building a spatiotemporal detail extraction module, using multi-layer stacked fine-grained spatiotemporal module and shared convolutional layer parameters, the model structure of the timing prediction layer is improved, the spatiotemporal feature representation ability is improved, the motion detail recurrent neural network MoDeRNN is realized, and the context states are enhanced. Prediction of correlation and fine-grained features.
The extraction and prediction of fine-grained spatiotemporal features by timing prediction networks has been significantly improved, and the accuracy of the model's detailed feature extraction and prediction in digital motion and meteorological predictions is improved, especially in the retention of texture details and object features.
Smart Images

Figure CN114445463B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to spatiotemporal prediction, and in particular to a lightweight fine-grained spatiotemporal sequence prediction method and system. Background Art
[0002] The spatiotemporal prediction task refers to predicting the changing trend of images in the future based on a given sequence of observed images, such as predicting the movement of objects, the changing trend of weather, etc. With the advancement and development of predictive learning, the spatiotemporal prediction task has broad practical application prospects. At present, the model structure based on the long short-term memory network is a common method in this task. Its model structure consists of an encoding layer, a time series prediction layer, and a decoding layer. Among them, the multi-layer stacked time series prediction network constitutes the basic structure of the time series prediction layer, which is used to extract high-level spatiotemporal features and predict subsequent states.
[0003] The time series prediction layer is usually composed of multiple layers of stacked recurrent neural networks (RNNs) or long short-term memory networks (LSTM) units. The current mainstream improvement direction is to improve the performance of the model in spatiotemporal prediction tasks by improving the representation ability of these RNN units for spatiotemporal features. Among them, the convolutional-long short-term memory network (ConvLSTM) is the most basic network. Most researchers design additional gating units based on this to improve the performance of the model.
[0004] Previous improved structures mostly use additional gating calculations to enhance the model's expressiveness in spatiotemporal features. However, network structures based on additional gating units usually introduce a large number of additional parameters, which limits the use of these models in complex scenarios.
[0005] It should be noted that the information disclosed in the above background technology section is only used for understanding the background of the present application, and therefore may include information that does not constitute prior art known to ordinary technicians in the field. Summary of the invention
[0006] Through further observation, it is found that these network models are usually difficult to predict and retain image detail texture features, and the prediction results of the model will show a significant trend of detail feature blurring over time. The present invention provides a lightweight fine-grained spatiotemporal sequence prediction method and system, which improves the spatiotemporal feature representation capability of the time series prediction layer through a lightweight model structure, enhances the deep interaction of context between spatiotemporal sequences, and thus achieves fine-grained spatiotemporal prediction goals.
[0007] To achieve the above object, the present invention adopts the following technical solutions:
[0008] A lightweight fine-grained spatiotemporal series prediction method, including:
[0009] Constructing a spatiotemporal detail extraction module, so that the spatiotemporal state updated by the spatiotemporal detail extraction module not only contains fine-grained features, but also improves the correlation between context states through deep interaction of contexts;
[0010] The spatiotemporal detail extraction module is used in the spatiotemporal prediction network to realize the motion detail recurrent neural network unit, and the deep context features are extracted with the help of multi-layer stacked fine-grained spatiotemporal modules to construct the final motion detail recurrent neural network MoDeRNN; wherein, by improving the model structure of the spatiotemporal prediction layer, the prediction results of the model in fine-grained features are improved.
[0011] Further:
[0012] The multi-layer stacked fine-grained spatiotemporal modules are stacked in a manner of sharing convolutional layer parameters.
[0013] The state transition equation of the motion detail recurrent neural network predicting the output state at the next moment is:
[0014]
[0015]
[0016]
[0017]
[0018]
[0019]
[0020]
[0021] Among them, Attn H represents the weight matrix, W (·) represents a convolutional layer of a specific filter size, represents the weight of the i×i convolutional layer of the filter, b (·) Indicates bias, X t Indicates the input status at the current moment. Indicates the updated input status at the current moment, H t-1 and C t-1 are the output state and memory state at time t-1, respectively, H t and C t are the output state and memory state at time t, s is a constant greater than or equal to 1, i t is the input gating, f t is the forget gate, g t is the input value, o tis the output gating, σ is the sigmoid activation function, k is the number of convolutional layers with different filter sizes, ★ refers to the convolution operation, is the Hadamard product of matrices.
[0022] The processing process of the spatiotemporal detail extraction module includes the following steps:
[0023] First, for the output state H at the previous moment t-1 First, multiple convolutional layers with different filter sizes are used to extract the spatiotemporal information in different neighborhoods at the previous moment, and then the addition operation and mean operation of the channel dimension are used to perform deep multi-scale feature fusion to obtain the fine-grained spatiotemporal feature z of the output state at the previous moment. H
[0024]
[0025] Where k represents the number of convolutional layers. Represents the weight of the i×i convolutional layer of the filter;
[0026] Subsequently, the fine-grained spatiotemporal features of the previous moment are used to generate a weight matrix, preferably mapped to a 0-1 weight matrix Attn through a Sigmoid activation function H , so that the area with larger weight reflects the more important part of the short-term memory state at the previous moment,
[0027]
[0028] With the help of the extracted weight matrix, the input state X at the current moment is multiplied by the channel dimension t Update the current state input information so that it can reflect the important part of the past state in the current input, thus obtaining the updated current input state
[0029]
[0030] Where s is a constant greater than or equal to 1;
[0031] For the updated current status Similarly, convolutional layers with different filter sizes are used to extract spatiotemporal information on different neighborhoods, and deep feature fusion is performed through the addition mean operation to obtain the fine-grained feature z of the current state. X
[0032]
[0033] in, Represents the weight of the i×i convolutional layer of the filter;
[0034] Then, the fine-grained spatiotemporal features of the current moment are mapped to a 0-1 weight matrix Attn through the Sigmoid activation function X , so that the area with a larger weight reflects the more important part of the input state at the current moment, that is, the potential movement trend of the model in the movement process at the subsequent moment,
[0035]
[0036] With the help of the extracted weight matrix, the output state H of the previous moment is multiplied by the channel dimension t-1 Update so that the input information of the current state can contain the potential movement trend of the current input state, so as to obtain the updated output state of the previous moment
[0037]
[0038] Through the above steps, the output state of the spatiotemporal detail extraction module is obtained. as well as
[0039] The convolutional layers with different receptive fields are used to extract the spatiotemporal features of the input state and the previous state in different neighborhoods.
[0040] The spatiotemporal prediction network includes a convolutional long short-term memory network ConvLSTM.
[0041] The lightweight fine-grained spatiotemporal sequence prediction method is used to predict the motion of objects.
[0042] The lightweight fine-grained spatiotemporal series prediction method is used to predict the changing trend of meteorological conditions, and preferably, is used for typhoon meteorological prediction.
[0043] A lightweight fine-grained spatiotemporal series prediction system, including:
[0044] processor;
[0045] A computer-readable storage medium, wherein the computer program stored therein, when executed by a processor, implements the lightweight fine-grained spatiotemporal sequence prediction method.
[0046] A computer-readable storage medium, wherein the computer program stored in the medium, when executed by a processor, implements the lightweight fine-grained spatiotemporal sequence prediction method.
[0047] The present invention has the following beneficial effects:
[0048] The present invention can effectively improve the existing solutions for the defect of lack of detailed features in the prediction results of the model, and significantly improves the extraction and prediction of fine-grained spatiotemporal features by the time series prediction network. The spatiotemporal detail extraction module of the present invention can significantly focus on the fine-grained change features in the spatiotemporal sequence, so that the model has better feature extraction of detailed features in the input sequence, and significantly enhances the expression of contextual features in detailed features.
[0049] The motion detail recurrent neural network provided by the present invention effectively improves the prediction results of the model on digital details in the digital motion dataset. By improving the fine-grained change characteristics between contextual spatiotemporal features, the model can focus on the more dominant change areas during the motion process. In addition, the motion detail recurrent neural network can significantly improve the prediction accuracy of meteorological texture details in the typhoon meteorological forecast dataset. BRIEF DESCRIPTION OF THE DRAWINGS
[0050] Figure 1 Schematic diagram of the structure of a spatiotemporal detail extraction module according to an embodiment of the present invention.
[0051] Figure 2 This is a diagram of the motion detail recurrent neural network structure of an embodiment of the present invention.
[0052] Figure 3 This is a comparison result of the embodiments of the present invention on a digital motion prediction standard data set.
[0053] Figure 4 This is the comparison result of the embodiment of the present invention on the typhoon meteorological forecast data set. DETAILED DESCRIPTION
[0054] The following is a detailed description of the embodiments of the present invention. It should be emphasized that the following description is only exemplary and is not intended to limit the scope and application of the present invention.
[0055] See also Figure 1 and Figure 2 The embodiment of the present invention provides a lightweight fine-grained spatiotemporal sequence prediction method, including: constructing a spatiotemporal detail extraction module, so that the spatiotemporal state updated by the spatiotemporal detail extraction module not only contains fine-grained features, but also enhances the correlation between contextual states through deep interaction of contexts; using the spatiotemporal detail extraction module in the spatiotemporal prediction network to implement a motion detail recurrent neural network unit, extracting deep contextual features with the help of multi-layer stacked fine-grained spatiotemporal modules, thereby constructing the final motion detail recurrent neural network MoDeRNN; wherein, by improving the model structure of the spatiotemporal prediction layer, the prediction result of the model in fine-grained features is improved.
[0056] The specific embodiments of the present invention are further described below.
[0057] The preferred embodiment of the present invention improves the extraction and prediction capabilities of traditional convolutional long short-term memory (ConvLSTM) time series features, focusing on improving the fine-grained spatiotemporal variation relationship between ConvLSTM context states.
[0058] Among them, the state transition equation of ConvLSTM predicting the output state at the next moment is:
[0059] g t =tanh(W xg ★X t +W hg ★H t-1 +b g )
[0060] i t =σ(W xi ★X t +W hi ★H t-1 +b i )
[0061] f t =σ(W xf ★X t +W hf ★H t-1 +b f )
[0062]
[0063] o t =σ(W xo ★X t +W ho ★H t-1 +h o )
[0064]
[0065] Among them, W (·) denotes a convolutional layer with a specific filter size, b ( · ) Indicates bias, X t Indicates the current input state, H t-1 and C t-1 are the output state and memory state of ConvLSTM at time t-1, H t and C t are the output state and memory state of ConvLSTM at time t, i t is the input gating, f t is the forget gate, g t is the input value, o tis the output gating, σ is the sigmoid activation function, ★ refers to the convolution operation, is the Hadamard product of matrices.
[0066] Previous experimental results show that since the input state and the output state of the previous moment are associated through a layer of convolution operation and an addition operation of the channel dimension during the process of updating the output state, it is difficult for the model to capture the fine-grained change characteristics between observation sequences, resulting in significant ambiguity in the prediction of texture details, and due to the gradual weakening of the correlation between contexts, the prediction results of the model are prone to incorrect predictions of target objects. The present invention constructs a spatiotemporal detail extraction module to realize a motion detail recurrent neural network unit, thereby effectively improving the model's prediction ability on the details of the observation sequence.
[0067] Among them, the structure of the spatiotemporal detail extraction module is as follows Figure 1 shown.
[0068] Specifically, a spatiotemporal detail extraction module consists of the following steps:
[0069] First, for the output state H at the previous moment t-1 We first use multiple convolutional layers with different filter sizes to extract the spatiotemporal information in different neighborhoods at the previous moment, and then use the addition and mean operations of the channel dimension to perform deep multi-scale feature fusion to obtain the fine-grained spatiotemporal features z of the output state at the previous moment. H .
[0070]
[0071] Where k represents the number of convolutional layers. Represents the weight of the filter of the i×i convolutional layer.
[0072] Then, we map the fine-grained spatiotemporal features of the previous moment into a 0-1 weight matrix Attn through the Sigmoid activation function H , so that the area (pixel) with a larger weight reflects the more important part of the short-term memory state at the previous moment, that is, the part that the model should focus on in subsequent state updates.
[0073]
[0074] With the help of the extracted weight matrix, we multiply the input state X at the current moment by the channel dimension t Update the current state input information so that it can reflect the important part of the past state in the current input, thus obtaining the updated current input state
[0075]
[0076] Among them, s is a constant greater than or equal to 1, which prevents the stacking of layers from causing Gradually decrease.
[0077] Similarly, for the updated current state We also use convolutional layers with different filter sizes to extract spatiotemporal information in different neighborhoods, and perform deep feature fusion through additive mean operations to obtain the fine-grained feature z of the current state. X .
[0078]
[0079] Where k represents the number of convolutional layers. Represents the weight of the filter of the i×i convolutional layer.
[0080] Then, we map the fine-grained spatiotemporal features of the current moment into a 0-1 weight matrix Attn through the Sigmoid activation function X , so that the area (pixel point) with a larger weight reflects the more important part of the input state at the current moment, that is, the potential movement trend of the model in the movement process at subsequent moments.
[0081]
[0082] With the help of the extracted weight matrix, we multiply the output state H at the previous moment by the channel dimension t-1 Update so that the input information of the current state can contain the potential movement trend of the current input state, so as to obtain the updated output state of the previous moment
[0083]
[0084] Through the above steps, we can get the output state of the spatiotemporal detail extraction module as well as Compared with the original state, the updated spatiotemporal state not only contains fine-grained features, but also improves the correlation between contextual states through deep interaction of the context.
[0085] In combination with the spatiotemporal detail extraction module, the preferred embodiment improves ConvLSTM, extracts deep context features with the help of n-layer stacked fine-grained spatiotemporal modules, and thus constructs the final motion detail recurrent neural network structure (MoDeRNN). The model structure diagram is shown in FIG. Figure 2As shown, in order to minimize the number of model parameters, multiple layers of stacked fine-grained spatiotemporal modules are stacked in a manner that shares convolutional layer parameters.
[0086] Its state transfer equation can be expressed as:
[0087]
[0088]
[0089]
[0090]
[0091]
[0092]
[0093]
[0094] Among them, Attn H represents the weight matrix, W (·) represents a convolutional layer with a specific filter size, represents the weight of the i×i convolutional layer of the filter, b (·) Indicates bias, X t Indicates the input status at the current moment. Indicates the updated input status at the current moment, H t-1 and C t-1 are the output state and memory state at time t-1, respectively, H t and C t are the output state and memory state at time t, s is a constant greater than or equal to 1, i t is the input gating, f t is the forget gate, g t is the input value, o t is the output gating, σ is the sigmoid activation function, k is the number of convolutional layers with different filter sizes, ★ refers to the convolution operation, is the Hadamard product of matrices.
[0095] By improving the model structure of the spatiotemporal prediction layer, the present invention can effectively improve the prediction results of the model in fine-grained features, especially significantly improving the preservation of texture details and object features.
[0096] Figure 3 This is a comparison result of the embodiments of the present invention on a digital motion prediction standard data set. Figure 4 This is the comparison result of the present invention on the typhoon meteorological forecast data set.
[0097] By comparing the network for spatiotemporal prediction constructed by an embodiment of the present invention with the previous benchmark network, it can be found that the motion detail recurrent neural network designed by the present invention effectively improves the prediction results of the model on digital details in the digital motion dataset. By improving the fine-grained change characteristics between contextual spatiotemporal features, the model can focus on the more dominant change areas during the motion process. In addition, the motion detail recurrent neural network can significantly improve the prediction accuracy on meteorological texture details in the typhoon meteorological forecast dataset.
[0098] The embodiment of the present invention extracts the spatiotemporal features of the input state and the state at the previous moment in different neighborhoods through convolutional layers with different receptive fields. In view of the shortcomings of insufficient interaction of context features in previous work, the present invention improves the extraction and expression of fine-grained spatiotemporal features between contexts through the constructed detail extraction module.
[0099] The motion detail recurrent neural network proposed in the present invention can be used as a backbone network for different downstream tasks and can be used in tasks other than spatiotemporal prediction tasks.
[0100] The spatiotemporal detail extraction module proposed in the present invention has excellent scalability. In addition to being combined with the basic ConvLSTM, it can also be combined with other spatiotemporal prediction networks to extract fine-grained contextual features and improve the performance of the model.
[0101] The present invention improves the interactivity and relevance of context by extracting spatiotemporal information of different scales. In addition to convolution kernels with different filter sizes, other structures can also achieve similar purposes and functions. In addition to the Sigmoid activation function, the weight matrix can also be generated through other transformations.
[0102] The background section of the present invention may contain background information about the problem or environment of the present invention, but does not necessarily describe the prior art. Therefore, the content included in the background section is not an admission by the applicant that the prior art is available.
[0103] The above content is a further detailed description of the present invention in combination with specific / preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, without departing from the concept of the present invention, it can also make several substitutions or modifications to these described embodiments, and these substitutions or modifications should be regarded as belonging to the protection scope of the present invention. In the description of this specification, the description of the reference terms "an embodiment", "some embodiments", "preferred embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in combination with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily target the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In the absence of mutual contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples. Although the embodiments of the present invention and their advantages have been described in detail, it should be understood that various changes, substitutions and modifications can be made herein without departing from the scope of protection of the patent application.
Claims
1. A lightweight fine-grained spatiotemporal sequence prediction method, characterized in that: include: Constructing a spatiotemporal detail extraction module, so that the spatiotemporal state updated by the spatiotemporal detail extraction module not only contains fine-grained features, but also improves the correlation between context states through deep interaction of contexts; The spatiotemporal detail extraction module is used in the spatiotemporal prediction network to realize the motion detail recurrent neural network unit, and the deep context features are extracted by means of multi-layer stacked fine-grained spatiotemporal modules, so as to construct the final motion detail recurrent neural network MoDeRNN; wherein, by improving the model structure of the spatiotemporal prediction layer, the prediction results of the model in fine-grained features are improved; The processing process of the spatiotemporal detail extraction module includes the following steps: First, for the output state H at the previous moment t-1 First, multiple convolutional layers with different filter sizes are used to extract the spatiotemporal information in different neighborhoods at the previous moment, and then the addition operation and mean operation of the channel dimension are used to perform deep multi-scale feature fusion to obtain the fine-grained spatiotemporal feature z of the output state at the previous moment. H Where k represents the number of convolutional layers. Represents the weight of the i×i convolutional layer of the filter; Then, the fine-grained spatiotemporal features of the previous moment are used to generate a weight matrix, which is mapped to a 0-1 weight matrix Attn through the Sigmoid activation function. H , so that the area with larger weight reflects the more important part of the short-term memory state at the previous moment, With the help of the extracted weight matrix, the input state X at the current moment is multiplied by the channel dimension t Update the current state input information so that it can reflect the important part of the past state in the current input, thus obtaining the updated current input state Where s is a constant greater than or equal to 1; For the updated current status Similarly, convolutional layers with different filter sizes are used to extract spatiotemporal information on different neighborhoods, and deep feature fusion is performed through the addition mean operation to obtain the fine-grained feature z of the current state. X in, Represents the weight of the i×i convolutional layer of the filter; Then, the fine-grained spatiotemporal features of the current moment are mapped to a 0-1 weight matrix Attn through the Sigmoid activation function X , so that the area with a larger weight reflects the more important part of the input state at the current moment, that is, the potential movement trend of the model in the movement process at the subsequent moment, With the help of the extracted weight matrix, the output state H of the previous moment is multiplied by the channel dimension t-1 Update so that the input information of the current state can contain the potential movement trend of the current input state, so as to obtain the updated output state of the previous moment Through the above steps, the output state of the spatiotemporal detail extraction module is obtained. as well as 2. The lightweight fine-grained spatiotemporal sequence prediction method according to claim 1, characterized in that: The multi-layer stacked fine-grained spatiotemporal modules are stacked in a manner of sharing convolutional layer parameters.
3. The lightweight fine-grained spatiotemporal sequence prediction method according to claim 1 or 2, characterized in that: The state transition equation of the motion detail recurrent neural network predicting the output state at the next moment is: Among them, Attn H represents the weight matrix, W (·) represents a convolutional layer with a specific filter size, represents the weight of the i×i convolutional layer of the filter, b (·) Indicates bias, X t Indicates the input status at the current moment. Indicates the updated input status at the current moment, H t-1 and C t-1 are the output state and memory state at time t-1, respectively, H t and C t are the output state and memory state at time t, s is a constant greater than or equal to 1, i t is the input gating, f t is the forget gate, g t is the input value, o t is the output gating, σ is the sigmoid activation function, k is the number of convolutional layers with different filter sizes, ★ represents the convolution operation, Represents the Hadamard product of matrices.
4. The lightweight fine-grained spatiotemporal sequence prediction method according to any one of claims 1 to 3, characterized in that: The convolutional layers with different receptive fields are used to extract the spatiotemporal features of the input state and the previous state in different neighborhoods.
5. The lightweight fine-grained spatiotemporal sequence prediction method according to any one of claims 1 to 3, characterized in that: The spatiotemporal prediction network includes a convolutional long short-term memory network ConvLSTM.
6. The lightweight fine-grained spatiotemporal sequence prediction method according to any one of claims 1 to 3, characterized in that: Used to predict the motion of objects.
7. The lightweight fine-grained spatiotemporal sequence prediction method according to any one of claims 1 to 3, characterized in that: Used to predict weather trends.
8. The lightweight fine-grained spatiotemporal sequence prediction method according to claim 7, characterized in that: Used for typhoon weather forecast.
9. A lightweight fine-grained spatiotemporal sequence prediction system, characterized in that: include: processor; A computer-readable storage medium, wherein the computer program stored therein, when executed by a processor, implements the lightweight fine-grained spatiotemporal series prediction method as described in any one of claims 1 to 8.
10. A computer-readable storage medium, characterized in that: When the computer program stored therein is executed by a processor, the lightweight fine-grained spatiotemporal sequence prediction method as described in any one of claims 1 to 8 is implemented.
Citation Information
Patent Citations
High-dimensional sequence data prediction method and system based on deep space-time memory network
CN107958044A
Short-term approaching rainfall forecasting method based on double-flow convolution long short-term memory network
CN113610329A
Method for multi-motion flow deep convolutional network model for video prediction
WO2020037965A1
