Cloud motion prediction method and system based on partial differential modeling

Abstract

The present invention relates to a cloud motion prediction method and system based on latent space decoupling and partial differential modeling. The method comprises: obtaining an original cloud image training set taken by an infrared camera and performing data preprocessing to obtain a preprocessed cloud image training set; constructing a dual-branch recursive neural network, including an image encoder, a partial differential unit, a long short-term memory unit, a variational mutual information network, and an image decoder; training the dual-branch recursive neural network using the cloud image training set; determining the cloud image of the future frame of the area to be predicted based on the cloud image of the current frame of the area to be predicted and the trained dual-branch recursive neural network. The method of the present invention can solve the high cost problem that traditional algorithms are only aimed at large-scale and long-term cloud motion prediction and require manual feature extraction, and realize low-cost, real-time, and accurate prediction of small-scale cloud motion by using atmospheric physical characteristics and deep neural networks.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and deep learning technology, and in particular to a cloud motion prediction method and system based on partial differential modeling. Background Art

[0002] With the continuous development of deep learning frameworks and algorithms, motion prediction has become a hot research field in computer vision in recent years. The purpose of cloud motion prediction is to predict the motion and change of clouds in the area in the future given the historical cloud images of the area to be predicted. Cloud motion prediction has a wide range of application scenarios in satellite laser communications, solar energy acquisition and utilization, real-time weather forecasts and other fields.

[0003] The existing traditional methods for cloud motion prediction include independent component analysis, slow feature analysis, random forest algorithm, physical modeling and other methods; however, these methods have some disadvantages. First, they need to obtain a large amount of physical data to create artificial features, which requires a lot of labor and computing costs; second, these methods are mostly used for large-scale and long-term fuzzy predictions of satellite cloud images, and cannot provide accurate, real-time and high-resolution small-scale cloud image predictions; finally, these methods do not have enough flexibility to capture the complexity of cloud image data. In recent years, deep learning methods have also been gradually applied to cloud motion prediction. However, the existing deep learning methods mainly capture the spatiotemporal characteristics of cloud motion through self-attention mechanisms or three-dimensional convolutions. Although these methods have good prediction effects, their shortcomings are also obvious, mainly reflected in the implicit network structure that cannot effectively utilize the physical characteristics of cloud motion, consumes a lot of computing resources and has slow inference speed. Summary of the invention

[0004] The purpose of the present invention is to provide a cloud motion prediction method and system based on partial differential modeling to solve the high cost problem that traditional algorithms are only used for large-scale and long-term cloud motion prediction and require manual feature extraction, and to use atmospheric physical properties and deep neural networks to achieve low-cost, real-time and accurate prediction of small-scale cloud motion.

[0005] To achieve the above object, the present invention provides the following solutions:

[0006] A cloud motion prediction method based on partial differential modeling, comprising:

[0007] Acquire an original cloud image training set taken by an infrared camera and perform data preprocessing to obtain a preprocessed cloud image training set; the cloud image training set includes cloud images of continuous frames;

[0008] Constructing a dual-branch recursive neural network; the dual-branch recursive neural network includes an image encoder, a partial differential unit, a long short-term memory unit, a variational mutual information network and an image decoder;

[0009] Using the cloud image training set to train the two-branch recursive neural network to obtain a trained two-branch recursive neural network;

[0010] Obtain the cloud image of the current frame of the area to be predicted;

[0011] The cloud layer image of the future frame of the area to be predicted is determined according to the cloud layer image of the current frame of the area to be predicted and the trained two-branch recursive neural network.

[0012] Optionally, the obtaining of an original cloud image training set taken by an infrared camera and performing data preprocessing to obtain a preprocessed cloud image training set specifically includes:

[0013] Acquire an original cloud image training set taken by an infrared camera; the original cloud image training set includes original cloud images of continuous frames;

[0014] The pixel value range of the original cloud image is normalized from [0, 255] to [0, 1], and the original cloud image is randomly horizontally flipped with a probability of 50% to obtain preprocessed continuous frames of cloud images to form the preprocessed cloud image training set.

[0015] Optionally, the constructing of a dual-branch recursive neural network specifically includes:

[0016] Constructing an image encoder; the image encoder is used to extract high-dimensional spatial features of a single-frame cloud image;

[0017] Constructing a partial differential unit and a long short-term memory unit respectively connected to the image encoder; the partial differential unit is used to obtain the predicted dynamic features of the future frame cloud image according to the high-dimensional spatial features of the cloud image; the long short-term memory unit is used to obtain the predicted non-dynamic features of the future frame cloud image according to the high-dimensional spatial features of the cloud image;

[0018] Constructing a variational mutual information network connected to the partial differential unit and the long short-term memory unit respectively; the variational mutual information network is used to approximate the conditional probability distribution of the predicted dynamic features and the predicted non-dynamic features of the future frame cloud image;

[0019] An image decoder connected to the partial differential unit and the long short-term memory unit respectively is constructed; the image decoder is used to predict the cloud image of the future frame according to the predicted dynamic features and predicted non-dynamic features of the cloud image of the future frame.

[0020] Optionally, the image encoder is used to extract high-dimensional spatial features of a single-frame cloud image, specifically including:

[0021] The preprocessed current frame cloud image u t Input into the image encoder of the dual-branch recurrent neural network, according to the encoder network formula h t =E(u t ) extract the high-dimensional spatial features h of the current frame cloud image t ; Where E represents the image encoder.

[0022] Optionally, the partial differential unit is used to obtain predicted dynamic features of future frame cloud images according to the high-dimensional spatial features of the cloud images, specifically including:

[0023] The high-dimensional spatial feature h of the current frame cloud layer image is t Enter the partial differential unit, according to the partial differential formula Obtain the predicted dynamic characteristics of future frame cloud images where f phycell Represents a partial differential unit.

[0024] Optionally, the long short-term memory unit is used to obtain predicted non-dynamic features of future frame cloud images according to the high-dimensional spatial features of the cloud images, specifically including:

[0025] The high-dimensional spatial feature h of the current frame cloud layer image is t Input the long short-term memory unit, according to the long short-term memory unit network formula Obtain predicted non-dynamic features of future frame cloud images where f lstmcell represents the long short-term memory unit, m t Represents the hidden state of the long short-term memory unit at the current moment.

[0026] Optionally, the variational mutual information network is used to approximate the conditional probability distribution of the predicted dynamic features and predicted non-dynamic features of the future frame cloud image, specifically including:

[0027] The predicted dynamic features of the future frame cloud image and predicting non-dynamic features Input the variational mutual information network, according to the variational mutual information network formula μ, Get the mean μ and variance σ of the conditional distribution; where f θ represents the variational mutual information network;

[0028] According to the mean μ and variance σ of the conditional distribution, using the formula Get the approximate conditional probability distribution in represents the normal distribution and I represents the identity matrix.

[0029] Optionally, the image decoder is used to predict a cloud image of a future frame according to the predicted dynamic features and the predicted non-dynamic features of the cloud image of the future frame, specifically comprising:

[0030] According to the formula Fusion of predicted dynamic features of the future frame cloud image and predicting non-dynamic features Get the fused feature h t+1 ;

[0031] The fused feature h t+1 Input the image into the decoder and use the decoder network formula Get the predicted cloud image for the future frame Where D represents the image decoder.

[0032] Optionally, the using the cloud image training set to train the two-branch recursive neural network to obtain a trained two-branch recursive neural network specifically includes:

[0033] Construct the momentum matrix loss function L during the dual-branch recursive neural network training process moment , mutual information minimization loss function L vCLUB And the reconstruction error loss function L recon ;

[0034] According to the momentum matrix loss function L moment , mutual information minimization loss function L vCLUB And the reconstruction error loss function L recon Determine the joint loss function L = L recon +L moment +L vCLUB ;

[0035] Using the joint loss function L = L recon +L moment +L vCLUB The dual-branch recurrent neural network is subjected to self-supervisory training until the joint loss function value L remains unchanged, thereby obtaining the trained dual-branch recurrent neural network.

[0036] A cloud motion prediction system based on partial differential modeling, comprising:

[0037] A data acquisition and preprocessing module is used to obtain an original cloud image training set taken by an infrared camera and perform data preprocessing to obtain a preprocessed cloud image training set; the cloud image training set includes cloud images of continuous frames;

[0038] A network construction module, used to construct a dual-branch recursive neural network; the dual-branch recursive neural network includes an image encoder, a partial differential unit, a long short-term memory unit, a variational mutual information network and an image decoder;

[0039] A network self-supervised training module, used to train the two-branch recursive neural network using the cloud image training set to obtain a trained two-branch recursive neural network;

[0040] The module for acquiring data to be predicted is used to acquire the cloud layer image of the current frame of the area to be predicted;

[0041] The cloud motion prediction module is used to determine the cloud image of the future frame of the area to be predicted based on the cloud image of the current frame of the area to be predicted and the trained two-branch recursive neural network.

[0042] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0043] The present invention provides a cloud motion prediction method and system based on partial differential modeling, the method comprising: obtaining an original cloud image training set taken by an infrared camera and performing data preprocessing to obtain a preprocessed cloud image training set; the cloud image training set includes cloud images of continuous frames; constructing a dual-branch recursive neural network; the dual-branch recursive neural network includes an image encoder, a partial differential unit, a long short-term memory unit, a variational mutual information network and an image decoder; using the cloud image training set to train the dual-branch recursive neural network to obtain a trained dual-branch recursive neural network; obtaining a cloud image of the current frame of the area to be predicted; determining the cloud image of the future frame of the area to be predicted based on the cloud image of the current frame of the area to be predicted and the trained dual-branch recursive neural network. The method of the present invention can solve the problem that traditional algorithms only predict large-scale and long-term cloud motion and require high costs for artificial feature extraction, and realize low-cost, real-time and accurate prediction of small-scale cloud motion by using atmospheric physical properties and deep neural networks. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0045] Figure 1 A flow chart of a cloud motion prediction method based on partial differential modeling provided by the present invention;

[0046] Figure 2A schematic diagram of the principle of a cloud motion prediction method based on partial differential modeling provided by the present invention;

[0047] Figure 3 A schematic diagram of the structure of a dual-branch recursive neural network provided by the present invention;

[0048] Figure 4 A schematic diagram of comparing a predicted image and a real image of cloud movement in a certain area to be predicted within a period of time in the future provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] The purpose of the present invention is to provide a cloud motion prediction method and system based on partial differential modeling to solve the high cost problem that traditional algorithms are only used for large-scale and long-term cloud motion prediction and require manual feature extraction, and to use atmospheric physical properties and deep neural networks to achieve low-cost, real-time and accurate prediction of small-scale cloud motion.

[0051] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0052] Figure 1 This is a flow chart of a cloud motion prediction method based on partial differential modeling of the present invention; Figure 2 A schematic diagram of the principle of a cloud motion prediction method based on partial differential modeling provided by the present invention. Figure 1 and Figure 2 The present invention provides a cloud motion prediction method based on partial differential modeling, which specifically includes:

[0053] Step 101: Obtain an original cloud image training set taken by an infrared camera and perform data preprocessing to obtain a preprocessed cloud image training set.

[0054] During the data collection and data preprocessing stages, an infrared camera was used to continuously photograph cloud movement in different weather conditions and at different time periods to obtain a training set of original cloud images.

[0055] The data preprocessing performed on the original cloud image training set is mainly data augmentation processing, including normalizing the cloud image pixel value range from [0, 255] to [0, 1] and randomly flipping the cloud image horizontally with a probability of 50%, thereby obtaining a preprocessed cloud image training set; the cloud image training set includes cloud images of continuous frames.

[0056] Step 102: Construct a dual-branch recursive neural network.

[0057] Figure 3 This is a schematic diagram of the structure of the dual-branch recursive neural network provided by the present invention. Figure 3 The dual-branch recurrent neural network constructed by the present invention includes an image encoder, a partial differential unit, a long short-term memory unit, a variational mutual information network and an image decoder. The dual-branch recurrent neural network includes: using the image encoder to extract the high-dimensional spatial features of a single-frame cloud image; using the partial differential unit and the long short-term memory unit to obtain the predicted dynamic features and predicted non-dynamic features of the future frame cloud image respectively according to the high-dimensional spatial features of the cloud image; using the variational mutual information network to approximate the conditional probability distribution of the predicted dynamic features and predicted non-dynamic features of the future frame cloud image; using the image decoder to predict the cloud image of the future frame according to the predicted dynamic features and predicted non-dynamic features of the future frame.

[0058] Therefore, the step 102 constructs a dual-branch recursive neural network, which specifically includes:

[0059] Step 2.1: construct an image encoder; the image encoder is used to extract high-dimensional spatial features of a single-frame cloud image.

[0060] The image encoder (also called encoder or encoder network) constructed by the present invention is composed of a multi-level CONV-GN-LAEKYRELU neural network in a linear order; wherein CONV represents a convolutional layer, GN represents a group normalization layer, and LEAKYRELU represents a linear rectification activation function.

[0061] The preprocessed current frame cloud image u t The input is sent to the encoder of the dual-branch recurrent neural network to extract the high-dimensional spatial features of the cloud image in the current frame. The process is abstracted as follows:

[0062]

[0063] Where E represents the encoder network, n, m and c represent the image domains respectively. The height, width and number of channels of n′, m′ and c′ represent the feature space domain The height, width, and number of channels of the

[0064] Therefore, the image encoder in step 2.1 is used to extract the high-dimensional spatial features of a single-frame cloud image, specifically including:

[0065] The preprocessed current frame cloud image u t Input into the image encoder of the dual-branch recurrent neural network, according to the encoder network formula h t =E(u t ) extract the high-dimensional spatial features h of the current frame cloud image t ; Where E represents the image encoder.

[0066] Step 2.2: Construct a partial differential unit and a long short-term memory unit respectively connected to the image encoder; the partial differential unit is used to obtain the predicted dynamic features of the future frame cloud image according to the high-dimensional spatial features of the cloud image; the long short-term memory unit is used to obtain the predicted non-dynamic features of the future frame cloud image according to the high-dimensional spatial features of the cloud image.

[0067] Get the high-dimensional spatial features h of the current frame cloud image t After that, h t The predicted dynamic features of the cloud images in future frames are obtained by sending them to the partial differential unit and the long short-term memory unit. and predicting non-dynamic features The calculation process can be described as the following formula:

[0068]

[0069] where f phycell and f lstmcell Respectively represent the partial differential unit and the long short-term memory unit, m t represents the hidden state of the long short-term memory unit at the current time t. N represents the order of partial differentiation, Indicates h t The differential terms of various orders, c i，j are the coefficients before the learnable differential terms of various orders.

[0070] Therefore, the partial differential unit in step 2.2 is used to obtain the predicted dynamic features of the future frame cloud image according to the high-dimensional spatial features of the cloud image, specifically including:

[0071] The high-dimensional spatial feature h of the current frame cloud layer image is t Enter the partial differential unit, according to the partial differential formula Obtain the predicted dynamic characteristics of future frame cloud images That is, the predicted dynamic characteristics of the cloud image at time t+1; where f phycell Represents a partial differential unit.

[0072] The long short-term memory unit in step 2.2 is used to obtain the predicted non-dynamic features of the future frame cloud image according to the high-dimensional spatial features of the cloud image, specifically including:

[0073] The high-dimensional spatial feature h of the current frame cloud layer image is t Input the long short-term memory unit, according to the long short-term memory unit network formula Obtain predicted non-dynamic features of future frame cloud images That is, the predicted non-dynamic features of the cloud image at time t+1; where f lstmcell represents the long short-term memory unit, m t Represents the hidden state of the long short-term memory unit at the current moment.

[0074] Step 2.3: construct a variational mutual information network connected to the partial differential unit and the long short-term memory unit respectively; the variational mutual information network is used to approximate the conditional probability distribution of the predicted dynamic features and the predicted non-dynamic features of the future frame cloud image.

[0075] The predicted dynamic features of the future frame cloud image and predicting non-dynamic features Send it into the variational mutual information network to get the conditional distribution The mean μ and variance σ of the , and transform it into a standard Gaussian distribution, the process can be abstracted as:

[0076]

[0077] where f θ represents the variational mutual information network, θ represents the optimizable parameters in the variational mutual information network, where represents the normal distribution and I represents the identity matrix.

[0078] Therefore, the variational mutual information network in step 2.3 is used to approximate the conditional probability distribution of the predicted dynamic features and predicted non-dynamic features of the future frame cloud image, specifically including:

[0079] The predicted dynamic features of the future frame cloud image and predicting non-dynamic features Input the variational mutual information network, according to the variational mutual information network formula μ, Get the mean μ and variance σ of the conditional distribution; where f θ represents the variational mutual information network;

[0080] According to the mean μ and variance σ of the conditional distribution, using the formula Get the approximate conditional probability distribution

[0081] Step 2.4: Construct an image decoder (also called a decoder or decoder network) connected to the partial differential unit and the long short-term memory unit respectively; the image decoder is used to predict the cloud image of the future frame based on the predicted dynamic features and predicted non-dynamic features of the cloud image of the future frame.

[0082] Fusion of predicted dynamic features of future frame cloud images and predicting non-dynamic features The predicted future frame cloud image is sent to the decoder The process can be abstracted as:

[0083]

[0084] Where D represents the decoder network, and the decoder is composed of a multi-level DECONV-GN-LAEKYRELU neural network in a linear order; where DECONV represents the deconvolution layer.

[0085] Therefore, the image decoder in step 2.4 is used to predict the cloud image of the future frame according to the predicted dynamic features and predicted non-dynamic features of the cloud image of the future frame, specifically including:

[0086] According to the formula Fusion of predicted dynamic features of the future frame cloud image and predicting non-dynamic features Get the fused feature h t+1 ;

[0087] The fused feature h t+1 Input the image into the decoder and use the decoder network formula Get the predicted cloud image for the future frame Where D represents the image decoder.

[0088] Step 103: Use the cloud image training set to train the two-branch recursive neural network to obtain a trained two-branch recursive neural network.

[0089] The step 103 uses the cloud image training set to train the two-branch recursive neural network to obtain a trained two-branch recursive neural network, which specifically includes:

[0090] Step 3.1: Construct the momentum matrix loss function L during the training of the dual-branch recursive neural network moment , mutual information minimization loss function L vCLUB And the reconstruction error loss function L recon ;

[0091] Construct momentum matrix loss function L moment , mutual information minimization loss function LvCLUB and the reconstruction error function L recon , train the network in a self-supervised manner.

[0092] The momentum matrix loss function L moment It is mainly used to constrain the convolution kernel in the partial differential network, so that the network convolution approximates the partial differential of the finite space. The construction process is as follows:

[0093] A convolution kernel w of size k×k has a moment matrix M(w) of size k×k, defined as:

[0094]

[0095] The Taylor expansion of the convolution of arbitrary functions h and w is expressed as:

[0096]

[0097] By imposing constraints on the moment matrix of the convolution kernel, we can obtain approximate values ​​of differentials of various orders. For example, if we want to approximate the differential operator It only needs the moment matrix to satisfy the condition M(w) i，j =0fori≠a and j≠b. represents a matrix of size k×k, where only the coordinate (i, j) is 1 and the rest of the coordinates are 0. Therefore, the bi-norm constraint can be imposed to make M(w) match the target matrix So that each convolution kernel Approximate corresponding partial differential terms So we can get the constraint L moment :

[0098]

[0099] In addition, in order to improve the decoupling effect of predicting dynamic features and non-dynamic features of future frame cloud images, a mutual information minimization constraint is constructed, and its construction process is as follows:

[0100] The sample pairs that conform to the distribution p(x, y) are expressed as The goal is to derive an upper bound on the mutual information I(x; y) based on these sample pairs. Assuming the two variables are x and y, the mutual information between the two variables is expressed as:

[0101]

[0102] Where p(x, y) represents the joint probability distribution, p(x) represents the probability distribution of variable x, and p(y) represents the probability distribution of variable y. I(x; y) represents the mutual information between x and y. The smaller the mutual information, the smaller the correlation between the two variables. For the conditional distribution p(y|x), the upper bound of the mutual information is defined as:

[0103] I CLUB (x;y) = E p(x,y) [logp(y|x)]-E p(x) E p(y) [logp(y|x)]

[0104] To prove I CLUB (x; y) is the upper bound of the mutual information I(x; y), and the difference between them is calculated:

[0105]

[0106] According to the definition of marginal distribution, p(y)=∫p(y|x)p(x)dx=E p(x) [logp(y]x)], since log(·) is a concave function, according to Jensen’s inequality, logp(y)=log(E p(x) [logp(y|x)])≥E p(x) [logp(y|x)] Substituting this inequality into the above equation, we can see that Δ is always non-negative. Therefore, I CLUB (x; y) is the upper bound of the mutual information I(x; y), and the equation holds if and only if the two variables x and y are independent of each other.

[0107] In this way, we get the sample pair The unbiased estimate of the upper bound of mutual information is:

[0108]

[0109] For known sample pairs Usually the specific function form of p(y|x) or p(x|y) is unknown, so a variational network q with parameter θ is used θ (y|x) approximates p(y|x). The upper bound of the variational mutual information becomes:

[0110] I vCLUB (x;y) = E p(x，y) [logq θ (y|x)]-E p(x) E p(y) [logq θ (y|x)](14)

[0111] Sample pair The unbiased estimate of the upper bound of the mutual information becomes:

[0112]

[0113] Therefore, in the present invention, the upper bound of mutual information is expressed as:

[0114]

[0115] in represents the predicted non-dynamic features of the i-th cloud image at time t+1, represents the predicted dynamic features of the i-th cloud layer image at time t+1, q θ represents the variational mutual information network, Represents the conditional probability distribution of cloud image prediction of dynamic features and prediction of non-dynamic features. Finally, the reconstruction error loss function is constructed using the cloud image reconstruction error:

[0116]

[0117] Among them, L recon is the scalar value of the two-norm loss function, is the reconstructed cloud image, χ is the input cloud image, and ||||2 is the vector norm.

[0118] Step 3.2: According to the momentum matrix loss function L moment , mutual information minimization loss function L vCLUB And the reconstruction error loss function L recon Determine the joint loss function L = L recon +L moment +L vCLUB ;

[0119] Step 3.3: Using the joint loss function L = L recon +L moment +L vCLUB The dual-branch recurrent neural network is subjected to self-supervisory training until the joint loss function value L remains unchanged, thereby obtaining the trained dual-branch recurrent neural network.

[0120] In the self-supervised training process, the present invention constructs a loss function The momentum matrix constraint is applied to the convolution kernel in the partial differential unit to realize the approximation of network convolution to each order of differentials. Where k represents the total order of differentials, i and j represent the order of differentials in the length and width directions respectively; represents the pth convolution kernel, which approximates the partial differential with order i in the length direction, order j in the width direction, and total order k; express The momentum matrix can be expressed as Represents a matrix of size k×k, where only the coordinate (i, j) is 1 and the rest of the coordinates are 0.

[0121] In the self-supervised training process, the present invention also imposes mutual information minimization constraints on the predicted dynamic features and predicted non-dynamic features of the future frame cloud image. This improves the decoupling effect of predicting dynamic features and predicting non-dynamic features of future frame cloud images.

[0122] In the self-supervised training process, the loss function constructed by the present invention using the cloud image reconstruction error is:

[0123] Using the joint loss function L = L recon +L moment +L vCLUB The two-branch recursive neural network is trained until the value of the joint loss function remains unchanged, thereby obtaining the trained two-branch recursive neural network.

[0124] Step 104: Obtain a cloud layer image of the current frame of the area to be predicted.

[0125] The cloud image of the current frame of the area to be predicted is also taken by an infrared camera.

[0126] Step 105: Determine the cloud image of the future frame of the area to be predicted based on the cloud image of the current frame of the area to be predicted and the trained two-branch recursive neural network.

[0127] The cloud image of the current frame of the area to be predicted is input into the trained two-branch recurrent neural network, and a new frame of cloud image in the future is generated using the trained two-branch recurrent neural network. The new frame of cloud image is then used as the input image for the next forward propagation. This cycle is repeated N times to obtain N consecutive predicted cloud images.

[0128] The method of the present invention comprises obtaining a cloud image training set through an infrared camera and performing data augmentation; constructing a dual-branch recursive neural network, including an image encoder, a partial differential unit, a long short-term memory unit, a variational mutual information network, and an image decoder; using the cloud image training set to train the parameters in the dual-branch recursive neural network; obtaining a cloud image of a current frame of a region to be predicted; and determining a cloud image of a future frame of the region to be predicted based on the cloud image of the current frame of the region to be predicted and the optimized dual-branch recursive neural network. The present invention adopts a self-supervisory algorithm, which can provide a prediction of the movement and change of the cloud layer in the region to be predicted over a period of time, improve the real-time and accuracy of cloud amount prediction, and truly reflect the distribution of the cloud layer. In view of the fact that most of the current mainstream methods are used for large-scale and long-term satellite cloud image prediction, the present invention adopts a combination of the physical characteristics of atmospheric motion and neural networks, and creatively divides the cloud motion process into dynamic and non-dynamic parts for modeling, thereby realizing low-cost, real-time and accurate prediction of small-scale cloud images, and solving the problems of the original method requiring manual feature extraction and high cost of collecting atmospheric data and the limited generalization of the original method to different weather conditions.

[0129] The method provided by the present invention uses a cloud image training set taken by an infrared camera to train a dual-branch recursive neural network, sends the cloud image of the current frame into the encoder of the neural network to obtain the high-dimensional spatial features of the cloud image; then the high-dimensional spatial features of the cloud image are respectively sent to the partial differential unit and the long short-term memory unit to obtain the predicted dynamic spatial features and predicted non-dynamic spatial features of the cloud image of the future frame, and cloud images of different time lengths in the future can be generated by controlling the partial differential unit and the long short-term memory unit; then the two parts of the features are sent to the variational mutual information network to approximate the conditional probability distribution of the predicted dynamic features and predicted non-dynamic features of the cloud image of the future frame; finally, the two parts of the spatial features are fused and sent to the image decoder to obtain the predicted cloud image of the future frame. When the method of the present invention is applied in practice, firstly, the cloud map of the area to be predicted is obtained, and sent to the trained dual-branch recursive neural network, that is, a real-time and accurate cloud motion prediction image of the area to be predicted within a period of time in the future can be obtained.

[0130] The following uses a specific embodiment to illustrate the technical effect of the method of the present invention. In the inference stage of the neural network, see Figure 4 As a specific embodiment, assume that there are 5 consecutive frames of current frame images as network input, represented as u 0：4 First, the first four frames of the current frame image u 0：3 Input the dual-branch recursive neural network in turn to obtain the predicted image of the input frame in turn Then the last frame input image u4 is input into the dual-branch recurrent neural network to obtain the first predicted image of the future frame Then As a new input, get the second predicted image of the future frame Repeat this process 5 times to get 5 consecutive predicted cloud images for the future frames. like Figure 4 As shown, the cloud image predicted by the method of the present invention is very similar to the real cloud image at that moment, so high that it cannot be distinguished by the naked eye, indicating that the method of the present invention has extremely high accuracy in predicting cloud motion. It can be seen that the method of realizing cloud motion prediction based on partial differential modeling proposed by the present invention can effectively capture the spatiotemporal information of cloud motion through the combination of the physical characteristics of atmospheric motion and deep neural network, and realize the prediction of cloud motion under different weather conditions efficiently and accurately.

[0131] Based on the method provided by the present invention, the present invention also provides a cloud motion prediction system based on partial differential modeling, the system comprising:

[0132] A data acquisition and preprocessing module is used to obtain an original cloud image training set taken by an infrared camera and perform data preprocessing to obtain a preprocessed cloud image training set; the cloud image training set includes cloud images of continuous frames;

[0133] A network construction module, used to construct a dual-branch recursive neural network; the dual-branch recursive neural network includes an image encoder, a partial differential unit, a long short-term memory unit, a variational mutual information network and an image decoder;

[0134] A network self-supervised training module, used to train the two-branch recursive neural network using the cloud image training set to obtain a trained two-branch recursive neural network;

[0135] The module for acquiring data to be predicted is used to acquire the cloud layer image of the current frame of the area to be predicted;

[0136] The cloud motion prediction module is used to determine the cloud image of the future frame of the area to be predicted based on the cloud image of the current frame of the area to be predicted and the trained two-branch recursive neural network.

[0137] The method and system of the present invention mainly include three stages:

[0138] The first stage is data collection and data preprocessing. In this stage, multiple infrared cameras are used to capture cloud images in multiple areas at different times and collect them as training data. Different cameras capture different areas, and cloud images under various weather conditions need to be collected. Data preprocessing includes normalizing the cloud image pixel value range [0,255] to [0,1] and randomly flipping the cloud image horizontally with a probability of 50%.

[0139] The second stage is the self-supervised training stage of the two-branch recurrent neural network. In this stage, after the two-branch recurrent neural network is built, the number of pictures that can be accommodated by the memory (or video memory) is randomly selected from the pre-trained data as the training data for one iteration. The selected training data is then passed through the encoder, partial differential unit, long short-term memory unit, variational mutual information network, and decoder to obtain the predicted cloud image of the future frame. Finally, the loss function is calculated through momentum matrix constraints, mutual information minimization constraints, and reconstruction error, and then after back propagation and gradient update, one iteration is completed. The above process is repeated until the network converges.

[0140] The third stage is the inference stage of the dual-branch recurrent neural network. In this stage, after the dual-branch recurrent neural network training is completed, the cloud layer image of the current frame of the area to be predicted is sent to the network as input to obtain the predicted cloud layer image of the future frame. The predicted cloud layer images of future frames of different time lengths can be generated by the number of forward propagations of the network.

[0141] The method and system of the present invention can realize low-cost, real-time and accurate prediction of small-scale cloud motion with only the cloud image of the current frame of the area to be predicted and no other artificial feature data, and can be widely used in satellite laser communication, solar energy acquisition and utilization, real-time weather forecasting and other fields.

[0142] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0143] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A cloud motion prediction method based on partial differential modeling, characterized in that: include: Acquire an original cloud image training set taken by an infrared camera and perform data preprocessing to obtain a preprocessed cloud image training set; the cloud image training set includes cloud images of continuous frames; Constructing a dual-branch recursive neural network; the dual-branch recursive neural network includes an image encoder, a partial differential unit, a long short-term memory unit, a variational mutual information network and an image decoder; The construction of a dual-branch recursive neural network specifically includes: Constructing an image encoder; the image encoder is used to extract high-dimensional spatial features of a single-frame cloud image; Constructing a partial differential unit and a long short-term memory unit respectively connected to the image encoder; the partial differential unit is used to obtain the predicted dynamic features of the future frame cloud image according to the high-dimensional spatial features of the cloud image; the long short-term memory unit is used to obtain the predicted non-dynamic features of the future frame cloud image according to the high-dimensional spatial features of the cloud image; Constructing a variational mutual information network connected to the partial differential unit and the long short-term memory unit respectively; the variational mutual information network is used to approximate the conditional probability distribution of the predicted dynamic features and the predicted non-dynamic features of the future frame cloud image; Constructing an image decoder connected to the partial differential unit and the long short-term memory unit respectively; the image decoder is used to predict the cloud image of the future frame according to the predicted dynamic features and predicted non-dynamic features of the cloud image of the future frame; Using the cloud image training set to train the two-branch recursive neural network to obtain a trained two-branch recursive neural network; Obtain the cloud image of the current frame of the area to be predicted; The cloud layer image of the future frame of the area to be predicted is determined according to the cloud layer image of the current frame of the area to be predicted and the trained two-branch recursive neural network.

2. The method according to claim 1, characterized in that The obtaining of the original cloud image training set taken by the infrared camera and performing data preprocessing to obtain the preprocessed cloud image training set specifically includes: Acquire an original cloud image training set taken by an infrared camera; the original cloud image training set includes original cloud images of continuous frames; The pixel value range of the original cloud image is normalized from [0, 255] to [0, 1], and the original cloud image is randomly horizontally flipped with a probability of 50% to obtain preprocessed continuous frames of cloud images to form the preprocessed cloud image training set.

3. The method according to claim 1, characterized in that The image encoder is used to extract high-dimensional spatial features of a single-frame cloud image, specifically including: The preprocessed current frame cloud image u t Input into the image encoder of the dual-branch recurrent neural network, according to the encoder network formula h t =E(u t ) extract the high-dimensional spatial features h of the current frame cloud image t ; Where E represents the image encoder.

4. The method according to claim 3, characterized in that The partial differential unit is used to obtain the predicted dynamic features of the future frame cloud image according to the high-dimensional spatial features of the cloud image, specifically including: The high-dimensional spatial feature h of the current frame cloud image t Enter the partial differential unit, according to the partial differential formula Obtain the predicted dynamic characteristics of future frame cloud images where f phycell Represents a partial differential unit.

5. The method according to claim 4, characterized in that The long short-term memory unit is used to obtain the predicted non-dynamic features of the future frame cloud image according to the high-dimensional spatial features of the cloud image, specifically including: The high-dimensional spatial feature h of the current frame cloud image t Input the long short-term memory unit, according to the long short-term memory unit network formula Obtain predicted non-dynamic features of future frame cloud images where f lstmcell represents the long short-term memory unit, m t Represents the hidden state of the long short-term memory unit at the current moment.

6. The method according to claim 5, characterized in that The variational mutual information network is used to approximate the conditional probability distribution of the predicted dynamic features and predicted non-dynamic features of the future frame cloud image, specifically including: The predicted dynamic features of the future frame cloud image and predicting non-dynamic features Input the variational mutual information network, according to the variational mutual information network formula μ, Get the mean μ and variance σ of the conditional distribution; where f θ represents the variational mutual information network; According to the mean μ and variance σ of the conditional distribution, using the formula Get the approximate conditional probability distribution in represents the normal distribution and I represents the identity matrix.

7. The method according to claim 6, characterized in that The image decoder is used to predict the cloud image of the future frame according to the predicted dynamic features and the predicted non-dynamic features of the cloud image of the future frame, specifically including: According to the formula Fusion of predicted dynamic features of the future frame cloud image and predicting non-dynamic features Get the fused feature h t+1 ; The fused feature h t+1 Input the image into the decoder and use the decoder network formula Get the predicted cloud image for the future frame Where D represents the image decoder.

8. The method according to claim 7, characterized in that The step of training the dual-branch recursive neural network using the cloud image training set to obtain a trained dual-branch recursive neural network specifically includes: Construct the momentum matrix loss function L in the dual-branch recursive neural network training process moment , mutual information minimization loss function L vCLUB And the reconstruction error loss function L recon ; Momentum Matrix Loss Function in Represents a matrix of size k×k, where only the coordinate (i, j) is 1 and the rest of the coordinates are 0; Represents a convolution kernel of size k×k; for The moment matrix of Mutual information minimization loss function in represents the predicted non-dynamic features of the i-th cloud image at time t+1, represents the predicted dynamic features of the i-th cloud layer image at time t+1, q θ represents the variational mutual information network; represents the predicted non-dynamic features of the j-th cloud image at time t+1, and N is the number of cloud images; Reconstruction error loss function Where χ is the input cloud image, || ||2 is the vector norm; According to the momentum matrix loss function L moment , mutual information minimization loss function L vCLUB And the reconstruction error loss function L recon Determine the joint loss function L = L recon +L moment +L vCLUB ; Using the joint loss function L = L recon +L moment +L vCLUB The dual-branch recurrent neural network is subjected to self-supervisory training until the value of the joint loss function L remains unchanged, thereby obtaining the trained dual-branch recurrent neural network.

9. A cloud motion prediction system based on partial differential modeling, characterized in that: include: A data acquisition and preprocessing module is used to obtain an original cloud image training set taken by an infrared camera and perform data preprocessing to obtain a preprocessed cloud image training set; the cloud image training set includes cloud images of continuous frames; A network construction module, used to construct a dual-branch recursive neural network; the dual-branch recursive neural network includes an image encoder, a partial differential unit, a long short-term memory unit, a variational mutual information network and an image decoder; The construction of a dual-branch recursive neural network specifically includes: Constructing an image encoder; the image encoder is used to extract high-dimensional spatial features of a single-frame cloud image; Constructing a partial differential unit and a long short-term memory unit respectively connected to the image encoder; the partial differential unit is used to obtain the predicted dynamic features of the future frame cloud image according to the high-dimensional spatial features of the cloud image; the long short-term memory unit is used to obtain the predicted non-dynamic features of the future frame cloud image according to the high-dimensional spatial features of the cloud image; Constructing a variational mutual information network connected to the partial differential unit and the long short-term memory unit respectively; the variational mutual information network is used to approximate the conditional probability distribution of the predicted dynamic features and the predicted non-dynamic features of the future frame cloud image; Constructing an image decoder connected to the partial differential unit and the long short-term memory unit respectively; the image decoder is used to predict the cloud image of the future frame according to the predicted dynamic features and predicted non-dynamic features of the cloud image of the future frame; A network self-supervised training module, used to train the two-branch recursive neural network using the cloud image training set to obtain a trained two-branch recursive neural network; The module for acquiring data to be predicted is used to acquire the cloud layer image of the current frame of the area to be predicted; The cloud motion prediction module is used to determine the cloud image of the future frame of the area to be predicted based on the cloud image of the current frame of the area to be predicted and the trained two-branch recursive neural network.

Images