Satellite multi-channel radiance image extrapolation method, device and storage medium

Abstract

The present invention provides a satellite multi-channel radiance image extrapolation method, device, and storage medium. The method includes: receiving an image extrapolation request; obtaining satellite multi-channel radiance images of a target area at different times; using the satellite multi-channel radiance images at different times to train an image extrapolation model; the image extrapolation model includes a convolutional recurrent neural network and a generative adversarial network; the convolutional recurrent neural network further includes: inputting the output result of the last ConvGRU unit in the previous time step into the first ConvGRU unit in the next time step; inputting the satellite multi-channel radiance images at different times into the image extrapolation model, and outputting a target extrapolated image at a future time according to the image extrapolation request. The extrapolation method provided by this solution can effectively and accurately obtain satellite multi-channel radiance extrapolated images over a long time range, while also improving the quality of the extrapolated images.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of image extrapolation technology, and in particular to a satellite multi-channel radiance image extrapolation method, device and storage medium. Background Art

[0002] Effective extrapolation of satellite multi-channel radiance image data is crucial for efficiently and accurately obtaining future meteorological parameter data. The essence of satellite multi-channel radiance image extrapolation is to extract, analyze, and summarize the features of image data in temporal and spatial sequences, and then predict and generate image data for future moments.

[0003] Currently, the main methods for extrapolating satellite multi-channel radiance images are optical flow methods, such as the multi-scale optical flow model and the semi-Lagrangian optical flow model. However, the constant brightness assumption of the optical flow method is not applicable to the extrapolation of satellite multi-channel radiance images; the estimation of the optical flow motion field can only be done using two adjacent satellite multi-channel radiance images. With the development of artificial intelligence technology, extrapolation methods based on convolutional recurrent neural networks have also been widely used to extrapolate future prediction data. However, existing network models still have problems with effective extrapolation for short time periods and poor image quality when extrapolating for long periods of time. Summary of the Invention

[0004] The present invention provides a satellite multi-channel radiance image extrapolation method, device and storage medium. The extrapolation method can effectively and accurately obtain multi-channel radiance extrapolated images within a long time range, while improving the quality of the extrapolated images.

[0005] In a first aspect, an embodiment of the present invention provides a satellite multi-channel radiance image extrapolation method, comprising:

[0006] Receiving an image extrapolation request for a target area at a future time;

[0007] Acquire satellite multi-channel radiance images of the target area at different times;

[0008] An image extrapolation model is trained using the satellite multi-channel radiance images at different times; wherein the image extrapolation model includes a convolutional recurrent neural network and a generative adversarial network; the convolutional recurrent neural network further includes: inputting the output result of the last ConvGRU unit of the previous time step into the first ConvGRU unit of the next time step;

[0009] The satellite multi-channel radiance images at different moments are input into the image extrapolation model, and a target extrapolated image at the future moment is output according to the image extrapolation request.

[0010] Optionally, the training of an image extrapolation model using the satellite multi-channel radiance images at different times includes:

[0011] Determine at least two sample sets based on the satellite multi-channel radiance image; wherein each sample set includes a satellite multi-channel radiance image at a historical moment as input and a satellite multi-channel radiance image at a current moment as output; the sample sets include: a training set and a validation set;

[0012] Based on the convolutional recurrent neural network and the generative adversarial network, an initial image extrapolation model is constructed; wherein, in each time step of the convolutional recurrent neural network, the output of each layer of the encoding ConvGRU unit is spliced to the input of the decoding ConvGRU unit of the corresponding layer;

[0013] The initial image extrapolation model is trained using the at least two groups of training sets to obtain the image extrapolation model.

[0014] Optionally, the training of an image extrapolation model using the satellite multi-channel radiance images at different times includes:

[0015] Introducing a loss function during training; the loss function consists of a mean absolute error loss function, a mean square error loss function, and a FocalLoss loss function;

[0016] When the value of the loss function is lower than a preset loss threshold, the training is terminated;

[0017] The loss function is expressed as follows:

[0018] L=λ1*MAE+λ2*MSE+λ3*FocalLoss

[0019] Among them, L is used to characterize the loss function; MAE is used to characterize the mean absolute error loss function; λ1 is used to characterize the weight coefficient of the MAE; MSE is used to characterize the mean square error loss function; λ2 is used to characterize the weight coefficient of the MSE; FocalLoss is used to characterize the FocalLoss loss function; λ3 is used to characterize the weight coefficient of the FocalLoss.

[0020] Optionally, the training the initial image extrapolation model using the at least two training sets to obtain the image extrapolation model includes:

[0021] After the initial image extrapolation model is trained using the at least two training sets, a first image extrapolation model is obtained;

[0022] Inputting the verification set into the first image extrapolation model to obtain a predicted satellite multi-channel radiance image at the current moment;

[0023] Determining a root mean square error, a critical success index, and an HSS score based on the predicted satellite multi-channel radiance image at the current moment and the satellite multi-channel radiance image at the current moment in the validation set;

[0024] Determining whether the root mean square error is greater than a first preset threshold, whether the critical success index is less than a second preset threshold, and whether the HSS score is less than a third preset threshold;

[0025] If the judgment result is no, the first image extrapolation model is determined to be the image extrapolation model; otherwise, the first image extrapolation model is optimized and adjusted.

[0026] Optionally, the training the initial image extrapolation model using the at least two training sets to obtain the image extrapolation model includes:

[0027] After the initial image extrapolation model is trained using the at least two training sets, a first image extrapolation model is obtained;

[0028] Inputting the verification set into the first image extrapolation model to obtain a predicted satellite multi-channel radiance image at the current moment;

[0029] Obtaining a root mean square error, a critical success index, and an HSS score based on the predicted satellite multi-channel radiance image at the current moment and the satellite multi-channel radiance image at the current moment in the validation set;

[0030] Determining whether the differences between five consecutive root mean square errors are all greater than a fourth preset threshold, whether the differences between five consecutive critical success indices are all greater than a fifth preset threshold, and whether the differences between five consecutive HSS scores are all greater than a sixth preset threshold;

[0031] If the judgment result is no, the first image extrapolation model is determined to be the image extrapolation model; otherwise, the first image extrapolation model is optimized and adjusted.

[0032] Optionally, the image extrapolation request includes a time range and a time granularity of a future moment.

[0033] Optionally, the satellite multi-channel radiance images at different moments are at least two satellite multi-channel radiance images arranged in time series;

[0034] The target extrapolated image is at least two satellite multi-channel radiance extrapolated images arranged in time series within the future time.

[0035] Optionally, acquiring satellite multi-channel radiance images of the target area at different times includes:

[0036] Obtain multi-channel radiance image data of cloud images at different times;

[0037] Converting the cloud image multi-channel radiance image data into a satellite multi-channel radiance image in JPG format;

[0038] The satellite multi-channel radiance image is selected according to the target area to obtain the satellite multi-channel radiance images at the different moments.

[0039] In a second aspect, an embodiment of the present invention further provides a satellite multi-channel radiance image extrapolation device, comprising:

[0040] A receiving module, configured to receive an image extrapolation request of a target area at a future time;

[0041] An acquisition module is used to acquire satellite multi-channel radiance images of the target area at different times;

[0042] A training generation module is used to train an image extrapolation model using the satellite multi-channel radiance images at different times; wherein the image extrapolation model includes a convolutional recurrent neural network and a generative adversarial network; the convolutional recurrent neural network further includes: inputting the output result of the last ConvGRU unit of the previous time step into the first ConvGRU unit of the next time step;

[0043] The extrapolation module is used to input the satellite multi-channel radiance images at different times into the image extrapolation model, and output the target extrapolated image at the future time according to the image extrapolation request.

[0044] In a third aspect, an embodiment of the present invention further provides a computing device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the satellite multi-channel radiance image extrapolation method described in any one of the above items is implemented.

[0045] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed in a computer, the computer is caused to execute any one of the satellite multi-channel radiance image extrapolation methods described above.

[0046] An embodiment of the present invention provides a satellite multi-channel radiance image extrapolation method, device, and storage medium. The method obtains historical satellite multi-channel radiance images of the corresponding target area at different times, and uses these images to train a fused convolutional recurrent neural network and a generative adversarial network to obtain an image extrapolation model. The image extrapolation model is then used to perform extrapolation prediction of the satellite multi-channel radiance image to obtain a target extrapolated image at a future time. The convolutional recurrent neural network also includes: inputting the output result of the last ConvGRU unit of the previous time step into the first ConvGRU unit of the next time step. In this way, the present invention effectively extends the prediction time range of the extrapolated image through the improved convolutional recurrent neural network, and further improves the quality of the predicted extrapolated image through the generative adversarial network, so that high-quality multi-channel radiance extrapolated images within a longer time range can be effectively and accurately obtained based on the image extrapolation model. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0048] Figure 1 This is a flow chart of a satellite multi-channel radiance image extrapolation method provided by one embodiment of the present invention;

[0049] Figure 2 1 is a schematic diagram of an optimized structure of a convolutional recurrent neural network provided by one embodiment of the present invention;

[0050] Figure 3 and Figure 4 An extrapolated image and a real image of a FY-4A satellite multi-channel radiance image are obtained based on an image extrapolation model provided by an embodiment of the present invention;

[0051] Figure 5 This is a hardware architecture diagram of a computing device provided by one embodiment of the present invention;

[0052] Figure 6 The present invention provides a structural diagram of a satellite multi-channel radiance image extrapolation device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0054] like Figure 1 As shown, an embodiment of the present invention provides a satellite multi-channel radiance image extrapolation method, the method comprising:

[0055] Step 100: receiving an image extrapolation request for a target area at a future time;

[0056] Step 102: Acquire satellite multi-channel radiance images of the target area at different times;

[0057] Step 104: Using satellite multi-channel radiance images at different times, an image extrapolation model is trained; wherein the image extrapolation model includes a convolutional recurrent neural network and a generative adversarial network; the convolutional recurrent neural network further includes: inputting the output result of the last ConvGRU unit in the previous time step into the first ConvGRU unit in the next time step;

[0058] Step 106: input the satellite multi-channel radiance images at different times into the image extrapolation model, and output the target extrapolated image at the future time according to the image extrapolation request.

[0059] In an embodiment of the present invention, historical satellite multi-channel radiance images of the corresponding target area at different times are obtained, and these images are used to train a fused convolutional recurrent neural network and a generative adversarial network to obtain an image extrapolation model. The image extrapolation model is then used to perform extrapolation prediction of the satellite multi-channel radiance image to obtain a target extrapolated image at a future time. In addition, the convolutional recurrent neural network further includes: inputting the output result of the last ConvGRU unit of the previous time step into the first ConvGRU unit of the next time step. In this way, the present invention effectively extends the prediction time range of the extrapolated image through the improved convolutional recurrent neural network, and further improves the quality of the predicted extrapolated image through the generative adversarial network, so that high-quality satellite multi-channel radiance extrapolated images within a longer time range can be effectively and accurately obtained based on the image extrapolation model.

[0060] Described below Figure 1 How to perform the steps shown.

[0061] First, for step 100, an image extrapolation request for a target area at a future time is received, which includes a time range and a time granularity of the future time.

[0062] Specifically, the time interval between adjacent images in the target extrapolated image acquired in the future is determined based on the time granularity. For example, an image extrapolation request is to obtain satellite multi-channel radiance extrapolated images of the South China Sea region for the next four hours starting from this moment, with a time granularity of 15 minutes; that is, the request is to obtain 16 satellite multi-channel radiance extrapolated images for the next four hours.

[0063] With respect to step 102, obtaining satellite multi-channel radiance images of the target area at different times includes:

[0064] Obtain multi-channel radiance image data of cloud images at different times;

[0065] Convert cloud image multi-channel radiance image data into satellite multi-channel radiance image in JPG format;

[0066] The satellite multi-channel radiance image is selected according to the target area to obtain the satellite multi-channel radiance images at different times.

[0067] In the present invention, the corresponding data product can be downloaded from the Fengyun Satellite Remote Sensing Data Service Network according to actual usage needs, obtaining historical cloud map multi-channel radiance image data at different times. This data format is then converted to JPG format. The corresponding data for the target area is selected based on the target area to obtain satellite multi-channel radiance images at different times. It should be noted that the satellite multi-channel radiance image includes the temporal and spatial characteristic information of the corresponding area.

[0068] Specifically, continuing with the previous example, users can enter the product interface of the Fengyun Satellite Remote Sensing Data Service website, select the satellite model as FY-4A and the data type as HDF format, and complete the 2018 full-year data download according to personal needs. First, use the edited Python program to extract the grid data from the HDF file while minimizing the loss of file information, and convert the HDF format data into JPG format to obtain satellite multi-channel radiance images at different times. Then, according to the target area, obtain the coordinate range of the area, and use the edited program to cut the South China Sea area from the above-mentioned satellite multi-channel radiance image large scene data to obtain the satellite multi-channel radiance images of the South China Sea area at different times.

[0069] In step 104, the image extrapolation model is trained using satellite multi-channel radiance images at different times, including:

[0070] Determine at least two sample sets based on the satellite multi-channel radiance image; wherein each sample set includes a satellite multi-channel radiance image at a historical moment as input and a satellite multi-channel radiance image at a current moment as output; the sample sets include: a training set and a validation set;

[0071] Based on a convolutional recurrent neural network and a generative adversarial network, an initial image extrapolation model is constructed. In each time step of the convolutional recurrent neural network, the output of each layer's encoding ConvGRU unit is spliced into the input of the corresponding layer's decoding ConvGRU unit.

[0072] Training the initial image extrapolation model using at least two training sets to obtain a first image extrapolation model;

[0073] Input the validation set into the first image extrapolation model to obtain the predicted satellite multi-channel radiance image at the current moment;

[0074] Determine the root mean square error, critical success index, and HSS score based on the predicted satellite multi-channel radiance image at the current moment and the satellite multi-channel radiance image at the current moment in the validation set;

[0075] determining whether the root mean square error is greater than a first preset threshold, whether the critical success index is less than a second preset threshold, and whether the HSS score is less than a third preset threshold;

[0076] If the judgment result is no, the first image extrapolation model is determined to be the image extrapolation model, and the image extrapolation model is obtained; otherwise, the first image extrapolation model is optimized and adjusted.

[0077] It should be noted that the sample set also includes a test set; preferably, the amount of data in the training set, test set, and validation set in the sample set is divided into 70%, 15%, and 15% ratios. Optimizing and adjusting the first image extrapolation model refers to adjusting the first image extrapolation model through iterative optimization until the results of all three judgments are negative, at which point the iterative optimization ends.

[0078] In a specific embodiment, the generative adversarial network uses a temporal discriminant model instead of a common binary classification model that ignores the temporal features in the generated sequence. In this way, the temporal discriminant model can be used to urge the generator to generate a realistic distribution image while also ensuring that the generated image conforms to the prediction of the input sequence.

[0079] In the present invention, the ConvGRU unit structure in the convolutional recurrent neural network is as follows: Figure 2As shown. First, compared with the original method of extrapolating images only in the decoding stage, this structure realizes the extrapolation of an image at each time step. In the input stage, it also includes inputting the output result of the last ConvGRU unit of the previous time step into the first ConvGRU unit of the next time step. In this way, when extrapolating the image, the input of the image extrapolation model not only has the hidden state of the previous time step, but also has the extrapolated image of the previous time step. In this way, such a structure allows spatiotemporal memory information to flow between different time points and different ConvGRU units. Second, for the network structure within a single time step, the output of each layer of encoding ConvGRU unit is spliced into the input of the corresponding decoding ConvGRU unit. By making full use of multi-scale features, the problem of inaccurate upsampling caused by information loss in the downsampling process is avoided. In the downsampling process, it is preferred to use a convolution kernel with a step size of 2 and Max Pooling to downsample the feature map respectively, and then splice the output. The upsampling process is completed using a deconvolution kernel. The feature map can retain the global information of the image by downsampling the convolution kernel, while the Max Pool can highlight the information of strong features in the image, avoiding the disappearance of strong features in the extrapolated image. Third, the original image information at multiple scales is added to the input of the ConvGRU units in different layers of the encoder. For example, the original image is directly downsampled to one-third, one-sixth, and one-twelfth of the original image and then directly appended to the ConvGRU unit input of the corresponding layer. In this way, the convolutional recurrent neural network can reduce the learning forgetting of the model during the training process, reduce feature loss, and slow down its forgetting speed, thereby effectively extending the extrapolation time of the image extrapolation model.

[0080] In the present invention, three quantitative image consistency indices are used to evaluate the accuracy of the extrapolated image. Specific evaluation methods include root mean square error (RMS), critical success index (HSS), and HSS score (Gilbert, 1984). The RMS mainly evaluates the overall error between the extrapolated image and the true image, while the other two evaluation indices are mainly used to evaluate the similarity between the extrapolated image and the true image. Thus, the quality of the extrapolated image is comprehensively evaluated using these three evaluation methods, and the image extrapolation model is further optimized and iterated based on these three evaluation methods, thereby further improving the quality of the extrapolated image output by the image extrapolation model.

[0081] In step 104, it includes: introducing a loss function during training; the loss function is composed of a mean absolute error loss function, a mean square error loss function and a FocalLoss loss function;

[0082] When the value of the loss function is lower than the preset loss threshold, the training ends;

[0083] The loss function is expressed as follows:

[0084] L=λ1*MAE+λ2*MSE+λ3*FocalLoss (1)

[0085] Among them, L is used to characterize the loss function; MAE is used to characterize the mean absolute error loss function; λ1 is used to characterize the weight coefficient of MAE; MSE is used to characterize the mean square error loss function; λ2 is used to characterize the weight coefficient of MSE; FocalLoss is used to characterize the FocalLoss loss function; λ3 is used to characterize the weight coefficient of FocalLoss.

[0086] It should be noted that λ1, λ2, and λ3 are determined by iterative optimization during the training process.

[0087] In the present invention, the adopted loss function is obtained by optimizing the perceptual loss and optical flow loss. Through the perceptual loss FocalLoss loss function, the loss calculation is performed on the feature image of higher dimension, and the optical flow loss is used to calculate the difference between the extrapolated image frame and the real image frame, and finally the optimized loss function as shown in formula (1) is obtained, thereby reducing the influence of the conventional loss calculation of the satellite multi-channel radiance image pixels on the image extrapolation model, thereby improving the quality of the extrapolated image.

[0088] In another specific embodiment, after determining the root mean square error, critical success index, and HSS score in step 104, the following operations may be further performed:

[0089] Determining whether the differences between five consecutive root mean square errors are all greater than a fourth preset threshold, whether the differences between five consecutive critical success indices are all greater than a fifth preset threshold, and whether the differences between five consecutive HSS scores are all greater than a sixth preset threshold;

[0090] If the judgment result is no, the first image extrapolation model is determined to be the image extrapolation model; otherwise, the first image extrapolation model is optimized and adjusted.

[0091] In the present invention, the three evaluation methods are also used to further optimize and iterate the image extrapolation model. However, the training is terminated when the performance no longer improves for five consecutive epochs on the validation set. This is beneficial to further improve the quality of the target extrapolated image output by the image extrapolation model obtained by the training, ensuring that the target extrapolated image has stable high quality.

[0092] More specifically, as described in the previous example, the ADS optimizer is used to train the optimized convolutional recurrent neural network and generative adversarial network using satellite multi-channel radiance images of the South China Sea at different times. The learning rate is set to 1e-4. When the performance of the trained model no longer improves after 5 consecutive epochs on the validation set, the training is terminated to obtain an image extrapolation model that integrates the convolutional recurrent neural network and the generative adversarial network.

[0093] Finally, in step 106 , the satellite multi-channel radiance images at different times are input into the image extrapolation model, and the target extrapolated image at the future time is output according to the image extrapolation request.

[0094] Specifically, following the previous example, the satellite multi-channel radiance image of the South China Sea region from 10:00 to 12:00 on September 6, 2018 is input into the image extrapolation model trained above. According to the image extrapolation request, 16 target extrapolated images arranged in time series from 12:15 to 16:00 are obtained, as shown in the following example: Figure 4 shown; among them, Figure 4 The extrapolated image of the target (i.e., extrapolated image) and the corresponding directly acquired real satellite multi-channel radiance image (i.e., real image) at different times are shown. Figure 4 The root mean square errors of the 16 target extrapolated images obtained are all no more than 0.07, the critical success index is no less than 0.7, and the HSS score is no less than 0.7, which indicates that the target extrapolated images within the next 4 hours predicted by the image extrapolation model are all of high quality.

[0095] The present invention designs and builds an intelligent network that integrates convolutional recurrent neural networks and generative adversarial networks, optimizes the convolutional recurrent neural network structure and related loss functions, and the trained image extrapolation model effectively and accurately realizes the extrapolation of satellite multi-channel radiance images over a longer time range, solving the problem that satellite multi-channel radiance images are difficult to effectively extrapolate and generate for a longer time, and the quality of image data generated by extrapolation for a longer time is poor. The main application directions of the satellite multi-channel radiance image extrapolation provided by the present invention include: special exercises, and effective and accurate acquisition of meteorological parameters of a certain area before drills (which can be obtained by inverting the extrapolated images), which improves the extrapolation time of satellite multi-channel radiance images and improves the quality of extrapolated image generation. In addition, satellite-captured images of natural disasters such as typhoons, floods, and wildfires can also be used for trend prediction. Other application directions, such as weather broadcasts and real-time weather forecasts, etc.

[0096] like Figure 5 、 Figure 6As shown, an embodiment of the present invention provides a satellite multi-channel radiance image extrapolation device. The device embodiment can be implemented by software, hardware, or a combination of software and hardware. From the hardware level, Figure 5 The figure shows a hardware architecture diagram of a computing device where a satellite multi-channel radiance image extrapolation device is located, in addition to Figure 5 In addition to the processor, memory, network interface, and non-volatile memory shown, the computing device in the embodiment may also include other hardware, such as a forwarding chip responsible for processing messages, etc. Taking software implementation as an example, Figure 6 As shown, as a logical device, the CPU of the computing device in which it is located reads the corresponding computer program in the non-volatile memory into the internal memory and runs it. This embodiment provides a satellite multi-channel radiance image extrapolation device, including: a receiving module 600, an acquisition module 602, a training generation module 604 and an extrapolation module 606;

[0097] A receiving module 600 is configured to receive an image extrapolation request of a target area at a future time;

[0098] An acquisition module 602 is used to acquire satellite multi-channel radiance images of the target area at different times;

[0099] The training generation module 604 is configured to train an image extrapolation model using satellite multi-channel radiance images at different times. The image extrapolation model includes a convolutional recurrent neural network and a generative adversarial network. The convolutional recurrent neural network further includes: inputting the output of the last ConvGRU unit in the previous time step into the first ConvGRU unit in the next time step.

[0100] The extrapolation module 606 is used to input the satellite multi-channel radiance images at different times into the image extrapolation model, and output the target extrapolated image at a future time according to the image extrapolation request.

[0101] In some specific implementations, the receiving module 600 may be used to perform the above step 100 , the acquiring module 602 may be used to perform the above step 102 , the training generation module 604 may be used to perform the above step 104 , and the extrapolation module 606 may be used to perform the above step 106 .

[0102] In some specific implementations, the training generation module 604 is further configured to perform the following operations:

[0103] Determine at least two sample sets based on the satellite multi-channel radiance image; wherein each sample set includes a satellite multi-channel radiance image at a historical moment as input and a satellite multi-channel radiance image at a current moment as output; the sample sets include: a training set and a validation set;

[0104] Based on a convolutional recurrent neural network and a generative adversarial network, an initial image extrapolation model is constructed. In each time step of the convolutional recurrent neural network, the output of each layer's encoding ConvGRU unit is spliced into the input of the corresponding layer's decoding ConvGRU unit.

[0105] The initial image extrapolation model is trained using at least two training sets to obtain an image extrapolation model.

[0106] In some specific implementations, the training generation module 604 is further configured to perform the following operations:

[0107] The loss function is introduced during training; the loss function consists of the mean absolute error loss function, the mean square error loss function and the FocalLoss loss function;

[0108] When the value of the loss function is lower than the preset loss threshold, the training ends;

[0109] The loss function is expressed as follows:

[0110] L=λ1*MAE+λ2*MSE+λ3*FocalLoss

[0111] Among them, L is used to characterize the loss function; MAE is used to characterize the mean absolute error loss function; λ1 is used to characterize the weight coefficient of MAE; MSE is used to characterize the mean square error loss function; λ2 is used to characterize the weight coefficient of MSE; FocalLoss is used to characterize the FocalLoss loss function; λ3 is used to characterize the weight coefficient of FocalLoss.

[0112] In some specific implementations, the training generation module 604 is further configured to perform the following operations:

[0113] After training the initial image extrapolation model using at least two training sets, a first image extrapolation model is obtained;

[0114] Input the validation set into the first image extrapolation model to obtain the predicted satellite multi-channel radiance image at the current moment;

[0115] Determine the root mean square error, critical success index, and HSS score based on the predicted satellite multi-channel radiance image at the current moment and the satellite multi-channel radiance image at the current moment in the validation set;

[0116] determining whether the root mean square error is greater than a first preset threshold, whether the critical success index is greater than a second preset threshold, and whether the HSS score is greater than a third preset threshold;

[0117] If the judgment result is no, the first image extrapolation model is determined to be the image extrapolation model; otherwise, the first image extrapolation model is optimized and adjusted.

[0118] In some specific implementations, the training generation module 604 is further configured to perform the following operations:

[0119] After training the initial image extrapolation model using at least two training sets, a first image extrapolation model is obtained;

[0120] Input the validation set into the first image extrapolation model to obtain the predicted satellite multi-channel radiance image at the current moment;

[0121] The root mean square error, critical success index, and HSS score are obtained based on the predicted satellite multi-channel radiance image at the current moment and the satellite multi-channel radiance image at the current moment in the validation set.

[0122] Determining whether the differences between five consecutive root mean square errors are all greater than a fourth preset threshold, whether the differences between five consecutive critical success indices are all greater than a fifth preset threshold, and whether the differences between five consecutive HSS scores are all greater than a sixth preset threshold;

[0123] If the judgment result is no, the first image extrapolation model is determined to be the image extrapolation model; otherwise, the first image extrapolation model is optimized and adjusted.

[0124] In some specific implementations, the image extrapolation request includes a time range and a time granularity of a future moment.

[0125] In some specific embodiments, the satellite multi-channel radiance images at different times are at least two satellite multi-channel radiance images arranged in time series;

[0126] The target extrapolated images are at least two satellite multi-channel radiance extrapolated images arranged in time series at a future time.

[0127] In some specific implementations, the acquisition module 602 is further configured to perform the following operations:

[0128] Obtain multi-channel radiance image data of cloud images at different times;

[0129] Convert cloud image multi-channel radiance image data into satellite multi-channel radiance image in JPG format;

[0130] The satellite multi-channel radiance image is selected according to the target area to obtain the satellite multi-channel radiance images at different times.

[0131] It should be understood that the structure illustrated in the embodiments of the present invention does not constitute a specific limitation on the satellite multi-channel radiance image extrapolation device. In other embodiments of the present invention, the satellite multi-channel radiance image extrapolation device may include more or fewer components than illustrated, or may combine or separate certain components, or employ different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of both.

[0132] The information interaction, execution process, etc. between the modules in the above-mentioned device are based on the same concept as the embodiment of the method of the present invention. For specific contents, please refer to the description in the embodiment of the method of the present invention and will not be repeated here.

[0133] An embodiment of the present invention further provides a computing device including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, a satellite multi-channel radiance image extrapolation method in any embodiment of the present invention is implemented.

[0134] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the processor executes a satellite multi-channel radiance image extrapolation method according to any embodiment of the present invention.

[0135] Specifically, a system or device equipped with a storage medium can be provided, on which software program codes that implement the functions of any of the above-mentioned embodiments are stored, and a computer (or CPU or MPU) of the system or device can be enabled to read and execute the program codes stored in the storage medium.

[0136] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute part of the present invention.

[0137] Examples of storage media for providing program code include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, the program code can be downloaded from a server computer via a communication network.

[0138] In addition, it should be clear that the functions of any of the above embodiments can be achieved not only by executing the program code read by the computer, but also by enabling the operating system operating on the computer to complete part or all of the actual operations based on the instructions of the program code.

[0139] In addition, it can be understood that the program code read from the storage medium is written into a memory provided in an expansion board inserted into the computer or into a memory provided in an expansion module connected to the computer, and then based on the instructions of the program code, a CPU installed on the expansion board or expansion module is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above embodiments.

[0140] It should be noted that, in this article, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises", "comprising" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprising a ..." do not exclude the presence of other identical factors in the process, method, article or device comprising the elements.

[0141] Those skilled in the art will understand that all or part of the steps of implementing the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk, etc. Various media that can store program codes.

[0142] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A satellite multi-channel radiance image extrapolation method, characterized in that: include: Receiving an image extrapolation request for a target area at a future time; Acquire satellite multi-channel radiance images of the target area at different times; An image extrapolation model is trained using the satellite multi-channel radiance images at different times; wherein the image extrapolation model includes a convolutional recurrent neural network and a generative adversarial network; the convolutional recurrent neural network further includes: inputting the output result of the last ConvGRU unit of the previous time step into the first ConvGRU unit of the next time step; The satellite multi-channel radiance images at different moments are input into the image extrapolation model, and a target extrapolated image at the future moment is output according to the image extrapolation request.

2. The method according to claim 1, characterized in that The method of training an image extrapolation model using the satellite multi-channel radiance images at different times includes: Determine at least two sample sets based on the satellite multi-channel radiance image; wherein each sample set includes a satellite multi-channel radiance image at a historical moment as input and a satellite multi-channel radiance image at a current moment as output; the sample sets include: a training set and a validation set; Based on the convolutional recurrent neural network and the generative adversarial network, an initial image extrapolation model is constructed; wherein, in each time step of the convolutional recurrent neural network, the output of each layer of the encoding ConvGRU unit is spliced to the input of the decoding ConvGRU unit of the corresponding layer; The initial image extrapolation model is trained using the at least two groups of training sets to obtain the image extrapolation model.

3. The method according to claim 1 or 2, characterized in that The method of training an image extrapolation model using the satellite multi-channel radiance images at different times includes: Introducing a loss function during training; the loss function consists of a mean absolute error loss function, a mean square error loss function, and a FocalLoss loss function; When the value of the loss function is lower than a preset loss threshold, the training is terminated; The loss function is expressed as follows: L=λ l *MAE+λ2+MSE+λ3*Focal Loss Among them, L is used to characterize the loss function; MAE is used to characterize the mean absolute error loss function; λ1 is used to characterize the weight coefficient of the MAE; MSE is used to characterize the mean square error loss function; λ2 is used to characterize the weight coefficient of the MSE; FocalLoss is used to characterize the FocalLoss loss function; λ3 is used to characterize the weight coefficient of the FocalLoss.

4. The method according to claim 2, characterized in that The step of training the initial image extrapolation model using the at least two training sets to obtain the image extrapolation model includes: After the initial image extrapolation model is trained using the at least two training sets, a first image extrapolation model is obtained; Inputting the verification set into the first image extrapolation model to obtain a predicted satellite multi-channel radiance image at the current moment; Determining a root mean square error, a critical success index, and an HSS score based on the predicted satellite multi-channel radiance image at the current moment and the satellite multi-channel radiance image at the current moment in the validation set; Determining whether the root mean square error is greater than a first preset threshold, whether the critical success index is less than a second preset threshold, and whether the HSS score is less than a third preset threshold; If the judgment result is no, the first image extrapolation model is determined to be the image extrapolation model; otherwise, the first image extrapolation model is optimized and adjusted.

5. The method according to claim 2, characterized in that The step of training the initial image extrapolation model using the at least two training sets to obtain the image extrapolation model includes: After the initial image extrapolation model is trained using the at least two training sets, a first image extrapolation model is obtained; Inputting the verification set into the first image extrapolation model to obtain a predicted satellite multi-channel radiance image at the current moment; Obtaining a root mean square error, a critical success index, and an HSS score based on the predicted satellite multi-channel radiance image at the current moment and the satellite multi-channel radiance image at the current moment in the validation set; Determining whether the differences between five consecutive root mean square errors are all greater than a fourth preset threshold, whether the differences between five consecutive critical success indices are all greater than a fifth preset threshold, and whether the differences between five consecutive HSS scores are all greater than a sixth preset threshold; If the judgment result is no, the first image extrapolation model is determined to be the image extrapolation model; otherwise, the first image extrapolation model is optimized and adjusted.

6. The method according to claim 1, characterized in that The image extrapolation request includes a time range and a time granularity of a future moment; and / or, The satellite multi-channel radiance images at different times are at least two satellite multi-channel radiance images arranged in time series; The target extrapolated image is at least two satellite multi-channel radiance extrapolated images arranged in time series within the future time.

7. The method according to claim 1, characterized in that The acquiring of satellite multi-channel radiance images of the target area at different times includes: Obtain multi-channel radiance image data of cloud images at different times; Converting the cloud image multi-channel radiance image data into a satellite multi-channel radiance image in JPG format; The satellite multi-channel radiance image is selected according to the target area to obtain the satellite multi-channel radiance images at the different moments.

8. A satellite multi-channel radiance image extrapolation device, characterized in that: include: A receiving module, configured to receive an image extrapolation request of a target area at a future time; An acquisition module is used to acquire satellite multi-channel radiance images of the target area at different times; A training generation module is used to train an image extrapolation model using the satellite multi-channel radiance images at different times; wherein the image extrapolation model includes a convolutional recurrent neural network and a generative adversarial network; the convolutional recurrent neural network further includes: inputting the output result of the last ConvGRU unit of the previous time step into the first ConvGRU unit of the next time step; The extrapolation module is used to input the satellite multi-channel radiance images at different times into the image extrapolation model, and output the target extrapolated image at the future time according to the image extrapolation request.

9. A computing device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 7.

Images