A Spatial Super-Resolution Reconstruction Model for Time-Varying Data and Its Training Method

Abstract

The present invention relates to a spatial super-resolution reconstruction model for time-varying data and a training method thereof. The model includes a first convolutional layer, a basic module, a first upsampling module, a second convolutional layer, a first InstanceNorm layer, a first LeakyReLU activation function layer, a third convolutional layer, and a first Sigmoid activation function layer connected in sequence. The basic module includes N basic units connected in sequence, and skip connections for low-resolution volume data and skip connections for high-resolution volume data are created between adjacent basic units. The basic unit includes a channel attention layer, a spatial attention layer, a second InstanceNorm layer, a second upsampling module, a downsampling module, and a second LeakyReLU activation function layer. The channel attention layer, the spatial attention layer, and the second InstanceNorm layer are connected in sequence, the second upsampling module, the downsampling module, and the second InstanceNorm layer are connected in sequence, and the second InstanceNorm layer is connected to the second LeakyReLU activation function layer. The present invention trains the model considering the temporal change information of time-varying data to obtain a model with good super-resolution reconstruction effect.

Description

Technical Field

[0001] The present invention belongs to the technical field of super-resolution reconstruction, and particularly relates to a spatial super-resolution reconstruction model for time-varying data and a training method therefor. Background Art

[0002] Time-varying data often arises in scientific simulations for various applications such as weather forecasting, computational fluid dynamics, combustion science, computational cosmology, climate model research, etc. These time-varying data usually contain complex and large-scale features and require the application of in-situ visualization techniques. In-situ visualization performs analysis and visualization by reducing features and data, and the super-resolution reconstruction technology of volume data plays an important role in the in-situ visualization of volume data and directly affects the results. Although many super-resolution reconstruction techniques have been proposed in the past, few studies have focused on the super-resolution reconstruction of time-varying data. At the same time, the rapid development of deep learning technology has provided new ideas and methods for super-resolution reconstruction technology, and it has achieved good results in the field of images.

[0003] For the super-resolution reconstruction of time-varying data, we need to consider the coherence of time-varying data in time series, and at the same time, we need to ensure the super-resolution reconstruction effect in visual perception. Summary of the Invention

[0004] Aiming at the above problems existing in the prior art, the present invention proposes a spatial super-resolution reconstruction model for time-varying data and a training method therefor. The reconstruction model can be trained using existing time-varying data, and then three temporally consecutive low-resolution volume data are input into the trained model. After processing, the model will output the corresponding three temporally consecutive reconstructed high-resolution data.

[0005] The present invention adopts the following technical solutions:

[0006] A spatial super-resolution reconstruction model for time-varying data, comprising an input layer, a first convolutional layer, a basic module, a first upsampling module, a second convolutional layer, a first InstanceNorm layer, a first LeakyReLU activation function layer, a third convolutional layer, a first Sigmoid activation function layer, and an output layer connected in sequence. The input layer is used to input low-resolution time-varying data, and the output layer is used to output reconstructed high-resolution time-varying data;

[0007] The basic module includes N basic units connected in sequence, and skip connections for low-resolution volume data and skip connections for high-resolution volume data are created between adjacent two basic units;

[0008] The basic unit includes a channel attention layer, a spatial attention layer, a second InstanceNorm layer, a second upsampling module, a downsampling module, and a second LeakyReLU activation function layer. The channel attention layer, the spatial attention layer, and the second InstanceNorm layer are connected in sequence. The second upsampling module, the downsampling module, and the second InstanceNorm layer are connected in sequence. The second InstanceNorm layer is connected to the second LeakyReLU activation function layer.

[0009] As a preferred solution, both the first upsampling module and the second upsampling module include at least one enlarged volume data structure. The enlarged volume structure includes a fourth convolutional layer, a PixelShuffle layer, and a third LeakyReLU activation function layer connected in sequence.

[0010] As a preferred solution, the downsampling module includes at least one reduced volume data structure. The reduced volume data structure includes a fifth convolutional layer and a fourth LeakyReLU activation function layer connected to each other.

[0011] As a preferred solution, the input of the basic unit consists of three parts, namely the volume data output by the previous basic unit, the high-resolution volume data generated by the second upsampling module in the previous basic unit, and the low-resolution volume data generated by the downsampling module in the previous basic unit.

[0012] As a preferred solution, a skip connection is established between the volume data output by the basic unit and the low-resolution volume data generated by the downsampling module in this basic unit, and after connection, it is input into the second upsampling module in the next basic unit;

[0013] A skip connection is established between the high-resolution volume data generated by the second upsampling module in the basic unit and the high-resolution volume data generated by the second upsampling module in the previous basic unit, and after connection, it is input into the downsampling module in this basic unit.

[0014] The present invention also provides a training method for a spatial super-resolution reconstruction model of time-varying data, including the steps of:

[0015] S1. The user customizes the number of training cycles, the relevant parameters of the RAdam optimizer, and the relevant parameters of the loss function;

[0016] S2. Preprocess the original high-resolution time-varying data to obtain the original low-resolution data spliced in the channel dimension, the original high-resolution data spliced in the channel dimension, and use the original low-resolution data spliced in the channel dimension as the training data;

[0017] S3. Divide the training data into a training set and a validation set;

[0018] S4. Shuffle the training set and extract three consecutive original low-resolution data concatenated in the channel dimension at time steps, and input them into the above spatial super-resolution reconstruction model, and output the reconstructed high-resolution data;

[0019] S5. Calculate the loss value based on the reconstructed high-resolution data, the original high-resolution data concatenated in the channel dimension, and the loss function;

[0020] S6. Perform backpropagation based on the loss value, and optimize the parameters of the super-resolution reconstruction model according to the gradient descent algorithm to complete the current training cycle;

[0021] S7. Verify the current model parameters based on the validation set. If the verification is successful, save the model parameters of the current cycle. If the verification fails, return to step S4 until the number of user-defined training cycles is reached.

[0022] As a preferred solution, step S2 includes the following steps:

[0023] S2.1. Normalize the original high-resolution time-varying data to obtain the normalized time-varying data;

[0024] S2.2. Customize the random cropping multiple and downsampling multiple of the volume data according to the super-resolution reconstruction multiple, the size of the normalized time-varying data, and the number of normalized time-varying data;

[0025] S2.3. Select three consecutive normalized time-varying data at time steps as a group, and randomly crop the three normalized time-varying data at the same position according to the random cropping multiple of the data;

[0026] S2.4. Downsample the three cropped normalized time-varying data using trilinear interpolation three times according to the downsampling multiple to obtain consecutive original low-resolution data;

[0027] S2.5. Concatenate the consecutive original low-resolution data and the corresponding consecutive normalized time-varying data in the channel dimension respectively to obtain the original low-resolution data concatenated in the channel dimension and the original high-resolution data concatenated in the channel dimension.

[0028] As a preferred solution, in step S5, the loss value includes content loss value, structural loss value, and perceptual loss value.

[0029] As a preferred solution, the content loss value is the mean square error value between the reconstructed high-resolution data and the corresponding original high-resolution data concatenated in the channel dimension;

[0030] The structural loss value is the structural similarity index between the reconstructed high-resolution data and the corresponding original high-resolution data concatenated in the channel dimension;

[0031] The perceived loss value is obtained by using a convolutional network to separately extract the high-dimensional features between the reconstructed high-resolution data and the original high-resolution data after splicing in the channel dimension, and calculating the mean square error between the two.

[0032] As a preferred solution, the formula of the loss function is:

[0033] L total = λ1L content + λ2L structure + λ3L perception ,

[0034] wherein, L content is the content loss value, L structure is the structure loss value, L perception is the perceived loss value, and λ1, λ2, and λ3 are user-defined and used to adjust the magnitudes of the corresponding loss values.

[0035] The beneficial effects of the present invention are:

[0036] The present invention provides a spatial super-resolution reconstruction model for time-varying data and its training method. Considering the time-series change information of time-varying data, a more suitable network model can be trained by adjusting the training parameters related to the model, so as to obtain a reconstruction model with better super-resolution reconstruction effect for time-varying data. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0038] Figure 1 is a structural schematic diagram of a spatial super-resolution reconstruction model for time-varying data according to the present invention;

[0039] Figure 2 is a structural schematic diagram of the basic unit according to the present invention;

[0040] Figure 3 is a structural schematic diagram of the upsampling module according to the present invention;

[0041] Figure 4 is a structural schematic diagram of the downsampling module according to the present invention;

[0042] Figure 5 is a flowchart of a training method for a spatial super-resolution reconstruction model for time-varying data according to the present invention;

[0043] Figure 6 It is the first comparison chart between the original data and the data after super-resolution reconstruction using the reconstruction model of the present invention.

[0044] Figure 7 It is the second comparison chart between the original data and the data after super-resolution reconstruction using the reconstruction model of the present invention. Detailed implementation manners

[0045] The following uses specific specific embodiments to illustrate the implementation manners of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific implementation manners. Various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, without conflict, the following embodiments and the features in the embodiments can be combined with each other.

[0046] Refer to Figure 1 As shown, this embodiment provides a spatial super-resolution reconstruction model for time-varying data, including an input layer, a first convolutional layer, a basic module, a first upsampling module, a second convolutional layer, a first InstanceNorm layer, a first LeakyReLU activation function layer, a third convolutional layer, a first Sigmoid activation function layer, and an output layer that are connected in sequence. The input layer is used to input low-resolution time-varying data, and the output layer is used to output reconstructed high-resolution time-varying data;

[0047] The basic module includes N basic units connected in sequence. Skip connections for low-resolution volume data and skip connections for high-resolution volume data are created between two adjacent basic units;

[0048] The basic unit includes a channel attention layer, a spatial attention layer, a second InstanceNorm layer, a second upsampling module, a downsampling module, and a second LeakyReLU activation function layer. The channel attention layer, the spatial attention layer, and the second InstanceNorm layer are connected in sequence. The second upsampling module, the downsampling module, and the second InstanceNorm layer are connected in sequence. The second InstanceNorm layer is connected to the second LeakyReLU activation function layer.

[0049] It should be noted that the time-varying data represents a series of three-dimensional data that is continuous in time, and specifically can be time-varying data of hurricane simulation, time-varying data of salt dissolution behavior simulation in water, time-varying data of hydrogen ion ionization instability simulation, time-varying data of incompressible flow field simulation, and so on.

[0050] Specifically:

[0051] Refer toFigure 3 As shown, the first upsampling module and the second upsampling module both include at least one enlarged volume data structure. The enlarged volume structure includes a fourth convolutional layer, a PixelShuffle layer, and a third LeakyReLU activation function layer connected in sequence. Assume that the depth of the input volume data is D, the height is H, the width is W, and the number of channels is C. First, the volume data is input into the convolutional layer, and volume data with a depth of D, a height of H, a width of W, and a number of channels of 2 3 C can be generated; then the PixelShuffl layer can extract the feature volume data in the channel dimension to double the depth, height, and width of the volume data, and the number of channels returns to C; finally, the volume data is input into the LeakyReLU activation function for calculation. Such a structure can be generated multiple times according to the target magnification factor. For example, if we want to magnify the low-resolution volume data by 4 times, then the upsampling module should have 2 such structures to magnify the volume data.

[0052] Refer to Figure 4 As shown, the downsampling module includes at least one reduced volume data structure. The reduced volume data structure includes a fifth convolutional layer and a fourth LeakyReLU activation function layer connected. It will reduce the input volume data by 2 times. Such a structure can be repeatedly generated according to the target reduction multiple. For example, if we want to reduce the volume data by 4 times, then the downsampling module should have 2 such basic structures to reduce the volume data.

[0053] Refer to Figure 5 As shown, based on the spatial super-resolution reconstruction model of time-varying data provided above, this embodiment also gives the training method of this model, including the steps:

[0054] S1. The user defines the number of training epochs, the relevant parameters of the RAdam optimizer, and the relevant parameters of the loss function;

[0055] S2. Preprocess the original high-resolution time-varying data to obtain the original low-resolution data concatenated in the channel dimension, the original high-resolution data concatenated in the channel dimension, and use the original low-resolution data concatenated in the channel dimension as the training data;

[0056] S3. Divide the training data into a training set and a validation set;

[0057] S4. After shuffling the training set, extract three original low-resolution data concatenated in the channel dimension that are consecutive in the time step and input them into the above-mentioned spatial super-resolution reconstruction model, and output the reconstructed high-resolution data;

[0058] S5. Calculate the loss value based on the reconstructed high-resolution data, the original high-resolution data concatenated in the channel dimension, and the loss function;

[0059] S6. Perform backpropagation based on the loss value, and optimize the super-resolution reconstruction model parameters according to the gradient descent algorithm to complete the current training cycle;

[0060] S7. Validate the current model parameters based on the validation set. If the validation is successful, save the model parameters of the current cycle. If the validation fails, return to step S4 until the number of cycles defined by the user for training is reached.

[0061] In step S2, the following steps are included:

[0062] S2.1. Normalize the original high-resolution temporal variant data, specifically, normalize the original high-resolution temporal variant data to the range [0, 1] using the global maximum and minimum values to obtain the normalized temporal variant data;

[0063] S2.2. The user customizes the random cropping multiple and the upsampling / downsampling multiple of the volume data according to the hardware conditions, the super-resolution reconstruction multiple, the size of the normalized temporal variant data, and the number of normalized temporal variant data. If the size of the volume data is (D, H, W), the random cropping multiple is F, and the upsampling / downsampling multiple is K, then the following conditions need to be met:

[0064] D % (F × K) = 0, H % (F × K) = 0, W % (F × K) = 0, K % 2 = 0;

[0065] where D, H, and W respectively represent the depth, height, and width of the volume data;

[0066] S2.3. Select three consecutive normalized temporal variant data at time steps as a group. If the user defines a random cropping multiple, first randomly crop these three normalized temporal variant data at the same position according to the multiple defined by the user;

[0067] S2.4. According to the downsampling multiple, perform downsampling on the three cropped normalized temporal variant data using trilinear interpolation three times to obtain continuous original low-resolution data;

[0068] S2.5. Concatenate the continuous original low-resolution data and the corresponding continuous normalized temporal variant data respectively in the channel dimension to obtain the original low-resolution data concatenated in the channel dimension and the original high-resolution data concatenated in the channel dimension.

[0069] In step S5, the loss value includes content loss value, structure loss value, and perceptual loss value.

[0070] The content loss value is the reconstructed high-resolution data and the corresponding original high-resolution data concatenated in the channel dimension The mean square error value between them, and the formula for the mean square error is:

[0071] MSE(x, y) = (x - y) 2 ,

[0072] Where h is high-resolution, representing that the data is high-resolution, m is model, representing the volume data output by the model, g is ground truth, that is, the real data, and x, y are the two input parameters of the MSE function.

[0073] So the formula for the content loss value is:

[0074]

[0075] The structure loss value is the difference in structure between the reconstructed high-resolution data and the original high-resolution data concatenated in the channel dimension. We use the structural similarity index to define it, and the formula for the structural similarity is;

[0076]

[0077] Where μ x is the average value of x, μ y is the average value of y, σ xy is the covariance of x and y, is the variance of x, is the variance of y, c1 = (k1L) 2 、c2 = (k2L) 2 are constants used for stabilization, L is the dynamic range of the voxel values, k1 = 0.01, k2 = 0.03, so the formula for the structure loss value is:

[0078]

[0079] The perceptual loss is defined as using a convolutional network to extract the high-dimensional features between the reconstructed high-resolution data output by the model and the original high-resolution data concatenated in the channel dimension respectively, and obtaining the loss value by calculating the mean square error between the two. In this technical solution, we use the VGG-19 network extended to three dimensions to extract the high-dimensional features of the data, so the formula for the perceptual loss value is:

[0080]

[0081] Furthermore, the formula for the loss function is:

[0082] L total = λ1L content + λ2L structure+λ3L perception ,

[0083] Among them, L content is the content loss value, L structure is the structure loss value, L perception is the perceptual loss value, and λ1, λ2, and λ3 are user-defined and used to adjust the magnitudes of the corresponding loss values.

[0084] In step S7, when a cycle of training is completed, the model will use the validation set for validation and calculate the PSNR or SSIM metric value. If the value of the metric rises, the model parameters of this cycle will be saved.

[0085] Referring to Figure 2 as shown, the specific way for the basic unit to process data is:

[0086] The input of the basic unit consists of three parts, namely the volume data X main (It should be noted here that if it is the first basic unit in the basic module, then this input is the output of the first convolutional layer), the high-resolution volume data X high generated by the second upsampling module in the previous basic unit, and the low-resolution volume data X low generated by the downsampling module in the previous basic unit;

[0087] First, we input X main into the channel attention layer to calculate the channel attention weight Attn c . Since, as described in step S2, 3 temporally consecutive volume data are concatenated along the channel dimension, the channel attention layer can learn the relationships of the volume data in the time series. Then we multiply Attn c and X main and obtain Attn c X main ;

[0088] Second, we input Attn c X main into the spatial attention layer to calculate the spatial attention weight Attn s . Similarly, we multiply Attn s and Attn c X main to obtain X attn , that is

[0089] Furthermore, create a skip connection for X main and X low , that is X main +X low , and input X main +Xlow It is input into the second upsampling module of the current basic unit for amplification to obtain X'. high , if the current basic unit is the first one in the basic module, there is no such skip connection, but X is directly main input into the second upsampling module of the current basic unit to obtain X'. high ;

[0090] Furthermore, a skip connection about X' high and X high is created, that is, X' high +X high . X' high +X high is input into the downsampling module of the current basic unit for reduction to obtain X'. low , if this basic unit is the first one in the basic module, there is no such skip connection, but X' higt is directly input into the downsampling module of the current basic unit for reduction to obtain X'. low ;

[0091] Furthermore, a skip connection about X' low and is created, that is it is input into the InstanceNorm layer for normalization processing to obtain X'. main ;

[0092] Finally, X' main is input into the LeakyReLU activation function for calculation, and then together with X' high and X' low is output as the input of the next basic unit.

[0093] Based on the above space super-resolution reconstruction model for time-varying data after training, the user inputs three consecutive low-resolution volume data at time steps as a group into the model for super-resolution reconstruction. The trained model can output the corresponding high-resolution volume data, and finally, visualization operations are performed on the output results. Specifically, volume rendering is used to visualize the generated high-resolution volume data in the form of an animation.

[0094] Referring to Figure 6 and Figure 7 shown, the results of reconstructing low-resolution volume data by the reconstruction model obtained by the training method of the present invention and the real data are respectively shown. It can be clearly seen from the figure the super-resolution reconstruction effect of the model. Among them, Figure 6 it is for the reconstruction of time-varying data of hydrogen ion ionization instability, Figure 7 and it is for the reconstruction of time-varying data of incompressible flow fields.

[0095] The embodiments described above are only descriptions of the preferred embodiments of the present invention, and do not limit the scope of the present invention. Without departing from the spirit of the present invention's design, various deformations and improvements made by those of ordinary skill in the art to the technical solutions of the present invention shall fall within the protection scope of the present invention.

Claims

1. A training method for a spatial super-resolution reconstruction model of time-varying data, characterized in that Including the steps: S1. The user defines the number of training cycles, the relevant parameters of the RAdam optimizer, and the relevant parameters of the loss function; S2. Preprocess the original high-resolution time-variant data to obtain the original low-resolution data concatenated in the channel dimension, the original high-resolution data concatenated in the channel dimension, and use the original low-resolution data concatenated in the channel dimension as the training data; S3. Divide the training data into a training set and a validation set; S4. After shuffling the training set, extract three consecutive original low-resolution data concatenated in the channel dimension at time steps and input them into the spatial super-resolution reconstruction model, and output the reconstructed high-resolution data; S5. Calculate the loss value based on the reconstructed high-resolution data, the original high-resolution data concatenated in the channel dimension, and the loss function; S6. Perform backpropagation based on the loss value, and optimize the parameters of the super-resolution reconstruction model according to the gradient descent algorithm to complete the current training cycle; S7. Validate the current model parameters based on the validation set. If the validation is successful, save the model parameters of the current cycle. If the validation fails, return to step S4 until the number of training cycles defined by the user is reached; The spatial super-resolution reconstruction model includes an input layer, a first convolutional layer, a basic module, a first upsampling module, a second convolutional layer, a first InstanceNorm layer, a first LeakyReLU activation function layer, a third convolutional layer, a first Sigmoid activation function layer, and an output layer connected in sequence. The input layer is used to input low-resolution time-variant data, and the output layer is used to output reconstructed high-resolution time-variant data; The basic module includes N consecutive basic units, and skip connections for low-resolution volume data and skip connections for high-resolution volume data are created between adjacent basic units; The basic unit includes a channel attention layer, a spatial attention layer, a second InstanceNorm layer, a second upsampling module, a downsampling module, and a second LeakyReLU activation function layer. The channel attention layer, the spatial attention layer, and the second InstanceNorm layer are connected in sequence. The second upsampling module, the downsampling module, and the second InstanceNorm layer are connected in sequence. The second InstanceNorm layer is connected to the second LeakyReLU activation function layer.

2. The training method of a spatial super-resolution reconstruction model for time-varying data according to claim 1, characterized in that Step S2 includes the following steps: S2.

1. Normalize the original high-resolution time-variant data to obtain normalized time-variant data; S2.

2. Customize the random cropping multiple and the downsampling multiple of the volume data according to the super-resolution reconstruction multiple, the size of the normalized time-variant data, and the number of normalized time-variant data; S2.

3. Select three consecutive normalized time-variant data at time steps as a group, and randomly crop the three normalized time-variant data at the same position according to the random cropping multiple of the data; S2.

4. According to the downsampling multiple, perform downsampling on the three cropped normalized time-variant data using trilinear interpolation three times to obtain consecutive original low-resolution data; S2.

5. Concatenate the continuous original low-resolution data and its corresponding continuous normalized time-variant data along the channel dimension to obtain the original low-resolution data concatenated along the channel dimension and the original high-resolution data concatenated along the channel dimension.

3. The training method of a spatio-temporal super-resolution reconstruction model for time-varying data according to claim 1, wherein In step S5, the loss value includes a content loss value, a structural loss value, and a perceptual loss value.

4. The training method of a spatial super-resolution reconstruction model for time-variant data according to claim 3, characterized in that, The content loss value is the mean square error value between the reconstructed high-resolution data and the corresponding original high-resolution data concatenated along the channel dimension. The structural loss value is the structural similarity index between the reconstructed high-resolution data and the corresponding original high-resolution data concatenated along the channel dimension. The perceptual loss value is obtained by using a convolutional network to extract the high-dimensional features between the reconstructed high-resolution data and the original high-resolution data concatenated along the channel dimension respectively, and calculating the mean square error between the two.

5. The training method of a spatio-temporal super-resolution reconstruction model for time-varying data according to claim 3, characterized in that, The formula of the loss function is: ， Among them, is the content loss value, is the structural loss value, is the perceptual loss value, , , are user-defined and used to adjust the magnitudes of the corresponding loss values.

6. The training method of a spatial super-resolution reconstruction model for time-varying data according to claim 1, characterized in that Both the first upsampling module and the second upsampling module include at least one enlarged volume data structure. The enlarged volume structure includes a fourth convolutional layer, a PixelShuffle layer, and a third LeakyReLU activation function layer connected in sequence.

7. The training method of a spatio-temporal super-resolution reconstruction model for time-varying data according to claim 1, characterized in that The downsampling module includes at least one reduced volume data structure. The reduced volume data structure includes a fifth convolutional layer and a fourth LeakyReLU activation function layer connected to each other.

8. The training method of a spatio-temporal super-resolution reconstruction model for time-variant data according to claim 1, characterized in that, The input of the basic unit consists of three parts, namely the volume data output by the previous basic unit, the high-resolution volume data generated by the second upsampling module in the previous basic unit, and the low-resolution volume data generated by the downsampling module in the previous basic unit.

9. The training method of a spatial super-resolution reconstruction model for time-varying data according to claim 8, wherein A skip connection is established between the volume data output by the basic unit and the low-resolution volume data generated by the downsampling module in this basic unit, and after connection, it is input into the second upsampling module in the next basic unit. A skip connection is established between the high-resolution volume data generated by the second upsampling module in the basic unit and the high-resolution volume data generated by the second upsampling module in the previous basic unit, and after connection, it is input into the downsampling module in this basic unit.

Images