Flow scene image super-resolution reconstruction method and device based on deep learning

Abstract

The invention discloses a flow scene image super-resolution reconstruction method and device based on deep learning, and the method comprises the steps: obtaining a fluid flow velocity data set, and enabling the data in the fluid flow velocity data set to have a time dimension and a geographical dimension; preprocessing the fluid flow rate data set to obtain a low-resolution image and a high-resolution image; building a super-resolution model based on a deconvolution neural network, wherein the super-resolution model comprises a deconvolution layer and a plurality of convolution layers; a loss function of the super-resolution model is improved, specifically, a piecewise function is added on the basis of L1 Charbonier loss, and the weight is adjusted according to a threshold value; training the super-resolution model by using the low-resolution image and the high-resolution image; and reconstructing a low-resolution image by using the trained super-resolution model to obtain a high-resolution flow field image. And the method has good effect and practical value for image super-resolution research in a flowing scene.

Description

Technical Field

[0001] The present application relates to the technical field of super-resolution image generation, and in particular to a method and device for super-resolution reconstruction of flow scene images based on deep learning. Background Art

[0002] In recent years, with the rapid development of deep learning technology in many fields, this technology has been increasingly used to deal with fluid mechanics related problems with complex physical characteristics and large amounts of data. Traditional numerical calculation methods can simulate high-precision flow fields, but they require huge computing power and hardware support; using interpolation fitting and other methods is difficult to capture the complex nonlinear relationships within the flow field; deep learning has powerful fitting capabilities and can mine effective information from large amounts of data, but directly using generative models to generate flow fields has problems such as the generated flow field not meeting physical constraints and deviating greatly from reality.

[0003] In actual research practice, layout images of flow field parameters such as velocity and temperature are needed to conduct meteorological research, aircraft power design and other research work. However, these images are usually difficult to obtain directly. Only the numerical modal information such as velocity and temperature obtained by sensors arranged at certain points is required. If simulation is performed using computational fluid dynamics and other methods, huge computing resources are required. If only the generative model is used, a flow field that does not conform to the actual physical constraints will be generated. Therefore, a deep learning method is needed to generate a high-resolution image reflecting the velocity of the entire flow field based on the discrete low-resolution information obtained by the sensors arranged at these points. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a method and device for super-resolution reconstruction of flow scene images based on deep learning, so as to solve the technical problems that traditional fluid mechanics simulation of high-resolution flow fields in related technologies requires huge computing resources, while directly using a generative model to generate flow fields does not conform to physical constraints and deviates greatly from reality. This application is different from the traditional numerical method of directly solving the high-resolution flow field. The method of first solving the low-resolution flow field and then using image reconstruction can effectively reduce the computational complexity and the cost of solving the problem, thus being more practical.

[0005] The purpose of the present invention is achieved by adopting the following technical solutions: In a first aspect, an embodiment of the present application provides a method for super-resolution reconstruction of flow scene images based on deep learning, comprising: Acquire a fluid velocity data set, wherein data in the fluid velocity data set has two dimensions: time and geography; Preprocessing the fluid velocity data set to obtain a low-resolution image and a high-resolution image; Build a super-resolution model based on deconvolutional neural network; The loss function of the super-resolution model is improved, specifically: a piecewise function is added on the basis of the L1 Charbonnier loss, and the weight size is adjusted according to the threshold; Training the super-resolution model using the low-resolution image and the high-resolution image; The trained super-resolution model is used to reconstruct the low-resolution image to obtain a high-resolution flow field image.

[0006] In a second aspect, an embodiment of the present application provides a flow scene image super-resolution reconstruction device based on deep learning, comprising: An acquisition module, used for acquiring a fluid velocity data set, wherein the data in the fluid velocity data set has two dimensions: time and geography; A preprocessing module, used for preprocessing the fluid velocity data set to obtain a low-resolution image and a high-resolution image; Model building module, used to build a super-resolution model based on deconvolutional neural network; A function improvement module is used to improve the loss function of the super-resolution model, specifically: adding a piecewise function on the basis of the L1Charbonnier loss and adjusting the weight size according to the threshold; A training module, used for training the super-resolution model using the low-resolution image and the high-resolution image; The reconstruction module is used to reconstruct the low-resolution image using the trained super-resolution model to obtain a high-resolution flow field image.

[0007] In a third aspect, an embodiment of the present application provides an electronic device, including: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in the first aspect.

[0008] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0009] The technical solution provided by the embodiments of the present application may have the following beneficial effects: Preprocessing the fluid velocity data set into an image overcomes the traditional interpolation method of performing super-resolution based only on local information, thereby being able to consider different local and global perspectives during super-resolution reconstruction.

[0010] A super-resolution model based on a deconvolutional neural network was built to overcome the problem that the internal relationships of the flow field are complex and difficult to fit using traditional interpolation methods. This allows the nonlinear relationships within the complex flow field to be fit while ensuring that the number of parameters is small.

[0011] The super-resolution model of the deconvolutional neural network takes low-resolution flow field images as input and introduces basic flow field information, overcoming the problems that the generated flow field directly generated by the generative model does not conform to physical constraints and deviates greatly from reality, thereby ensuring that the generated flow field can conform to the constraints of the actual physical world.

[0012] In addition, in order to address the shortcomings of large data volume and poor model robustness in the field of flow field reconstruction, this application adds a piecewise function based on the L1Charbonnier loss, adjusts the weight size according to the threshold, enhances the stability of the loss function in algorithm training, and effectively improves the training effect.

[0013] Through the design of this method, a faster and more effective flow field image reconstruction can be achieved, which can be applied to atmospheric wind speed prediction scenarios, solving the problems of complex solution process and high computing power consumption of traditional CFD methods.

[0014] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0016] Figure 1 The present invention is a flowchart of a method for super-resolution reconstruction of flow scene images based on deep learning according to an exemplary embodiment.

[0017] Figure 2 It is a structural diagram of a deconvolution network model according to an exemplary embodiment.

[0018] Figure 3 is original flow field image information shown according to an exemplary embodiment.

[0019] Figure 4 It is a model image generation result shown according to an exemplary embodiment, wherein the “6-layer DCNN” column is the model generation result proposed in this example, and the “8-layer and 10-layer DCNN” are the generation results after changing the model structure, and the actual image is a schematic diagram of the original image.

[0020] Fig. 5 is a block diagram of a device for super-resolution reconstruction of flow scene images based on deep learning according to an exemplary embodiment. DETAILED DESCRIPTION

[0021] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0022] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0023] Figure 1 is a flowchart of a method for super-resolution reconstruction of flow scene images based on deep learning according to an exemplary implementation, such as Figure 1 As shown, the method may include the following steps: S1: Obtain a fluid velocity data set, the data set includes training data and test data, and the data has two dimensions of time and geographic coordinates; this step includes the following sub-steps: S11: Obtain a wind field velocity data set. The required data set should be wind speed data sampled according to time.

[0024] Specifically, we use an atmospheric latitudinal wind field dataset from a research team of the Chinese Academy of Sciences, which has a scale of (428, 40, 40) and two dimensions: time and geographic coordinates.

[0025] S12: Decompose the data set according to the time dimension so that the data can be represented as an array matrix of different geographical coordinates, i.e., different longitudes and altitudes at the same time; Specifically, at a fixed time, images of wind speed with respect to latitude and altitude were plotted, and a total of 428 sets of image data were obtained for subsequent work.

[0026] The data in the data set is randomly divided into training data and test data; specifically, in this example, the data in the data set is randomly divided into training data and test data according to a ratio of 8:2, but of course it is not limited to this ratio.

[0027] S2: Preprocessing the fluid velocity data set to obtain a low-resolution image and a high-resolution image, including: S21: Determine the color scale of the drawing based on the maximum and minimum wind speed values ​​in the data, use the contour map method to draw the velocity image, and convert the matrix into an image as a high-resolution image; Specifically, in this example, the minimum wind speed is -81.1 and the maximum is 117.6. Based on these values, the number of color levels with a step size of 40 is set, and a contour map is drawn. Three samples are selected to draw the example map as shown below. Figure 3 As shown in the figure, in meteorology and fluid mechanics, contour maps are widely used to depict cross-sections of wind field velocity. It can provide information such as wind speed and direction, show the spatial variation of the wind field, and facilitate the observation of data comparison and differences. According to the outline and density of the contour lines, the spatial difference in wind field velocity can be intuitively seen, and it is also convenient to observe the difference between the generated image and the actual image. The colors range from blue to red, representing wind speeds from low to high. After observing the data set, the maximum and minimum values ​​are obtained, and the color scale of the contour lines is formulated based on this, and then the color scale is fixed to ensure that different images are drawn using the same color scale.

[0028] S22: Perform grid sampling with an equal spacing of step size n on the fluid velocity image to obtain a pixel scale of The low-resolution input images are paired with high-resolution images to form a fluid velocity dataset.

[0029] Specifically, in this example, the scale of the original image is 40×40, and sampling with a step size of 4 is performed to obtain a 10×10 low-resolution input image, which is paired with a high-resolution image to form a data set.

[0030] S3: Build a super-resolution model based on a deconvolutional neural network, including: S31: Deconvolution layers are introduced into the model to achieve upsampling from low-resolution features to high-resolution images; Specifically, the deconvolution layer is similar to the convolution layer. It transposes the convolution kernel matrix, swaps the input and output, and obtains the image from the features. Assuming the input size is 2×2, the convolution kernel size is 3×3, the step size is 1, and the padding is 0, according to the implementation process of deconvolution, the output size is 4×4, that is, through deconvolution, the 2×2 feature value is extended to 4×4 picture pixels.

[0031] S32: After upsampling, build several convolutional layers to further optimize the generated image. After each convolutional layer and deconvolution layer, use PReLU as the activation function;

[0032] Specifically, the convolution layer is used to refine the features of the generated image, and through convolution calculation, the pixel values ​​at different positions on the image are obtained to restore the image. PReLU is used as the activation function to retain the feature information of the negative part.

[0033] The network structure is as follows Figure 2 As shown in the figure, in terms of model parameters, in the deconvolution layer, the size of the convolution kernel is set to 3×3, the input is 1 channel, the output is 16 channels, the step size is 4, the output padding is 1, the Adam optimizer is used, the batch size is 16, there are 5 convolution layers, the convolution kernel size is 3×3, the padding is 1, and the training rounds are 30 rounds.

[0034] The input low-resolution image matrix is ​​input into the model, and after upsampling in the deconvolution layer and convolution calculations in each layer, the final high-resolution generated image is obtained, and the result can be compared and analyzed with the original flow field image. In this way, layer-by-layer deconvolution overcomes the traditional interpolation method of super-resolution based only on local information, and can consider different local and global perspectives in super-resolution reconstruction.

[0035] S4: improving the loss function of the super-resolution model, specifically: adding a piecewise function on the basis of L1 Charbonnier loss, and adjusting the weight size according to the threshold; including: Specifically, a piecewise function is added to the L1 Charbonnier loss, and a threshold t is set. When the error is less than the threshold t, the L1 Charbonnier loss is used; when the error value is greater than the threshold, the slope of the function is reduced and a smaller weight is used, such as one-fourth of the original weight. The improved loss function The expression is:

[0036] in, is the deviation between the calculated value and the true value, is a smoothing parameter, is the threshold value, is the reduction coefficient of the gradient. In this example, the threshold t is set to 5 and the slope coefficient Set to 0.8. This improves the L1 Charbonnier loss, enhances the stability of the loss function in algorithm training, and effectively improves the training effect.

[0037] S5: training the super-resolution model using the low-resolution image and the high-resolution image; Specifically, the deconvolution network is trained using the training data and the test data. Here, parameters such as SSIM and PSNR can also be combined to judge the quality and accuracy of the generated image.

[0038] The above zonal wind field dataset is used for data processing and neural network training. Since the dataset used in this example is not large, after experiments, the training rounds are set to 30 to reduce the risk of overfitting.

[0039] In this example, 4-layer, 6-layer, 8-layer and 10-layer network structures are used. The same low-resolution image is used as input for model calculation. The high-resolution flow field images obtained by the super-resolution models with different network layers and the actual images are compared. Figure 4 As shown in the figure, it can be found that after training comparison, the effect of using a 6-layer or 8-layer convolutional network is better. Taking into account the training time and inference time, it is concluded that the 6-layer network has the best performance; Parameters such as PSNR and SSIM are introduced to evaluate the image generation results. PSNR is used to measure the quality of image reconstruction, and SSIM is used to judge the similarity between two images.

[0040] Specifically, this example uses a neural network with 1 deconvolution layer and 5 convolution layers. After training, the PSNR and SSIM parameters of the generated image and the original image are calculated on the test set, and the results are: PSNR 41.88, SSIM 0.9966.

[0041] S6: Use the trained super-resolution model to reconstruct the low-resolution image to obtain a high-resolution flow field image, including: S61: obtaining a low-resolution flow field image that needs to be super-resolution reconstructed, and decomposing the image according to the time dimension; S62: Determine the color scale of the drawing based on the maximum and minimum wind speed values ​​in the data, and use the contour map method to draw the flow velocity image; Specifically, in this example, the minimum value of the wind speed is -81.1 and the maximum value is 117.6. According to these values, the number of color levels is set with a step size of 40, and a contour map is drawn.

[0042] S63: Input the flow field image into the image super-resolution model for reconstruction to obtain a high-resolution flow field image.

[0043] Specifically, in this example, the flow field image is input into the trained neural network, the input image size is 10×10, and the output image size is 40×40.

[0044] It can be seen from the above embodiments that the deconvolution network model built by this application proposes a flow field reconstruction method, which uses multimodal image generation and super-resolution related technologies to establish a connection between the sensor flow velocity measurement value of the numerical mode and the image mode, so as to achieve more accurate and effective data prediction. Specifically, this paper analyzes and uses the data set of the atmospheric equilibrium wind field, determines the image drawing method and input and output size, uses multimodal image generation and super-resolution related technologies, designs a deconvolution neural network algorithm, realizes a fast and effective data prediction method, and makes a comparative analysis with the existing flow field prediction algorithm, and tests and evaluates the algorithm. In addition, this paper proposes a new loss function to address the shortcomings of large data volume and poor model robustness in the field of flow field reconstruction, enhances the stability of the loss function in algorithm training, and effectively improves the training effect. Through the design of this method, it is possible to achieve more rapid and effective flow field image reconstruction, which is applied in the atmospheric wind speed prediction scenario, and solves the problems of complex solution process and high computing power consumption of traditional CFD methods.

[0045] Corresponding to the aforementioned embodiment of the method for super-resolution reconstruction of flow scene images based on deep learning, the present application also provides an embodiment of a device for super-resolution reconstruction of flow scene images based on deep learning.

[0046] FIG5 is a block diagram of a device for super-resolution reconstruction of flow scene images based on deep learning according to an exemplary embodiment. Referring to FIG5 , the device includes:

[0047] An acquisition module 1 is used to acquire a fluid velocity data set, wherein the data in the fluid velocity data set has two dimensions: time and geography; A preprocessing module 2, used for preprocessing the fluid velocity data set to obtain a low-resolution image and a high-resolution image; A model building module 3 is used to build a super-resolution model based on a deconvolutional neural network, wherein the super-resolution model includes a deconvolution layer and several convolution layers; Function improvement module 4 is used to improve the loss function of the super-resolution model, specifically: adding a piecewise function on the basis of L1Charbonnier loss and adjusting the weight size according to the threshold; A training module 5, configured to train the super-resolution model using the low-resolution image and the high-resolution image; The reconstruction module 6 is used to reconstruct the low-resolution image using the trained super-resolution model to obtain a high-resolution flow field image.

[0048] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0049] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present application scheme. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0050] Correspondingly, the present application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned deep learning-based flow scene image super-resolution reconstruction method.

[0051] Correspondingly, the present application also provides a computer-readable storage medium on which computer instructions are stored, and when the instructions are executed by a processor, the above-mentioned deep learning-based super-resolution reconstruction method for flow scene images is implemented.

[0052] Those skilled in the art will readily appreciate other embodiments of the present application after considering the description and practicing the contents disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The description and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the claims.

[0053] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A method for super-resolution reconstruction of flow scene images based on deep learning, characterized in that: include: Acquire a fluid velocity data set, wherein data in the fluid velocity data set has two dimensions: time and geography; Preprocessing the fluid velocity data set to obtain a low-resolution image and a high-resolution image; Build a super-resolution model based on deconvolutional neural network; The loss function of the super-resolution model is improved, specifically: a piecewise function is added on the basis of the L1 Charbonnier loss, and the weight size is adjusted according to the threshold; Training the super-resolution model using the low-resolution image and the high-resolution image; The trained super-resolution model is used to reconstruct the low-resolution image to obtain a high-resolution flow field image.

2. The method according to claim 1, characterized in that Acquire fluid velocity data sets, including: Sampling wind speed datasets by time; The wind speed data set is decomposed according to the time dimension so that the data can be expressed as an array matrix with two dimensions, thereby obtaining a fluid flow velocity data set.

3. The method according to claim 1, characterized in that Preprocessing the fluid velocity data set includes: Determine the color scale of the drawing according to the maximum and minimum values ​​of the wind speed in the fluid flow velocity data set, and use the contour map method to draw the fluid flow velocity image as a high-resolution image; The fluid velocity image is subjected to grid sampling with an equal spacing of step length n to obtain a pixel scale of The low-resolution input images are paired with high-resolution images to form a fluid velocity dataset.

4. The method according to claim 1, characterized in that: The deconvolution layer is introduced to achieve upsampling from low-resolution features to high-resolution images. After upsampling, several convolution layers are built to further optimize the generated image. After each convolution layer and deconvolution layer, PReLU is used as the activation function to retain the feature information of the negative part.

5. The method according to claim 1, characterized in that The loss function of the super-resolution model is improved, including: On the basis of L1 Charbonnier loss, a piecewise function is added and a threshold t is set. When the error is less than the threshold t, L1 Charbonnier loss is used; when the error value is greater than the threshold, the slope of the function is reduced.

6. The method according to claim 5, characterized in that Improved loss function The expression is: in, is the deviation between the calculated value and the true value, is a smoothing parameter, is the threshold value, is the reduction factor of the gradient.

7. The method according to claim 1, characterized in that When the super-resolution model is trained using the low-resolution image and the high-resolution image, the method further includes: The PSNR and SSIM parameter indicators are introduced to evaluate the image generation results, where PSNR is used to measure the quality of image reconstruction and SSIM is used to judge the similarity between two images, so as to select the super-resolution model with the highest evaluation result.

8. A super-resolution reconstruction device for flow scene images based on deep learning, characterized in that: include: An acquisition module, used for acquiring a fluid velocity data set, wherein the data in the fluid velocity data set has two dimensions: time and geography; A preprocessing module, used for preprocessing the fluid velocity data set to obtain a low-resolution image and a high-resolution image; Model building module, used to build a super-resolution model based on deconvolutional neural network; A function improvement module is used to improve the loss function of the super-resolution model, specifically: adding a piecewise function on the basis of the L1Charbonnier loss and adjusting the weight size according to the threshold; A training module, used for training the super-resolution model using the low-resolution image and the high-resolution image; The reconstruction module is used to reconstruct the low-resolution image using the trained super-resolution model to obtain a high-resolution flow field image.

9. An electronic device, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the instruction is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images