Deep learning based method and device for super-resolution reconstruction of flow scene image

Abstract

The application discloses a kind of based on deep learning's flow scene image super-resolution reconstruction method and device, including obtaining fluid flow rate dataset, data in the fluid flow rate dataset has two dimensions of time and geography;The fluid flow rate dataset is preprocessed, and low-resolution image and high-resolution image are obtained;Build the super-resolution model based on deconvolution neural network, the super-resolution model includes a deconvolution layer and several convolution layers;The loss function of the super-resolution model is improved, specifically as follows: on the basis of L1 Charbonnier loss, increase sectional function, according to threshold adjustment weight size;The low-resolution image and high-resolution image are used to train the super-resolution model;The low-resolution image is reconstructed by the trained super-resolution model, and high-resolution flow field image is obtained.The image super-resolution under flow scene has good effect and practical value.

Description

Technical Field

[0001] The present application relates to the technical field of super-resolution image generation, and in particular to a method and device for super-resolution reconstruction of flow scene images based on deep learning. Background Art

[0002] In recent years, with the rapid development of deep learning technology in many fields, it has been increasingly used to address fluid dynamics problems involving complex physical properties and large amounts of data. Traditional numerical calculation methods can simulate flow fields with high precision, but they require enormous computing power and hardware support. Interpolation and fitting methods struggle to capture the complex nonlinear relationships within the flow field. Deep learning, with its powerful fitting capabilities, can extract valuable information from large amounts of data. However, directly generating flow fields using generative models presents problems such as inconsistent physical constraints and significant deviations from reality.

[0003] In actual research, images of flow field parameters such as velocity and temperature are often required for meteorological studies and aircraft power design. However, these images are often difficult to obtain directly. Sensors placed at certain locations can only capture numerical modal information such as velocity and temperature. Simulating these images using computational fluid dynamics (CFD) requires enormous computing resources. Using only generative models will produce flow fields that do not conform to actual physical constraints. Therefore, a deep learning approach is needed to generate high-resolution images of the entire flow field's velocity based on the discrete, low-resolution information captured by these point-based sensors. Summary of the Invention

[0004] The purpose of the embodiments of the present application is to provide a method and apparatus for super-resolution reconstruction of flow scene images based on deep learning, in order to address the technical issues in related technologies, such as the huge computational resources required for traditional fluid mechanics simulations of high-resolution flow fields, and the fact that directly using a generative model to generate flow fields does not conform to physical constraints and deviates significantly from reality. Unlike traditional numerical methods that directly calculate high-resolution flow fields, this application first solves the low-resolution flow field and then uses image reconstruction to effectively reduce computational complexity and solution costs, thereby being more practical.

[0005] The purpose of the present invention is achieved by adopting the following technical solutions:

[0006] In a first aspect, an embodiment of the present application provides a method for super-resolution reconstruction of flow scene images based on deep learning, comprising:

[0007] Acquire a fluid velocity dataset, wherein the data in the fluid velocity dataset has two dimensions: time and geography;

[0008] Preprocessing the fluid velocity data set to obtain a low-resolution image and a high-resolution image;

[0009] Build a super-resolution model based on deconvolutional neural network;

[0010] The loss function of the super-resolution model is improved by adding a piecewise function based on the L1 Charbonnier loss and adjusting the weight according to the threshold.

[0011] Training the super-resolution model using the low-resolution image and the high-resolution image;

[0012] The trained super-resolution model is used to reconstruct the low-resolution image to obtain a high-resolution flow field image.

[0013] In a second aspect, an embodiment of the present application provides a deep learning-based super-resolution reconstruction device for flow scene images, comprising:

[0014] An acquisition module, configured to acquire a fluid velocity dataset, wherein the data in the fluid velocity dataset has two dimensions: time and geography;

[0015] A preprocessing module, configured to preprocess the fluid velocity data set to obtain a low-resolution image and a high-resolution image;

[0016] Model building module, used to build a super-resolution model based on deconvolutional neural network;

[0017] A function improvement module is used to improve the loss function of the super-resolution model, specifically by adding a piecewise function on the basis of the L1 Charbonnier loss and adjusting the weight according to the threshold;

[0018] A training module, configured to train the super-resolution model using the low-resolution image and the high-resolution image;

[0019] The reconstruction module is used to reconstruct the low-resolution image using the trained super-resolution model to obtain a high-resolution flow field image.

[0020] In a third aspect, an embodiment of the present application provides an electronic device, including:

[0021] one or more processors;

[0022] a memory for storing one or more programs;

[0023] When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in the first aspect.

[0024] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0025] The technical solutions provided by the embodiments of the present application may have the following beneficial effects:

[0026] Preprocessing the fluid velocity dataset into an image overcomes the traditional interpolation method of performing super-resolution based only on local information, thereby being able to consider different local and global perspectives during super-resolution reconstruction.

[0027] A super-resolution model based on a deconvolutional neural network was built to overcome the problem that the internal relationships of the flow field are complex and difficult to fit using traditional interpolation methods. This allows the nonlinear relationships within the complex flow field to be fit while ensuring a small number of parameters.

[0028] The super-resolution model of the deconvolutional neural network takes low-resolution flow field images as input and introduces basic flow field information, overcoming the problems of the generative model directly generating high-resolution flow fields, such as the generated flow fields not conforming to physical constraints and deviating greatly from reality, thereby ensuring that the generated flow fields can conform to the constraints of the actual physical world.

[0029] In addition, to address the shortcomings of large data volume and poor model robustness in the field of flow field reconstruction, this application adds a piecewise function based on the L1Charbonnier loss, adjusts the weight size according to the threshold, enhances the stability of the loss function in algorithm training, and effectively improves the training effect.

[0030] Through the design of this method, faster and more effective flow field image reconstruction can be achieved, which can be applied to atmospheric wind speed prediction scenarios, solving the problems of complex solution process and high computing power consumption of traditional CFD methods.

[0031] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0033] Figure 1 The present invention is a flowchart of a method for super-resolution reconstruction of flow scene images based on deep learning according to an exemplary embodiment.

[0034] Figure 2 It is a structural diagram of a deconvolution network model according to an exemplary embodiment.

[0035] Figure 3 is original flow field image information shown according to an exemplary embodiment.

[0036] Figure 4 This figure shows the model image generation result according to an exemplary embodiment, where the "6-layer DCNN" column is the model generation result proposed in this example, and the "8-layer and 10-layer DCNN" are the generation results after changing the model structure. The actual image is a schematic diagram of the original image.

[0037] Figure 5 The present invention is a block diagram of a device for super-resolution reconstruction of flow scene images based on deep learning according to an exemplary embodiment. DETAILED DESCRIPTION

[0038] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0039] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. As used in this application and the appended claims, the singular forms "a," "an," "the," and "the" are intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0040] Figure 1 FIG. 1 is a flow chart showing a method for super-resolution reconstruction of flow scene images based on deep learning according to an exemplary embodiment. Figure 1 As shown, the method may include the following steps:

[0041] S1: Obtain a fluid velocity dataset, which includes training data and test data, and the data has two dimensions: time and geographic coordinates. This step includes the following sub-steps:

[0042] S11: Obtain a wind field velocity data set. The required data set should be wind velocity data sampled according to time.

[0043] Specifically, we use an atmospheric latitudinal wind field dataset from a research team of the Chinese Academy of Sciences, which has a size of (428, 40, 40) and two dimensions: time and geographic coordinates.

[0044] S12: Decompose the data set according to the time dimension so that the data can be represented as an array matrix with different geographical coordinates, i.e., different longitudes and altitudes, at the same time;

[0045] Specifically, at a fixed time, images of wind speed relative to latitude and altitude were plotted, resulting in a total of 428 sets of image data for subsequent work.

[0046] The data in the dataset is randomly divided into training data and test data. Specifically, in this example, the data in the dataset is randomly divided into training data and test data in a ratio of 8:2, but the ratio is not limited to this.

[0047] S2: Preprocessing the fluid velocity dataset to obtain a low-resolution image and a high-resolution image, including:

[0048] S21: Determine the color scale of the drawing based on the maximum and minimum wind speed values ​​in the data, use the contour map method to draw the velocity image, and convert the matrix into an image as a high-resolution image;

[0049] Specifically, in this example, the minimum value of wind speed is -81.1 and the maximum value is 117.6. Based on these values, the number of color levels with a step size of 40 is set, and a contour map is drawn. Three samples are selected to draw the example map as shown below. Figure 3 As shown in the figure, in meteorology and fluid dynamics, contour maps are widely used to depict cross-sections of wind field velocity. They can provide information such as wind speed magnitude and direction, showing the spatial variation of the wind field, and facilitate the observation of data comparison and differences. The contour lines and density can intuitively see the spatial differences in wind field velocity, and also facilitate the observation of the difference between the generated image and the actual image. The colors range from blue to red, representing wind speeds from low to high. The maximum and minimum values ​​are obtained by observing the data set, and the color scale of the contour lines is then determined based on this. The color scale is then fixed to ensure that different images are drawn using the same color scale.

[0050] S22: Perform grid sampling of the fluid velocity image with an equal spacing of step size n to obtain a pixel scale of The low-resolution input images are paired with high-resolution images to form a fluid velocity dataset.

[0051] Specifically, in this example, the scale of the original image is 40×40, and sampling with a step size of 4 is performed to obtain a 10×10 low-resolution input image, which is paired with a high-resolution image to form a data set.

[0052] S3: Build a super-resolution model based on a deconvolutional neural network, including:

[0053] S31: Deconvolution layers are introduced into the model to achieve upsampling from low-resolution features to high-resolution images;

[0054] Specifically, the deconvolution layer is similar to the convolution layer. It transposes the convolution kernel matrix, swaps the input and output, and extracts the image from the features. Assuming the input size is 2×2, the convolution kernel size is 3×3, the stride is 1, and the padding is 0, according to the deconvolution implementation process, the output size is 4×4. That is, through deconvolution, the 2×2 feature values ​​are extended to 4×4 image pixels.

[0055] S32: After upsampling, several convolutional layers are built to further optimize the generated image. After each convolutional and deconvolutional layer, PReLU is used as the activation function.

[0056] Specifically, the convolutional layer refines the features of the generated image. Through convolution calculations, the pixel values ​​at different locations on the image are obtained, thereby restoring the image. PReLU is used as the activation function to retain the feature information of the negative part.

[0057] The network structure is as follows Figure 2 As shown in the figure, in terms of model parameters, in the deconvolution layer, the size of the convolution kernel is set to 3×3, the input is 1 channel, the output is 16 channels, the stride is 4, the output padding is 1, the Adam optimizer is used, the batch size is 16, there are 5 convolution layers, the convolution kernel size is 3×3, the padding is 1, and the training rounds are 30 rounds.

[0058] The input low-resolution image matrix is ​​input into the model, and after upsampling in the deconvolution layer and convolution calculations in each layer, the final high-resolution generated image is obtained. The obtained result can be compared and analyzed with the original flow field image. In this way, the layer-by-layer deconvolution overcomes the traditional interpolation method that only performs super-resolution based on local information, and can consider different local and global perspectives in super-resolution reconstruction.

[0059] S4: Improve the loss function of the super-resolution model, specifically by adding a piecewise function based on the L1 Charbonnier loss and adjusting the weight according to the threshold; including:

[0060] Specifically, a piecewise function is added to the L1 Charbonnier loss, and a threshold t is set. When the error is less than the threshold t, the L1 Charbonnier loss is used; when the error value is greater than the threshold, the slope of the function is reduced and a smaller weight is used, such as one-fourth of the original weight. The improved loss function The expression is:

[0061]

[0062] in, is the deviation between the calculated value and the true value, is a smoothing parameter, is the threshold, is the reduction coefficient of the gradient. In this example, the threshold t is set to 5 and the slope coefficient Set to 0.8. This improves the L1 Charbonnier loss, enhances the stability of the loss function in algorithm training, and effectively improves the training effect.

[0063] S5: training the super-resolution model using the low-resolution image and the high-resolution image;

[0064] Specifically, the deconvolution network is trained using the training data and the test data. Parameters such as SSIM and PSNR can also be used to determine the quality and accuracy of the generated image.

[0065] The aforementioned zonal wind field dataset was used for data processing and neural network training. Since the dataset used in this example is not large, after experimentation, the number of training rounds was set to 30 to reduce the risk of overfitting.

[0066] In this example, 4-layer, 6-layer, 8-layer and 10-layer network structures are used. The same low-resolution image is used as input for model calculation. The high-resolution flow field images obtained by the super-resolution models with different network layers are compared with the actual images. Figure 4 As shown in the figure, it can be found that after training comparison, the use of 6-layer or 8-layer convolutional networks has better results. Taking into account the training time and inference time, it is concluded that the 6-layer network has the best performance;

[0067] We introduce parameters such as PSNR and SSIM to evaluate the image generation results. PSNR is used to measure the quality of image reconstruction, and SSIM is used to determine the similarity between two images.

[0068] Specifically, this example uses a neural network with one deconvolution layer and five convolution layers. After training, the PSNR and SSIM parameters of the generated and original images are calculated on the test set, and the results are: PSNR 41.88, SSIM 0.9966.

[0069] S6: Use the trained super-resolution model to reconstruct the low-resolution image to obtain a high-resolution flow field image, including:

[0070] S61: Obtain a low-resolution flow field image that needs to be super-resolution reconstructed, and decompose it according to the time dimension;

[0071] S62: Determine the color scale of the drawing with the maximum and minimum values of the wind speed in the data, and draw the flow rate image using the method of contour map;

[0072] Specifically, in this example, the minimum value of the wind speed is -81.1, and the maximum value is 117.6. According to these values, the number of color scales with a step size of 40 is set, and a contour map is drawn.

[0073] S63: Input the flow field image into the image super-resolution model for reconstruction to obtain a high-resolution flow field image.

[0074] Specifically, in this example, the flow field image is input into the trained neural network, and the input image size is 10x10, and the output image size is 40x40.

[0075] From the above embodiments, it can be seen that the deconvolution network model built by the present application proposes a flow field reconstruction method, which uses multi-modal image generation and super-resolution related technologies to establish a connection between numerical modal sensor flow rate measurement values and image modal, and realizes more accurate and effective data prediction. Specifically, this paper analyzes and uses the data set of atmospheric equilibrium wind field, determines the image drawing method and input and output size, uses multi-modal image generation and super-resolution related technologies, designs a deconvolution neural network algorithm, realizes a fast and effective data prediction method, and makes a comparative analysis with the existing flow field prediction algorithm, tests and evaluates the algorithm. In addition, in view of the shortcomings of large amount of data and poor model robustness in the field of flow field reconstruction, a new loss function is proposed, which enhances the stability of the loss function in the algorithm training, effectively improves the training effect. Through the design of this method, more rapid and effective flow field image reconstruction can be realized, which is applied to the atmospheric wind speed prediction scene, and solves the problems of complex solving process and large algorithm consumption of traditional CFD method.

[0076] Corresponding to the foregoing embodiments of the deep learning-based flow scene image super-resolution reconstruction method, the present application also provides embodiments of a deep learning-based flow scene image super-resolution reconstruction device.

[0077] Figure 5 is a deep learning-based flow scene image super-resolution reconstruction device block diagram according to an exemplary embodiment. Referring to Figure 5 , the device comprises:

[0078] The acquisition module 1 is configured to acquire a fluid flow rate data set, wherein the data in the fluid flow rate data set has two dimensions of time and geography;

[0079] The preprocessing module 2 is configured to preprocess the fluid flow rate data set to obtain a low-resolution image and a high-resolution image;

[0080] Model construction module 3, used to build a super-resolution model based on a deconvolutional neural network, wherein the super-resolution model includes a deconvolution layer and several convolution layers;

[0081] Function improvement module 4 is used to improve the loss function of the super-resolution model, specifically by adding a piecewise function on the basis of L1 Charbonnier loss and adjusting the weight according to the threshold;

[0082] A training module 5 is configured to train the super-resolution model using the low-resolution image and the high-resolution image;

[0083] The reconstruction module 6 is used to reconstruct the low-resolution image using the trained super-resolution model to obtain a high-resolution flow field image.

[0084] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0085] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present application scheme. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0086] Accordingly, the present application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned deep learning-based flow scene image super-resolution reconstruction method.

[0087] Accordingly, the present application also provides a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the above-mentioned deep learning-based super-resolution reconstruction method for flow scene images.

[0088] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the contents disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of the present application and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered merely as exemplary, and the true scope and spirit of the present application are indicated by the claims.

[0089] It should be understood that the present application is not limited to the exact structures described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A method for super-resolution reconstruction of flow scene images based on deep learning, characterized in that: include: Acquire a fluid velocity dataset, wherein the data in the fluid velocity dataset has two dimensions: time and geography; Preprocessing the fluid velocity data set to obtain a low-resolution image and a high-resolution image; Build a super-resolution model based on deconvolutional neural network; The loss function of the super-resolution model is improved by adding a piecewise function based on the L1 Charbonnier loss and adjusting the weight according to the threshold. Training the super-resolution model using the low-resolution image and the high-resolution image; Use the trained super-resolution model to reconstruct the low-resolution image to obtain a high-resolution flow field image; The preprocessing of the fluid velocity data set includes: Determining the color scale of the drawing based on the maximum and minimum wind speed values ​​in the fluid flow velocity data set, and using a contour map method to draw a fluid flow velocity image as a high-resolution image; The fluid velocity image is sampled in a grid with an equal spacing of n steps to obtain a pixel scale of The low-resolution input image is paired with the high-resolution image to form the fluid velocity dataset; Improved loss function The expression is: in, is the deviation between the calculated value and the true value, is a smoothing parameter, is the threshold, is the gradient reduction factor.

2. The method according to claim 1, characterized in that Acquire fluid velocity datasets, including: Sampling wind speed datasets by time; The wind speed dataset is decomposed according to the time dimension so that the data can be represented as an array matrix with two dimensions, thereby obtaining a fluid flow velocity dataset.

3. The method according to claim 1, characterized in that The deconvolution layer is introduced to achieve upsampling from low-resolution features to high-resolution images. After upsampling, several convolution layers are built to further optimize the generated image. After each convolution layer and deconvolution layer, PReLU is used as the activation function to retain the feature information of the negative part.

4. The method according to claim 1, wherein When the super-resolution model is trained using the low-resolution image and the high-resolution image, the method further includes: The PSNR and SSIM parameter indicators are introduced to evaluate the image generation results. PSNR is used to measure the quality of image reconstruction, and SSIM is used to judge the similarity between two images, so as to select the super-resolution model with the highest evaluation result.

5. A super-resolution reconstruction device for flow scene images based on deep learning, characterized in that: include: An acquisition module, configured to acquire a fluid velocity dataset, wherein the data in the fluid velocity dataset has two dimensions: time and geography; A preprocessing module, configured to preprocess the fluid velocity data set to obtain a low-resolution image and a high-resolution image; Model building module, used to build a super-resolution model based on deconvolutional neural network; A function improvement module is used to improve the loss function of the super-resolution model, specifically by adding a piecewise function on the basis of the L1 Charbonnier loss and adjusting the weight according to the threshold; A training module, configured to train the super-resolution model using the low-resolution image and the high-resolution image; The reconstruction module is used to reconstruct the low-resolution image using the trained super-resolution model to obtain a high-resolution flow field image; The preprocessing of the fluid velocity data set includes: Determining the color scale of the drawing based on the maximum and minimum wind speed values ​​in the fluid flow velocity data set, and using a contour map method to draw a fluid flow velocity image as a high-resolution image; The fluid velocity image is sampled in a grid with an equal spacing of n steps to obtain a pixel scale of The low-resolution input image is paired with the high-resolution image to form the fluid velocity dataset; Improved loss function The expression is: in, is the deviation between the calculated value and the true value, is a smoothing parameter, is the threshold, is the gradient reduction factor.

6. An electronic device, characterized in that: include: one or more processors; a memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 4.

7. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the instruction is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images