A Schlieren flow field reconstruction method based on multi-input feature fusion ConvLSTM model

Abstract

The present invention provides a Schlieren flow field reconstruction method based on a multi-input feature fusion ConvLSTM model, comprising the following steps: S1: parameter initialization; S2: data preprocessing, cropping and grayscale processing of the image, and normalization of the pressure data; S3: data generation and segmentation, data set division; S4: model construction and compilation, using the ConvLSTM model to build a multi-input neural network; S5: model training and callback, setting the optimizer and model optimization function; S6: result visualization and prediction, including drawing loss curves and model prediction and result preservation. Beneficial effects: This method fully utilizes the spatial and temporal information in the spatiotemporal sequence by applying the ConvLSTM technology to the processing of Schlieren flow field video frames, avoids the performance degradation problem of traditional methods when processing long sequences, makes the reconstruction results more continuous and consistent, improves the accuracy and robustness of the reconstruction effect, greatly reduces the demand for computing resources, and has better real-time performance.

Description

Technical Field

[0001] The present invention relates to a time-series video frame image prediction and reconstruction method, in particular to a schlieren flow field video frame image reconstruction method based on a multi-input feature fusion ConvLSTM model, and belongs to the technical field of computer vision. Background Art

[0002] The current development of automobile engines and aircraft engines mainly relies on experiments to obtain performance index data. Each test requires a lot of manpower and material resources, and produces a lot of harmful gases and waste to pollute the environment. The image information obtained from a single test is highly dependent on the performance of the camera. In order to obtain the data of the entire engine process, it is often necessary to repeat multiple tests and integrate segmented shooting to obtain target information.

[0003] With the rapid development of deep learning technology, spatiotemporal sequence data processing tasks are increasingly widely used in the field of computer vision. Traditional convolutional neural networks (CNNs) perform well in processing image data, but their limitations in processing time series data are gradually emerging. LSTM can capture long-range dependencies in sequence data by introducing a "gate" mechanism in its architecture, thereby effectively solving the gradient vanishing problem of traditional RNN (recurrent neural network) when processing long time series. In order to better capture the temporal dependencies in sequence data, researchers introduced the convolutional long short-term memory network (ConvLSTM), a model architecture that combines LSTM (long short-term memory network) with convolutional neural networks. ConvLSTM expands the input, hidden state, and output of LSTM from one-dimensional data to two-dimensional matrices, enabling the network to model the spatiotemporal features in time series image sequences. Specifically, the core of ConvLSTM is that it replaces the linear operations in traditional LSTM with convolution operations, allowing the network to learn spatial features in the time dimension. The main parts of the ConvLSTM model architecture include the input gate, forget gate, output gate and cell state. These parts work together through convolution operations to form a memory unit that can capture spatiotemporal features. Unlike traditional fully connected LSTM, ConvLSTM can not only retain information in time series, but also maintain spatial information in images, which makes it perform well in the field of video processing.

[0004] As an important optical measurement method, the Schlieren method is widely used in the fields of fluid mechanics and aerodynamics to study the motion characteristics of fluids. Traditional Schlieren flow field video frame reconstruction methods mainly rely on numerical simulation based on physical models and algorithms based on image processing. These methods mainly use fluid mechanics equations or image processing techniques to extract flow field information from continuous Schlieren image sequences, such as the Navier-Stokes equations or optical flow method, phase averaging method, etc. By combining boundary conditions and initial conditions, the evolution of the flow field is numerically solved or the pixel changes between adjacent frames are analyzed to infer the movement of the fluid. However, this type of method has extremely high requirements for computing resources, and the computational complexity increases significantly with the complexity of the flow field. In addition, the accuracy of numerical simulation depends on the accuracy of initial and boundary conditions, and it is difficult to handle the uncertainty and complex changes in real scenes. Methods such as the optical flow method usually assume small movements and constant brightness, and it is difficult to handle large displacements and brightness changes in actual flow fields. Image noise and complex backgrounds will also interfere with the performance of the algorithm, resulting in a decrease in reconstruction accuracy.

[0005] In recent years, with the popularity of deep learning, data-driven methods have begun to be applied to flow field reconstruction. Although these methods have shown excellent performance in some scenarios, most methods are still limited to processing single frames or short time series, and it is difficult to effectively capture the complex dynamic features in long time series. Summary of the invention

[0006] Purpose of the invention: The purpose of the present invention is to provide a Schlieren flow field reconstruction method based on the ConvLSTM model of multi-input feature fusion in view of the deficiencies in the prior art. This method applies ConvLSTM technology to the processing of Schlieren flow field video frames, makes full use of the spatial and temporal information in the time series, avoids the performance degradation problem of traditional methods when processing long sequences, makes the reconstruction results more continuous and consistent, improves the accuracy and robustness of the reconstruction effect, greatly reduces the demand for computing resources, and has better real-time performance.

[0007] Technical solution: A Schlieren flow field reconstruction method based on a multi-input feature fusion ConvLSTM model, comprising the following steps:

[0008] S1, parameter initialization, defining the learning rate, decay rate, batch size, regularization factor, and time step required for model training;

[0009] S2, data preprocessing, first manually select the pressure parameters and image data to be trained, grayscale all original images, and crop the images to 1024×256 pixels while retaining the original image features, and normalize the original pressure data;

[0010] S3, data generation and segmentation, divide the data set into training set and validation set, and customize the data generator to batch load and process data during training to avoid loading all data into memory at once;

[0011] S4. Model construction and compilation. Use the ConvLSTM model to construct a multi-input neural network, including an image input part, a pressure data input part, a feature fusion and an output part. The training set of the multi-input neural network model contains two parts: a processed image and a corresponding normalized pressure parameter. The image is input to the image input part of the multi-input neural network, and the pressure parameter is input to the pressure data input part of the multi-input neural network. After that, the outputs of the image input part and the pressure input part will be subjected to feature cross-learning in the feature fusion part before the model output.

[0012] S5, model training and callback, set the optimizer, loss function, callback function, define early stop, checkpoint saving and learning rate scheduler, and use the custom generator and callback function after starting model training;

[0013] S6. Result visualization and prediction, including drawing loss curves and saving model predictions and results, drawing loss curves during training and validation, observing the performance of the model, making predictions on the validation set, and saving the prediction results as image files.

[0014] The S2 is specifically:

[0015] Read the file path containing image data and pressure parameters, arrange the input image data and pressure data in chronological order, align the time points of image data and pressure data at intervals of 1 ms, and adjust the shape of pressure data to the four-dimensional tensor form required by the model, including the number of samples, time steps, 1, and the number of features, to match the shape of image data, and provide a standardized, time-point-matched data set for subsequent training and verification.

[0016] The data set in S3 is specifically:

[0017] The data set is divided into a training set and a validation set by group, the data set includes pressure data and image data, and the pressure data and image data corresponding to each other after the alignment moment are combined into one group, which is divided into 14 groups of combined data, which are combined into a data set for ConvLSTM model training, among which the first 13 groups of combined data are divided into training sets, and the 14th group of combined data is divided into a validation set.

[0018] The custom data generator in S3 is specifically:

[0019] The custom data generator specifically extracts 32 original images and their corresponding pressure data for each training batch from a library that stores preprocessed image paths and corresponding pressure data. The image size is 1024×256, and the number of time steps is 2630. The custom data generator outputs the image data and pressure data at the same moment respectively, and extracts the features of the image data and pressure data at the same moment respectively through the ConvLSTM model.

[0020] The S4 is specifically:

[0021] The multi-input feature fusion model is divided into three parts: the part that uses the convolution layer to process the image data, the part that uses the time series layer to process the pressure input, and the feature fusion output prediction part;

[0022] The image processing module contains 5 ConvLSTM2D layers and a Flatten layer. The ConvLSTM2D layer can process time and space information at the same time. Each layer has 64 filters and the convolution kernel size is 3×3. Each ConvLSTM2D layer sets 'return_sequences=True' so that each layer can output a sequence containing the time dimension to retain the characteristics of the time step. A batch normalization layer is inserted after each ConvLSTM2D layer to accelerate the training process and improve the stability of the model. After the last ConvLSTM2D layer, the Flatten layer is used to flatten the output tensor into a one-dimensional vector to prepare for combining with the output of the pressure data. The basic formula for single-step prediction of ConvLSTM is as follows:

[0023]

[0024]

[0025] Where: g Represents the sigmoid function, tanh g represents the hyperbolic tangent activation function, X t is the input of the current time step, H t-1 is the hidden state of the previous time step, C t-1 is the memory unit of the previous time step, I t is the input gate, F t For the forget gate, O t is the output gate, C t is a memory unit, W and b are weight matrix and bias matrix respectively, is the product, * is the convolution operator, bi is the bias matrix of the input gate, bf is the bias matrix of the forget gate, bo is the bias matrix of the output gate, and bc is the memory unit C t The bias matrix.

[0026] The pressure data processing module includes a Flatten layer and a Dense fully connected layer. The Flatten layer flattens the input layer of the pressure data and converts it into a one-dimensional vector, merging the pressure data of each time step into a vector. The Dense fully connected layer processes the flattened pressure data. The fully connected layer contains 64 neurons and uses the ReLU activation function to extract the features of the pressure data by adding nonlinear transformations.

[0027] The feature fusion and prediction output modules are Concatenate layer, Dense fully connected layer, Dropout layer, and Dense layer in order. The Concatenate layer merges the output (x) of the image processing module and the output (y) of the pressure data processing module. This fused output contains the features extracted from the image and pressure data. The merged output passes through a fully connected layer Dense again to further extract mixed features. Then, the Dropout layer is used to randomly discard a certain number of neurons to prevent overfitting. Finally, a fully connected layer is used to output the prediction result.

[0028] Define the entire model through 'tf.keras.Model', specify the input layer as image and pressure data, and the output layer as predicted value, so that the model integrates the processing pipeline of image and pressure data and can consider the characteristics of these two types of data at the same time;

[0029] Use Adam optimizer as the adaptive learning rate optimization algorithm;

[0030] The update rules of the Adam optimizer are as follows:

[0031] m t =β1m t-1 +(1-β1)g t

[0032]

[0033] Where: g t is the gradient of the parameter, β1 and β2 are the decay coefficients of the two exponentially weighted averages, and is the bias-corrected moving average of the gradient, θ t+1 is the updated parameter, η is the learning rate, and ε is a constant greater than 0.

[0034] The S5 is specifically:

[0035] The model training and callback part mainly includes a model training module and a model callback module. The training module sets the model training process according to the training data, verification data, and the number of training rounds, sets the training data generator to dynamically generate each batch of training data, and sets the verification data generator to dynamically generate each batch of verification data;

[0036] The model callback module is divided into three parts: model save callback function, early stop callback function, and weight check callback function. The model save callback function automatically saves the model during the training process. It monitors the loss value on the validation set and saves the model weight when the loss value reaches a new low. The model is only saved when the validation set loss decreases. The saving expression is:

[0037] if((val_loss) t <(best_val_loss)),save_model

[0038] Where (val_loss) is the loss value of the validation set, t is the training round, and (best_val_loss) is the minimum loss value among all previous training batches.

[0039] The early stopping callback function stops training early when the validation set loss no longer decreases. The early stopping expression is:

[0040] if((val_loss) t >min(val_loss) t-p )for p epochs, then stop the training

[0041] Among them, p is the patience parameter, min(val_loss) t-p is the minimum loss value in the past p rounds,

[0042] The weight check callback function checks and outputs the statistical information of each layer's weight. The weight check expression is:

[0043] weights i min =min(weights i ),weights i max =max(weights i )

[0044] Among them, weight expresses the weight parameter, weights i Represents the weight parameter of the i-th layer, weights i max Indicates the maximum value of the weight of the i-th layer, weightsi min Represents the minimum value of the weight of the i-th layer.

[0045] The S6 is specifically:

[0046] The trained ConvLSTM model is called and input into the test data generator, which provides test data in batches, dynamically loads test data, and outputs the predicted images of the model on these test data. Images are generated one by one by looping through all prediction results to form continuous video frame images.

[0047] Beneficial effects: The present invention uses the schlieren image sequence and related pressure parameters as multiple inputs, combined with the spatiotemporal feature capture capability of ConvLSTM, to construct a flow field reconstruction model that can process long time series and fuse multiple input features. ConvLSTM can effectively capture the spatial features in the flow field image and the dynamic changes in the time series at the same time, so that the model can better reflect the actual evolution of the flow field, thereby improving the reconstruction accuracy. The ability of ConvLSTM to process long time series data avoids the performance degradation problem of traditional methods when processing long sequences. By introducing a memory mechanism, ConvLSTM can capture the long-range dependencies in the evolution of the flow field, making the reconstruction results more continuous and consistent. Through the method of multi-input feature fusion, the ConvLSTM model simultaneously processes the schlieren images and pressure parameters for feature fusion training, so that the model can use information from more dimensions and further improve the accuracy and robustness of reconstruction. Compared with the traditional physical model numerical simulation method, the schlieren flow field reconstruction method based on the ConvLSTM model with multi-input feature fusion maintains high accuracy while greatly reducing the demand for computing resources and has better real-time performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0049] Figure 1 This is a flow chart of the Schlieren flow field reconstruction method based on the ConvLSTM model of multi-input feature fusion in the present invention.

[0050] Figure 2 The image after the image preprocessing step and the original image of the present invention.

[0051] Figure 3 The predicted images and original images of the validation set output by the present invention. DETAILED DESCRIPTION

[0052] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] In the description of the present invention, it is necessary to understand that the terms "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship are based on the orientation or position relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0054] In the present invention, unless otherwise clearly specified and limited, a first feature being "above" or "below" a second feature may include that the first and second features are in direct contact, or may include that the first and second features are not in direct contact but are in contact through another feature between them. Moreover, a first feature being "above", "above" and "above" a second feature includes that the first feature is directly above and obliquely above the second feature, or simply indicates that the first feature is higher in level than the second feature. A first feature being "below", "below" and "below" a second feature includes that the first feature is directly below and obliquely below the second feature, or simply indicates that the first feature is lower in level than the second feature.

[0055] A Schlieren flow field reconstruction method based on a multi-input feature fusion ConvLSTM model comprises the following steps:

[0056] S1, parameter initialization, defining the learning rate, decay rate, batch size, regularization factor, and time step required for model training;

[0057] S2, data preprocessing, first manually select the pressure parameters and image data to be trained, grayscale all original images, and crop the images to 1024×256 pixels while retaining the original image features, and normalize the original pressure data;

[0058] S3, data generation and segmentation, divide the data set into training set and validation set, and customize the data generator to batch load and process data during training to avoid loading all data into memory at once;

[0059] S4. Model construction and compilation. Use the ConvLSTM model to construct a multi-input neural network, including an image input part, a pressure data input part, a feature fusion and an output part. The training set of the multi-input neural network model contains two parts: a processed image and a corresponding normalized pressure parameter. The image is input to the image input part of the multi-input neural network, and the pressure parameter is input to the pressure data input part of the multi-input neural network. After that, the outputs of the image input part and the pressure input part will be subjected to feature cross-learning in the feature fusion part before the model output.

[0060] S5, model training and callback, set the optimizer, loss function, callback function, define early stop, checkpoint saving and learning rate scheduler, and use the custom generator and callback function after starting model training;

[0061] S6. Result visualization and prediction, including drawing loss curves and saving model predictions and results, drawing loss curves during training and validation, observing the performance of the model, making predictions on the validation set, and saving the prediction results as image files.

[0062] The S2 is specifically:

[0063] Read the file path containing image data and pressure parameters, arrange the input image data and pressure data in chronological order, align the time points of image data and pressure data at intervals of 1 ms, and adjust the shape of pressure data to the four-dimensional tensor form required by the model, including the number of samples, time steps, 1, and the number of features, to match the shape of image data, and provide a standardized, time-point-matched data set for subsequent training and verification.

[0064] The data set in S3 is specifically:

[0065] The data set is divided into a training set and a validation set by group, the data set includes pressure data and image data, and the pressure data and image data corresponding to each other after the alignment moment are combined into one group, which is divided into 14 groups of combined data, which are combined into a data set for ConvLSTM model training, among which the first 13 groups of combined data are divided into training sets, and the 14th group of combined data is divided into a validation set.

[0066] The custom data generator in S3 is specifically:

[0067] The custom data generator specifically extracts 32 original images and their corresponding pressure data for each training batch from a library that stores preprocessed image paths and corresponding pressure data. The image size is 1024×256, and the number of time steps is 2630. The custom data generator outputs the image data and pressure data at the same moment respectively, and extracts the features of the image data and pressure data at the same moment respectively through the ConvLSTM model.

[0068] Specific operations include:

[0069] Build a data generator, use the '__init__' method to initialize the generator object, customize the library that has preprocessed and stored the image path and corresponding pressure data, the number of samples extracted from the data set each time, the image size (width, height), and the number of time steps, use the generator's '__len__' method to define the number of batches in an epoch, ensure that the generator can evenly distribute data at the end of each epoch, use the generator's '__getitem__' method to initialize two empty numpy arrays to store the current batch of image data and pressure data respectively, and automatically traverse the image path and the corresponding pressure data before each batch of training and store them in empty numpy arrays for training.

[0070] The generator's '__getitem__' method determines how to get each batch of data. First, an empty numpy array is initialized through 'batch_images' to store the image data of the current batch. The size is (batch_size, num_timesteps, image_height, image_width, channels), where batch_size is the batch size, num_timesteps is the time step, and channels is the number of channels. The number of channels of a normal RGB image is 3, and the number of channels of a binary image is 1. The 'batch_pressure' is initialized. Initialize another empty numpy array to store the current batch of pressure data, with dimensions of (batch_size, num_timesteps, 1, num_features), where num_features is the number of features. During training, the model will load and process the corresponding batch of images and pressure data, normalize the image data to floating point numbers between [0, 1], and increase the dimension to adapt to the input requirements of the model. Extract the pressure data with the custom time step from the preprocessed pressure database and store it in batch_pressure. Finally, '__getitem__' returns the numpy array of image and pressure data.

[0071] The S4 is specifically:

[0072] The multi-input feature fusion model is divided into three parts: the part that uses the convolution layer to process the image data, the part that uses the time series layer to process the pressure input, and the feature fusion output prediction part;

[0073] The image processing module contains 5 ConvLSTM2D layers and a Flatten layer. The ConvLSTM2D layer can process time and space information at the same time. Each layer has 64 filters and the convolution kernel size is 3×3. Each ConvLSTM2D layer sets 'return_sequences=True' so that each layer can output a sequence containing the time dimension to retain the characteristics of the time step. A batch normalization layer is inserted after each ConvLSTM2D layer to accelerate the training process and improve the stability of the model. After the last ConvLSTM2D layer, the Flatten layer is used to flatten the output tensor into a one-dimensional vector to prepare for combining with the output of the pressure data. The basic formula for single-step prediction of ConvLSTM is as follows:

[0074]

[0075] Where: g Represents the sigmoid function, tanh g represents the hyperbolic tangent activation function, X t is the input of the current time step, H t-1 is the hidden state of the previous time step, C t-1 is the memory unit of the previous time step, I t is the input gate, F t For the forget gate, O t is the output gate, C t is the memory unit, W and b are the weight matrix and bias matrix respectively, ⊙ is the product, * is the convolution operator, is the bias matrix of the input gate, bf is the bias matrix of the forget gate, bo is the bias matrix of the output gate, and bc is the memory unit C t The bias matrix.

[0076] The pressure data processing module includes a Flatten layer and a Dense fully connected layer. The Flatten layer flattens the input layer of the pressure data and converts it into a one-dimensional vector, merging the pressure data of each time step into a vector. The Dense fully connected layer processes the flattened pressure data. The fully connected layer contains 64 neurons and uses the ReLU activation function to extract the features of the pressure data by adding nonlinear transformations.

[0077] The feature fusion and prediction output modules are Concatenate layer, Dense fully connected layer, Dropout layer, and Dense layer in order. The Concatenate layer merges the output (x) of the image processing module and the output (y) of the pressure data processing module. This fused output contains the features extracted from the image and pressure data. The merged output passes through a fully connected layer Dense again to further extract mixed features. Then, the Dropout layer is used to randomly discard a certain number of neurons to prevent overfitting. Finally, a fully connected layer is used to output the prediction result.

[0078] Define the model training logic through 'tf.keras.Model', specifically specify the image data processing module input layer as image data, the pressure data processing module input layer as pressure data, and the output layer as the prediction value array, so that the model integrates the image and pressure data processing pipelines and considers the characteristics of these two types of data at the same time;

[0079] Use Adam optimizer as the adaptive learning rate optimization algorithm;

[0080] The update rules of the Adam optimizer are as follows:

[0081] m t =β1m t-1 +(1-β1)g t

[0082]

[0083]

[0084] Where: g t is the gradient of the parameter, β1 and β2 are the decay coefficients of the two exponentially weighted averages, and is the bias-corrected moving average of the gradient, θ t+1 is the updated parameter, η is the learning rate, and ε is a constant greater than 0.

[0085] The S5 is specifically:

[0086] The model training and callback part mainly includes a model training module and a model callback module. The training module sets the model training process according to the training data, verification data, and the number of training rounds, sets the training data generator to dynamically generate each batch of training data, and sets the verification data generator to dynamically generate each batch of verification data;

[0087] The model callback module is divided into three parts: model save callback function, early stop callback function, and weight check callback function. The model save callback function automatically saves the model during the training process. It monitors the loss value on the validation set and saves the model weight when the loss value reaches a new low. The model is only saved when the validation set loss decreases. The saving expression is:

[0088] if((val_loss) t <(best_val_loss)),save_model

[0089] Where (val_loss) is the loss value of the validation set, t is the training round, and (best_val_loss) is the minimum loss value among all previous training batches.

[0090] The early stopping callback function stops training early when the validation set loss no longer decreases. The early stopping expression is:

[0091] if((val_loss) t >min(val_loss) t-p )for p epochs, then stop the training

[0092] Among them, p is the patience parameter, min(val_loss) t-p is the minimum loss value in the past p rounds,

[0093] The weight check callback function checks and outputs the statistical information of each layer's weight. The weight check expression is:

[0094] weights i min =min(weights i ),weights i max =max(weights i )

[0095] Among them, weight expresses the weight parameter, weights i Represents the weight parameter of the i-th layer, weights i max Indicates the maximum value of the weight of the i-th layer, weights i min Represents the minimum value of the weight of the i-th layer.

[0096] Specific operations include:

[0097] The core of the model training module is based on Kerasa's model.fit() function, which trains the model based on the given training data, validation data, number of training epochs, and callback function. The model.fit() function contains a training data generator, a validation data generator, a specified number of training epochs, and a callback function list. The training data generator dynamically generates training data for each batch. The data does not need to be loaded into the memory at one time, but is generated on demand, effectively saving memory. The validation data generator generates validation set data. During the training process, model.fit() will return a History object, which contains information such as the loss value, accuracy, validation loss, and validation accuracy of each round of training.

[0098] The model callback module is mainly composed of the ModelCheckpoint callback function, the EarlyStopping callback function and the LearningRateScheduler callback function. The ModelCheckpoint callback function and the EarlyStopping callback function are both mechanisms provided by Keras, which can perform some specific operations at different stages of training. The ModelCheckpoint callback function is used to automatically save the model during training. It monitors the loss (val_loss) on the validation set and saves the model weights when the loss reaches a new low. By setting 'save_best_only=True', the model only saves the model with the lowest loss, that is, the model is saved only when the loss of the validation set decreases, avoiding the situation in which the model may degrade in performance due to overfitting in the later stages of training.

[0099] The EarlyStopping callback function is used to stop training in advance when the validation set loss no longer decreases, to prevent the model from overfitting the training data after too many training rounds. By setting the patience parameter (patience), when the validation set loss does not decrease within the set value (patience), training stops to prevent the model from overfitting. By setting 'restore_best_weights=True', after stopping training, the model weights are restored to the minimum validation set loss, so that the model is in the optimal state.

[0100] The LearningRateScheduler callback function is used to dynamically adjust the learning rate. The learning rate will gradually decrease during the training process, so that it can quickly approach the optimal solution in the early stage and fine-tune the model weights in the later stage. 'lr_schedule' is the learning rate scheduler function. According to the number of training rounds (epoch), the function will return the current learning rate value. The LearningRateScheduler callback function is used to improve the convergence speed of the model to avoid jumping out of the optimal solution due to too large a learning rate or slow training speed due to too small a learning rate during training.

[0101] The S6 is specifically:

[0102] The trained ConvLSTM model is called and input into the test data generator, which provides test data in batches, dynamically loads test data, and outputs the predicted images of the model on these test data. Images are generated one by one by looping through all prediction results to form continuous video frame images.

[0103] Specific operations include:

[0104] The result visualization and prediction part is equipped with the 'model.predict()' function to generate prediction results. The input is the test data generator, and the output is the prediction result of the model on these test data. In order to prevent the program from terminating due to memory overflow, the 'test_data_generator' test data generator is set to provide test data in batches and dynamically load test data to prevent memory overflow. Since the prediction output of the model for the test data is an array containing the predicted value of each input sample,

[0105] Set 'for iin range(len(predictions))' to process each prediction output one by one by traversing the index of the prediction results, so that each prediction result generates an image, and generate images one by one by looping through all prediction results to form continuous video frame images.

[0106] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0107] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A Schlieren flow field reconstruction method based on a multi-input feature fusion ConvLSTM model, characterized in that , including the following steps: S1, parameter initialization, defining the learning rate, decay rate, batch size, regularization factor, and time step required for model training; S2, data preprocessing, first manually select the pressure parameters and image data to be trained, grayscale all original images, and crop the images to 1024×256 pixels while retaining the original image features, and normalize the original pressure data; S3, data generation and segmentation, divide the data set into training set and validation set, and customize the data generator to batch load and process data during training to avoid loading all data into memory at once; S4. Model construction and compilation. Use the ConvLSTM model to construct a multi-input neural network, including an image input part, a pressure data input part, a feature fusion and an output part. The training set of the multi-input neural network model contains two parts: a processed image and a corresponding normalized pressure parameter. The image is input to the image input part of the multi-input neural network, and the pressure parameter is input to the pressure data input part of the multi-input neural network. After that, the outputs of the image input part and the pressure input part will be subjected to feature cross-learning in the feature fusion part before the model output. S5, model training and callback, set the optimizer, loss function, callback function, define early stop, checkpoint saving and learning rate scheduler, and use the custom generator and callback function after starting model training; S6, result visualization and prediction, including drawing loss curves and model prediction and result saving, drawing loss curves during training and validation, observing the performance of the model, and making predictions on the validation set, and saving the prediction results as image files; The S4 is specifically: The multi-input feature fusion model is divided into three parts: the part that uses the convolution layer to process the image data, the part that uses the time series layer to process the pressure input, and the feature fusion output prediction part; The image processing module contains 5 ConvLSTM2D layers and a Flatten layer. The ConvLSTM2D layer can process time and space information at the same time. Each layer has 64 filters and the convolution kernel size is 3×3. Each ConvLSTM2D layer sets 'return_sequences=True' so that each layer can output a sequence containing the time dimension to retain the characteristics of the time step. A batch normalization layer is inserted after each ConvLSTM2D layer to accelerate the training process and improve the stability of the model. After the last ConvLSTM2D layer, the Flatten layer is used to flatten the output tensor into a one-dimensional vector to prepare for combining with the output of the pressure data. The basic formula for single-step prediction of ConvLSTM is as follows: Where: g Represents the sigmoid function, tanh g represents the hyperbolic tangent activation function, X t is the input of the current time step, H t-1 is the hidden state of the previous time step, C t-1 is the memory unit of the previous time step, I t is the input gate, F t For the forget gate, O t is the output gate, C t is a memory unit, W and b are weight matrix and bias matrix respectively, ° is the product, * is the convolution operator, bi is the bias matrix of input gate, bf is the bias matrix of forget gate, bo is the bias matrix of output gate, bc is the memory unit C t The bias matrix of The pressure data processing module includes a Flatten layer and a Dense fully connected layer. The Flatten layer flattens the input layer of the pressure data and converts it into a one-dimensional vector, merging the pressure data of each time step into a vector. The Dense fully connected layer processes the flattened pressure data. The fully connected layer contains 64 neurons and uses the ReLU activation function to extract the features of the pressure data by adding nonlinear transformations. The feature fusion and prediction output modules are Concatenate layer, Dense fully connected layer, Dropout layer, and Dense layer in order. The Concatenate layer merges the output (x) of the image processing module and the output (y) of the pressure data processing module. This fused output contains the features extracted from the image and pressure data. The merged output passes through a fully connected layer Dense again to further extract mixed features. Then, the Dropout layer is used to randomly discard a certain number of neurons to prevent overfitting. Finally, a fully connected layer is used to output the prediction result. Define the entire model through 'tf.keras.Model', specify the input layer as image and pressure data, and the output layer as predicted value, so that the model integrates the processing pipeline of image and pressure data and can consider the characteristics of these two types of data at the same time; Use Adam optimizer as the adaptive learning rate optimization algorithm; The update rules of the Adam optimizer are as follows: m t =β1m t-1 +(1-β1)g t Where: g t is the gradient of the parameter, β1 and β2 are the decay coefficients of the two exponentially weighted averages, and is the bias-corrected moving average of the gradient, θ t+1 is the updated parameter, η is the learning rate, and ε is a constant greater than 0.

2. The Schlieren flow field reconstruction method based on the multi-input feature fusion ConvLSTM model according to claim 1 is characterized in that: The S2 is specifically: Read the file path containing image data and pressure parameters, arrange the input image data and pressure data in chronological order, align the time points of image data and pressure data at intervals of 1 ms, and adjust the shape of pressure data to the four-dimensional tensor form required by the model, including the number of samples, time steps, 1, and the number of features, to match the shape of image data, and provide a standardized, time-point-matched data set for subsequent training and verification.

3. The Schlieren flow field reconstruction method based on the multi-input feature fusion ConvLSTM model according to claim 2 is characterized in that: The data set in S3 is specifically: The data set is divided into a training set and a validation set by group, the data set includes pressure data and image data, and the pressure data and image data corresponding to each other after the alignment moment are combined into one group, which is divided into 14 groups of combined data, which are combined into a data set for ConvLSTM model training, among which the first 13 groups of combined data are divided into training sets, and the 14th group of combined data is divided into a validation set.

4. The Schlieren flow field reconstruction method based on the multi-input feature fusion ConvLSTM model according to claim 3 is characterized in that: The custom data generator in S3 is specifically: The custom data generator specifically extracts 32 original images and their corresponding pressure data for each training batch from a library that stores preprocessed image paths and corresponding pressure data. The image size is 1024×256, and the number of time steps is 2630. The custom data generator outputs the image data and pressure data at the same moment respectively, and extracts the features of the image data and pressure data at the same moment respectively through the ConvLSTM model.

5. The Schlieren flow field reconstruction method based on the multi-input feature fusion ConvLSTM model according to claim 4 is characterized in that: The S5 is specifically: The model training and callback part mainly includes a model training module and a model callback module. The training module sets the model training process according to the training data, verification data, and the number of training rounds, sets the training data generator to dynamically generate each batch of training data, and sets the verification data generator to dynamically generate each batch of verification data; The model callback module is divided into three parts: model save callback function, early stop callback function, and weight check callback function. The model save callback function automatically saves the model during the training process. It monitors the loss value on the validation set and saves the model weight when the loss value reaches a new low. The model is only saved when the validation set loss decreases. The saving expression is: if((val_loss) t <(best_val_loss)),save_model Where (val_loss) is the loss value of the validation set, t is the training round, and (best_val_loss) is the minimum loss value among all previous training batches. The early stopping callback function stops training early when the validation set loss no longer decreases. The early stopping expression is: if((val_loss) t >min(val_loss) t-p )for p epochs,then stop the training Among them, p is the patience parameter, min(val_loss) t-p is the minimum loss value in the past p rounds, The weight check callback function checks and outputs the statistical information of each layer's weight. The weight check expression is: weights i min =min(weights i ),weights i max =max(weights i ) Among them, weight expresses the weight parameter, weights i Represents the weight parameter of the i-th layer, weights i max Indicates the maximum value of the weight of the i-th layer, weights i min Represents the minimum value of the weight of the i-th layer.

6. The Schlieren flow field reconstruction method based on the multi-input feature fusion ConvLSTM model according to claim 5 is characterized in that: The S6 is specifically: The trained ConvLSTM model is called and input into the test data generator, which provides test data in batches, dynamically loads test data, and outputs the predicted images of the model on these test data. Images are generated one by one by looping through all prediction results to form continuous video frame images.

Images