A high-resolution particle image velocimetry method based on a lightweight model

Abstract

The present invention discloses a high-resolution particle image velocimetry method based on a lightweight model, comprising the following steps: S1: obtaining a data set; S2: constructing an RFPA-PIV model and training the RFPA-PIV model based on the data set, wherein the RFPA-PIV model includes a feature encoder, a 4D correlation body establishment module, a decoder, and an upsampling module; and S3: obtaining an optimized RFPA-PIV model for predicting a velocity field from a particle image. The present invention extracts a resolution feature map of the particle image through the feature encoder in the RFPA-PIV model, thereby reducing the computational complexity of the RFPA-PIV model while maintaining the performance of the RFPA-PIV model. The decoder iteratively updates a set initial optical flow based on a correlation pyramid established by the 4D correlation body establishment module and a resolution feature map extracted by the feature encoder, generating an accurate velocity field through iteration, thereby reducing the number of parameters and the memory consumption of the RFPA-PIV model during training and inference, thereby improving computational efficiency.

Description

Technical Field

[0001] The present invention relates to the technical field of experimental fluid mechanics, and in particular to a high-resolution particle image velocimetry method based on a lightweight model. Background Art

[0002] Particle Image Velocimetry (PIV) is a non-perturbation, non-contact visualization flow field measurement technique that can analyze the global flow velocity field distribution from continuous images, and has important research value and significance. The basic principle of PIV technology is to sprinkle tracer particles into the flow field, illuminate the fluid area with a light sheet generated by lasers and other optical components, and then use a camera to continuously capture particle images at extremely short intervals. Finally, algorithmic analysis is used to determine the velocity field of the particles. Therefore, it is crucial to predict the velocity field with high accuracy and resolution from continuous particle images.

[0003] Velocity field estimation based on traditional PIV methods currently exhibits significant limitations in terms of accuracy, spatial resolution, and robustness, making it difficult to meet the higher demands of current scientific research and engineering applications. Deep learning technology has been proven to offer advantages in processing particle velocity field images, including high prediction accuracy and strong feature fitting capabilities. In recent years, related work has resulted in the development of neural networks for processing PIV images, such as PIV-LiteFlowNet-en, LightPIVNet, and RFMFlowNet. However, current deep learning methods still suffer from low computational efficiency and high video memory consumption when processing high-resolution images. Therefore, a lightweight deep learning model is urgently needed to improve the performance of particle image velocimetry methods. Summary of the Invention

[0004] The present invention provides a high-resolution particle image velocimetry method based on a lightweight model to overcome the technical problems of low computational efficiency and high video memory consumption in existing deep learning methods when processing high-resolution images.

[0005] In order to achieve the above object, the technical solution of the present invention is:

[0006] A high-resolution particle image velocimetry method based on a lightweight model, comprising the following steps:

[0007] S1: Acquire particle image velocimetry simulation data set;

[0008] S2: Constructing an RFPA-PIV model and training the RFPA-PIV model based on a particle image velocimetry simulation dataset. The RFPA-PIV model includes a feature encoder, a 4D correlator building module, a decoder, and an upsampling module.

[0009] The feature encoder is used to extract the resolution feature maps of two consecutive frames of original particle images in the particle image velocimetry simulation data set, and transmit them to the 4D correlation volume establishment module and the decoder module;

[0010] The 4D correlation volume building module constructs a 4D correlation volume based on the two input resolution feature maps, and performs a pooling operation on the constructed 4D correlation volume to obtain a correlation pyramid;

[0011] The decoder is used to iteratively update the set initial optical flow according to the relevant pyramid and the resolution feature map to obtain a predicted optical flow image;

[0012] The upsampling module is used to upsample the predicted optical flow image, thereby restoring the predicted optical flow image to full resolution, and obtaining a velocity field based on the optical flow image restored to full resolution;

[0013] S3: During the training of the RFPA-PIV model, a loss function is set to optimize the RFPA-PIV model, and finally an optimized RFPA-PIV model is obtained for predicting the velocity field from the particle image.

[0014] Furthermore, the feature encoder includes a first downsampling module, a first InceptionNeXt module, a second downsampling module, a second InceptionNeXt module, a third downsampling module, a third InceptionNeXt module, a feature pyramid module and a multi-scale feature aggregation module;

[0015] The first downsampling module is used to downsample the input image to obtain a downsampled image and transmit it to the first InceptionNeXt module. The first InceptionNeXt module is used to extract a resolution feature map of the downsampled image, and then splice the extracted resolution feature map with the downsampled image to obtain a first resolution feature map, and transmit it to the second downsampling module and the feature pyramid module;

[0016] The second downsampling module is used to downsample the first resolution feature map to obtain a first downsampling feature map, and transmit it to the second InceptionNeXt module. The second InceptionNeXt module is used to extract a resolution feature map of the first downsampling feature map, and then splice the extracted resolution feature map with the first downsampling feature map to obtain a second resolution feature map, and transmit it to the third downsampling module and the feature pyramid module;

[0017] The third downsampling module is used to downsample the second resolution feature map to obtain a second downsampling feature map, and transmit it to the third InceptionNeXt module. The third InceptionNeXt module is used to extract a resolution feature map of the second downsampling feature map, and then splice the extracted resolution feature map with the second downsampling feature map to obtain a third resolution feature map, and transmit it to the feature pyramid module;

[0018] The feature pyramid module establishes a feature pyramid based on the first resolution feature map, the second resolution feature map and the third resolution feature map, and transmits the pyramid to the multi-scale feature aggregation module;

[0019] The multi-scale feature aggregation module performs feature aggregation on the feature pyramid to obtain an integrated feature map, and transmits the integrated feature map to the 4D correlation volume establishment module.

[0020] Furthermore, each of the InceptionNeXt modules includes: a depthwise separable convolutional layer, a normalization layer, a multi-scale feature extraction layer, a GELU activation function layer, and a first convolutional layer;

[0021] The depthwise separable convolution layer is used to perform depthwise convolution and pointwise convolution operations on the input image in sequence to obtain a preliminary extracted feature map and transmit it to the normalization layer;

[0022] The normalization layer is used to perform a normalization operation on the initially extracted feature map to obtain a normalized feature map and transmit it to the multi-scale feature extraction layer;

[0023] The multi-scale feature extraction layer includes a second convolutional layer, a third convolutional layer, and a fourth convolutional layer, wherein the second convolutional layer, the third convolutional layer, and the fourth convolutional layer are respectively used to perform feature extraction on the normalized feature map to obtain three feature maps of different scales, and the three feature maps of different scales are spliced ​​in the channel dimension according to their respective channel numbers to obtain a spliced ​​feature map, and the spliced ​​feature map is transmitted to the GELU activation function layer;

[0024] The GELU activation function layer is used to perform nonlinear activation on the concatenated feature map to obtain an activated feature map and transmit it to the first convolutional layer;

[0025] The first convolutional layer is used to reduce the dimension of the activated feature map to obtain a reduced-dimensional feature map.

[0026] Furthermore, the calculation formula of the normalization layer is:

[0027]

[0028] Where x n,c,h,ωis the activation value of the neurons in this layer; is the normalized activation value of the neurons in this layer; n,c,h,ω is the normalized activation value after scaling and offset operations; n and c represent the sample batch size and number of channels, respectively; h and ω represent the height and width, respectively; μ n,h,ω is the mean of all neuron activation values ​​in this layer; ε is a constant used to ensure that the denominator is not zero. is the variance of the activation values ​​of all neurons in this layer; c is the scaling factor used to control the scale after normalization, β c is the offset coefficient, which is used to control the normalized displacement.

[0029] Furthermore, the 4D correlation volume building module constructs a 4D correlation volume based on the two resolution feature maps, and performs a pooling operation on the constructed 4D correlation volume to obtain a correlation pyramid in the following process:

[0030] The dot product operation is performed on the feature vectors between the corresponding pixel pairs of the two-frame resolution feature maps after feature aggregation to construct a 4D correlation volume. The formula for the dot product operation is:

[0031]

[0032] Among them, g θ (I1) and g θ (I2) are the resolution feature maps of the original particle images I1 and I2, respectively, and g θ (I1)∈R H *W*D , g θ (I2)∈R H*W*D ; i and j are the width and height of the first frame particle image, k and l are the width and height of the second frame particle image, and p is the number of channels of the resolution feature map;

[0033] The high-resolution features of the first two dimensions of the constructed 4D correlation body are retained, and the average pooling operation is performed on the last two dimensions of the 4D correlation body according to different kernel sizes to obtain the first tensor, the second tensor, the third tensor and the fourth tensor. A four-layer feature pyramid is established based on the first tensor, the second tensor, the third tensor and the fourth tensor. The tensor dimension calculation formula of the k-th layer of the feature pyramid is:

[0034] C k =H*W*H / 2 k *W / 2 k (6)

[0035] In the formula, H represents the height of the feature map, W represents the width of the feature map, k represents the kth layer of the feature pyramid, and C kRepresents the dimension of the correlation volume at the kth level of the feature pyramid.

[0036] Furthermore, the decoder includes N decoding modules, each decoding module includes a fourth InceptionNeXt module and a decoding convolution layer; the decoder iteratively updates the set initial optical flow according to the relevant pyramid and the resolution feature map, and the process of obtaining the predicted optical flow image is:

[0037] S41: Obtain a query feature map in the relevant pyramid based on the bilinear sampling bilinear_sampler function;

[0038] S42: Splicing the query feature map and the second frame resolution feature map extracted by the feature encoder, and transmitting the spliced ​​features to a first decoding module. The first decoding module processes the input data, outputs a first residual optical flow, and adds the first residual optical flow to the set initial optical flow to obtain an updated optical flow.

[0039] S43: Extracting a feature map of the input data through the second InceptionNeXt module in the first decoding module;

[0040] S44: updating the obtained query feature graph;

[0041] S45: The feature map extracted in S43 is concatenated with the updated query feature map and transmitted to the second decoding module. The second decoding module processes the input data and outputs a second residual optical flow. The second residual optical flow is added to the optical flow output by the first decoding module to obtain an updated optical flow.

[0042] S46: Switch to the next decoding module and update the data input to the next decoding module according to the processing flow corresponding to S43 and S44. The updated optical flow is output by the next decoding module. Iterate N-2 times to obtain the optical flow output by the last decoding module, which is the predicted optical flow image.

[0043] Furthermore, in S41, the process of obtaining the query feature map in the relevant pyramid based on the bilinear sampling bilinear_sampler function is:

[0044] S411: Generate row coordinates and column coordinates based on the height and width of the second frame resolution feature map extracted by the feature encoder, and splice the row coordinates and column coordinates to obtain a first grid coordinate system;

[0045] S412: Divide the first grid coordinate system by the scaling factor 2 of each level of the relevant pyramid i , get the first grid coordinate system after resizing, i is the number of the current level in the relevant pyramid;

[0046] S413: Setting the query radius in the relevant pyramid to r;

[0047] S414: Use the torch.linspace function to generate displacement vectors dx and dy according to the query radius r. The value range of dx and dy is (-r, r).

[0048] S415: Concatenate the displacement vectors dx and dy using the torch.meshgrid function to obtain a second grid coordinate system.

[0049] S416: Adding the resized first grid coordinate system to the second grid coordinate system to obtain a third grid coordinate system;

[0050] S417: using the bilinear_sampler function, sampling each layer of the relevant pyramid according to the sampling points determined by the third grid coordinate system;

[0051] S418: Concatenate the sampling results of all levels to obtain a query feature map;

[0052] In S44, the process of updating the query feature graph is as follows:

[0053] The query feature map obtained in S418 is spliced ​​with the second frame resolution feature map extracted by the feature encoder, and the first grid coordinate system is updated based on the height and width of the spliced ​​feature map. Similarly, each time the output query feature map is spliced ​​with the feature map last used to update the first grid coordinate system, thereby updating the first grid coordinate system.

[0054] Furthermore, the loss function is an L1 distance loss function, and the formula is:

[0055]

[0056] Where, f i To predict the flow field, f gt is a true value.

[0057] Beneficial effects: The present invention establishes an RFPA-PIV model including a feature encoder, a 4D correlation body establishment module, a decoder and an upsampling module. The feature encoder extracts the resolution feature map of the particle image, which can reduce the computational complexity of the RFPA-PIV model while maintaining the performance of the RFPA-PIV model; the decoder iteratively updates the set initial optical flow according to the correlation pyramid established by the 4D correlation body establishment module and the resolution feature map extracted by the feature encoder, and generates an accurate velocity field in an iterative manner, thereby reducing the number of parameters and the video memory consumption of the RFPA-PIV model during training and inference, and improving computational efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following is a brief introduction to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0059] Figure 1 This is a flow chart of a high-resolution particle image velocimetry method based on a lightweight model in the present invention;

[0060] Figure 2 Schematic diagram of the structure of the RFPA-PIV model in an embodiment of the present invention;

[0061] Figure 3 Schematic diagram of the structure of the InceptionNeXt module according to an embodiment of the present invention;

[0062] Figure 4 A velocity field comparison diagram drawn using this method and other existing methods based on a public particle data set in an embodiment of the present invention;

[0063] Figure 5 A velocity field comparison diagram drawn using this method and other existing methods based on a real-world standard flow field in an embodiment of the present invention;

[0064] Figure 6 This is a comparison diagram of the velocity field drawn using this method and other existing methods based on the real-world eddy flow field in an embodiment of the present invention. DETAILED DESCRIPTION

[0065] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0066] This embodiment provides a high-resolution particle image velocimetry method based on a lightweight model, such as Figure 1 As shown, the specific steps include:

[0067] S1: Acquire particle image velocimetry simulation data set;

[0068] S2: Build Figure 2The recursive feature aggregation pyramid particle image velocimetry (RFPA-PIV) model shown is trained based on a particle image velocimetry simulation dataset. The RFPA-PIV model includes a feature encoder, a 4D correlator building module, a decoder, and an upsampling module.

[0069] The feature encoder is used to extract the resolution feature maps of two consecutive frames of original particle images in the particle image velocimetry simulation data set, and transmit them to the 4D correlation volume establishment module and the decoder module;

[0070] Specifically, in this embodiment, the RFPA-PIV model can only process two consecutive frames of images at a time, and multiple images need to be processed multiple times.

[0071] In this embodiment, feature maps with a 1 / 8 resolution of two consecutive frames of original particle images I1 and I2 are respectively extracted by a feature encoder.

[0072] The 4D correlation volume building module constructs a 4D correlation volume based on the two input resolution feature maps, and performs a pooling operation on the constructed 4D correlation volume to obtain a correlation pyramid;

[0073] The decoder is used to iteratively update the set initial optical flow according to the relevant pyramid and the resolution feature map to obtain a predicted optical flow image;

[0074] The upsampling module is used to upsample the predicted optical flow image, thereby restoring the predicted optical flow image to full resolution, and obtaining a velocity field based on the optical flow image restored to full resolution;

[0075] Specifically, in this embodiment, the decoder outputs a 1 / 8 resolution optical flow after iterative update, and the 1 / 8 resolution optical flow is upsampled 8 times by the upsampling module to restore it to the full resolution, and the corresponding pixel points in the optical flow image restored to the full resolution are replaced with the set color according to the numerical value to obtain the full-resolution velocity field.

[0076] S3: During the training of the RFPA-PIV model, a loss function is set to optimize the RFPA-PIV model, and finally an optimized RFPA-PIV model is obtained for predicting the velocity field from the particle image.

[0077] In a specific embodiment, the feature encoder includes a first downsampling module, a first InceptionNeXt module, a second downsampling module, a second InceptionNeXt module, a third downsampling module, a third InceptionNeXt module, a feature pyramid module and a multi-scale feature aggregation module;

[0078] The first downsampling module is used to downsample the input image to obtain a downsampled image and transmit it to the first InceptionNeXt module. The first InceptionNeXt module is used to extract a resolution feature map of the downsampled image, and then splice the extracted resolution feature map with the downsampled image to obtain a first resolution feature map, and transmit it to the second downsampling module and the feature pyramid module;

[0079] In this embodiment, the first resolution feature map is Resolution feature map.

[0080] The second downsampling module is used to downsample the first resolution feature map to obtain a first downsampling feature map, and transmit it to the second InceptionNeXt module. The second InceptionNeXt module is used to extract a resolution feature map of the first downsampling feature map, and then splice the extracted resolution feature map with the first downsampling feature map to obtain a second resolution feature map, and transmit it to the third downsampling module and the feature pyramid module;

[0081] In this embodiment, the second resolution feature map is Resolution feature map.

[0082] The third downsampling module is used to downsample the second resolution feature map to obtain a second downsampling feature map, and transmit it to the third InceptionNeXt module. The third InceptionNeXt module is used to extract a resolution feature map of the second downsampling feature map, and then splice the extracted resolution feature map with the second downsampling feature map to obtain a third resolution feature map, and transmit it to the feature pyramid module;

[0083] In this embodiment, the third resolution feature map is Resolution feature map.

[0084] The feature pyramid module establishes a feature pyramid based on the first resolution feature map, the second resolution feature map and the third resolution feature map, where the first resolution feature map is the first layer of the feature pyramid, the second resolution feature map is the second layer of the feature pyramid, and the third resolution feature map is the third layer of the feature pyramid, and transmits the result to the multi-scale feature aggregation module;

[0085] The multi-scale feature aggregation module Multi-scale Feature Aggregation (MFA) performs feature aggregation on the feature pyramid to obtain an integrated feature map, and transmits it to the 4D correlation volume establishment module.

[0086] Specifically, by aggregating multi-scale features, the RFPA-PIV model can be made more robust to the size variation of the input image.

[0087] In a specific embodiment, Figure 3 As shown, each of the InceptionNeXt modules includes: a depth-separable convolution layer, a normalization layer (Layer Normalization), a multi-scale feature extraction layer, a GELU (Gaussian Error Linear Unit) activation function layer and a first convolution layer;

[0088] The depthwise separable convolution layer is used to perform depthwise convolution and pointwise convolution operations on the input image in sequence to obtain a preliminary extracted feature map and transmit it to the normalization layer;

[0089] The normalization layer is used to perform a normalization operation on the initially extracted feature map to obtain a normalized feature map and transmit it to the multi-scale feature extraction layer;

[0090] The multi-scale feature extraction layer includes a second convolutional layer, a third convolutional layer, and a fourth convolutional layer, wherein the second convolutional layer, the third convolutional layer, and the fourth convolutional layer are respectively used to perform feature extraction on the normalized feature map to obtain three feature maps of different scales, and the three feature maps of different scales are spliced ​​in the channel dimension according to their respective channel numbers to obtain a spliced ​​feature map, and the spliced ​​feature map is transmitted to the GELU activation function layer;

[0091] Specifically, in the feature encoder module, the second convolution layer, the third convolution layer and the fourth convolution layer in the multi-scale feature extraction layer are 1×1 convolution layer, 3×3 convolution layer and 5×5 convolution layer, respectively.

[0092] The GELU activation function layer is used to perform nonlinear activation on the concatenated feature map to increase the nonlinear expression capability, obtain the activated feature map, and transmit it to the first convolutional layer;

[0093] The first convolutional layer is used to reduce the dimension of the activated feature map and integrate channel information to obtain a reduced-dimensional feature map.

[0094] Specifically, in the feature encoder module, the first convolutional layer is a 3×3 convolutional layer.

[0095] In a specific embodiment, the calculation formula of the normalization layer is:

[0096]

[0097] Where x n,c,h,ω is the activation value of the neurons in this layer; is the normalized activation value of the neurons in this layer; n,c,h,ω is the normalized activation value after scaling and offset operations; n and c represent the sample batch size and number of channels, respectively; h and ω represent the height and width, respectively; μ n,h,ω is the mean of all neuron activation values ​​in this layer; ε is a constant used to ensure that the denominator is not zero. is the variance of the activation values ​​of all neurons in this layer; c is the scaling factor used to control the scale after normalization, β c is the offset coefficient, which is used to control the normalized displacement, and γ c and β c are all learnable parameters.

[0098] In a specific embodiment, the 4D correlation volume building module constructs a 4D correlation volume based on two resolution feature maps, and performs a pooling operation on the constructed 4D correlation volume to obtain a correlation pyramid in the following process:

[0099] The dot product operation is performed on the feature vectors between the corresponding pixel pairs of the two-frame resolution feature maps after feature aggregation to construct a 4D correlation volume. The formula for the dot product operation is:

[0100]

[0101] Among them, g θ (I1) and g θ (I2) are the resolution feature maps of the original particle images I1 and I2, respectively, and g θ (I1)∈R H *W*D , g θ (I2)∈R H*W*D ; i and j are the width and height of the first frame particle image, k and l are the width and height of the second frame particle image, and p is the number of channels of the resolution feature map;

[0102] Specifically, the 4D correlation volume is established by the dot product operation of two feature maps: the dot product operation is performed on two feature maps with a resolution of H*W to form a 4D correlation volume of H*W*H*W, and the high-resolution features of the first two dimensions of the constructed 4D correlation volume are retained. The last two dimensions of the 4D correlation volume are average pooled according to different kernel sizes. Specifically, the kernel sizes of the pooling layer are 1, 2, 4 and 8, and the first tensor, the second tensor, the third tensor and the fourth tensor are obtained. A four-layer feature pyramid is established based on the first tensor, the second tensor, the third tensor and the fourth tensor. The tensor dimension calculation formula of the kth layer of the feature pyramid is:

[0103] C k =H*W*H / 2 k *W / 2 k (6)

[0104] In the formula, H represents the height of the feature map, W represents the width of the feature map, k represents the kth layer of the feature pyramid, and C k Represents the dimension of the correlation volume at the kth level of the feature pyramid.

[0105] Specifically, each layer in the correlation pyramid is obtained by performing pooling operations of different scales on the last two dimensions of the original 4D correlation volume. Since the first two dimensions of the 4D correlation volume remain unchanged, high-resolution information is retained; after the last two dimensions are pooled, small displacement motion can be tracked. Therefore, the correlation pyramid can represent visual similarity at different scales, from high resolution to low resolution, which helps to more accurately estimate the optical flow field in subsequent iterative updates.

[0106] In a specific embodiment, in S4, the decoder includes N decoding modules, each decoding module includes a fourth InceptionNeXt module and a decoding convolutional layer; the decoder iteratively updates the set initial optical flow according to the relevant pyramid and the resolution feature map, and the process of obtaining the predicted optical flow image is:

[0107] S41: Obtain a query feature map in the relevant pyramid based on the bilinear sampling bilinear_sampler function;

[0108] S42: Splicing the query feature map and the second frame resolution feature map extracted by the feature encoder, and transmitting the spliced ​​features to a first decoding module. The first decoding module processes the input data, outputs a first residual optical flow, and adds the first residual optical flow to the set initial optical flow to obtain an updated optical flow.

[0109] Specifically, the initial optical flow is set to 0.

[0110] S43: Extracting a feature map of the input data through the second InceptionNeXt module in the first decoding module;

[0111] S44: updating the obtained query feature graph;

[0112] S45: The feature map extracted in S43 is concatenated with the updated query feature map and transmitted to the second decoding module. The second decoding module processes the input data and outputs a second residual optical flow. The second residual optical flow is added to the optical flow output by the first decoding module to obtain an updated optical flow.

[0113] S46: Switch to the next decoding module and update the data input to the next decoding module according to the processing flow corresponding to S43 and S44. The updated optical flow is output by the next decoding module. Based on N-2 decoding modules, iterate N-2 times according to S43-S45 to obtain the optical flow output by the last decoding module, which is the predicted optical flow image.

[0114] Specifically, based on the updated feature map and residual optical flow Δf output by the decoding module after each iteration, the residual optical flow is applied to the current optical flow f i To update the optical flow, the update formula is:

[0115] f i+1 =f i +Δf

[0116] Therefore, the decoder generates a series of estimated optical flows {f1,…,f N}.

[0117] Specifically, in this embodiment, the RFPA-PIV model uses the InceptionNeXt module as an optical flow updater to generate an accurate velocity field in an iterative manner. Due to the reduction in the number of parameters, the InceptionNeXt module helps to reduce the model's video memory consumption during training and inference, thereby improving computational efficiency.

[0118] In a specific embodiment, in S41, the process of obtaining the query feature map in the relevant pyramid based on the bilinear sampling bilinear_sampler function is:

[0119] S411: Generate row coordinates and column coordinates based on the height and width of the second frame resolution feature map extracted by the feature encoder, respectively, and concatenate the row coordinates and column coordinates to obtain a first grid coordinate system for representing coordinate information of each pixel point in the second frame resolution feature map;

[0120] S412: Divide the first grid coordinate system by the scaling factor 2 of each level of the relevant pyramid i, get the first grid coordinate system after resizing, i is the number of layers of the current level in the relevant pyramid, to adapt to the feature map size of the current level;

[0121] S413: Setting the query radius in the correlation pyramid to r. In this embodiment, r is set to 4. The query radius r defines the size of the area in the other frame image where matching features need to be searched for each pixel when extracting and calculating the correlation feature map.

[0122] S414: Use the torch.linspace function to generate displacement vectors dx and dy based on the query radius r. The value range of dx and dy is (-r, r). dx and dy represent the horizontal and vertical offsets in the relevant pyramid, respectively. Therefore, the values ​​from -r to r are considered, a total of 2×r+1 pixels.

[0123] S415: Concatenate the displacement vectors dx and dy using the torch.meshgrid function to obtain a second grid coordinate system, where the second grid coordinates represent the offset of the bilinear_sampler function when sampling around each coordinate point.

[0124] S416: Add the resized first grid coordinate system to the second grid coordinate system to obtain a third grid coordinate system. The third grid coordinate system indicates that sampling is performed in an area of ​​(2×r+1)×(2×r+1) around each sampling coordinate point at each level.

[0125] S417: Use the bilinear_sampler function to sample each layer of the relevant pyramid according to the sampling points determined by the third grid coordinate system.

[0126] S418: Concatenate the sampling results of all levels to obtain a query feature map, which shows how the feature vector in the first frame matches the feature vector in the second frame at each position. This feature map is used to estimate the motion of each point in the image.

[0127] In S44, the process of updating the query feature graph is as follows:

[0128] The query feature map obtained in S418 is spliced ​​with the second frame resolution feature map extracted by the feature encoder, and the first grid coordinate system is updated based on the height and width of the spliced ​​feature map. Similarly, each time the output query feature map is spliced ​​with the feature map last used to update the first grid coordinate system, thereby updating the first grid coordinate system.

[0129] In a specific embodiment, the loss function is an L1 distance loss function, and the formula is:

[0130]

[0131] Where, f i To predict the flow field, f gt is a true value.

[0132] In order to verify the effectiveness of the RFPA-PIV model proposed in this embodiment, the velocity field of the public PIV dataset, the real standard flow field, and the real eddy flow field are estimated using the RFPA-PIV model. Figure 4 As shown in the figure, the velocity field estimation results of the public PIV dataset using the RFPA-PIV model and the real ground, HS model, WIDIM model and PIV-RAFT model are compared. It can be seen from the figure that the velocity field image obtained by the RFPA-PIV model ( Figure 4 (e)) and the real ground velocity field image ( Figure 4 (a)) is the closest, and the average Euclidean distance error (AEE) is the lowest among several square models, which is 4.7. Figure 5 As shown in the figure, the velocity field estimation results of the real standard flow field using the RFPA-PIV model, the WIDIM model, the HS model, the PIV-DCNN model, the PIV-LightFlowNet-en model and the PIV-RAFT model are compared. It can be seen from the figure that the RFPA-PIV model can capture the movement of particles more accurately. Figure 5 Velocity field images obtained using the RFPA-PIV model within the red box selection range of (d), (e) and (f) Figure 5 (f)) in the above is the most clear and accurate. Figure 6 As shown in the figure, the velocity field estimation results of the real eddy flow field using the RFPA-PIV model, the HS model, the PIV-LightFlowNet-en model and the PIV-RAFT model are compared. It can be seen from the figure that Figure 6 The velocity field images obtained by the RFPA-PIV model are shown in the white boxes of (b), (c) and (d). Figure 6 The particle motion shown in (d) is more regular and more consistent with the motion of particles in a real eddy current field.

[0133] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A high-resolution particle image velocimetry method based on a lightweight model, characterized in that: The specific steps include: S1: Acquire particle image velocimetry simulation data set; S2: Constructing an RFPA-PIV model and training the RFPA-PIV model based on a particle image velocimetry simulation dataset. The RFPA-PIV model includes a feature encoder, a 4D correlator building module, a decoder, and an upsampling module. The feature encoder is used to extract the resolution feature maps of two consecutive frames of original particle images in the particle image velocimetry simulation data set, and transmit them to the 4D correlation volume establishment module and the decoder module; The 4D correlation volume building module constructs a 4D correlation volume based on the two input resolution feature maps, and performs a pooling operation on the constructed 4D correlation volume to obtain a correlation pyramid; The decoder is used to iteratively update the set initial optical flow according to the relevant pyramid and the resolution feature map to obtain a predicted optical flow image; The upsampling module is used to upsample the predicted optical flow image, thereby restoring the predicted optical flow image to full resolution, and obtaining a velocity field based on the optical flow image restored to full resolution; S3: During the training of the RFPA-PIV model, a loss function is set to optimize the RFPA-PIV model, and finally an optimized RFPA-PIV model is obtained for predicting the velocity field from the particle image.

2. The high-resolution particle image velocimetry method based on a lightweight model according to claim 1, characterized in that: The feature encoder includes a first downsampling module, a first InceptionNeXt module, a second downsampling module, a second InceptionNeXt module, a third downsampling module, a third InceptionNeXt module, a feature pyramid module and a multi-scale feature aggregation module; The first downsampling module is used to downsample the input image to obtain a downsampled image and transmit it to the first InceptionNeXt module. The first InceptionNeXt module is used to extract a resolution feature map of the downsampled image, and then splice the extracted resolution feature map with the downsampled image to obtain a first resolution feature map, and transmit it to the second downsampling module and the feature pyramid module; The second downsampling module is used to downsample the first resolution feature map to obtain a first downsampling feature map, and transmit it to the second InceptionNeXt module. The second InceptionNeXt module is used to extract a resolution feature map of the first downsampling feature map, and then splice the extracted resolution feature map with the first downsampling feature map to obtain a second resolution feature map, and transmit it to the third downsampling module and the feature pyramid module; The third downsampling module is used to downsample the second resolution feature map to obtain a second downsampling feature map, and transmit it to the third InceptionNeXt module. The third InceptionNeXt module is used to extract a resolution feature map of the second downsampling feature map, and then splice the extracted resolution feature map with the second downsampling feature map to obtain a third resolution feature map, and transmit it to the feature pyramid module; The feature pyramid module establishes a feature pyramid based on the first resolution feature map, the second resolution feature map and the third resolution feature map, and transmits the pyramid to the multi-scale feature aggregation module; The multi-scale feature aggregation module performs feature aggregation on the feature pyramid to obtain an integrated feature map, and transmits the integrated feature map to the 4D correlation volume establishment module.

3. The high-resolution particle image velocimetry method based on a lightweight model according to claim 2, characterized in that: Each of the InceptionNeXt modules includes: a depth-separable convolutional layer, a normalization layer, a multi-scale feature extraction layer, a GELU activation function layer, and a first convolutional layer; The depthwise separable convolution layer is used to perform depthwise convolution and pointwise convolution operations on the input image in sequence to obtain a preliminary extracted feature map and transmit it to the normalization layer; The normalization layer is used to perform a normalization operation on the initially extracted feature map to obtain a normalized feature map and transmit it to the multi-scale feature extraction layer; The multi-scale feature extraction layer includes a second convolutional layer, a third convolutional layer, and a fourth convolutional layer, wherein the second convolutional layer, the third convolutional layer, and the fourth convolutional layer are respectively used to perform feature extraction on the normalized feature map to obtain three feature maps of different scales, and the three feature maps of different scales are spliced ​​in the channel dimension according to their respective channel numbers to obtain a spliced ​​feature map, and the spliced ​​feature map is transmitted to the GELU activation function layer; The GELU activation function layer is used to perform nonlinear activation on the concatenated feature map to obtain an activated feature map and transmit it to the first convolutional layer; The first convolutional layer is used to reduce the dimension of the activated feature map to obtain a reduced-dimensional feature map.

4. The high-resolution particle image velocimetry method based on a lightweight model according to claim 3, characterized in that: The calculation formula of the normalization layer is: Where x n,c,h,ω is the activation value of the neurons in this layer; is the normalized activation value of the neurons in this layer; n,c,h,ω is the normalized activation value after scaling and offset operations; n and c represent the sample batch size and number of channels, respectively; h and ω represent the height and width, respectively; μ n,h,ω is the mean of all neuron activation values ​​in this layer; ε is a constant used to ensure that the denominator is not zero, is the variance of the activation values ​​of all neurons in this layer; c is the scaling factor used to control the scale after normalization, β c is the offset coefficient, which is used to control the normalized displacement.

5. The high-resolution particle image velocimetry method based on a lightweight model according to claim 1, characterized in that: The 4D correlation volume building module constructs a 4D correlation volume based on two resolution feature maps, and performs a pooling operation on the constructed 4D correlation volume to obtain a correlation pyramid. The process is as follows: The dot product operation is performed on the feature vectors between the corresponding pixel pairs of the two-frame resolution feature maps after feature aggregation to construct a 4D correlation volume. The formula for the dot product operation is: Among them, g θ (I1) and g θ (I2) are the resolution feature maps of the original particle images I1 and I2, respectively, and g θ (I1)∈R H*W*D , g θ (I2)∈R H*W*D ; i and j are the width and height of the first frame particle image, k and l are the width and height of the second frame particle image, and p is the number of channels of the resolution feature map; The high-resolution features of the first two dimensions of the constructed 4D correlation body are retained, and the average pooling operation is performed on the last two dimensions of the 4D correlation body according to different kernel sizes to obtain the first tensor, the second tensor, the third tensor and the fourth tensor. A four-layer feature pyramid is established based on the first tensor, the second tensor, the third tensor and the fourth tensor. The tensor dimension calculation formula of the k-th layer of the feature pyramid is: C k =H*W*H / 2 k *W / 2 k (6) In the formula, H represents the height of the feature map, W represents the width of the feature map, k represents the kth layer of the feature pyramid, and C k Represents the dimension of the correlation volume at the kth level of the feature pyramid.

6. The high-resolution particle image velocimetry method based on a lightweight model according to claim 2, characterized in that: The decoder includes N decoding modules, each of which includes a fourth InceptionNeXt module and a decoding convolutional layer; the decoder iteratively updates the set initial optical flow according to the relevant pyramid and resolution feature map to obtain a predicted optical flow image in the following process: S41: Obtain a query feature map in the relevant pyramid based on the bilinear sampling bilinear_sampler function; S42: Splicing the query feature map and the second frame resolution feature map extracted by the feature encoder, and transmitting the spliced ​​features to a first decoding module. The first decoding module processes the input data, outputs a first residual optical flow, and adds the first residual optical flow to the set initial optical flow to obtain an updated optical flow. S43: Extracting a feature map of the input data through the second InceptionNeXt module in the first decoding module; S44: updating the obtained query feature graph; S45: The feature map extracted in S43 is concatenated with the updated query feature map and transmitted to the second decoding module. The second decoding module processes the input data and outputs a second residual optical flow. The second residual optical flow is added to the optical flow output by the first decoding module to obtain an updated optical flow. S46: Switch to the next decoding module and update the data input to the next decoding module according to the processing flow corresponding to S43 and S44. The updated optical flow is output by the next decoding module. Iterate N-2 times to obtain the optical flow output by the last decoding module, which is the predicted optical flow image.

7. The high-resolution particle image velocimetry method based on a lightweight model according to claim 2, characterized in that: In S41, the process of obtaining the query feature map in the relevant pyramid based on the bilinear sampling bilinear_sampler function is: S411: Generate row coordinates and column coordinates based on the height and width of the second frame resolution feature map extracted by the feature encoder, and splice the row coordinates and column coordinates to obtain a first grid coordinate system; S412: Divide the first grid coordinate system by the scaling factor 2 of each level of the relevant pyramid i , get the first grid coordinate system after resizing, i is the number of the current level in the relevant pyramid; S413: Setting the query radius in the relevant pyramid to r; S414: Use the torch.linspace function to generate displacement vectors dx and dy according to the query radius r. The value range of dx and dy is (-r, r). S415: Concatenate the displacement vectors dx and dy using the torch.meshgrid function to obtain a second grid coordinate system. S416: Adding the resized first grid coordinate system to the second grid coordinate system to obtain a third grid coordinate system; S417: using the bilinear_sampler function, sampling each layer of the relevant pyramid according to the sampling points determined by the third grid coordinate system; S418: Concatenate the sampling results of all levels to obtain a query feature map; In S44, the process of updating the query feature graph is as follows: The query feature map obtained in S418 is spliced ​​with the second frame resolution feature map extracted by the feature encoder, and the first grid coordinate system is updated based on the height and width of the spliced ​​feature map. Similarly, each time the output query feature map is spliced ​​with the feature map last used to update the first grid coordinate system, thereby updating the first grid coordinate system.

8. The high-resolution particle image velocimetry method based on a lightweight model according to claim 1, characterized in that: The loss function is the L1 distance loss function, and the formula is: Where, f i To predict the flow field, f gt is the true value.

Images