High-resolution particle image velocity measurement method based on lightweight model

Abstract

The invention discloses a high-resolution particle image velocity measurement method based on a lightweight model. The method comprises the following steps: S1, acquiring a data set; s2, an RFPA-PIV model is constructed, the RFPA-PIV model is trained based on the data set, and the RFPA-PIV model comprises a feature encoder, a 4D correlator establishment module, a decoder and an up-sampling module; and S3, obtaining an optimized RFPA-PIV model so as to predict a velocity field from the particle image. According to the method, the resolution feature map of the particle image is extracted through the feature encoder in the RFPA-PIV model, so that the calculation complexity of the RFPA-PIV model can be reduced, and meanwhile, the performance of the RFPA-PIV model is kept; the decoder iteratively updates the set initial optical flow according to the correlation pyramid established by the 4D correlation body establishment module and the resolution feature map extracted by the feature encoder, and generates an accurate velocity field through an iteration mode, thereby reducing the parameter quantity and the video memory consumption of the RFPA-PIV model in the training and reasoning process, and improving the calculation efficiency.

Description

Technical Field

[0001] The invention relates to the technical field of experimental fluid mechanics, and in particular to a high-resolution particle image velocimetry method based on a lightweight model. Background Art

[0002] Particle Image Velocimetry (PIV) is a non-disturbance, non-contact visualization flow field measurement technology that can analyze and obtain the global flow velocity field distribution from continuous images, and has important research value and significance. The basic principle of PIV technology is: scatter tracer particles in the flow field, obtain a light source through lasers and other optical elements to illuminate the fluid area, and then use a camera to continuously shoot in a very short time interval to obtain particle images, and finally obtain the velocity field of particle motion through algorithm analysis. Therefore, how to predict the velocity field with high precision and high resolution from continuous particle images is very critical.

[0003] At present, velocity field estimation based on traditional PIV methods has shown great limitations in terms of accuracy, spatial resolution and robustness, and it is difficult to meet the higher requirements in current scientific research and engineering applications. Deep learning technology has been proven to have the advantages of high prediction accuracy and strong feature fitting ability when processing particle velocity field images. In recent years, neural networks such as PIV-LiteFlowNet-en, LightPIVNet, and RFMFlowNet have been born in related work for processing PIV images. However, the current deep learning methods still have the problems of low computational efficiency and high video memory consumption when processing high-resolution images. Therefore, there is an urgent need for a lightweight deep learning model to improve the performance of particle image velocimetry methods. Summary of the invention

[0004] The present invention provides a high-resolution particle image velocimetry method based on a lightweight model to overcome the technical problems of low computational efficiency and high video memory consumption in existing deep learning methods when processing high-resolution images.

[0005] In order to achieve the above object, the technical solution of the present invention is:

[0006] A high-resolution particle image velocimetry method based on a lightweight model, the specific steps comprising:

[0007] S1: Obtain particle image velocimetry simulation data set;

[0008] S2: constructing an RFPA-PIV model and training the RFPA-PIV model based on a particle image velocimetry simulation data set, wherein the RFPA-PIV model includes a feature encoder, a 4D correlation body building module, a decoder, and an upsampling module;

[0009] The feature encoder is used to extract the resolution feature maps of two consecutive frames of original particle images in the particle image velocimetry simulation data set, and transmit them to the 4D correlation volume establishment module and the decoder module;

[0010] The 4D correlation volume building module constructs a 4D correlation volume based on the two input resolution feature maps, and performs a pooling operation on the constructed 4D correlation volume to obtain a correlation pyramid;

[0011] The decoder is used to iteratively update the set initial optical flow according to the relevant pyramid and the resolution feature map to obtain a predicted optical flow image;

[0012] The upsampling module is used to upsample the predicted optical flow image, thereby restoring the predicted optical flow image to full resolution, and obtaining a velocity field based on the optical flow image restored to full resolution;

[0013] S3: During the training of the RFPA-PIV model, a loss function is set to optimize the RFPA-PIV model, and finally an optimized RFPA-PIV model is obtained for predicting the velocity field from the particle image.

[0014] Further, the feature encoder includes a first downsampling module, a first InceptionNeXt module, a second downsampling module, a second InceptionNeXt module, a third downsampling module, a third InceptionNeXt module, a feature pyramid module and a multi-scale feature aggregation module;

[0015] The first downsampling module is used to downsample the input image to obtain a downsampled image, and transmit it to the first InceptionNeXt module. The first InceptionNeXt module is used to extract a resolution feature map of the downsampled image, and then splice the extracted resolution feature map with the downsampled image to obtain a first resolution feature map, and transmit it to the second downsampling module and the feature pyramid module;

[0016] The second downsampling module is used to downsample the first resolution feature map to obtain a first downsampling feature map, and transmit it to the second InceptionNeXt module. The second InceptionNeXt module is used to extract a resolution feature map of the first downsampling feature map, and then splice the extracted resolution feature map with the first downsampling feature map to obtain a second resolution feature map, and transmit it to the third downsampling module and the feature pyramid module;

[0017] The third down-sampling module is used to down-sample the second resolution feature map to obtain a second down-sampling feature map, and transmit it to the third InceptionNeXt module. The third InceptionNeXt module is used to extract a resolution feature map of the second down-sampling feature map, and then splice the extracted resolution feature map with the second down-sampling feature map to obtain a third resolution feature map, and transmit it to the feature pyramid module;

[0018] The feature pyramid module establishes a feature pyramid based on the first resolution feature map, the second resolution feature map and the third resolution feature map, and transmits the pyramid to the multi-scale feature aggregation module;

[0019] The multi-scale feature aggregation module performs feature aggregation on the feature pyramid to obtain an integrated feature map, and transmits the integrated feature map to the 4D correlation volume establishment module.

[0020] Furthermore, each of the InceptionNeXt modules includes: a depth-separable convolutional layer, a normalization layer, a multi-scale feature extraction layer, a GELU activation function layer and a first convolutional layer;

[0021] The depthwise separable convolution layer is used to perform depthwise convolution and pointwise convolution operations on the input image in sequence to obtain a preliminarily extracted feature map, and transmit it to the normalization layer;

[0022] The normalization layer is used to perform a normalization operation on the initially extracted feature map to obtain a normalized feature map, and transmit it to the multi-scale feature extraction layer;

[0023] The multi-scale feature extraction layer includes a second convolution layer, a third convolution layer and a fourth convolution layer, wherein the second convolution layer, the third convolution layer and the fourth convolution layer are respectively used to perform feature extraction on the normalized feature map to obtain three feature maps of different scales, and the three feature maps of different scales are spliced ​​in the channel dimension according to their respective channel numbers to obtain a spliced ​​feature map, and transmitted to the GELU activation function layer;

[0024] The GELU activation function layer is used to perform nonlinear activation on the concatenated feature map to obtain the activated feature map, and transmit it to the first convolutional layer;

[0025] The first convolutional layer is used to reduce the dimension of the activated feature map to obtain a reduced-dimensional feature map.

[0026] Furthermore, the calculation formula of the normalization layer is:

[0027]

[0028]

[0029]

[0030]

[0031] In the formula, x n,c,h,ω is the activation value of the neurons in this layer; is the normalized activation value of the neurons in this layer; n,c,h,ω is the normalized activation value after scaling and offset operations; n and c represent the sample batch size and the number of channels, respectively; h and ω represent the height and width, respectively; μ n,h,ω is the mean of all neuron activation values ​​in this layer; ε is a constant used to ensure that the denominator is not zero. is the variance of the activation values ​​of all neurons in this layer; c is the scaling factor used to control the scale after normalization, β c is the offset coefficient, which is used to control the normalized displacement.

[0032] Furthermore, the 4D correlation volume building module constructs a 4D correlation volume based on two resolution feature maps, and performs a pooling operation on the constructed 4D correlation volume to obtain a correlation pyramid process as follows:

[0033] The dot product operation is performed on the feature vectors between the corresponding pixel pairs of the two frame resolution feature maps after feature aggregation to construct a 4D correlation volume. The formula for the dot product operation is:

[0034]

[0035] Among them, g θ (I 1 ) and g θ (I 2 ) are the original particle images I 1 ,I 2 The resolution feature map of g θ (I 1 )∈R H *W*D , g θ (I 2 )∈R H*W*D ; i and j are the width and height of the first frame particle image, k and l are the width and height of the second frame particle image, and p is the number of channels of the resolution feature map;

[0036] The high-resolution features of the first two dimensions of the constructed 4D correlation body are retained, and the average pooling operation is performed on the last two dimensions of the 4D correlation body according to different kernel sizes to obtain the first tensor, the second tensor, the third tensor and the fourth tensor, and a four-layer feature pyramid is established based on the first tensor, the second tensor, the third tensor and the fourth tensor. The tensor dimension calculation formula of the kth layer of the feature pyramid is:

[0037] C k =H*W*H / 2 k *W / 2 k (6)

[0038] In the formula, H represents the height of the feature map, W represents the width of the feature map, k represents the kth layer of the feature pyramid, and C k Represents the dimension of the correlation volume at the kth level of the feature pyramid.

[0039] Furthermore, the decoder includes N decoding modules, each decoding module includes a fourth InceptionNeXt module and a decoding convolution layer; the decoder iteratively updates the set initial optical flow according to the relevant pyramid and the resolution feature map, and the process of obtaining the predicted optical flow image is:

[0040] S41: Obtain a query feature map in the relevant pyramid based on a bilinear sampling bilinear_sampler function;

[0041] S42: splicing the query feature map and the second frame resolution feature map extracted by the feature encoder, and transmitting the splicing to the first decoding module, the first decoding module processes the input data, outputs a first residual optical flow, and adds the first residual optical flow to the set initial optical flow to obtain an updated optical flow;

[0042] S43: extracting a feature map of the input data through the second InceptionNeXt module in the first decoding module;

[0043] S44: updating the obtained query feature graph;

[0044] S45: splicing the feature map extracted in S43 and the updated query feature map, and transmitting them to the second decoding module, the second decoding module processes the input data, outputs a second residual optical flow, and adds the second residual optical flow to the optical flow output by the first decoding module to obtain an updated optical flow;

[0045] S46: Switch to the next decoding module, and update the data input to the next decoding module according to the processing flow corresponding to S43 and S44, output the updated optical flow through the next decoding module, iterate N-2 times, and obtain the optical flow output by the last decoding module, which is the predicted optical flow image.

[0046] Furthermore, in S41, the process of obtaining the query feature map in the relevant pyramid based on the bilinear sampling bilinear_sampler function is:

[0047] S411: Generate row coordinates and column coordinates based on the height and width of the second frame resolution feature map extracted by the feature encoder, respectively, and concatenate the row coordinates and the column coordinates to obtain a first grid coordinate system;

[0048] S412: Divide the first grid coordinate system by the scaling factor 2 of each level of the relevant pyramid i , get the first grid coordinate system after resizing, i is the number of the current level in the relevant pyramid;

[0049] S413: Setting the query radius in the relevant pyramid to r;

[0050] S414: Use the torch.linspace function to generate displacement vectors dx and dy according to the query radius r, and the value range of dx and dy is (-r, r);

[0051] S415: Concatenate the displacement vectors dx and dy through the torch.meshgrid function to obtain a second grid coordinate system;

[0052] S416: Add the resized first grid coordinate system to the second grid coordinate system to obtain a third grid coordinate system;

[0053] S417: using the bilinear_sampler function, sampling each layer of the relevant pyramid according to the sampling points determined by the third grid coordinate system;

[0054] S418: concatenating the sampling results of all levels to obtain a query feature map;

[0055] In S44, the process of updating the query feature graph is as follows:

[0056] The query feature map obtained in S418 is spliced ​​with the second frame resolution feature map extracted by the feature encoder, and the first grid coordinate system is updated based on the height and width of the spliced ​​feature map. Similarly, each time the output query feature map is spliced ​​with the feature map last used to update the first grid coordinate system, thereby updating the first grid coordinate system.

[0057] Furthermore, the loss function is L 1 Distance loss function, the formula is:

[0058]

[0059] In the formula, fi To predict the flow field, f gt is true value.

[0060] Beneficial effects: The present invention establishes an RFPA-PIV model including a feature encoder, a 4D correlation body establishment module, a decoder and an upsampling module. The feature encoder extracts the resolution feature map of the particle image, which can reduce the computational complexity of the RFPA-PIV model while maintaining the performance of the RFPA-PIV model; the decoder iteratively updates the set initial optical flow according to the correlation pyramid established by the 4D correlation body establishment module and the resolution feature map extracted by the feature encoder, and generates an accurate velocity field in an iterative manner, thereby reducing the number of parameters and the video memory consumption of the RFPA-PIV model during training and inference, and improving the computational efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0062] Figure 1 It is a flow chart of a high-resolution particle image velocimetry method based on a lightweight model in the present invention;

[0063] Figure 2 It is a structural schematic diagram of the RFPA-PIV model in an embodiment of the present invention;

[0064] Figure 3 Schematic diagram of the structure of the InceptionNeXt module according to an embodiment of the present invention;

[0065] Figure 4 A velocity field comparison diagram drawn by the present method and other existing methods based on a common particle data set in an embodiment of the present invention;

[0066] Figure 5 A velocity field comparison diagram drawn by the present method and other existing methods based on a real-world standard flow field in an embodiment of the present invention;

[0067] Figure 6 This is a velocity field comparison diagram drawn by using this method and other existing methods based on the real-world eddy flow field in an embodiment of the present invention. DETAILED DESCRIPTION

[0068] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0069] This embodiment provides a high-resolution particle image velocimetry method based on a lightweight model, such as Figure 1 As shown, the specific steps include:

[0070] S1: Obtain particle image velocimetry simulation data set;

[0071] S2: Build Figure 2 The recursive feature aggregation pyramid particle image velocimetry RFPA-PIV (Recurrent Feature Pyramid Aggregation) model shown in the figure is trained based on a particle image velocimetry simulation data set, wherein the RFPA-PIV model includes a feature encoder, a 4D correlation body building module, a decoder and an upsampling module;

[0072] The feature encoder is used to extract the resolution feature maps of two consecutive frames of original particle images in the particle image velocimetry simulation data set, and transmit them to the 4D correlation volume establishment module and the decoder module;

[0073] Specifically, in this embodiment, the RFPA-PIV model can only process two consecutive frames of images at a time, and multiple images need to be processed multiple times.

[0074] In this embodiment, two consecutive frames of original particle images I are extracted by the feature encoder. 1 and I 2 The feature map is 1 / 8 resolution.

[0075] The 4D correlation volume building module constructs a 4D correlation volume based on the two input resolution feature maps, and performs a pooling operation on the constructed 4D correlation volume to obtain a correlation pyramid;

[0076] The decoder is used to iteratively update the set initial optical flow according to the relevant pyramid and the resolution feature map to obtain a predicted optical flow image;

[0077] The upsampling module is used to upsample the predicted optical flow image, thereby restoring the predicted optical flow image to full resolution, and obtaining a velocity field based on the optical flow image restored to full resolution;

[0078] Specifically, in this embodiment, the decoder outputs a 1 / 8 resolution optical flow after iterative updating, and the 1 / 8 resolution optical flow is upsampled 8 times through the upsampling module to restore it to the full resolution, and the corresponding pixel points in the optical flow image restored to the full resolution are replaced with the set color according to the numerical value to obtain the full resolution velocity field.

[0079] S3: During the training of the RFPA-PIV model, a loss function is set to optimize the RFPA-PIV model, and finally an optimized RFPA-PIV model is obtained for predicting the velocity field from the particle image.

[0080] In a specific embodiment, the feature encoder includes a first downsampling module, a first InceptionNeXt module, a second downsampling module, a second InceptionNeXt module, a third downsampling module, a third InceptionNeXt module, a feature pyramid module, and a multi-scale feature aggregation module;

[0081] The first downsampling module is used to downsample the input image to obtain a downsampled image, and transmit it to the first InceptionNeXt module. The first InceptionNeXt module is used to extract a resolution feature map of the downsampled image, and then splice the extracted resolution feature map with the downsampled image to obtain a first resolution feature map, and transmit it to the second downsampling module and the feature pyramid module;

[0082] In this embodiment, the first resolution feature map is Resolution feature map.

[0083] The second downsampling module is used to downsample the first resolution feature map to obtain a first downsampling feature map, and transmit it to the second InceptionNeXt module. The second InceptionNeXt module is used to extract a resolution feature map of the first downsampling feature map, and then splice the extracted resolution feature map with the first downsampling feature map to obtain a second resolution feature map, and transmit it to the third downsampling module and the feature pyramid module;

[0084] In this embodiment, the second resolution feature map is Resolution feature map.

[0085] The third down-sampling module is used to down-sample the second resolution feature map to obtain a second down-sampling feature map, and transmit it to the third InceptionNeXt module. The third InceptionNeXt module is used to extract a resolution feature map of the second down-sampling feature map, and then splice the extracted resolution feature map with the second down-sampling feature map to obtain a third resolution feature map, and transmit it to the feature pyramid module;

[0086] In this embodiment, the third resolution feature map is Resolution feature map.

[0087] The feature pyramid module establishes a feature pyramid based on the first resolution feature map, the second resolution feature map and the third resolution feature map, the first resolution feature map is the first layer of the feature pyramid, the second resolution feature map is the second layer of the feature pyramid, and the third resolution feature map is the third layer of the feature pyramid, and transmits it to the multi-scale feature aggregation module;

[0088] The multi-scale feature aggregation module Multi-scale Feature Aggregation (MFA) performs feature aggregation on the feature pyramid to obtain an integrated feature map, and transmits it to the 4D correlation volume establishment module.

[0089] Specifically, by aggregating multi-scale features, the RFPA-PIV model can be made more robust to the size variation of the input image.

[0090] In a specific embodiment, Figure 3 As shown, each of the InceptionNeXt modules includes: a depth-separable convolutional layer, a normalization layer (Layer Normalization), a multi-scale feature extraction layer, a GELU (Gaussian Error Linear Unit) activation function layer and a first convolutional layer;

[0091] The depthwise separable convolution layer is used to perform depthwise convolution and pointwise convolution operations on the input image in sequence to obtain a preliminary extracted feature map and transmit it to the normalization layer;

[0092] The normalization layer is used to perform a normalization operation on the initially extracted feature map to obtain a normalized feature map, and transmit it to the multi-scale feature extraction layer;

[0093] The multi-scale feature extraction layer includes a second convolution layer, a third convolution layer and a fourth convolution layer, wherein the second convolution layer, the third convolution layer and the fourth convolution layer are respectively used to perform feature extraction on the normalized feature map to obtain three feature maps of different scales, and the three feature maps of different scales are spliced ​​in the channel dimension according to their respective channel numbers to obtain a spliced ​​feature map, and transmitted to the GELU activation function layer;

[0094] Specifically, in the feature encoder module, the second convolution layer, the third convolution layer and the fourth convolution layer in the multi-scale feature extraction layer are respectively a 1×1 convolution layer, a 3×3 convolution layer and a 5×5 convolution layer.

[0095] The GELU activation function layer is used to perform nonlinear activation on the concatenated feature map to increase the nonlinear expression capability, obtain the activated feature map, and transmit it to the first convolution layer;

[0096] The first convolutional layer is used to reduce the dimension of the activated feature map and integrate channel information to obtain a reduced-dimensional feature map.

[0097] Specifically, in the feature encoder module, the first convolution layer is a 3×3 convolution layer.

[0098] In a specific embodiment, the calculation formula of the normalization layer is:

[0099]

[0100]

[0101]

[0102]

[0103] In the formula, x n,c,h,ω is the activation value of the neurons in this layer; is the normalized activation value of the neurons in this layer; n,c,h,ω is the normalized activation value after scaling and offset operations; n and c represent the sample batch size and the number of channels, respectively; h and ω represent the height and width, respectively; μ n,h,ω is the mean of all neuron activation values ​​in this layer; ε is a constant used to ensure that the denominator is not zero. is the variance of the activation values ​​of all neurons in this layer; c is the scaling factor used to control the scale after normalization, β c is the offset coefficient, which is used to control the normalized displacement, and γ c With β c are all learnable parameters.

[0104] In a specific embodiment, the 4D correlation volume building module constructs a 4D correlation volume based on two resolution feature maps, and performs a pooling operation on the constructed 4D correlation volume to obtain a correlation pyramid in the following process:

[0105] The dot product operation is performed on the feature vectors between the corresponding pixel pairs of the two frame resolution feature maps after feature aggregation to construct a 4D correlation volume. The formula for the dot product operation is:

[0106]

[0107] Among them, g θ (I 1 ) and g θ (I 2 ) are the original particle images I 1 ,I 2 The resolution feature map of g θ (I 1 )∈R H *W*D , g θ (I 2 )∈R H*W*D ; i and j are the width and height of the first frame particle image, k and l are the width and height of the second frame particle image, and p is the number of channels of the resolution feature map;

[0108] Specifically, the 4D correlation body is established by the dot product operation of two feature maps: the dot product operation of two feature maps with a resolution of H*W is performed to form a 4D correlation body of H*W*H*W, the high-resolution features of the first two dimensions of the constructed 4D correlation body are retained, and the last two dimensions of the 4D correlation body are average pooled according to different kernel sizes. Specifically, the kernel size of the pooling layer is 1, 2, 4 and 8, and the first tensor, the second tensor, the third tensor and the fourth tensor are obtained. A four-layer feature pyramid is established based on the first tensor, the second tensor, the third tensor and the fourth tensor. The tensor dimension calculation formula of the kth layer of the feature pyramid is:

[0109] C k =H*W*H / 2 k *W / 2 k (6)

[0110] In the formula, H represents the height of the feature map, W represents the width of the feature map, k represents the kth layer of the feature pyramid, and C k Represents the dimension of the correlation volume at the kth level of the feature pyramid.

[0111] Specifically, each layer in the correlation pyramid is obtained by performing pooling operations of different scales on the last two dimensions of the original 4D correlation volume. Since the first two dimensions of the 4D correlation volume remain unchanged, high-resolution information can be retained; after the last two dimensions are pooled, small displacement motion can be tracked. Therefore, the correlation pyramid can represent visual similarities of different scales from high resolution to low resolution, which helps to estimate the optical flow field more accurately in subsequent iterative updates.

[0112] In a specific embodiment, in S4, the decoder includes N decoding modules, each decoding module includes a fourth InceptionNeXt module and a decoding convolution layer; the decoder iteratively updates the set initial optical flow according to the relevant pyramid and the resolution feature map, and the process of obtaining the predicted optical flow image is:

[0113] S41: Obtain a query feature map in the relevant pyramid based on a bilinear sampling bilinear_sampler function;

[0114] S42: splicing the query feature map and the second frame resolution feature map extracted by the feature encoder, and transmitting the splicing to the first decoding module, the first decoding module processes the input data, outputs a first residual optical flow, and adds the first residual optical flow to the set initial optical flow to obtain an updated optical flow;

[0115] Specifically, the initial optical flow is set to 0.

[0116] S43: extracting a feature map of the input data through the second InceptionNeXt module in the first decoding module;

[0117] S44: updating the obtained query feature graph;

[0118] S45: splicing the feature map extracted in S43 and the updated query feature map, and transmitting them to the second decoding module, the second decoding module processes the input data, outputs a second residual optical flow, and adds the second residual optical flow to the optical flow output by the first decoding module to obtain an updated optical flow;

[0119] S46: Switch to the next decoding module, and update the data input to the next decoding module according to the processing flow corresponding to S43 and S44, output the updated optical flow through the next decoding module, iterate N-2 times according to S43-S45 based on N-2 decoding modules, and obtain the optical flow output by the last decoding module, which is the predicted optical flow image.

[0120] Specifically, based on the updated feature map and residual optical flow Δf output by the decoding module after each iteration, the residual optical flow is applied to the current optical flow f iTo update the optical flow, the update formula is:

[0121] f i+1 =f i +Δf

[0122] Therefore, the decoder generates a series of estimated optical flows {f 1 ,…,f N}.

[0123] Specifically, in this embodiment, the RFPA-PIV model uses the InceptionNeXt module as an optical flow updater to generate an accurate velocity field in an iterative manner. Due to the reduction in the number of parameters, the InceptionNeXt module helps to reduce the video memory consumption of the model during training and inference, and improve computing efficiency.

[0124] In a specific embodiment, in S41, the process of obtaining the query feature map in the relevant pyramid based on the bilinear sampling bilinear_sampler function is:

[0125] S411: Generate row coordinates and column coordinates based on the height and width of the second frame resolution feature map extracted by the feature encoder, respectively, and concatenate the row coordinates and the column coordinates to obtain a first grid coordinate system for representing coordinate information of each pixel point in the second frame resolution feature map;

[0126] S412: Divide the first grid coordinate system by the scaling factor 2 of each level of the relevant pyramid i , get the first grid coordinate system after resizing, i is the number of layers of the current level in the relevant pyramid, to adapt to the feature map size of the current level;

[0127] S413: setting the query radius in the correlation pyramid to r. In this embodiment, r is set to 4. The query radius r defines the size of the area for searching for matching features in another frame image for each pixel when extracting and calculating the correlation feature map.

[0128] S414: Use torch.linspace function to generate displacement vectors dx and dy according to the query radius r. The value range of dx and dy is (-r, r). dx and dy represent the horizontal and vertical offsets in the relevant pyramid, respectively. Therefore, the values ​​from -r to r are considered, with a total of 2×r+1 pixels.

[0129] S415: concatenate the displacement vectors dx and dy through the torch.meshgrid function to obtain a second grid coordinate system, where the second grid coordinates represent the offset when the bilinear_sampler function samples around each coordinate point;

[0130] S416: Add the resized first grid coordinate system to the second grid coordinate system to obtain a third grid coordinate system, where the third grid coordinate system indicates that sampling will be performed in an area of ​​(2×r+1)×(2×r+1) around each sampling coordinate point at each level.

[0131] S417: Use the bilinear_sampler function to sample each layer of the relevant pyramid according to the sampling points determined by the third grid coordinate system.

[0132] S418: splicing the sampling results of all levels to obtain a query feature map, which represents how the feature vector in the first frame matches the feature vector in the second frame at each position. This feature map is used to estimate the motion of each point in the image;

[0133] In S44, the process of updating the query feature graph is as follows:

[0134] The query feature map obtained in S418 is spliced ​​with the second frame resolution feature map extracted by the feature encoder, and the first grid coordinate system is updated based on the height and width of the spliced ​​feature map. Similarly, each time the output query feature map is spliced ​​with the feature map last used to update the first grid coordinate system, thereby updating the first grid coordinate system.

[0135] In a specific embodiment, the loss function is L 1 Distance loss function, the formula is:

[0136]

[0137] In the formula, f i To predict the flow field, f gt is true value.

[0138] In order to verify the effectiveness of the RFPA-PIV model proposed in this embodiment, the RFPA-PIV model is used to estimate the velocity field of the public PIV dataset, the real standard flow field, and the real eddy flow field, as shown in the following figure: Figure 4 As shown in the figure, the velocity field estimation results of the public PIV dataset using the RFPA-PIV model and the real ground, HS model, WIDIM model and PIV-RAFT model are compared. It can be seen from the figure that the velocity field image ( Figure 4 (e)) and the real ground velocity field image ( Figure 4 (a)) is the closest, and the average Euclidean distance error (AEE) is the lowest among several square models, which is 4.7. Figure 5As shown in the figure, the velocity field estimation results of the real standard flow field using the RFPA-PIV model, the WIDIM model, the HS model, the PIV-DCNN model, the PIV-LightFlowNet-en model and the PIV-RAFT model are compared. It can be seen from the figure that the RFPA-PIV model can capture the movement of particles more accurately. Figure 5 (d) Figure 5 (e) Figure 5 In the red box of (f), the velocity field image obtained by using the RFPA-PIV model ( Figure 5 (f)) is the clearest and most accurate. Figure 6 As shown in the figure, the velocity field estimation results of the real eddy flow field using the RFPA-PIV model, the HS model, the PIV-LightFlowNet-en model and the PIV-RAFT model are compared. It can be seen from the figure that Figure 6 (b) Figure 6 (c) Figure 6 In the white box of (d), the velocity field image obtained by using the RFPA-PIV model ( Figure 6 (d)) shows a more regular particle motion, which is more consistent with the motion of particles in a real eddy current field.

[0139] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A high-resolution particle image velocimetry method based on a lightweight model, characterized in that: The specific steps include: S1: Obtain particle image velocimetry simulation data set; S2: constructing an RFPA-PIV model and training the RFPA-PIV model based on a particle image velocimetry simulation data set, wherein the RFPA-PIV model includes a feature encoder, a 4D correlation body building module, a decoder, and an upsampling module; The feature encoder is used to extract the resolution feature maps of two consecutive frames of original particle images in the particle image velocimetry simulation data set, and transmit them to the 4D correlation volume establishment module and the decoder module; The 4D correlation volume building module constructs a 4D correlation volume based on the two input resolution feature maps, and performs a pooling operation on the constructed 4D correlation volume to obtain a correlation pyramid; The decoder is used to iteratively update the set initial optical flow according to the relevant pyramid and the resolution feature map to obtain a predicted optical flow image; The upsampling module is used to upsample the predicted optical flow image, thereby restoring the predicted optical flow image to full resolution, and obtaining a velocity field based on the optical flow image restored to full resolution; S3: During the training of the RFPA-PIV model, a loss function is set to optimize the RFPA-PIV model, and finally an optimized RFPA-PIV model is obtained for predicting the velocity field from the particle image.

2. The high-resolution particle image velocimetry method based on a lightweight model according to claim 1, characterized in that: The feature encoder includes a first downsampling module, a first InceptionNeXt module, a second downsampling module, a second InceptionNeXt module, a third downsampling module, a third InceptionNeXt module, a feature pyramid module and a multi-scale feature aggregation module; The first downsampling module is used to downsample the input image to obtain a downsampled image, and transmit it to the first InceptionNeXt module. The first InceptionNeXt module is used to extract a resolution feature map of the downsampled image, and then splice the extracted resolution feature map with the downsampled image to obtain a first resolution feature map, and transmit it to the second downsampling module and the feature pyramid module; The second downsampling module is used to downsample the first resolution feature map to obtain a first downsampling feature map, and transmit it to the second InceptionNeXt module. The second InceptionNeXt module is used to extract a resolution feature map of the first downsampling feature map, and then splice the extracted resolution feature map with the first downsampling feature map to obtain a second resolution feature map, and transmit it to the third downsampling module and the feature pyramid module; The third down-sampling module is used to down-sample the second resolution feature map to obtain a second down-sampling feature map, and transmit it to the third InceptionNeXt module. The third InceptionNeXt module is used to extract a resolution feature map of the second down-sampling feature map, and then splice the extracted resolution feature map with the second down-sampling feature map to obtain a third resolution feature map, and transmit it to the feature pyramid module; The feature pyramid module establishes a feature pyramid based on the first resolution feature map, the second resolution feature map and the third resolution feature map, and transmits the pyramid to the multi-scale feature aggregation module; The multi-scale feature aggregation module performs feature aggregation on the feature pyramid to obtain an integrated feature map, and transmits the integrated feature map to the 4D correlation volume establishment module.

3. The high-resolution particle image velocimetry method based on a lightweight model according to claim 2, characterized in that: Each of the InceptionNeXt modules includes: a depth-separable convolutional layer, a normalization layer, a multi-scale feature extraction layer, a GELU activation function layer and a first convolutional layer; The depthwise separable convolution layer is used to perform depthwise convolution and pointwise convolution operations on the input image in sequence to obtain a preliminarily extracted feature map, and transmit it to the normalization layer; The normalization layer is used to perform a normalization operation on the initially extracted feature map to obtain a normalized feature map, and transmit it to the multi-scale feature extraction layer; The multi-scale feature extraction layer includes a second convolution layer, a third convolution layer and a fourth convolution layer, wherein the second convolution layer, the third convolution layer and the fourth convolution layer are respectively used to perform feature extraction on the normalized feature map to obtain three feature maps of different scales, and the three feature maps of different scales are spliced ​​in the channel dimension according to their respective channel numbers to obtain a spliced ​​feature map, and transmitted to the GELU activation function layer; The GELU activation function layer is used to perform nonlinear activation on the concatenated feature map to obtain the activated feature map, and transmit it to the first convolutional layer; The first convolutional layer is used to reduce the dimension of the activated feature map to obtain a reduced-dimensional feature map.

4. The high-resolution particle image velocimetry method based on a lightweight model according to claim 3, characterized in that: The calculation formula of the normalization layer is: In the formula, x n,c,h,ω is the activation value of the neurons in this layer; is the normalized activation value of the neurons in this layer; n,c,h,ω is the normalized activation value after scaling and offset operations; n and c represent the sample batch size and the number of channels, respectively; h and ω represent the height and width, respectively; μ n,h,ω is the mean of all neuron activation values ​​in this layer; ε is a constant used to ensure that the denominator is not zero. is the variance of the activation values ​​of all neurons in this layer; c is the scaling factor used to control the scale after normalization, β c is the offset coefficient, which is used to control the normalized displacement.

5. The high-resolution particle image velocimetry method based on a lightweight model according to claim 1, characterized in that: The 4D correlation volume building module constructs a 4D correlation volume based on two resolution feature maps, and performs a pooling operation on the constructed 4D correlation volume to obtain a correlation pyramid process as follows: The dot product operation is performed on the feature vectors between the corresponding pixel pairs of the two frame resolution feature maps after feature aggregation to construct a 4D correlation volume. The formula for the dot product operation is: Among them, g θ (I1) and g θ (I2) are the resolution feature maps of the original particle images I1 and I2, respectively, and g θ (I1)∈R H*W*D , g θ (I2)∈R H*W*D ; i and j are the width and height of the first frame particle image, k and l are the width and height of the second frame particle image, and p is the number of channels of the resolution feature map; The high-resolution features of the first two dimensions of the constructed 4D correlation body are retained, and the average pooling operation is performed on the last two dimensions of the 4D correlation body according to different kernel sizes to obtain the first tensor, the second tensor, the third tensor and the fourth tensor, and a four-layer feature pyramid is established based on the first tensor, the second tensor, the third tensor and the fourth tensor. The tensor dimension calculation formula of the kth layer of the feature pyramid is: C k =H*W*H / 2 k *W / 2 k (6) In the formula, H represents the height of the feature map, W represents the width of the feature map, k represents the kth layer of the feature pyramid, and C k Represents the dimension of the correlation volume at the kth level of the feature pyramid.

6. The high-resolution particle image velocimetry method based on a lightweight model according to claim 2, characterized in that: The decoder includes N decoding modules, each of which includes a fourth InceptionNeXt module and a decoding convolution layer; the decoder iteratively updates the set initial optical flow according to the relevant pyramid and the resolution feature map, and the process of obtaining the predicted optical flow image is: S41: Obtain a query feature map in the relevant pyramid based on a bilinear sampling bilinear_sampler function; S42: splicing the query feature map and the second frame resolution feature map extracted by the feature encoder, and transmitting the splicing to the first decoding module, the first decoding module processes the input data, outputs a first residual optical flow, and adds the first residual optical flow to the set initial optical flow to obtain an updated optical flow; S43: extracting a feature map of the input data through the second InceptionNeXt module in the first decoding module; S44: updating the obtained query feature graph; S45: splicing the feature map extracted in S43 and the updated query feature map, and transmitting them to the second decoding module, the second decoding module processes the input data, outputs a second residual optical flow, and adds the second residual optical flow to the optical flow output by the first decoding module to obtain an updated optical flow; S46: Switch to the next decoding module, and update the data input to the next decoding module according to the processing flow corresponding to S43 and S44, output the updated optical flow through the next decoding module, iterate N-2 times, and obtain the optical flow output by the last decoding module, which is the predicted optical flow image.

7. The high-resolution particle image velocimetry method based on a lightweight model according to claim 2, characterized in that: In S41, the process of obtaining the query feature map in the relevant pyramid based on the bilinear sampling bilinear_sampler function is: S411: Generate row coordinates and column coordinates based on the height and width of the second frame resolution feature map extracted by the feature encoder, respectively, and concatenate the row coordinates and the column coordinates to obtain a first grid coordinate system; S412: Divide the first grid coordinate system by the scaling factor 2 of each level of the relevant pyramid i , get the first grid coordinate system after resizing, i is the number of the current level in the relevant pyramid; S413: Setting the query radius in the relevant pyramid to r; S414: Use the torch.linspace function to generate displacement vectors dx and dy according to the query radius r, and the value range of dx and dy is (-r, r); S415: Concatenate the displacement vectors dx and dy through the torch.meshgrid function to obtain a second grid coordinate system; S416: Add the resized first grid coordinate system to the second grid coordinate system to obtain a third grid coordinate system; S417: using the bilinear_sampler function, sampling each layer of the relevant pyramid according to the sampling points determined by the third grid coordinate system; S418: concatenating the sampling results of all levels to obtain a query feature map; In S44, the process of updating the query feature graph is as follows: The query feature map obtained in S418 is spliced ​​with the second frame resolution feature map extracted by the feature encoder, and the first grid coordinate system is updated based on the height and width of the spliced ​​feature map. Similarly, each time the output query feature map is spliced ​​with the feature map last used to update the first grid coordinate system, thereby updating the first grid coordinate system.

8. The high-resolution particle image velocimetry method based on a lightweight model according to claim 1, characterized in that: The loss function is the L1 distance loss function, and the formula is: In the formula, f i To predict the flow field, f gt is true value.

Images