A multi-scale pyramid optical flow estimation method for PIV

Abstract

The application provides a PIV-oriented multi-scale pyramid optical flow estimation method, comprising: performing adaptive image preprocessing on an image pair; inputting the processed image into a lightweight feature extraction network to obtain multi-scale feature maps and context feature maps of two frames of images; using a traditional dense optical flow algorithm to calculate an initial optical flow and down-sample the initial optical flow to the deepest layer resolution; using the initial optical flow to locally sample the second frame feature map, and calculating a dot product of the first frame feature map and the second frame feature map to obtain a correlation body; fusing the correlation body and the optical flow information through a multi-scale motion encoder to regress a residual optical flow; performing weighted up-sampling on the residual optical flow based on neighboring pixels; iteratively performing the above steps layer by layer until a full-resolution optical flow estimation is output. Through the fusion of a traditional optical flow prior and a multi-scale pyramid iterative network, the application solves the problem of low optical flow estimation accuracy caused by large changes in PIV environment illumination, similar foreground and background, and transparent objects, reduces the calculation complexity, and meets the real-time requirement.

Description

Technical Field

[0001] This invention relates to the field of signal processing technology, and more particularly to a multi-scale pyramid optical flow estimation method for PIV. Background Technology

[0002] Optical flow estimation is a fundamental problem in computer vision, aiming to calculate the motion vector of each pixel between two consecutive frames. In a PIV (Particle Image Velocimetry) environment, the background is simple, but there are problems such as complex particle distribution, uneven lighting, and transparent bubbles causing high similarity between the target and the background, making it difficult to distinguish them.

[0003] Traditional optical flow methods are based on variational optimization (such as Horn-Schunck and Lucas-Kanade) or matching cost aggregation. Some traditional methods have high computational complexity and poor robustness to occlusion and large displacements. Methods with good real-time performance are prone to block artifacts, resulting in low accuracy, and are also sensitive to illumination. With the development of deep learning, optical flow estimation methods based on convolutional neural networks (CNNs) have achieved significant breakthroughs. Since FlowNet first introduced the end-to-end learning approach to optical flow estimation, FlowNet 2.0, PWC-Net, and RAFT networks have undergone multiple iterations, achieving state-of-the-art performance on various datasets.

[0004] However, these methods are not applicable to environments like PIV where there are large changes in lighting, simple backgrounds, and a lot of noise and occlusion. They still have the following shortcomings: related calculations are often limited to a single scale or a fixed neighborhood, making it difficult to capture large and small displacements simultaneously; single-resolution iterative estimation of optical flow is prone to bias in scenes with occlusion and changes in lighting; the use of combined features from two frames of images in multiple scale directions to obtain motion information to guide optical flow estimation; the iterative optimization process has high computational overhead and the number of iterations limits the real-time performance of the network, making it difficult to balance accuracy and speed.

[0005] Furthermore, in the PIV environment, particles are distributed three-dimensionally, and there is occlusion between particles, which affects the accuracy of optical flow calculation. Due to light scattering, particles outside the laser sheet are also more obvious, interfering with optical flow calculation. Bubbles are transparent, which makes bubble features difficult to distinguish from background features. Ignoring the guidance provided by traditional methods in deep learning-based optical flow estimation greatly affects the real-time performance of the network. Summary of the Invention

[0006] To address the technical problems of existing technologies, such as large illumination variations, similar foreground and background, and low optical flow estimation accuracy due to the presence of transparent objects in PIV environments, as well as the high computational complexity and poor real-time performance of traditional deep learning optical flow methods, this invention provides a multi-scale pyramid optical flow estimation method for PIV. This invention primarily achieves high-precision real-time estimation of complex flow fields through multi-scale image preprocessing, improved correlation volume calculation, and a multi-level feature fusion network that integrates traditional optical flow constraints with deep learning.

[0007] The technical means employed in this invention are as follows:

[0008] A multi-scale pyramid optical flow estimation method for PIV includes: S1. Obtain the original particle image pair and perform adaptive image preprocessing on the original particle image pair to obtain the preprocessed first frame image and second frame image, so as to suppress random illumination and enhance target features. S2. Input the preprocessed first frame image and second frame image into a lightweight feature extraction network for multi-scale feature extraction, and extract multi-scale first frame feature map and multi-scale second frame feature map respectively, and extract the context feature map of the corresponding level of the first frame image. S3. Calculate the initial optical flow for the preprocessed first frame image and second frame image using the traditional dense optical flow algorithm, and downsample the initial optical flow to the deepest layer resolution; S4. Locally sample the second frame feature map of the current layer using the initial optical flow of the current layer resolution, and perform a dot product operation between the sampled features and the first frame feature map of the current layer to obtain the current layer correlation volume. S5. Using a multi-scale motion encoder, the current layer correlation volume and the initial optical flow information of the current layer resolution are fused together, and the residual optical flow is regressed by combining the context feature map of the current layer to obtain the residual optical flow of the current layer. S6. Perform weighted upsampling on the residual optical flow of the current layer based on neighboring pixels to obtain the upsampled optical flow of the current layer; S7. Determine whether the optical flow resolution after upsampling of the current layer reaches the original resolution: If yes, use the optical flow after upsampling of the current layer as the final optical flow output; if no, use the optical flow after upsampling of the current layer as the initial optical flow of the next layer, return to step S4, use the initial optical flow of the next layer to locally sample the multi-scale second frame feature map of the next layer, and calculate the correlation volume of the next layer. Repeat steps S5 to S7 until the full resolution optical flow estimate is output.

[0009] Further, step S1 includes: S11. Use a multi-scale inverse adaptive contrast enhancement algorithm to enhance the image; S12. Enhance the image using a Gaussian-Laplacian pyramid, and use brightness region adaptive suppression and multi-scale frequency enhancement within the Gaussian-Laplacian pyramid.

[0010] Further, step S11 includes: S111. The original particle image pair is processed using two different scales of inverse adaptive contrast enhancement algorithms. The first scale is used to enhance local details and textures to enhance small target features, and the second scale is used to improve overall contrast and suppress background features, resulting in the first frame image and the second frame image after contrast enhancement, wherein the first scale is smaller than the second scale. S112. A multi-scale inverse adaptive contrast enhancement algorithm is used to invert the pixel values ​​of the first frame image and the second frame image. CLAHE processing is performed on the inverted image to reduce the contrast of the dark areas of the original image. The processed image is then inverted back, thereby suppressing details in the dark and bright areas.

[0011] Further, step S12 includes: S121. Using Gaussian-Laplacian pyramid decomposition combined with multi-scale gain adjustment, the image is decomposed into multiple Gaussian pyramids and then converted into Laplacian pyramids. The gain coefficients are calculated according to adaptive weights, and weighted enhancement is applied to different levels of the Laplacian pyramid to achieve targeted enhancement of different frequencies. S122. Dynamically adjust the weights based on the brightness characteristics of different regions of the image, and dynamically adjust the brightness threshold based on the maximum brightness of the image.

[0012] Further, step S122 includes: S1221. Calculate the maximum brightness of the image using the following formula:

[0013] in, Indicates the image height. Indicates the image width. Indicates the image at position The pixel brightness value at that location; S1222, Based on the maximum brightness of the image Dynamically adjust the brightness threshold, taking the maximum brightness of the image. The high brightness threshold is the highest brightness value of the image. Threshold for dark areas; S1223. For bright areas, the weights are dynamically adjusted based on the average brightness. The weight calculation formula is as follows:

[0014] in, This indicates that the brightness value is greater than or equal to the high-brightness threshold. The average brightness of all pixels, This indicates the total number of pixels in the highlighted area. This represents the weighting coefficient of the highlighted area; S1224. For dark areas, the weights are dynamically adjusted based on the average brightness. The weight calculation formula is as follows:

[0015] in, This indicates that the brightness value is less than or equal to the dark area threshold. The average brightness of all pixels, Indicates the weighting coefficients for dark areas; S1225. For areas with normal brightness, the weights are dynamically adjusted based on the average brightness. The weight calculation formula is as follows:

[0016] in, Indicates that the brightness value is at The average brightness of all pixels in the range The weighting coefficient represents the normal brightness area.

[0017] S1226. Based on the weights of the bright area, dark area and normal brightness area, and combined with the average brightness, dynamically adjust the gain of each area, and reconstruct the Laplacian pyramid after weighted enhancement of each level to obtain the preprocessed first frame image and second frame image.

[0018] Further, step S2 includes: S21. Construct a lightweight feature extraction network with an encoder-decoder structure. The encoder part contains multiple downsampling layers. Each downsampling layer reduces spatial resolution and increases the number of feature channels through convolution and pooling operations. The decoder part is used for subsequent optical flow prediction. The lightweight feature extraction network is used to reduce the number of parameters and computation while maintaining feature representation capabilities, so that the inference speed can reach more than 30fps. S22. Input the preprocessed first frame image and second frame image into the lightweight feature extraction network respectively. Through multi-layer convolution and downsampling operations, extract feature information of different scales layer by layer to obtain multi-scale first frame feature map fmap1 and multi-scale second frame feature map fmap2 of each level. S23. Perform multi-scale feature extraction on the preprocessed first frame image to obtain a context feature map cmap at the same level as the first frame feature map fmap1. The context feature map is used to provide spatial context information in the optical flow prediction stage to help constrain the geometric consistency and spatial smoothness of optical flow estimation.

[0019] Further, step S3 includes: S31. The traditional dense optical flow algorithm is used to perform dense optical flow calculation on the preprocessed first frame image and second frame image. The displacement vector field is calculated by polynomial expansion to approximate the image neighborhood and obtain the full-resolution initial optical flow. S32. Determine the target downsampling resolution based on the resolution of the deepest feature map; S33. The full-resolution initial optical flow is downsampled from the original resolution to the deepest layer resolution to obtain an initial optical flow that matches the spatial resolution of the deepest feature map of the pyramid.

[0020] Further, step S4 includes: S41. Sample the feature map of the second frame using the initial optical flow, for the deepest layer position. The corresponding initial optical flow at the location The second frame feature map fmap2 is obtained using quadratic interpolation. Centered on, with a scope of of Region feature vector; S42, the above The region feature vector and the first frame feature map fmap1 are in Perform a dot product operation on the eigenvectors at point , and obtain The correlation matrix of size ultimately forms a matrix of size. The deepest related matrix.

[0021] Further, step S5 includes: S51. Different motion encoders are used for the relevant volume matrix and context features at different scales. The first motion encoder is used for the relevant volume and context features at lower resolution levels, and the second motion encoder is used for the relevant volume and context features at higher resolution levels. S52. Using the selected motion encoder, the current layer related volume, the initial optical flow information of the current layer resolution, and the current layer context feature map cmap are fused and encoded. The encoded features are then input into the optical flow prediction head for residual optical flow regression to obtain the current layer residual optical flow.

[0022] Further, step S6 includes: S61. For residual optical flow at different scales, different neighborhood pixel weighted upsampling modules are used. For residual optical flow at lower resolution levels, the first neighborhood pixel weighted upsampling module is used, and for residual optical flow at higher resolution levels, the second neighborhood pixel weighted upsampling module is used. S62. Use the selected neighboring pixel weighted upsampling module to upsample the current layer residual optical flow to the next layer resolution to obtain the current layer upsampled optical flow.

[0023] Furthermore, the method is applicable to complex fluid scenarios such as PIV gas-liquid two-phase flow, particle image velocimetry, and digital image correlation, and is used for preprocessing illumination, distinguishing foreground from background, and calculating displacement fields or strains containing transparent objects with high real-time requirements.

[0024] This invention also provides a multi-scale pyramid optical flow estimation system based on the above-mentioned multi-scale pyramid optical flow estimation method, comprising: The image acquisition and preprocessing module is used to acquire the original particle image pairs and perform adaptive image preprocessing on the original particle image pairs to obtain the preprocessed first frame image and second frame image, so as to suppress random illumination and enhance target features. The multi-scale feature extraction module is used to input the preprocessed first frame image and second frame image into the lightweight feature extraction network for multi-scale feature extraction, and extract multi-scale first frame feature map and multi-scale second frame feature map respectively, and extract the context feature map of the corresponding level of the first frame image. The initial optical flow calculation module is used to calculate the initial optical flow of the preprocessed first frame image and the second frame image using a traditional dense optical flow algorithm, and to downsample the initial optical flow to the current layer resolution; The correlation volume calculation module is used to locally sample the multi-scale second frame feature map of the current layer using the initial optical flow of the current layer resolution, and perform a dot product operation between the sampled features and the multi-scale first frame feature map of the current layer to obtain the correlation volume of the current layer. The residual optical flow regression module is used to fuse the current layer correlation volume with the initial optical flow information of the current layer resolution using a multi-scale motion encoder, and perform residual optical flow regression in combination with the context feature map of the current layer to obtain the residual optical flow of the current layer. The neighborhood pixel weighted upsampling module is used to perform weighted upsampling of the current layer residual optical flow based on neighborhood pixels to obtain the current layer upsampled optical flow; The iterative update module is used to determine whether the optical flow resolution after upsampling of the current layer reaches the original resolution: if yes, the optical flow after upsampling of the current layer is used as the final optical flow output; if no, the optical flow after upsampling of the current layer is used as the initial optical flow of the next layer, and the module returns to the correlation volume calculation module. The module uses the initial optical flow of the next layer to locally sample the multi-scale second frame feature map of the next layer, and calculates the correlation volume of the next layer. The residual optical flow regression module, the neighborhood pixel weighted upsampling module, and the iterative update module are repeatedly executed until the full-resolution optical flow estimate is output.

[0025] Compared with the prior art, the present invention has the following advantages: 1. The multi-scale pyramid optical flow estimation method for PIV provided by this invention suppresses the influence of highlights on the image through inverse adaptive image preprocessing and reduces the visibility of dark areas, and has high robustness to changes in overall brightness and local brightness.

[0026] 2. This invention enhances the texture and edge information of the target through Gaussian pyramids, distinguishing foreground target information from background information, and has a particularly good enhancement effect on transparent objects.

[0027] 3. The improved network structure of this invention has strong robustness to occlusion scenarios due to its pyramid structure. The modified network can meet the real-time requirements of engineering applications and achieve an inference speed of 30fps for a single 4090 card.

[0028] 4. In PIV environments where optical flow changes drastically and adjacent particles exhibit significant differences in velocity magnitude and direction, this invention allows for the estimation of richer flow field details and more accurate optical flow estimation through multiple upsampling compared to single upsampling.

[0029] In summary, the technical solution of this invention improves particle recognition and optical flow calculation accuracy in PIV environments through targeted improvements in image preprocessing and neural networks. Modifications to the correlation volume calculation method ensure the algorithm can be deployed and run on edge devices, enhancing its real-time performance and meeting the online computing requirements of PIV environments. Therefore, the technical solution of this invention solves the problems of low optical flow estimation accuracy caused by large illumination variations, similar foreground and background, and the presence of transparent objects in PIV environments, as well as the high computational complexity and poor real-time performance of traditional deep learning optical flow methods.

[0030] Based on the above reasons, this invention can be widely applied in fields such as PIV gas-liquid two-phase flow, particle image velocimetry, and digital image correlation. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1 This is a schematic diagram of the overall process of a multi-scale pyramid optical flow estimation method for PIV according to the present invention.

[0033] Figure 2 This is a schematic flowchart of the image preprocessing method proposed in this invention.

[0034] Figure 3 The image shows the optical flow estimation effect of simulated particle images provided in an embodiment of the present invention.

[0035] Figure 4 This is an image showing the enhancement result of a real image provided in an embodiment of the present invention.

[0036] Figure 5 This is a diagram illustrating the optical flow estimation effect on a real image provided in an embodiment of the present invention. Detailed Implementation

[0037] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0038] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0039] like Figure 1 As shown, this invention provides a multi-scale pyramid optical flow estimation method for PIV, comprising: S1. Obtain the original particle image pair and perform adaptive image preprocessing on the original particle image pair to obtain the preprocessed first frame image and second frame image, so as to suppress random illumination and enhance target features. S2. Input the preprocessed first frame image and second frame image into a lightweight feature extraction network for multi-scale feature extraction, and extract multi-scale first frame feature map and multi-scale second frame feature map respectively, and extract the context feature map of the corresponding level of the first frame image. S3. Calculate the initial optical flow for the preprocessed first frame image and second frame image using the traditional dense optical flow algorithm, and downsample the initial optical flow to the deepest layer resolution; S4. Locally sample the second frame feature map of the current layer using the initial optical flow of the current layer resolution, and perform a dot product operation between the sampled features and the first frame feature map of the current layer to obtain the current layer correlation volume. S5. Using a multi-scale motion encoder, the current layer correlation volume and the initial optical flow information of the current layer resolution are fused together, and the residual optical flow is regressed by combining the context feature map of the current layer to obtain the residual optical flow of the current layer. S6. Perform weighted upsampling on the residual optical flow of the current layer based on neighboring pixels to obtain the upsampled optical flow of the current layer; S7. Determine whether the optical flow resolution after upsampling of the current layer reaches the original resolution: If yes, use the optical flow after upsampling of the current layer as the final optical flow output; if no, use the optical flow after upsampling of the current layer as the initial optical flow of the next layer, return to step S4, use the initial optical flow of the next layer to locally sample the multi-scale second frame feature map of the next layer, and calculate the correlation volume of the next layer. Repeat steps S5 to S7 until the full resolution optical flow estimate is output.

[0040] In a specific implementation, as a preferred embodiment of the present invention, step S1 includes: S11. Use a multi-scale inverse adaptive contrast enhancement algorithm to enhance the image; S12. Enhance the image using a Gaussian-Laplacian pyramid, and use brightness region adaptive suppression and multi-scale frequency enhancement within the Gaussian-Laplacian pyramid.

[0041] In specific implementation, as a preferred embodiment of the present invention, such as Figure 2 As shown, step S11 includes: S111. To address the issue of similarity between the foreground and background of transparent objects, a first scale is used to enhance local details and textures to improve small target features, while a second scale is used to improve overall contrast and suppress background features, resulting in a first frame image and a second frame image with enhanced contrast, wherein the first scale is smaller than the second scale; in this embodiment, the first scale is preferably 4×4 and the second scale is 16×16. S112. In order to suppress bright and dark areas without affecting the normally lit areas, the pixel values ​​of the first frame image and the second frame image after contrast enhancement are reversed. The reversed image is subjected to CLAHE processing to reduce the contrast of the dark areas of the original image. Then the processed image is reversed back, thereby suppressing the details of the dark and bright areas.

[0042] In specific implementation, as a preferred embodiment of the present invention, please refer to [reference needed]. Figure 2 Step S12 includes: S121. To address edge artifacts and block effects caused by local enhancement, Gaussian-Laplacian pyramid decomposition combined with multi-scale gain adjustment is used. The image is decomposed into multiple Gaussian pyramids and then converted into Laplacian pyramids. Gain coefficients are calculated based on adaptive weights, and weighted enhancement is applied to different levels of the Laplacian pyramid to achieve targeted enhancement at different frequencies. S122. To address the issues of overall illumination variation and local illumination variation between adjacent frames, the weights are dynamically adjusted based on the brightness characteristics of different regions of the image, and the brightness threshold is dynamically adjusted based on the maximum brightness of the image.

[0043] In a specific implementation, as a preferred embodiment of the present invention, step S122 includes: S1221. Calculate the maximum brightness of the image using the following formula:

[0044] in, Indicates the image height. Indicates the image width. Indicates the image at position The pixel brightness value at that location; S1222, Based on the maximum brightness of the image Dynamically adjust the brightness threshold, taking the maximum brightness of the image. The high brightness threshold is the highest brightness value of the image. The threshold value is set for the dark area; in this embodiment, it is preferred to use a threshold value set for the dark area. ,

[0045] S1223. For bright areas, the weights are dynamically adjusted based on the average brightness. The weight calculation formula is as follows:

[0046] in, This indicates that the brightness value is greater than or equal to the high-brightness threshold. The average brightness of all pixels, This indicates the total number of pixels in the highlighted area. The weighting coefficient represents the highlighted area; in this embodiment, preferably... ; S1224. For dark areas, the weights are dynamically adjusted based on the average brightness. The weight calculation formula is as follows:

[0047] in, This indicates that the brightness value is less than or equal to the dark area threshold. The average brightness of all pixels, The weighting coefficients represent the dark regions; in this embodiment, preferably... ; S1225. For areas with normal brightness, the weights are dynamically adjusted based on the average brightness. The weight calculation formula is as follows:

[0048] in, Indicates that the brightness value is at The average brightness of all pixels in the range The weighting coefficient represents the normal brightness region. In this embodiment, preferably... ; S1226. Based on the weights of the bright area, dark area and normal brightness area, and combined with the average brightness, dynamically adjust the gain of each area, and reconstruct the Laplacian pyramid after weighted enhancement of each level to obtain the preprocessed first frame image and second frame image.

[0049] In a specific implementation, as a preferred embodiment of the present invention, step S2 includes: S21. Construct a lightweight feature extraction network with an encoder-decoder structure. The encoder part contains multiple downsampling layers. Each downsampling layer reduces spatial resolution and increases the number of feature channels through convolution and pooling operations. The decoder part is used for subsequent optical flow prediction. The lightweight feature extraction network is used to reduce the number of parameters and computation while maintaining feature representation capabilities, so that the inference speed can reach more than 30fps. S22. Input the preprocessed first frame image and second frame image into the lightweight feature extraction network respectively. Through multi-layer convolution and downsampling operations, extract feature information of different scales layer by layer to obtain multi-scale first frame feature map fmap1 and multi-scale second frame feature map fmap2 of each level. S23. Perform multi-scale feature extraction on the preprocessed first frame image to obtain a context feature map cmap at the same level as the first frame feature map fmap1. The context feature map is used to provide spatial context information in the optical flow prediction stage to help constrain the geometric consistency and spatial smoothness of optical flow estimation.

[0050] In a specific implementation, as a preferred embodiment of the present invention, step S3 includes: S31. The traditional dense optical flow algorithm is used to perform dense optical flow calculation on the preprocessed first frame image and second frame image. The displacement vector field is calculated by polynomial expansion to approximate the image neighborhood and obtain the full-resolution initial optical flow. S32. Determine the target downsampling resolution based on the resolution of the deepest feature map; S33. The full-resolution initial optical flow is downsampled from the original resolution to the deepest layer resolution to obtain an initial optical flow that matches the spatial resolution of the deepest feature map of the pyramid.

[0051] In a specific implementation, as a preferred embodiment of the present invention, step S4 includes: S41. Sample the feature map of the second frame using the initial optical flow, for the deepest layer position. The corresponding initial optical flow at the location The second frame feature map fmap2 is obtained using quadratic interpolation. Centered on, with a scope of of Region feature vector; S42, the above The region feature vector and the first frame feature map fmap1 are in Perform a dot product operation on the eigenvectors at point , and obtain The correlation matrix of size ultimately forms a matrix of size. The deepest related matrix.

[0052] In a specific implementation, as a preferred embodiment of the present invention, step S5 includes: S51. Different motion encoders are used for correlation volume matrices and context features at different scales. A first motion encoder is used for correlation volume and context features at lower resolution levels, and a second motion encoder is used for correlation volume and context features at higher resolution levels. In this embodiment, it is preferred that the correlation volume at lower resolution levels be the correlation volume at 1 / 16 and 1 / 8 resolution of the original resolution, and the correlation volume at higher resolution levels be the correlation volume at 1 / 4 and 1 / 2 resolution of the original resolution. S52. Using the selected motion encoder, the current layer related volume, the initial optical flow information of the current layer resolution, and the current layer context feature map cmap are fused and encoded. The encoded features are then input into the optical flow prediction head for residual optical flow regression to obtain the current layer residual optical flow.

[0053] In a specific implementation, as a preferred embodiment of the present invention, step S6 includes: S61. For residual optical flow at different scales, different neighborhood pixel weighted upsampling modules are used. For residual optical flow at lower resolution levels, the first neighborhood pixel weighted upsampling module is used, and for residual optical flow at higher resolution levels, the second neighborhood pixel weighted upsampling module is used. In this embodiment, it is preferred that the residual optical flow at lower resolution levels be the residual optical flow at 1 / 16 and 1 / 8 resolution of the original resolution, and the residual optical flow at higher resolution levels be the residual optical flow at 1 / 4 and 1 / 2 resolution of the original resolution.

[0054] S62. Use the selected neighboring pixel weighted upsampling module to upsample the current layer residual optical flow to the next layer resolution to obtain the current layer upsampled optical flow.

[0055] In specific implementation, as a preferred embodiment of the present invention, the method is applicable to complex fluid scenarios such as PIV gas-liquid two-phase flow, particle image velocimetry, and digital image correlation. It is used for preprocessing illumination, distinguishing foreground and background, and calculating displacement fields or strains that include transparent objects and have high real-time requirements.

[0056] Corresponding to the multi-scale pyramid optical flow estimation method in this application, this application also provides a multi-scale pyramid optical flow estimation system for PIV, including: The image acquisition and preprocessing module is used to acquire the original particle image pairs and perform adaptive image preprocessing on the original particle image pairs to obtain the preprocessed first frame image and second frame image, so as to suppress random illumination and enhance target features. The multi-scale feature extraction module is used to input the preprocessed first frame image and second frame image into the lightweight feature extraction network for multi-scale feature extraction, and extract multi-scale first frame feature map and multi-scale second frame feature map respectively, and extract the context feature map of the corresponding level of the first frame image. The initial optical flow calculation module is used to calculate the initial optical flow of the preprocessed first frame image and the second frame image using a traditional dense optical flow algorithm, and to downsample the initial optical flow to the current layer resolution; The correlation volume calculation module is used to locally sample the multi-scale second frame feature map of the current layer using the initial optical flow of the current layer resolution, and perform a dot product operation between the sampled features and the multi-scale first frame feature map of the current layer to obtain the correlation volume of the current layer. The residual optical flow regression module is used to fuse the current layer correlation volume with the initial optical flow information of the current layer resolution using a multi-scale motion encoder, and perform residual optical flow regression in combination with the context feature map of the current layer to obtain the residual optical flow of the current layer. The neighborhood pixel weighted upsampling module is used to perform weighted upsampling of the current layer residual optical flow based on neighborhood pixels to obtain the current layer upsampled optical flow; The iterative update module is used to determine whether the optical flow resolution after upsampling of the current layer reaches the original resolution: if yes, the optical flow after upsampling of the current layer is used as the final optical flow output; if no, the optical flow after upsampling of the current layer is used as the initial optical flow of the next layer, and the module returns to the correlation volume calculation module. The module uses the initial optical flow of the next layer to locally sample the multi-scale second frame feature map of the next layer, and calculates the correlation volume of the next layer. The residual optical flow regression module, the neighborhood pixel weighted upsampling module, and the iterative update module are repeatedly executed until the full-resolution optical flow estimate is output.

[0057] The embodiments of the present invention are described simply because they correspond to those in the embodiments above. For any similarities, please refer to the descriptions in the embodiments above, which will not be elaborated here.

[0058] Example 1: Optical Flow Estimation from Simulated Particle Images A first frame of simulated particle image is generated. A simulated optical flow is then generated using turbulence as the target flow field. The first frame of particle image is then distorted based on the simulated optical flow to generate a second frame. The original images of the two frames are shown below. Figure 3 As shown, the particle image does not require preprocessing. Assume the particle image size is... The network first extracts features from the particles to obtain... , , , Four pairs of feature matrices are represented by fmap1 and fmap2, respectively, along with a set of context feature matrices cmap of the same size. Optical flow is calculated from two frames using the Farneback algorithm and downsampled to [the desired size]. As the initial optical flow, take To calculate the correlation volume for the lowest-level feature map, fmap2 is first sampled based on the initial optical flow, and then calculated through quadratic interpolation. A sampled feature matrix of size [size missing] is generated by dot product of the sampled matrix and fmap1. The size correlation volume matrix, after being encoded by the motion encoder, is combined with cmap and input to the optical flow prediction module to obtain the residual optical flow. The residual optical flow is then upsampled to obtain... The upsampled optical flow map will be used as the new initial optical flow guide size. The fmap2 sampling is performed, and the sampled feature matrix is ​​then dot-producted with fmap1 to form a new local correlation volume. This process is repeated until the size of the optical flow map obtained after upsampling is [size missing]. As the final optical flow output, the final output optical flow estimation diagram is as follows: Figure 3 As shown, it is particularly emphasized that in the context of and The optical flow prediction module and the neighborhood pixel weighted upsampling module used in the feature map processing of the size are the same module. and The optical flow prediction module and the neighborhood pixel weighted upsampling module used in the feature map processing of the size are the same module.

[0059] Example 2: Real Image Preprocessing and Optical Flow Estimation Realistic PIV environment images were obtained by photographing the experimental platform. The captured images are as follows: Figure 5As shown, the image is preprocessed. First, the image is inverted, and then CLAHE (clahe_16bit_suppress_dark) is applied to it inversely. The processed image is then inverted back and combined with the original image. Next, a Gaussian-Laplacian pyramid is constructed. Regions are automatically distinguished based on the image's brightness, and adaptive weights are calculated based on the average brightness of each region. Gain is applied to each level of the pyramid based on these calculated weights to smooth the image and enhance texture and edge features. After reconstructing the results of the multi-level pyramids, the image is normalized. Finally, regular CLAHE is applied to enhance the image, resulting in the enhanced image, as shown. Figure 4 As shown in the figure. The image after image enhancement is input into the deep learning network, and the specific effect is the same as in Example 1. The final optical flow estimation map is shown in the figure. Figure 5 As shown.

[0060] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-scale pyramid optical flow estimation method for PIV, characterized in that, include: S1. Obtain the original particle image pair and perform adaptive image preprocessing on the original particle image pair to obtain the preprocessed first frame image and second frame image, so as to suppress random illumination and enhance target features. S2. Input the preprocessed first frame image and second frame image into a lightweight feature extraction network for multi-scale feature extraction, and extract multi-scale first frame feature map and multi-scale second frame feature map respectively, and extract the context feature map of the corresponding level of the first frame image. S3. Calculate the initial optical flow for the preprocessed first frame image and second frame image using the traditional dense optical flow algorithm, and downsample the initial optical flow to the deepest layer resolution; S4. Locally sample the second frame feature map of the current layer using the initial optical flow of the current layer resolution, and perform a dot product operation between the sampled features and the first frame feature map of the current layer to obtain the current layer correlation volume. S5. Using a multi-scale motion encoder, the current layer correlation volume and the initial optical flow information of the current layer resolution are fused together, and the residual optical flow is regressed by combining the context feature map of the current layer to obtain the residual optical flow of the current layer. S6. Perform weighted upsampling on the residual optical flow of the current layer based on neighboring pixels to obtain the upsampled optical flow of the current layer; S7. Determine whether the optical flow resolution after upsampling of the current layer reaches the original resolution: if so, then use the optical flow after upsampling of the current layer as the final optical flow output; If not, the optical flow upsampled in the current layer is used as the initial optical flow for the next layer. Return to step S4, use the initial optical flow of the next layer to locally sample the multi-scale second frame feature map of the next layer, and calculate the correlation volume of the next layer. Repeat steps S5 to S7 until the full-resolution optical flow estimate is output.

2. The multi-scale pyramid optical flow estimation method for PIV as described in claim 1, characterized in that, Step S1 includes: S11. Use a multi-scale inverse adaptive contrast enhancement algorithm to enhance the image; S12. Enhance the image using a Gaussian-Laplacian pyramid, and use brightness region adaptive suppression and multi-scale frequency enhancement within the Gaussian-Laplacian pyramid.

3. The multi-scale pyramid optical flow estimation method for PIV as described in claim 2, characterized in that, Step S11 includes: S111. The original particle image pair is processed using two different scales of inverse adaptive contrast enhancement algorithms. The first scale is used to enhance local details and textures to enhance small target features, and the second scale is used to improve overall contrast and suppress background features, resulting in the first frame image and the second frame image after contrast enhancement, wherein the first scale is smaller than the second scale. S112. A multi-scale inverse adaptive contrast enhancement algorithm is used to invert the pixel values ​​of the first frame image and the second frame image. CLAHE processing is performed on the inverted image to reduce the contrast of the dark areas of the original image. The processed image is then inverted back, thereby suppressing details in the dark and bright areas.

4. The multi-scale pyramid optical flow estimation method for PIV as described in claim 2, characterized in that, Step S12 includes: S121. Using Gaussian-Laplacian pyramid decomposition combined with multi-scale gain adjustment, the image is decomposed into multiple Gaussian pyramids and then converted into Laplacian pyramids. The gain coefficients are calculated according to adaptive weights, and weighted enhancement is applied to different levels of the Laplacian pyramid to achieve targeted enhancement of different frequencies. S122. Dynamically adjust the weights based on the brightness characteristics of different regions of the image, and dynamically adjust the brightness threshold based on the maximum brightness of the image.

5. The multi-scale pyramid optical flow estimation method for PIV as described in claim 2, characterized in that, Step S122 includes: S1221. Calculate the maximum brightness of the image using the following formula: in, Indicates the image height. Indicates the image width. Indicates the image at position The pixel brightness value at that location; S1222, Based on the maximum brightness of the image Dynamically adjust the brightness threshold, taking the maximum brightness of the image. The high brightness threshold is the highest brightness value of the image. Threshold for dark areas; S1223. For bright areas, the weights are dynamically adjusted based on the average brightness. The weight calculation formula is as follows: in, This indicates that the brightness value is greater than or equal to the high-brightness threshold. The average brightness of all pixels, This indicates the total number of pixels in the highlighted area. This represents the weighting coefficient of the highlighted area; S1224. For dark areas, the weights are dynamically adjusted based on the average brightness. The weight calculation formula is as follows: in, This indicates that the brightness value is less than or equal to the dark area threshold. The average brightness of all pixels, Indicates the weighting coefficients for dark areas; S1225. For areas with normal brightness, the weights are dynamically adjusted based on the average brightness. The weight calculation formula is as follows: in, Indicates that the brightness value is at The average brightness of all pixels in the range The weighting coefficient represents the normal brightness area. S1226. Based on the weights of the bright area, dark area and normal brightness area, and combined with the average brightness, dynamically adjust the gain of each area, and reconstruct the Laplacian pyramid after weighted enhancement of each level to obtain the preprocessed first frame image and second frame image.

6. The multi-scale pyramid optical flow estimation method for PIV as described in claim 1, characterized in that, Step S2 includes: S21. Construct a lightweight feature extraction network with an encoder-decoder structure. The encoder part contains multiple downsampling layers. Each downsampling layer reduces spatial resolution and increases the number of feature channels through convolution and pooling operations. The decoder part is used for subsequent optical flow prediction. The lightweight feature extraction network is used to reduce the number of parameters and computation while maintaining feature representation capabilities, so that the inference speed can reach more than 30fps. S22. Input the preprocessed first frame image and second frame image into the lightweight feature extraction network respectively. Through multi-layer convolution and downsampling operations, extract feature information of different scales layer by layer to obtain multi-scale first frame feature map fmap1 and multi-scale second frame feature map fmap2 of each level. S23. Perform multi-scale feature extraction on the preprocessed first frame image to obtain a context feature map cmap at the same level as the first frame feature map fmap1. The context feature map is used to provide spatial context information in the optical flow prediction stage to help constrain the geometric consistency and spatial smoothness of optical flow estimation.

7. The multi-scale pyramid optical flow estimation method for PIV as described in claim 1, characterized in that, Step S3 includes: S31. The traditional dense optical flow algorithm is used to perform dense optical flow calculation on the preprocessed first frame image and second frame image. The displacement vector field is calculated by polynomial expansion to approximate the image neighborhood and obtain the full-resolution initial optical flow. S32. Determine the target downsampling resolution based on the resolution of the deepest feature map; S33. The full-resolution initial optical flow is downsampled from the original resolution to the deepest layer resolution to obtain an initial optical flow that matches the spatial resolution of the deepest feature map of the pyramid.

8. The multi-scale pyramid optical flow estimation method for PIV as described in claim 1, characterized in that, Step S4 includes: S41. Sample the feature map of the second frame using the initial optical flow, for the deepest layer position. The corresponding initial optical flow at the location The second frame feature map fmap2 is obtained using quadratic interpolation. Centered on, with a scope of of Region feature vector; S42, the above The region feature vector and the first frame feature map fmap1 are in Perform a dot product operation on the eigenvectors at point , and obtain The correlation matrix of size ultimately forms a matrix of size. The deepest related matrix.

9. The multi-scale pyramid optical flow estimation method for PIV as described in claim 1, characterized in that, Step S5 includes: S51. Different motion encoders are used for the relevant volume matrix and context features at different scales. The first motion encoder is used for the relevant volume and context features at lower resolution levels, and the second motion encoder is used for the relevant volume and context features at higher resolution levels. S52. Using the selected motion encoder, the current layer related volume, the initial optical flow information of the current layer resolution, and the current layer context feature map cmap are fused and encoded. The encoded features are then input into the optical flow prediction head for residual optical flow regression to obtain the current layer residual optical flow.

10. The multi-scale pyramid optical flow estimation method for PIV according to claim 1, characterized in that, Step S6 includes: S61. For residual optical flow at different scales, different neighborhood pixel weighted upsampling modules are used. For residual optical flow at lower resolution levels, the first neighborhood pixel weighted upsampling module is used, and for residual optical flow at higher resolution levels, the second neighborhood pixel weighted upsampling module is used. S62. Use the selected neighboring pixel weighted upsampling module to upsample the current layer residual optical flow to the next layer resolution to obtain the current layer upsampled optical flow.

11. The multi-scale pyramid optical flow estimation method for PIV as described in claim 1, characterized in that, The method is applicable to complex fluid scenarios such as PIV gas-liquid two-phase flow, particle image velocimetry, and digital image correlation. It is used for preprocessing illumination, distinguishing foreground from background, and calculating displacement fields or strains containing transparent objects with high real-time requirements.

12. A multi-scale pyramid optical flow estimation system based on the multi-scale pyramid optical flow estimation method according to any one of claims 1-11, characterized in that, include: The image acquisition and preprocessing module is used to acquire the original particle image pairs and perform adaptive image preprocessing on the original particle image pairs to obtain the preprocessed first frame image and second frame image, so as to suppress random illumination and enhance target features. The multi-scale feature extraction module is used to input the preprocessed first frame image and second frame image into the lightweight feature extraction network for multi-scale feature extraction, and extract multi-scale first frame feature map and multi-scale second frame feature map respectively, and extract the context feature map of the corresponding level of the first frame image. The initial optical flow calculation module is used to calculate the initial optical flow of the preprocessed first frame image and the second frame image using a traditional dense optical flow algorithm, and to downsample the initial optical flow to the current layer resolution; The correlation volume calculation module is used to locally sample the multi-scale second frame feature map of the current layer using the initial optical flow of the current layer resolution, and perform a dot product operation between the sampled features and the multi-scale first frame feature map of the current layer to obtain the correlation volume of the current layer. The residual optical flow regression module is used to fuse the current layer correlation volume with the initial optical flow information of the current layer resolution using a multi-scale motion encoder, and perform residual optical flow regression in combination with the context feature map of the current layer to obtain the residual optical flow of the current layer. The neighborhood pixel weighted upsampling module is used to perform weighted upsampling of the current layer residual optical flow based on neighborhood pixels to obtain the current layer upsampled optical flow; The iterative update module is used to determine whether the optical flow resolution after upsampling of the current layer reaches the original resolution: if so, the optical flow after upsampling of the current layer is used as the final optical flow output. If not, the optical flow upsampled in the current layer is used as the initial optical flow for the next layer. The correlation volume calculation module is then returned to the module. The initial optical flow of the next layer is used to locally sample the multi-scale second frame feature map of the next layer, and the correlation volume of the next layer is calculated. The residual optical flow regression module, the neighborhood pixel weighted upsampling module, and the iterative update module are executed repeatedly until the full-resolution optical flow estimate is output.

Images