Multi-frame particle image denoising method based on spatial-temporal feature interaction

Abstract

The invention discloses a multi-frame particle image denoising method based on spatio-temporal feature interaction. The method comprises the following steps: S1, constructing an enhanced data set comprising continuous multi-frame enhanced particle images; s2, constructing a spatio-temporal feature interaction network model, and training the spatio-temporal feature interaction network model based on the enhanced data set to obtain a trained spatio-temporal feature interaction network model; the spatio-temporal feature interaction network model comprises a spatial denoising module and a spatio-temporal denoising module; and S3, based on the trained spatio-temporal feature interaction network model, denoising processing is carried out on actual continuous multi-frame noise-containing particle images. According to the invention, parallel processing is carried out through the space denoising module, then stacking is carried out in the time dimension, denoising enhancement is carried out through the space-time denoising module, continuous multi-frame enhanced particle images are output, the calculation efficiency is improved, and thus high-quality images are provided for subsequent PIV estimation.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a multi-frame particle image denoising method based on spatiotemporal feature interaction. Background Art

[0002] Particle image velocimetry (PIV), a widely used experimental fluid dynamics technique, can extract velocity field information from continuous image pairs, thereby obtaining fluid motion information and revealing its laws. In the field of shipbuilding and ocean engineering, PIV technology can be used to non-contactly measure the flow velocity field of ships and underwater vehicles, thereby capturing the complex flow field evolution characteristics around bows, propellers, and tail fins, and has important applications. Current methods for calculating velocity fields, such as cross-correlation algorithms and optical flow algorithms, require high-quality particle images. However, in actual underwater PIV experiments, factors such as uneven illumination, background interference, low contrast, Gaussian random noise, and moving objects can hinder the acquisition of clear particle images. These interferences in particle images negatively impact velocity field calculations. Therefore, it is necessary to remove this unnecessary interference while preserving the particle information to enhance the particle image and achieve accurate velocity field calculation. Due to the powerful feature extraction and nonlinear fitting capabilities of convolutional neural networks, they can effectively learn noise information in images and reconstruct high-quality images, making them a mainstream method for particle image denoising.

[0003] Existing deep learning-based image enhancement methods are mainly divided into single-stage image enhancement and two-stage image enhancement algorithms. Single-stage image enhancement, including DnCNN, FFDNet, and CBDNet, has the advantage of fast inference speed. However, since these algorithms do not consider the spatiotemporal correlation between consecutive images, when applied to particle image enhancement, if the grayscale values of the two output frames change significantly, it will lead to incorrect cross-correlation matching, thus affecting the accuracy of the estimated velocity field. Two-stage image enhancement, including FastDVDNet, FITVNet, and ViDeNN, considers the spatiotemporal correlation between consecutive images. However, they all enhance the output image using information from adjacent frames of the target frame, inputting multiple consecutive frames but outputting only one enhanced frame, resulting in low computational efficiency. Furthermore, these methods do not adequately account for the small, uneven, and highly similar characteristics of particles in particle images, as well as the interference of illumination noise and complex underwater background information during particle image capture, resulting in poor particle enhancement results. Summary of the Invention

[0004] The present invention provides a multi-frame particle image denoising method based on the interaction of spatiotemporal features to overcome the technical problems that existing image denoising methods cannot effectively remove the interference of complex noise, and may cause the grayscale value of particles to decrease when removing noise, making it difficult to provide high-quality particle images for subsequent PIV estimation, resulting in reduced calculation accuracy.

[0005] In order to achieve the above object, the technical solution of the present invention is:

[0006] A multi-frame particle image denoising method based on the interaction of spatiotemporal features, comprising the following steps:

[0007] S1: Construct an enhanced dataset including multiple frames of continuous enhanced particle images;

[0008] S2: constructing a spatiotemporal feature interaction network model, and training the spatiotemporal feature interaction network model based on the enhanced dataset to obtain a trained spatiotemporal feature interaction network model;

[0009] The spatiotemporal feature interaction network model includes a spatial denoising module and a spatiotemporal denoising module;

[0010] The spatial denoising module is used to perform preliminary denoising on the continuous multi-frame enhanced particle images, and stack the continuous multi-frame enhanced particle images that have undergone preliminary denoising in the time dimension to obtain a spatial denoising feature map, and transmit it to the spatiotemporal denoising module;

[0011] The spatiotemporal denoising module is used to perform spatiotemporal feature interaction on the spatial denoising feature map to further denoise and obtain denoised continuous multi-frame particle images;

[0012] S3: De-noising is performed on actual continuous multiple frames of particle images containing noise based on the trained spatiotemporal feature interaction network model.

[0013] Furthermore, the spatial denoising module includes several spatial denoising submodules and a stacking module, and each spatial denoising submodule is constructed based on an encoder-decoder structure;

[0014] The encoder of the spatial denoising submodule includes a first standard convolutional layer, a first hybrid feature enhancement module, a second hybrid feature enhancement module, a second standard convolutional layer, a first Res2Net module, a second Res2Net module and a third standard convolutional layer;

[0015] The decoder of the spatial denoising submodule includes a first feature reconstruction module, a second feature reconstruction module and a fourth standard convolutional layer;

[0016] A single frame of enhanced particle images in the continuous multi-frame enhanced particle images is preliminarily denoised in parallel by using a plurality of the spatial denoising submodules. The preliminarily denoising process based on each of the spatial denoising submodules includes:

[0017] Adjusting the dimension of the single-frame enhanced particle image through the first standard convolution layer, and transmitting the enhanced particle image with adjusted dimension to the first hybrid feature enhancement module;

[0018] extracting, for the first time, the shape and distribution features of the particles in the enhanced particle image with adjusted dimension by the first hybrid feature enhancement module, and transmitting the shape and distribution features of the particles to the second hybrid feature enhancement module;

[0019] extracting the particle shape and distribution features from the particle shape and distribution features for a second time through the second mixed feature enhancement module, and transmitting the second extracted particle shape and distribution features to the second standard convolutional layer;

[0020] The particle shape and distribution features extracted for the second time are upgraded through the second standard convolutional layer, and the upgraded particle features are transmitted to the first Res2Net module;

[0021] Extracting multi-scale particle features from the dimensionally upgraded particle features through the first Res2Net module, and transmitting the multi-scale particle features to the second Res2Net module;

[0022] After refining the multi-scale particle features through the second Res2Net module, the refined multi-scale particle features are transmitted to the third standard convolutional layer;

[0023] Performing dimensionality reduction on the refined multi-scale particle features through the third standard convolutional layer, and transmitting the reduced dimensionality multi-scale particle features to the first feature reconstruction module;

[0024] Performing a first reconstruction of particle features on the multi-scale particle features after dimensionality reduction by the first feature reconstruction module, and transmitting the first reconstructed particle features to the second feature reconstruction module;

[0025] Performing a second particle feature reconstruction on the first reconstructed particle features through the second feature reconstruction module, and transmitting the second reconstructed particle features to the fourth standard convolutional layer;

[0026] The dimension of the particle features reconstructed for the second time is adjusted to three channels through the fourth standard convolution layer to obtain a single-frame enhanced particle image with preliminary denoising;

[0027] The stacking module is used to stack several preliminarily denoised single-frame enhanced particle images in the time dimension to obtain a spatial denoising feature map.

[0028] Furthermore, each hybrid feature enhancement module includes:

[0029] The first stage module includes: the first normalization layer, the first 1×1 convolution layer, the 5×5 convolution layer, the first depth-wise separable convolution layer, the second depth-wise separable convolution layer, the third depth-wise separable convolution layer, and a channel-wise convolution group;

[0030] The channel-by-channel convolution group includes a second 1×1 convolution layer, a GELU activation layer, and a third 1×1 convolution layer;

[0031] The second stage module includes: the first average pooling layer, the second average pooling layer, the 1×1 convolution group, the first Sigmod activation layer, and the second Sigmod activation layer;

[0032] The 1×1 convolution group includes a fourth 1×1 convolution layer, a second normalization layer, and a first ReLU activation layer;

[0033] The third stage module includes: the first branch dilated convolution layer, the second branch dilated convolution layer, the third branch dilated convolution layer, the fourth branch dilated convolution layer and the second ReLU activation layer;

[0034] The first branch hole convolution layer includes a fourth 1×1 convolution layer and a first 3×3 convolution layer;

[0035] The second branch hole convolution layer includes a fifth 1×1 convolution layer, a 1×3 convolution layer, a 3×1 convolution layer and a first 3×3 hole convolution layer;

[0036] The third branch hole convolution layer includes a sixth 1×1 convolution layer, a second 3×3 convolution layer, a seventh 1×1 convolution layer and a second 3×3 hole convolution layer;

[0037] The fourth branch hole convolution layer includes an eighth 1×1 convolution layer;

[0038] The step of processing the input features by the hybrid feature enhancement module includes:

[0039] The input features are normalized by the first normalization layer and transmitted to the first 1×1 convolutional layer;

[0040] The output features of the first normalization layer are subjected to inter-channel information exchange through the first 1×1 convolutional layer and transmitted to the 5×5 convolutional layer;

[0041] Capturing the particle image features in the output features of the first 1×1 convolutional layer through the 5×5 convolutional layer, and transmitting them as the first particle image features to the first depthwise separable convolutional layer, the second depthwise separable convolutional layer, and the third depthwise separable convolutional layer respectively;

[0042] The output features of the 5×5 convolutional layer are respectively subjected to separate spatial convolution and channel-by-channel convolution by the first depth-wise separable convolutional layer, the second depth-wise separable convolutional layer, and the third depth-wise separable convolutional layer, and then transmitted to the channel-by-channel convolution group;

[0043] The output features of the first depth-wise separable convolution layer, the second depth-wise separable convolution layer, and the third depth-wise separable convolution layer are spliced in the channel dimension through the channel-by-channel convolution group, and the splicing result is processed by the second 1×1 convolution layer for information interaction between channels and increase of feature dimension, and then transmitted to the GELU activation layer, and the output features of the second 1×1 convolution layer are processed by the GELU activation layer for nonlinear activation, and then transmitted to the third 1×1 convolution layer, and the output features of the GELU activation layer are processed for feature dimension restoration through the third 1×1 convolution layer, and then the output features of the third 1×1 convolution layer are added to the input features of the first normalization layer to obtain the first-stage output feature map, and transmitted to the second-stage module;

[0044] The first stage output feature map is average pooled in the H direction and the second average pooling layer respectively, and then transmitted to the 1×1 convolution group;

[0045] After splicing the output features of the first average pooling layer and the second average pooling layer in the channel dimension through the 1×1 convolution group, the splicing result is subjected to information interaction processing in the H direction and the W direction through the fourth 1×1 convolution layer, and the result is transmitted to the second normalization layer. After normalizing the output features of the fourth 1×1 convolution layer through the second normalization layer, the result is transmitted to the first ReLU activation layer. After performing nonlinear activation processing on the output features of the second normalization layer through the first ReLU activation layer, the output features of the first ReLU activation layer are evenly divided into a first part of the feature map and a second part of the feature map according to the channel dimension, the first part of the feature map is transmitted to the first Sigmod activation layer, and the second part of the feature map is transmitted to the second Sigmod activation layer;

[0046] After activating the first part of the feature map through the first Sigmod activation layer, multiplying the output result by the input feature of the second stage module to obtain a first feature map;

[0047] After activating the second part of the feature map through the second Sigmod activation layer, multiplying the output result by the input feature of the second stage module to obtain a second feature map;

[0048] Multiplying the first feature map and the second feature map to obtain a second-stage output feature map, and transmitting it to the third-stage module;

[0049] The fourth 1×1 convolutional layer in the first branch dilated convolutional layer performs information interaction and feature channel adjustment on the second stage output feature map, and then transmits it to the first 3×3 convolutional layer. The first 3×3 convolutional layer performs standard convolution on the output features of the fourth 1×1 convolutional layer to obtain the third feature map;

[0050] The fifth 1×1 convolutional layer in the second-branch dilated convolutional layer performs information interaction and feature channel adjustment on the second-stage output feature map, and then transmits it to the 1×3 convolutional layer. The 1×3 convolutional layer performs convolution processing on the output features of the fifth 1×1 convolutional layer in the H direction, and then transmits it to the 3×1 convolutional layer. The 3×1 convolutional layer performs convolution processing on the output features of the 1×3 convolutional layer in the W direction, and then transmits it to the first 3×3 dilated convolutional layer. The first 3×3 dilated convolutional layer performs dilated convolution processing on the output features of the 3×1 convolutional layer, and obtains the fourth feature map.

[0051] After performing information interaction and feature channel adjustment processing on the second-stage output feature map through the sixth 1×1 convolution layer in the third branch dilated convolution layer, the feature map is transmitted to the second 3×3 convolution layer. After performing standard convolution processing on the output features of the sixth 1×1 convolution layer through the second 3×3 convolution layer, the feature map is transmitted to the seventh 1×1 convolution layer. After performing information interaction between feature channels on the output features of the second 3×3 convolution layer through the seventh 1×1 convolution layer, the feature map is transmitted to the second 3×3 dilated convolution layer. After performing dilated convolution processing on the output features of the seventh 1×1 convolution layer through the second 3×3 dilated convolution layer, a fifth feature map is obtained.

[0052] After performing information interaction between feature channels and adjustment processing on the feature map output from the second stage through the eighth 1×1 convolutional layer in the fourth branch dilated convolutional layer, a sixth feature map is obtained;

[0053] After splicing the third feature map, the fourth feature map, the fifth feature map, and the sixth feature map in the channel dimension, nonlinear activation processing is performed on the splicing result through the second ReLU activation layer to obtain a third-stage output feature map.

[0054] Furthermore, each of the feature reconstruction modules includes a 3×3 residual convolution group and an adaptive fine-grained channel attention mechanism module;

[0055] The 3×3 residual convolution group includes a third 3×3 convolution layer, a third normalization layer and a third ReLU activation layer;

[0056] The adaptive fine-grained channel attention mechanism module includes an adaptive average pooling layer, a 1×1 convolution layer for band operation, a 1×1 convolution layer for diag operation, a third Sigmod activation layer, a fourth Sigmod activation layer, and a fifth Sigmod activation layer;

[0057] The step of processing the input features by the feature reconstruction module includes:

[0058] After performing standard convolution processing on the input features through the third 3×3 convolution layer in the 3×3 residual convolution group, the input features are transmitted to the third normalization layer; after normalizing the output features of the third 3×3 convolution layer through the third normalization layer, the output features are transmitted to the third ReLU activation layer; after performing nonlinear activation processing on the output features of the third normalization layer through the third ReLU activation layer, a feature map with a dimension of C×H×W is output, and the feature map with a dimension of C×H×W is added to the input features of the 3×3 residual convolution group to obtain a seventh feature map with a dimension of C×H×W, which is transmitted to the adaptive fine-grained channel attention mechanism module;

[0059] Performing adaptive average pooling on the seventh feature map of dimension C×H×W through the adaptive average pooling layer in the adaptive fine-grained channel attention mechanism module to compress the dimension of the seventh feature map into an eighth feature map of dimension C×1×1 while retaining channel information;

[0060] A 1×1 convolutional layer for band operation is used to perform a band operation on the eighth feature map of dimension C×1×1 to capture the long-range dependency between channels, and a first eigenvector of dimension C×1 is obtained. A 1×1 convolutional layer for diag operation is used to perform a diag operation on the eighth feature map of dimension C×1×1 to learn the local relationship between channels, and a second eigenvector of dimension C×1 is obtained.

[0061] Perform matrix multiplication on the first eigenvector of dimension C×1 to obtain the channel similarity matrix U1 of dimension C×C;

[0062] Perform a matrix multiplication operation on the second eigenvectors of dimension C×1 after transposing them, and obtain a bidirectionally associated matrix U2 of dimension C×C;

[0063] Perform a single-dimensional sum operation on U1 and U2 to obtain the third and fourth eigenvectors with dimensions of C×1;

[0064] Activate the third eigenvector and the fourth eigenvector of dimension C×1 through the third Sigmod activation layer and the fourth Sigmod activation layer, respectively, to obtain a fifth eigenvector and a sixth eigenvector;

[0065] After adding the fifth eigenvector and the sixth eigenvector, the addition result is activated by the fifth Sigmod activation layer to obtain a seventh eigenvector with a dimension of C×1;

[0066] The seventh eigenvector is multiplied by the input feature of the 3×3 residual convolution group to obtain a reconstructed feature map.

[0067] Furthermore, the spatiotemporal denoising module is constructed based on an encoder-decoder structure;

[0068] The encoder of the spatiotemporal denoising module includes: a first spatiotemporal feature encoding module, a second spatiotemporal feature encoding module, a third spatiotemporal feature encoding module and a fourth spatiotemporal feature encoding module;

[0069] The decoder of the spatiotemporal denoising module includes: a first spatiotemporal feature decoding module, a second spatiotemporal feature decoding module, a third spatiotemporal feature decoding module and a standard 1×3×3 convolutional layer;

[0070] The specific steps of further denoising the spatial denoising feature map by the encoder and decoder of the spatiotemporal denoising module include:

[0071] Extracting the spatiotemporal correlation features of the spatial denoising feature map for the first time through the first spatiotemporal feature encoding module to obtain a first spatiotemporal denoising feature map, and transmitting the first spatiotemporal denoising feature map to the second spatiotemporal feature encoding module and the third spatiotemporal feature decoding module respectively;

[0072] Extracting the spatiotemporal correlation features of the first spatiotemporal denoising feature map for a second time through the second spatiotemporal feature encoding module to obtain a second spatiotemporal denoising feature map, and transmitting the second spatiotemporal denoising feature map to the third spatiotemporal feature encoding module and the second spatiotemporal feature decoding module respectively;

[0073] Extracting the spatiotemporal correlation features of the second spatiotemporal denoising feature map for a third time through the third spatiotemporal feature encoding module to obtain a third spatiotemporal denoising feature map, and transmitting the third spatiotemporal denoising feature map to the fourth spatiotemporal feature encoding module and the first spatiotemporal feature decoding module respectively;

[0074] Extracting the spatiotemporal correlation features of the third spatiotemporal denoising feature map for the fourth time by the fourth spatiotemporal feature encoding module to obtain a fourth spatiotemporal denoising feature map, and transmitting the fourth spatiotemporal denoising feature map to the first spatiotemporal feature decoding module;

[0075] Adding the fourth spatiotemporal denoised feature map and the third spatiotemporal denoised feature map through the first spatiotemporal feature decoding module to obtain a fifth spatiotemporal denoised feature map, performing a first decoding on the fifth spatiotemporal denoised feature map, and transmitting the first decoded fifth spatiotemporal denoised feature map to the second spatiotemporal feature decoding module;

[0076] The fifth spatiotemporal denoised feature map after the first decoding is added to the second spatiotemporal denoised feature map by the second spatiotemporal feature decoding module to obtain a sixth spatiotemporal denoised feature map, and the sixth spatiotemporal denoised feature map is decoded for a second time, and the sixth spatiotemporal denoised feature map obtained by the second decoding is transmitted to the third spatiotemporal feature decoding module;

[0077] Adding the sixth spatiotemporal denoising feature map decoded for the second time to the first spatiotemporal denoising feature map through the third spatiotemporal feature decoding module to obtain a seventh spatiotemporal denoising feature map, performing a final decoding on the seventh spatiotemporal denoising feature map, and transmitting the finally decoded seventh spatiotemporal denoising feature map to the standard 1×3×3 convolutional layer;

[0078] The dimension of the seventh spatiotemporal denoising feature map finally decoded is adjusted to three channels through the standard 1×3×3 convolutional layer to obtain three consecutive frames of enhanced particle image denoising feature maps, and the three consecutive frames of enhanced particle image denoising feature maps are separated in the time dimension to obtain three consecutive frames of denoised particle images.

[0079] Furthermore, the first spatiotemporal feature encoding module, the second spatiotemporal feature encoding module, the third spatiotemporal feature encoding module and the fourth spatiotemporal feature encoding module each include: a first 1×3×3 convolutional layer, a first LeakReLU activation layer and a first 3D convolutional group;

[0080] The first 3D convolution group includes a first 1×1×1 convolution layer, a second 1×3×3 convolution layer, a second LeakReLU activation layer, a first 3×1×1 convolution layer, a first BatchNorm3d layer, a third 1×3×3 convolution layer, a third LeakReLU activation layer, a first 3×3×3 convolution layer, a fourth LeakReLU activation layer, and a second BatchNorm3d layer.

[0081] The process of each spatiotemporal feature encoding module processing the input feature map includes:

[0082] After adjusting the dimension of the spatial denoising feature map through the first 1×3×3 convolutional layer, it is transmitted to the first LeakReLU activation layer;

[0083] The output of the first 1×3×3 convolutional layer is nonlinearly activated by the first LeakReLU activation layer and then transmitted to the first 3D convolution group;

[0084] The output of the first LeakReLU activation layer is passed to the second 1×3×3 convolutional layer after performing information exchange between feature channels in time and space through the first 1×1×1 convolutional layer in the first 3D convolutional group.

[0085] The output of the first 1×1×1 convolutional layer is subjected to standard convolution processing in the spatial dimension by the second 1×3×3 convolutional layer, and information interaction between channels in the temporal dimension is performed before being transmitted to the second LeakReLU activation layer;

[0086] The output of the second 1×3×3 convolutional layer is nonlinearly activated by the second LeakReLU activation layer and then transmitted to the first 3×1×1 convolutional layer;

[0087] The output of the second LeakReLU activation layer is subjected to standard convolution processing in the time dimension and information interaction between channels in the spatial dimension through the first 3×1×1 convolutional layer, and then transmitted to the first BatchNorm3d layer;

[0088] The output of the first 3×1×1 convolutional layer is batch normalized in time and space dimensions by the first BatchNorm3d layer and then transmitted to the third 1×3×3 convolutional layer;

[0089] The output of the first BatchNorm3d layer is subjected to standard convolution processing in the spatial dimension and information interaction between channels in the temporal dimension through the third 1×3×3 convolutional layer, and then transmitted to the third LeakReLU activation layer;

[0090] After performing nonlinear activation processing on the output of the third 1×3×3 convolutional layer through the third LeakReLU activation layer, the output is transmitted to the first 3×3×3 convolutional layer;

[0091] The output of the third LeakReLU activation layer is subjected to standard convolution processing in time and space by the first 3×3×3 convolutional layer, and then transmitted to the fourth LeakReLU activation layer;

[0092] After performing nonlinear activation processing on the output of the first 3×3×3 convolutional layer through the fourth LeakReLU activation layer, the output is transmitted to the second BatchNorm3d layer;

[0093] The output of the fourth LeakReLU activation layer is batch normalized in time and space dimensions by the second BatchNorm3d layer;

[0094] The output of the second BatchNorm3d layer is added to the output of the first 3×1×1 convolutional layer to obtain the corresponding spatiotemporal denoising feature map.

[0095] Furthermore, the fourth spatiotemporal feature encoding module further includes: a second 1×1×1 convolutional layer and a fifth LeakReLU activation layer;

[0096] The process of the fourth spatiotemporal feature encoding module processing the input feature map further includes:

[0097] The output of the first 3D convolution group in the fourth spatiotemporal feature encoding module is subjected to information exchange between feature channels in time and space by the second 1×1×1 convolution, and then transmitted to the fifth LeakReLU activation layer;

[0098] The output of the second 1×1×1 convolution is subjected to nonlinear activation processing through the fifth LeakReLU activation layer to obtain the fourth spatiotemporal denoising feature map.

[0099] Furthermore, the first and second spatiotemporal feature decoding modules each include a second 3D convolution group, a third 1×1×1 convolution layer, and a sixth LeakReLU activation layer;

[0100] The process of each spatiotemporal feature decoding module processing the input spatiotemporal denoising feature map includes:

[0101] After processing the input spatiotemporal denoising feature map through the second 3D convolution group, it is transmitted to the third 1×1×1 convolution layer;

[0102] After the output of the second 3D convolution group is subjected to information exchange between feature channels in time and space through the third 1×1×1 convolution layer, it is transmitted to the sixth LeakReLU activation layer;

[0103] After the output of the third 1×1×1 convolutional layer is processed by the sixth LeakReLU activation layer with nonlinear activation, the corresponding spatiotemporal denoising feature map is obtained.

[0104] Furthermore, the third spatiotemporal feature decoding module includes a third 3D convolution group, a fourth 1×3×3 convolution layer, a seventh LeakReLU activation layer, a second 3×3×3 convolution layer and a third BatchNorm3d layer;

[0105] The process of the third spatiotemporal feature decoding module processing the seventh spatiotemporal denoising feature map includes:

[0106] After processing the seventh spatiotemporal denoising feature map through the third 3D convolution group, the map is transmitted to the fourth 1×3×3 convolution;

[0107] The output of the third 3D convolution group is subjected to standard convolution processing in the spatial dimension and information interaction between channels in the temporal dimension by the fourth 1×3×3 convolution layer, and then transmitted to the seventh LeakReLU activation layer;

[0108] After performing nonlinear activation processing on the output of the fourth 1×3×3 convolutional layer through the seventh LeakReLU activation layer, it is transmitted to the second 3×3×3 convolutional layer;

[0109] The output of the seventh LeakReLU activation layer is subjected to standard convolution processing in both time and space by the second 3×3×3 convolutional layer, and then passed to the third BatchNorm3d layer;

[0110] After batch normalization processing is performed on the output of the second 3×3×3 convolutional layer in the time and space dimensions through the third BatchNorm3d layer, the seventh spatiotemporal denoising feature map that is finally decoded is obtained.

[0111] Beneficial effects: The present invention designs a spatiotemporal feature interaction network model, including a spatial denoising module and a spatiotemporal denoising module; the spatial denoising module is used to perform preliminary denoising on continuous multi-frame enhanced particle images, and stack the preliminary denoised continuous multi-frame enhanced particle images in the time dimension to obtain a spatial denoising feature map, and transmit it to the spatiotemporal denoising module; the spatiotemporal denoising module is used to perform spatiotemporal feature interaction on the spatial denoising feature map to further denoise and obtain denoised continuous multi-frame particle images. The present invention improves computational efficiency by stacking continuous multi-frame enhanced particle images in the time dimension after parallel processing by the spatial denoising module, and then performing denoising and enhancement by the spatiotemporal denoising module, thereby outputting high-quality images for subsequent PIV estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0112] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following is a brief introduction to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0113] Figure 1 This is a flow chart of a multi-frame particle image denoising method based on the interaction of spatiotemporal features in the present invention;

[0114] Figure 2 Schematic diagram of a multi-frame particle image enhancement model constructed in an embodiment of the present invention;

[0115] Figure 3 1 is an overall structural diagram of the spatial denoising module and the spatiotemporal denoising module in an embodiment of the present invention;

[0116] Figure 4 Schematic diagram of the structure of the hybrid feature enhancement module in an embodiment of the present invention;

[0117] Figure 5 Schematic diagram of the module structure of the feature reconstruction model and adaptive fine-grained channel attention mechanism in an embodiment of the present invention;

[0118] Figure 6 A schematic diagram of multi-frame particle image enhancement constructed in an embodiment of the present invention;

[0119] Figure 7 Schematic diagram of comparison between an enhanced image and a high-quality labeled image of a dataset constructed in an embodiment of the present invention;

[0120] Figure 8 This is a comparison diagram of SSIM curves after particle images are enhanced using different methods in an embodiment of the present invention. DETAILED DESCRIPTION

[0121] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0122] This embodiment provides a multi-frame particle image denoising method based on the interaction of spatiotemporal features. Figure 1 As shown, the specific steps include:

[0123] S1: Build Figure 6 The enhanced dataset shown includes multiple consecutive frames of enhanced particle images;

[0124] Specifically, this embodiment uses a particle image generator (PIG) to generate synthetic particle images. It first constructs high-quality, noise-free, high-contrast continuous multi-frame particle images using the DNS and Cylinder flow fields. It then uses the cross-correlation algorithm WIDIM to estimate the flow field, ensuring that the estimated velocity field error is within an acceptable range. After obtaining high-quality particle images, various types of interference (Gaussian noise, random illumination interference, and some common underwater background interference) are added. The images are then partitioned into training, validation, and test sets with a 7:2:1 ratio. During training, the dataset is preprocessed using data augmentation methods such as random horizontal and vertical flips and random rotations.

[0125] Specifically, if Figure 7As shown in the figure, a schematic diagram of the comparison between the images in the enhanced dataset and the high-quality labeled images is shown, where input is the image input to the model, output is the image of the low-quality image after being processed by the model, and label is the high-quality labeled image; Lowcontrast, noisy, lite, background and all are the added interference information, namely: low contrast processing, superimposed Gaussian noise, superimposed random illumination interference, superimposed underwater background interference and superimposed Gaussian noise + random illumination interference + underwater background interference.

[0126] S2: Construct a spatiotemporal feature interaction network model SFINet, and train the spatiotemporal feature interaction network model based on the enhanced dataset to obtain a trained spatiotemporal feature interaction network model;

[0127] The spatiotemporal feature interaction network model includes a spatial denoising module and a spatiotemporal denoising module;

[0128] The spatial denoising module is used to perform preliminary denoising on the continuous multi-frame enhanced particle images, and stack the continuous multi-frame enhanced particle images that have undergone preliminary denoising in the time dimension to obtain a spatial denoising feature map, and transmit it to the spatiotemporal denoising module;

[0129] The spatiotemporal denoising module is used to perform spatiotemporal feature interaction on the spatial denoising feature map to further denoise and obtain denoised continuous multi-frame particle images;

[0130] Specifically, in the process of training the spatiotemporal feature interaction network model based on the enhanced dataset, the loss function selects the joint loss, which is expressed as:

[0131] loss=α×MSEloss+β×correlationloss+γ×SSIMloss

[0132] Wherein, MSEloss, correlationloss and SSIMloss are the mean square error loss function, correlation loss function and structural similarity index loss function, respectively; α, β and γ are the weights of the mean square error loss function, correlation loss function and structural similarity index loss function, respectively; the optimal weights are obtained through the optimization algorithm in the training process; in this embodiment, the normal distribution is used to initialize the model parameters.

[0133] Specifically, in this embodiment, the optimizer selects the Adam optimizer based on the gradient descent algorithm. In each training iteration, data is input into the model for forward propagation, the loss function value is calculated, and then backpropagation is performed to calculate the gradient, and finally the model parameters are updated according to the gradient. This process continues for multiple training rounds until the model converges. During the training process, the model is verified using a validation set to check the performance of the model on unseen data. The optimal weight is obtained based on the performance of the model on the validation set. When the performance of the model on the validation set no longer improves or begins to decline, the training can be stopped and the corresponding model parameters can be selected as the optimal weight. After obtaining the optimal weight, it can be deployed on the relevant equipment, and the continuous multi-frame particle images containing interference information are enhanced, and then the velocity field is estimated using the cross-correlation algorithm WIDIM or the optical flow method.

[0134] S3: De-noising is performed on actual continuous multiple frames of particle images containing noise based on the trained spatiotemporal feature interaction network model.

[0135] Specifically, if Figure 2 As shown, this embodiment adopts a multi-frame parallel processing strategy, that is, three consecutive noisy frames (t-1frame, tframe and t+1frame) are input, and after parallel processing by the spatial denoising module, they are stacked in the time dimension and then denoised and enhanced by the spatiotemporal denoising module, and three frames of high-quality images are output, which improves the computational efficiency and provides high-quality images for subsequent PIV estimation.

[0136] In a specific embodiment, the spatial denoising module and the spatiotemporal denoising module are both constructed based on an encoder-decoder structure.

[0137] In a specific embodiment, Figure 2 As shown, the spatial denoising module includes several spatial denoising submodules and stacking modules, and each spatial denoising submodule is constructed based on an encoder-decoder structure;

[0138] like Figure 3 As shown, the encoder of the spatial denoising submodule includes a first standard convolutional layer, a first mixed feature enhancement module (MIX module), a second mixed feature enhancement module, a second standard convolutional layer, a first Res2Net module, a second Res2Net module and a third standard convolutional layer;

[0139] The decoder of the spatial denoising submodule includes a first feature reconstruction module, a second feature reconstruction module and a fourth standard convolutional layer;

[0140] A single frame of enhanced particle images in the continuous multi-frame enhanced particle images is preliminarily denoised in parallel by using a plurality of the spatial denoising submodules. The preliminarily denoising process based on each of the spatial denoising submodules includes:

[0141] Adjusting the dimension of the single-frame enhanced particle image through the first standard convolution layer, and transmitting the enhanced particle image with adjusted dimension to the first hybrid feature enhancement module;

[0142] extracting, for the first time, the shape and distribution features of the particles in the enhanced particle image with adjusted dimension by the first hybrid feature enhancement module, and transmitting the shape and distribution features of the particles to the second hybrid feature enhancement module;

[0143] extracting the particle shape and distribution features from the particle shape and distribution features for a second time through the second mixed feature enhancement module, and transmitting the second extracted particle shape and distribution features to the second standard convolutional layer;

[0144] The particle shape and distribution features extracted for the second time are upgraded through the second standard convolutional layer, and the upgraded particle features are transmitted to the first Res2Net module;

[0145] Extracting multi-scale particle features from the dimensionally upgraded particle features through the first Res2Net module, and transmitting the multi-scale particle features to the second Res2Net module;

[0146] After refining the multi-scale particle features through the second Res2Net module, the refined multi-scale particle features are transmitted to the third standard convolutional layer;

[0147] Performing dimensionality reduction on the refined multi-scale particle features through the third standard convolutional layer, and transmitting the reduced dimensionality multi-scale particle features to the first feature reconstruction module;

[0148] Performing a first reconstruction of particle features on the multi-scale particle features after dimensionality reduction by the first feature reconstruction module, and transmitting the first reconstructed particle features to the second feature reconstruction module;

[0149] Performing a second particle feature reconstruction on the first reconstructed particle features through the second feature reconstruction module, and transmitting the second reconstructed particle features to the fourth standard convolutional layer;

[0150] The dimension of the particle features reconstructed for the second time is adjusted to three channels through the fourth standard convolution layer to obtain a single-frame enhanced particle image with preliminary denoising;

[0151] Specifically, the three channels correspond to the RGB channels.

[0152] The stacking module is used to stack several preliminarily denoised single-frame enhanced particle images in the time dimension to obtain a spatial denoising feature map.

[0153] In a specific embodiment, Figure 4 As shown, each hybrid feature enhancement module includes:

[0154] The first stage module includes: the first normalization layer, the first 1×1 convolution layer, the 5×5 convolution layer, the first depth-wise separable convolution layer, the second depth-wise separable convolution layer, the third depth-wise separable convolution layer, and a channel-wise convolution group;

[0155] The channel-by-channel convolution group includes a second 1×1 convolution layer, a GELU activation layer, and a third 1×1 convolution layer;

[0156] Specifically, the formula for the first stage module is:

[0157] F1=Conv 1×1 [concat(DWconv 3_19 (F0),DWconv 3_13 (F0),DWconv 3_7 (F0))] (1)

[0158] Among them, Conv 1×1 is a channel-by-channel convolution group, concat indicates concatenation in dimension, DWconv(·) indicates depthwise separable convolution, and the subscripts 3_19, 3_13, and 3_7 denote convolution kernels with a dilation rate of 3 and effective receptive field sizes of 19, 13, and 7, respectively.

[0159] The second stage module includes: the first average pooling layer, the second average pooling layer, the 1×1 convolution group, the first Sigmod activation layer, and the second Sigmod activation layer;

[0160] The 1×1 convolution group includes a fourth 1×1 convolution layer, a second normalization layer, and a first ReLU activation layer;

[0161] Specifically, the formula for the second-stage module is:

[0162] F H =mean(F1(:,:,j,:)) (2)

[0163] F W =mean(F1(:,:,:,i)) (3)

[0164]

[0165] Where ⊙ represents element-by-element multiplication;

[0166] The third stage module includes: the first branch dilated convolution layer, the second branch dilated convolution layer, the third branch dilated convolution layer, the fourth branch dilated convolution layer and the second ReLU activation layer;

[0167] The first branch hole convolution layer includes a fourth 1×1 convolution layer and a first 3×3 convolution layer;

[0168] The second branch hole convolution layer includes a fifth 1×1 convolution layer, a 1×3 convolution layer, a 3×1 convolution layer and a first 3×3 hole convolution layer;

[0169] The third branch hole convolution layer includes a sixth 1×1 convolution layer, a second 3×3 convolution layer, a seventh 1×1 convolution layer and a second 3×3 hole convolution layer;

[0170] The fourth branch hole convolution layer includes an eighth 1×1 convolution layer;

[0171] Specifically, in this embodiment, the first stage combines different types of convolution operations (depthwise separable convolution and standard convolution) to more effectively extract multi-scale image features, especially in detail recovery. At the same time, by adopting depthwise separable convolution, the amount of calculation and parameters are reduced; the second stage constructs position attention to help the model pay more attention to small targets such as particles in spatial position; the third stage increases feature richness through a multi-branch void convolution structure, expands the local perception ability of the network, and improves the semantic information expression of small targets.

[0172] The step of processing the input features by the hybrid feature enhancement module includes:

[0173] The input features are normalized by the first normalization layer and transmitted to the first 1×1 convolutional layer;

[0174] The output features of the first normalization layer are subjected to inter-channel information exchange through the first 1×1 convolutional layer and transmitted to the 5×5 convolutional layer;

[0175] The particle image features in the output features of the first 1×1 convolutional layer are captured by the 5×5 convolutional layer and used as the first particle image features, where the first particle image features are particle image features of a larger range, and the first particle image features are transmitted to the first depthwise separable convolutional layer, the second depthwise separable convolutional layer, and the third depthwise separable convolutional layer respectively;

[0176] The output features of the 5×5 convolutional layer are respectively subjected to separate spatial convolution and channel-by-channel convolution by the first depth-wise separable convolutional layer, the second depth-wise separable convolutional layer, and the third depth-wise separable convolutional layer, and then transmitted to the channel-by-channel convolution group;

[0177] The output features of the first depth-wise separable convolution layer, the second depth-wise separable convolution layer, and the third depth-wise separable convolution layer are spliced in the channel dimension through the channel-by-channel convolution group, and the splicing result is processed by the second 1×1 convolution layer for information interaction between channels and feature dimension increase, and then transmitted to the GELU activation layer. The output features of the second 1×1 convolution layer are processed by the GELU activation layer for nonlinear activation, and then transmitted to the third 1×1 convolution layer. After the output features of the GELU activation layer are restored in feature dimension through the third 1×1 convolution layer, the output features of the third 1×1 convolution layer are added to the input features of the first normalization layer to obtain the first-stage output feature map F1, and transmitted to the second-stage module;

[0178] After the first stage output feature map F1 is average pooled in the H direction and the W direction by the first average pooling layer and the second average pooling layer, F is obtained. H and F W , and transmitted to the 1×1 convolution group;

[0179] The output features F of the first and second average pooling layers are processed by the 1×1 convolution group. H and F W After splicing in the channel dimension, the splicing result is processed by the fourth 1×1 convolution layer for information interaction in the H and W directions to realize the model's attention to the characteristics of small particles in the spatial position, and is transmitted to the second normalization layer. The output features of the fourth 1×1 convolution layer are normalized by the second normalization layer and then transmitted to the first ReLU activation layer. The output features of the second normalization layer are nonlinearly activated by the first ReLU activation layer, and the output features of the first ReLU activation layer are evenly divided (Split) into the first and second feature maps according to the channel dimension. The first feature map is transmitted to the first Sigmod activation layer, and the second feature map is transmitted to the second Sigmod activation layer.

[0180] After the first Sigmod activation layer activates the first feature map, the output result F H * Multiplying it with the input features of the second stage module, we get the first feature map, which focuses on the fine features of the particles in the H direction.

[0181] After the second part of the feature map is activated by the second Sigmod activation layer, the output result F W * Multiplying it with the input features of the second stage module, we get the second feature map, which focuses on the fine features of particles in the W direction.

[0182] Multiply the first feature map and the second feature map to achieve the model's attention to the small features of the particles in space, obtain the second-stage output feature map F2, and transmit it to the third-stage module;

[0183] The fourth 1×1 convolutional layer in the first branch dilated convolutional layer performs information interaction and feature channel adjustment on the second stage output feature map, and then transmits it to the first 3×3 convolutional layer. The first 3×3 convolutional layer performs standard convolution on the output features of the fourth 1×1 convolutional layer to obtain the third feature map;

[0184] The fifth 1×1 convolutional layer in the second-branch dilated convolutional layer performs information interaction and feature channel adjustment on the second-stage output feature map, and then transmits it to the 1×3 convolutional layer. The 1×3 convolutional layer performs convolution processing on the output features of the fifth 1×1 convolutional layer in the H direction, and then transmits it to the 3×1 convolutional layer. The 3×1 convolutional layer performs convolution processing on the output features of the 1×3 convolutional layer in the W direction, and then transmits it to the first 3×3 dilated convolutional layer. The first 3×3 dilated convolutional layer performs dilated convolution processing on the output features of the 3×1 convolutional layer, and obtains the fourth feature map.

[0185] After performing information interaction and feature channel adjustment processing on the second-stage output feature map through the sixth 1×1 convolution layer in the third branch dilated convolution layer, the feature map is transmitted to the second 3×3 convolution layer. After performing standard convolution processing on the output features of the sixth 1×1 convolution layer through the second 3×3 convolution layer, the feature map is transmitted to the seventh 1×1 convolution layer. After performing information interaction between feature channels on the output features of the second 3×3 convolution layer through the seventh 1×1 convolution layer, the feature map is transmitted to the second 3×3 dilated convolution layer. After performing dilated convolution processing on the output features of the seventh 1×1 convolution layer through the second 3×3 dilated convolution layer, a fifth feature map is obtained.

[0186] After performing information interaction between feature channels and adjustment processing on the feature map output from the second stage through the eighth 1×1 convolutional layer in the fourth branch dilated convolutional layer, a sixth feature map is obtained;

[0187] After splicing the third feature map, the fourth feature map, the fifth feature map, and the sixth feature map in the channel dimension, nonlinear activation processing is performed on the splicing result through the second ReLU activation layer to obtain a third-stage output feature map.

[0188] The final output feature map of the MIX module is F out , F out The expression is as follows:

[0189] F out=ReLU[concat(branch1(F2),branch2(F2),branch3(F2),Conv 1×1 (F2))] (6)

[0190] Among them, branch1(·), branch2(·), and branch3(·) respectively represent Figure 3 The upper part of the third stage has three branches, Conv 1×1 (·) indicates the fourth branch.

[0191] The formulas for different branch(·) are as follows:

[0192] branch1(x)=Conv 3×3 (Conv 1×1 (x)) (7)

[0193] branch2(x)=AtrousConv 3×3 (Conv 3×1 (Conv 1×3 (Conv 1×1 (x)))) (8)

[0194] branch2(x)=AtrousConv 3×3 (Conv 1×1 (Conv 3×3 (Conv 1×1 (x)))) (9)

[0195] Where AtrousConv(·) represents the dilation rate of 5.

[0196] In a specific embodiment, Figure 5 As shown, each of the feature reconstruction modules includes a 3×3 residual convolution group and an adaptive fine-grained channel attention mechanism module (FCA module);

[0197] The 3×3 residual convolution group includes a third 3×3 convolution layer, a third normalization layer and a third ReLU activation layer;

[0198] The adaptive fine-grained channel attention mechanism module includes an adaptive average pooling layer, a 1×1 convolution layer for band operation, a 1×1 convolution layer for diag operation, a third Sigmod activation layer, a fourth Sigmod activation layer, and a fifth Sigmod activation layer;

[0199] The step of processing the input features by the feature reconstruction module includes:

[0200] After performing standard convolution processing on the input features through the third 3×3 convolution layer in the 3×3 residual convolution group, the input features are transmitted to the third normalization layer; after normalizing the output features of the third 3×3 convolution layer through the third normalization layer, the output features are transmitted to the third ReLU activation layer; after performing nonlinear activation processing on the output features of the third normalization layer through the third ReLU activation layer, a feature map with a dimension of C×H×W is output, and the feature map with a dimension of C×H×W is added to the input features of the 3×3 residual convolution group to obtain a seventh feature map with a dimension of C×H×W, which is transmitted to the adaptive fine-grained channel attention mechanism module;

[0201] Specifically, in the feature reconstruction process, this embodiment improves the flow efficiency of particle feature information by introducing residual connections. The formula of residual connections is as follows:

[0202] y=x+f(x) (10)where f(x) represents a 3×3 residual convolution group: f(x)=ReLU(BN(Conv 3×3 (x)));

[0203] Performing adaptive average pooling on the seventh feature map of dimension C×H×W through the adaptive average pooling layer in the adaptive fine-grained channel attention mechanism module to compress the dimension of the seventh feature map into an eighth feature map of dimension C×1×1 while retaining channel information;

[0204] Perform a band operation on the eighth feature map of dimension C×1×1 through a 1×1 convolutional layer for band operation to capture the long-range dependencies between channels, and obtain the first feature vector of dimension C×1;

[0205] Perform a diag operation on the eighth feature map of dimension C×1×1 through a 1×1 convolutional layer for diag operation to learn the channel local relationship and obtain a second feature vector of dimension C×1;

[0206] Perform matrix multiplication on the first eigenvector of dimension C×1 to obtain the channel similarity matrix U1 of dimension C×C;

[0207] Perform a matrix multiplication operation on the second eigenvectors of dimension C×1 after transposing them, and obtain a bidirectionally associated matrix U2 of dimension C×C;

[0208] Perform a single-dimensional sum operation on U1 and U2 to obtain the third and fourth eigenvectors with dimensions of C×1;

[0209] Activate the third eigenvector and the fourth eigenvector of dimension C×1 through the third Sigmod activation layer and the fourth Sigmod activation layer, respectively, to obtain a fifth eigenvector X1 and a sixth eigenvector X2;

[0210] After adding the fifth eigenvector and the sixth eigenvector, the addition result is activated by the fifth Sigmod activation layer to obtain a seventh eigenvector with a dimension of C×1;

[0211] Multiply the seventh eigenvector by the input feature of the 3×3 residual convolution group to obtain the reconstructed feature map F' out .

[0212] Specifically, F' out The relationship between X1, X2 and θ is as follows:

[0213] F′ out = F in ×[σ(θ)×X1+ (1-σ(θ)×X2)] (11)

[0214] Where, σ(·) is the Sigmod activation layer; F in The stacking module is a spatial denoising feature map obtained by stacking the denoising feature maps of several single-frame enhanced particle images in the time dimension. in =stack(F out1 ,F oit2 ,F out3 ).

[0215] In a specific embodiment, the spatiotemporal denoising module is constructed based on an encoder-decoder structure;

[0216] The encoder of the spatiotemporal denoising module includes: a first spatiotemporal feature encoding module, a second spatiotemporal feature encoding module, a third spatiotemporal feature encoding module and a fourth spatiotemporal feature encoding module;

[0217] The decoder of the spatiotemporal denoising module includes: a first spatiotemporal feature decoding module, a second spatiotemporal feature decoding module, a third spatiotemporal feature decoding module and a standard 1×3×3 convolutional layer;

[0218] The specific steps of further denoising the spatial denoising feature map by the encoder and decoder of the spatiotemporal denoising module include:

[0219] Extracting the spatiotemporal correlation features of the spatial denoising feature map for the first time through the first spatiotemporal feature encoding module to obtain a first spatiotemporal denoising feature map, and transmitting the first spatiotemporal denoising feature map to the second spatiotemporal feature encoding module and the third spatiotemporal feature decoding module respectively;

[0220] Extracting the spatiotemporal correlation features of the first spatiotemporal denoising feature map for a second time through the second spatiotemporal feature encoding module to obtain a second spatiotemporal denoising feature map, and transmitting the second spatiotemporal denoising feature map to the third spatiotemporal feature encoding module and the second spatiotemporal feature decoding module respectively;

[0221] Extracting the spatiotemporal correlation features of the second spatiotemporal denoising feature map for a third time through the third spatiotemporal feature encoding module to obtain a third spatiotemporal denoising feature map, and transmitting the third spatiotemporal denoising feature map to the fourth spatiotemporal feature encoding module and the first spatiotemporal feature decoding module respectively;

[0222] Extracting the spatiotemporal correlation features of the third spatiotemporal denoising feature map for the fourth time by the fourth spatiotemporal feature encoding module to obtain a fourth spatiotemporal denoising feature map, and transmitting the fourth spatiotemporal denoising feature map to the first spatiotemporal feature decoding module;

[0223] Adding the fourth spatiotemporal denoised feature map and the third spatiotemporal denoised feature map through the first spatiotemporal feature decoding module to obtain a fifth spatiotemporal denoised feature map, performing a first decoding on the fifth spatiotemporal denoised feature map, and transmitting the first decoded fifth spatiotemporal denoised feature map to the second spatiotemporal feature decoding module;

[0224] The fifth spatiotemporal denoised feature map after the first decoding is added to the second spatiotemporal denoised feature map by the second spatiotemporal feature decoding module to obtain a sixth spatiotemporal denoised feature map, and the sixth spatiotemporal denoised feature map is decoded for a second time, and the sixth spatiotemporal denoised feature map obtained by the second decoding is transmitted to the third spatiotemporal feature decoding module;

[0225] Adding the further decoded sixth spatiotemporal denoising feature map to the first spatiotemporal denoising feature map through the third spatiotemporal feature decoding module to obtain a seventh spatiotemporal denoising feature map, performing a final decoding on the seventh spatiotemporal denoising feature map, and transmitting the finally decoded seventh spatiotemporal denoising feature map to the standard 1×3×3 convolutional layer;

[0226] The dimension of the seventh spatiotemporal denoising feature map finally decoded is adjusted to three channels through the standard 1×3×3 convolutional layer to obtain three consecutive frames of enhanced particle image denoising feature maps, and the three consecutive frames of enhanced particle image denoising feature maps are separated in the time dimension to obtain three consecutive frames of denoised particle images.

[0227] In a specific embodiment, the first spatiotemporal feature encoding module, the second spatiotemporal feature encoding module, the third spatiotemporal feature encoding module, and the fourth spatiotemporal feature encoding module each include: a first 1×3×3 convolutional layer, a first LeakReLU activation layer, and a first 3D convolutional group;

[0228] The first 3D convolution group includes a first 1×1×1 convolution layer, a second 1×3×3 convolution layer, a second LeakReLU activation layer, a first 3×1×1 convolution layer, a first BatchNorm3d layer, a third 1×3×3 convolution layer, a third LeakReLU activation layer, a first 3×3×3 convolution layer, a fourth LeakReLU activation layer, and a second BatchNorm3d layer.

[0229] The process of each spatiotemporal feature encoding module processing the input feature map includes:

[0230] After adjusting the dimension of the spatial denoising feature map through the first 1×3×3 convolutional layer, it is transmitted to the first LeakReLU activation layer;

[0231] The output of the first 1×3×3 convolutional layer is nonlinearly activated by the first LeakReLU activation layer and then transmitted to the first 3D convolution group;

[0232] The output of the first LeakReLU activation layer is passed to the second 1×3×3 convolutional layer after performing information exchange between feature channels in time and space through the first 1×1×1 convolutional layer in the first 3D convolutional group.

[0233] The output of the first 1×1×1 convolutional layer is subjected to standard convolution processing in the spatial dimension by the second 1×3×3 convolutional layer, and information interaction between channels in the temporal dimension is performed before being transmitted to the second LeakReLU activation layer;

[0234] The output of the second 1×3×3 convolutional layer is nonlinearly activated by the second LeakReLU activation layer and then transmitted to the first 3×1×1 convolutional layer;

[0235] The output of the second LeakReLU activation layer is subjected to standard convolution processing in the time dimension and information interaction between channels in the spatial dimension through the first 3×1×1 convolutional layer, and then transmitted to the first BatchNorm3d layer;

[0236] The output of the first 3×1×1 convolutional layer is batch normalized in time and space dimensions by the first BatchNorm3d layer and then transmitted to the third 1×3×3 convolutional layer;

[0237] The output of the first BatchNorm3d layer is subjected to standard convolution processing in the spatial dimension and information interaction between channels in the temporal dimension through the third 1×3×3 convolutional layer, and then transmitted to the third LeakReLU activation layer;

[0238] After performing nonlinear activation processing on the output of the third 1×3×3 convolutional layer through the third LeakReLU activation layer, the output is transmitted to the first 3×3×3 convolutional layer;

[0239] The output of the third LeakReLU activation layer is subjected to standard convolution processing in time and space by the first 3×3×3 convolutional layer, and then transmitted to the fourth LeakReLU activation layer;

[0240] After performing nonlinear activation processing on the output of the first 3×3×3 convolutional layer through the fourth LeakReLU activation layer, the output is transmitted to the second BatchNorm3d layer;

[0241] The output of the fourth LeakReLU activation layer is batch normalized in time and space dimensions by the second BatchNorm3d layer;

[0242] The output of the second BatchNorm3d layer is added to the output of the first 3×1×1 convolutional layer to obtain the corresponding spatiotemporal denoising feature map.

[0243] In a specific embodiment, the fourth spatiotemporal feature encoding module further includes: a second 1×1×1 convolutional layer and a fifth LeakReLU activation layer;

[0244] The process of the fourth spatiotemporal feature encoding module processing the input feature map further includes:

[0245] The output of the first 3D convolution group in the fourth spatiotemporal feature encoding module is subjected to information exchange between feature channels in time and space by the second 1×1×1 convolution, and then transmitted to the fifth LeakReLU activation layer;

[0246] The output of the second 1×1×1 convolution is subjected to nonlinear activation processing through the fifth LeakReLU activation layer to obtain the fourth spatiotemporal denoising feature map.

[0247] The first and second spatiotemporal feature decoding modules each include a second 3D convolution group, a third 1×1×1 convolution layer, and a sixth LeakReLU activation layer;

[0248] The process of each spatiotemporal feature decoding module processing the input spatiotemporal denoising feature map includes:

[0249] After processing the input spatiotemporal denoising feature map through the second 3D convolution group, it is transmitted to the third 1×1×1 convolution layer;

[0250] After the output of the second 3D convolution group is subjected to information exchange between feature channels in time and space through the third 1×1×1 convolution layer, it is transmitted to the sixth LeakReLU activation layer;

[0251] After the output of the third 1×1×1 convolutional layer is processed by the sixth LeakReLU activation layer with nonlinear activation, the corresponding spatiotemporal denoising feature map is obtained.

[0252] In a specific embodiment, the third spatiotemporal feature decoding module includes a third 3D convolution group, a fourth 1×3×3 convolution layer, a seventh LeakReLU activation layer, a second 3×3×3 convolution layer, and a third BatchNorm3d layer;

[0253] The process of the third spatiotemporal feature decoding module processing the seventh spatiotemporal denoising feature map includes:

[0254] After processing the seventh spatiotemporal denoising feature map through the third 3D convolution group, the map is transmitted to the fourth 1×3×3 convolution;

[0255] The output of the third 3D convolution group is subjected to standard convolution processing in the spatial dimension and information interaction between channels in the temporal dimension by the fourth 1×3×3 convolution layer, and then transmitted to the seventh LeakReLU activation layer;

[0256] After performing nonlinear activation processing on the output of the fourth 1×3×3 convolutional layer through the seventh LeakReLU activation layer, it is transmitted to the second 3×3×3 convolutional layer;

[0257] The output of the seventh LeakReLU activation layer is subjected to standard convolution processing in both time and space by the second 3×3×3 convolutional layer, and then passed to the third BatchNorm3d layer;

[0258] After batch normalization processing is performed on the output of the second 3×3×3 convolutional layer in the time and space dimensions through the third BatchNorm3d layer, the seventh spatiotemporal denoising feature map that is finally decoded is obtained.

[0259] Specifically, in this embodiment, a hybrid feature enhancement module is designed that combines standard convolution, depthwise separable convolution, and channel-by-channel convolution to extract small target detail features and reduce computational complexity. Through the positional attention mechanism, average pooling and element-by-element multiplication are performed in the H and W directions to accurately locate the particle area. Multi-branch dilated convolution is used to expand the receptive field and enhance the semantic expression of particles in complex backgrounds. The residual connection in the feature reconstruction module improves the efficiency of information flow. The cross-correlation matrix in the adaptive fine-grained channel attention dynamically fuses global and local information to optimize channel weight allocation. The three-dimensional spatiotemporal convolution in the spatiotemporal denoising module directly captures the spatiotemporal correlation between consecutive frames.

[0260] Specifically, the spatiotemporal denoising module uses three-dimensional spatiotemporal convolution. Unlike traditional 2D convolution, spatiotemporal convolution can directly extract features in the time dimension, which enables the module to make full use of the spatiotemporal correlation information in the time series.

[0261] Using spatiotemporal convolution kernels of varying sizes (e.g., 1×3×3, 3×1×1, and 3×3×3), we extract multi-level spatiotemporal features. This is particularly important for processing particle images with spatiotemporal correlations, as it better captures the timing information between consecutive frames. Furthermore, using residual connections within the network accelerates training and information flow, effectively alleviating the vanishing gradient problem during training.

[0262] To demonstrate the effectiveness of SFINet, Tables 1 and 2 compare the image enhancement performance of SFINet with that of classic deep learning image enhancement methods and traditional PIV particle image enhancement methods. The results are expressed in terms of peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM), with higher values indicating better image denoising. #01, #02, #03, #04, and #05 represent low contrast, superimposed Gaussian noise, superimposed random illumination interference, superimposed complex underwater images, and superimposed multiple types of interference, respectively. As can be seen from the table, SFINet achieves superior results in both PSNR and SSIM.

[0263] Table 1 PSNR of different methods on the test set

[0264]

[0265] Table 2 SSIM of different methods on the test set

[0266]

[0267]

[0268] like Figure 8The figure shows the SSIM curves of 100 particle images generated based on the test set after adding various interferences. These 100 images are then enhanced using different methods using a sliding window (window size of 3). As can be seen from the figure, among deep learning methods, SFINet has the highest SSIM curve with minimal fluctuation, followed by FITVNet, and finally DnCNN. This demonstrates the importance of temporal and spatial feature interactions for enhancing continuous multi-frame particle images. However, for traditional particle image enhancement methods, histogram equalization (CLAHE) and high-pass filtering (HPF), CLAHE and HPF are often used together to process particle images. The SSIM curve of CLAHE+HPF is higher than that of HPF, but both methods show a downward trend, and the particle image enhancement effect is poor, and it cannot effectively process particle images containing multiple complex background interferences.

[0269] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-frame particle image denoising method based on the interaction of spatiotemporal features, characterized in that: The specific steps include: S1: Construct an enhanced dataset including multiple consecutive frames of enhanced particle images; S2: constructing a spatiotemporal feature interaction network model, and training the spatiotemporal feature interaction network model based on the enhanced dataset to obtain a trained spatiotemporal feature interaction network model; The spatiotemporal feature interaction network model includes a spatial denoising module and a spatiotemporal denoising module; The spatial denoising module is used to perform preliminary denoising on the continuous multi-frame enhanced particle images, and stack the continuous multi-frame enhanced particle images that have undergone preliminary denoising in the time dimension to obtain a spatial denoising feature map, and transmit it to the spatiotemporal denoising module; The spatiotemporal denoising module is used to perform spatiotemporal feature interaction on the spatial denoising feature map to further denoise and obtain denoised continuous multi-frame particle images; S3: De-noising is performed on actual continuous multiple frames of particle images containing noise based on the trained spatiotemporal feature interaction network model.

2. The multi-frame particle image denoising method based on spatiotemporal feature interaction according to claim 1 is characterized in that: The spatial denoising module includes several spatial denoising submodules and a stacking module, each of which is constructed based on an encoder-decoder structure; The encoder of the spatial denoising submodule includes a first standard convolutional layer, a first hybrid feature enhancement module, a second hybrid feature enhancement module, a second standard convolutional layer, a first Res2Net module, a second Res2Net module and a third standard convolutional layer; The decoder of the spatial denoising submodule includes a first feature reconstruction module, a second feature reconstruction module and a fourth standard convolutional layer; A single frame of enhanced particle images in the continuous multi-frame enhanced particle images is preliminarily denoised in parallel by using a plurality of the spatial denoising submodules. The preliminarily denoising process based on each of the spatial denoising submodules includes: Adjusting the dimension of the single-frame enhanced particle image through the first standard convolution layer, and transmitting the enhanced particle image with adjusted dimension to the first hybrid feature enhancement module; extracting, for the first time, the shape and distribution features of the particles in the enhanced particle image with adjusted dimension by the first hybrid feature enhancement module, and transmitting the shape and distribution features of the particles to the second hybrid feature enhancement module; extracting the particle shape and distribution features from the particle shape and distribution features for a second time through the second mixed feature enhancement module, and transmitting the second extracted particle shape and distribution features to the second standard convolutional layer; The particle shape and distribution features extracted for the second time are upgraded through the second standard convolutional layer, and the upgraded particle features are transmitted to the first Res2Net module; Extracting multi-scale particle features from the dimensionally upgraded particle features through the first Res2Net module, and transmitting the multi-scale particle features to the second Res2Net module; After refining the multi-scale particle features through the second Res2Net module, the refined multi-scale particle features are transmitted to the third standard convolutional layer; Performing dimensionality reduction on the refined multi-scale particle features through the third standard convolutional layer, and transmitting the reduced dimensionality multi-scale particle features to the first feature reconstruction module; Performing a first reconstruction of particle features on the multi-scale particle features after dimensionality reduction by the first feature reconstruction module, and transmitting the first reconstructed particle features to the second feature reconstruction module; Performing a second particle feature reconstruction on the first reconstructed particle features through the second feature reconstruction module, and transmitting the second reconstructed particle features to the fourth standard convolutional layer; The dimension of the particle features reconstructed for the second time is adjusted to three channels through the fourth standard convolution layer to obtain a single-frame enhanced particle image with preliminary denoising; The stacking module is used to stack several preliminarily denoised single-frame enhanced particle images in the time dimension to obtain a spatial denoising feature map.

3. The multi-frame particle image denoising method based on spatiotemporal feature interaction according to claim 2 is characterized in that: Each hybrid feature enhancement module includes: The first stage module includes: the first normalization layer, the first 1×1 convolution layer, the 5×5 convolution layer, the first depth-wise separable convolution layer, the second depth-wise separable convolution layer, the third depth-wise separable convolution layer, and a channel-wise convolution group; The channel-by-channel convolution group includes a second 1×1 convolution layer, a GELU activation layer, and a third 1×1 convolution layer; The second stage module includes: the first average pooling layer, the second average pooling layer, the 1×1 convolution group, the first Sigmod activation layer, and the second Sigmod activation layer; The 1×1 convolution group includes a fourth 1×1 convolution layer, a second normalization layer, and a first ReLU activation layer; The third stage module includes: the first branch dilated convolution layer, the second branch dilated convolution layer, the third branch dilated convolution layer, the fourth branch dilated convolution layer and the second ReLU activation layer; The first branch hole convolution layer includes a fourth 1×1 convolution layer and a first 3×3 convolution layer; The second branch hole convolution layer includes a fifth 1×1 convolution layer, a 1×3 convolution layer, a 3×1 convolution layer and a first 3×3 hole convolution layer; The third branch hole convolution layer includes a sixth 1×1 convolution layer, a second 3×3 convolution layer, a seventh 1×1 convolution layer and a second 3×3 hole convolution layer; The fourth branch hole convolution layer includes an eighth 1×1 convolution layer; The step of processing the input features by the hybrid feature enhancement module includes: The input features are normalized by the first normalization layer and transmitted to the first 1×1 convolutional layer; The output features of the first normalization layer are subjected to inter-channel information exchange through the first 1×1 convolutional layer and transmitted to the 5×5 convolutional layer; Capturing the particle image features in the output features of the first 1×1 convolutional layer through the 5×5 convolutional layer, and transmitting them as the first particle image features to the first depthwise separable convolutional layer, the second depthwise separable convolutional layer, and the third depthwise separable convolutional layer respectively; The output features of the 5×5 convolutional layer are respectively subjected to separate spatial convolution and channel-by-channel convolution by the first depth-wise separable convolutional layer, the second depth-wise separable convolutional layer, and the third depth-wise separable convolutional layer, and then transmitted to the channel-by-channel convolution group; The output features of the first depth-wise separable convolution layer, the second depth-wise separable convolution layer, and the third depth-wise separable convolution layer are spliced in the channel dimension through the channel-by-channel convolution group, and the splicing result is processed by the second 1×1 convolution layer for information interaction between channels and increase of feature dimension, and then transmitted to the GELU activation layer, and the output features of the second 1×1 convolution layer are processed by the GELU activation layer for nonlinear activation, and then transmitted to the third 1×1 convolution layer, and the output features of the GELU activation layer are processed for feature dimension restoration through the third 1×1 convolution layer, and then the output features of the third 1×1 convolution layer are added to the input features of the first normalization layer to obtain the first-stage output feature map, and transmitted to the second-stage module; The first stage output feature map is average pooled in the H direction and the second average pooling layer respectively, and then transmitted to the 1×1 convolution group; After splicing the output features of the first average pooling layer and the second average pooling layer in the channel dimension through the 1×1 convolution group, the splicing result is subjected to information interaction processing in the H direction and the W direction through the fourth 1×1 convolution layer, and the result is transmitted to the second normalization layer. After normalizing the output features of the fourth 1×1 convolution layer through the second normalization layer, the result is transmitted to the first ReLU activation layer. After performing nonlinear activation processing on the output features of the second normalization layer through the first ReLU activation layer, the output features of the first ReLU activation layer are evenly divided into a first part of the feature map and a second part of the feature map according to the channel dimension, the first part of the feature map is transmitted to the first Sigmod activation layer, and the second part of the feature map is transmitted to the second Sigmod activation layer; After activating the first part of the feature map through the first Sigmod activation layer, multiplying the output result by the input feature of the second stage module to obtain a first feature map; After activating the second part of the feature map through the second Sigmod activation layer, multiplying the output result by the input feature of the second stage module to obtain a second feature map; Multiplying the first feature map and the second feature map to obtain a second-stage output feature map, and transmitting it to the third-stage module; The fourth 1×1 convolutional layer in the first branch dilated convolutional layer performs information interaction and feature channel adjustment on the second stage output feature map, and then transmits it to the first 3×3 convolutional layer. The first 3×3 convolutional layer performs standard convolution on the output features of the fourth 1×1 convolutional layer to obtain the third feature map; The fifth 1×1 convolutional layer in the second-branch dilated convolutional layer performs information interaction and feature channel adjustment on the second-stage output feature map, and then transmits it to the 1×3 convolutional layer. The 1×3 convolutional layer performs convolution processing on the output features of the fifth 1×1 convolutional layer in the H direction, and then transmits it to the 3×1 convolutional layer. The 3×1 convolutional layer performs convolution processing on the output features of the 1×3 convolutional layer in the W direction, and then transmits it to the first 3×3 dilated convolutional layer. The first 3×3 dilated convolutional layer performs dilated convolution processing on the output features of the 3×1 convolutional layer, and obtains the fourth feature map. After performing information interaction and feature channel adjustment processing on the second-stage output feature map through the sixth 1×1 convolution layer in the third branch dilated convolution layer, the feature map is transmitted to the second 3×3 convolution layer. After performing standard convolution processing on the output features of the sixth 1×1 convolution layer through the second 3×3 convolution layer, the feature map is transmitted to the seventh 1×1 convolution layer. After performing information interaction between feature channels on the output features of the second 3×3 convolution layer through the seventh 1×1 convolution layer, the feature map is transmitted to the second 3×3 dilated convolution layer. After performing dilated convolution processing on the output features of the seventh 1×1 convolution layer through the second 3×3 dilated convolution layer, a fifth feature map is obtained. After performing information interaction between feature channels and adjustment processing on the feature map output from the second stage through the eighth 1×1 convolutional layer in the fourth branch dilated convolutional layer, a sixth feature map is obtained; After splicing the third feature map, the fourth feature map, the fifth feature map, and the sixth feature map in the channel dimension, nonlinear activation processing is performed on the splicing result through the second ReLU activation layer to obtain a third-stage output feature map.

4. The multi-frame particle image denoising method based on spatiotemporal feature interaction according to claim 3 is characterized in that: Each of the feature reconstruction modules includes a 3×3 residual convolution group and an adaptive fine-grained channel attention mechanism module; The 3×3 residual convolution group includes a third 3×3 convolution layer, a third normalization layer and a third ReLU activation layer; The adaptive fine-grained channel attention mechanism module includes an adaptive average pooling layer, a 1×1 convolution layer for band operation, a 1×1 convolution layer for diag operation, a third Sigmod activation layer, a fourth Sigmod activation layer, and a fifth Sigmod activation layer; The step of processing the input features by the feature reconstruction module includes: After performing standard convolution processing on the input features through the third 3×3 convolution layer in the 3×3 residual convolution group, the input features are transmitted to the third normalization layer; after normalizing the output features of the third 3×3 convolution layer through the third normalization layer, the output features are transmitted to the third ReLU activation layer; after performing nonlinear activation processing on the output features of the third normalization layer through the third ReLU activation layer, a feature map with a dimension of C×H×W is output, and the feature map with a dimension of C×H×W is added to the input features of the 3×3 residual convolution group to obtain a seventh feature map with a dimension of C×H×W, which is transmitted to the adaptive fine-grained channel attention mechanism module; Performing adaptive average pooling on the seventh feature map of dimension C×H×W through the adaptive average pooling layer in the adaptive fine-grained channel attention mechanism module to compress the dimension of the seventh feature map into an eighth feature map of dimension C×1×1 while retaining channel information; A 1×1 convolutional layer for band operation is used to perform a band operation on the eighth feature map of dimension C×1×1 to capture the long-range dependency between channels, and a first eigenvector of dimension C×1 is obtained. A 1×1 convolutional layer for diag operation is used to perform a diag operation on the eighth feature map of dimension C×1×1 to learn the local relationship between channels, and a second eigenvector of dimension C×1 is obtained. Perform matrix multiplication on the first eigenvector of dimension C×1 to obtain the channel similarity matrix U1 of dimension C×C; Perform a matrix multiplication operation on the second eigenvectors of dimension C×1 after transposing them, and obtain a bidirectionally associated matrix U2 of dimension C×C; Perform a single-dimensional sum operation on U1 and U2 to obtain the third and fourth eigenvectors with dimensions of C×1; Activate the third eigenvector and the fourth eigenvector of dimension C×1 through the third Sigmod activation layer and the fourth Sigmod activation layer, respectively, to obtain a fifth eigenvector and a sixth eigenvector; After adding the fifth eigenvector and the sixth eigenvector, the addition result is activated by the fifth Sigmod activation layer to obtain a seventh eigenvector with a dimension of C×1; The seventh eigenvector is multiplied by the input feature of the 3×3 residual convolution group to obtain a reconstructed feature map.

5. The multi-frame particle image denoising method based on spatiotemporal feature interaction according to claim 4 is characterized in that: The spatiotemporal denoising module is constructed based on an encoder-decoder structure; The encoder of the spatiotemporal denoising module includes: a first spatiotemporal feature encoding module, a second spatiotemporal feature encoding module, a third spatiotemporal feature encoding module and a fourth spatiotemporal feature encoding module; The decoder of the spatiotemporal denoising module includes: a first spatiotemporal feature decoding module, a second spatiotemporal feature decoding module, a third spatiotemporal feature decoding module and a standard 1×3×3 convolutional layer; The specific steps of further denoising the spatial denoising feature map by the encoder and decoder of the spatiotemporal denoising module include: Extracting the spatiotemporal correlation features of the spatial denoising feature map for the first time through the first spatiotemporal feature encoding module to obtain a first spatiotemporal denoising feature map, and transmitting the first spatiotemporal denoising feature map to the second spatiotemporal feature encoding module and the third spatiotemporal feature decoding module respectively; Extracting the spatiotemporal correlation features of the first spatiotemporal denoising feature map for a second time through the second spatiotemporal feature encoding module to obtain a second spatiotemporal denoising feature map, and transmitting the second spatiotemporal denoising feature map to the third spatiotemporal feature encoding module and the second spatiotemporal feature decoding module respectively; Extracting the spatiotemporal correlation features of the second spatiotemporal denoising feature map for a third time through the third spatiotemporal feature encoding module to obtain a third spatiotemporal denoising feature map, and transmitting the third spatiotemporal denoising feature map to the fourth spatiotemporal feature encoding module and the first spatiotemporal feature decoding module respectively; Extracting the spatiotemporal correlation features of the third spatiotemporal denoising feature map for the fourth time by the fourth spatiotemporal feature encoding module to obtain a fourth spatiotemporal denoising feature map, and transmitting the fourth spatiotemporal denoising feature map to the first spatiotemporal feature decoding module; Adding the fourth spatiotemporal denoised feature map and the third spatiotemporal denoised feature map through the first spatiotemporal feature decoding module to obtain a fifth spatiotemporal denoised feature map, performing a first decoding on the fifth spatiotemporal denoised feature map, and transmitting the first decoded fifth spatiotemporal denoised feature map to the second spatiotemporal feature decoding module; The fifth spatiotemporal denoised feature map after the first decoding is added to the second spatiotemporal denoised feature map by the second spatiotemporal feature decoding module to obtain a sixth spatiotemporal denoised feature map, and the sixth spatiotemporal denoised feature map is decoded for a second time, and the sixth spatiotemporal denoised feature map obtained by the second decoding is transmitted to the third spatiotemporal feature decoding module; Adding the sixth spatiotemporal denoising feature map decoded for the second time to the first spatiotemporal denoising feature map through the third spatiotemporal feature decoding module to obtain a seventh spatiotemporal denoising feature map, performing a final decoding on the seventh spatiotemporal denoising feature map, and transmitting the finally decoded seventh spatiotemporal denoising feature map to the standard 1×3×3 convolutional layer; The dimension of the seventh spatiotemporal denoising feature map finally decoded is adjusted to three channels through the standard 1×3×3 convolutional layer to obtain three consecutive frames of enhanced particle image denoising feature maps, and the three consecutive frames of enhanced particle image denoising feature maps are separated in the time dimension to obtain three consecutive frames of denoised particle images.

6. The multi-frame particle image denoising method based on spatiotemporal feature interaction according to claim 5, characterized in that: The first spatiotemporal feature encoding module, the second spatiotemporal feature encoding module, the third spatiotemporal feature encoding module and the fourth spatiotemporal feature encoding module each include: a first 1×3×3 convolutional layer, a first LeakReLU activation layer and a first 3D convolutional group; The first 3D convolution group includes a first 1×1×1 convolution layer, a second 1×3×3 convolution layer, a second LeakReLU activation layer, a first 3×1×1 convolution layer, a first BatchNorm3d layer, a third 1×3×3 convolution layer, a third LeakReLU activation layer, a first 3×3×3 convolution layer, a fourth LeakReLU activation layer, and a second BatchNorm3d layer. The process of each spatiotemporal feature encoding module processing the input feature map includes: After adjusting the dimension of the spatial denoising feature map through the first 1×3×3 convolutional layer, it is transmitted to the first LeakReLU activation layer; The output of the first 1×3×3 convolutional layer is nonlinearly activated by the first LeakReLU activation layer and then transmitted to the first 3D convolution group; The output of the first LeakReLU activation layer is passed to the second 1×3×3 convolutional layer after performing information exchange between feature channels in time and space through the first 1×1×1 convolutional layer in the first 3D convolutional group. The output of the first 1×1×1 convolutional layer is subjected to standard convolution processing in the spatial dimension by the second 1×3×3 convolutional layer, and information interaction between channels in the temporal dimension is performed before being transmitted to the second LeakReLU activation layer; The output of the second 1×3×3 convolutional layer is nonlinearly activated by the second LeakReLU activation layer and then transmitted to the first 3×1×1 convolutional layer; The output of the second LeakReLU activation layer is subjected to standard convolution processing in the time dimension and information interaction between channels in the spatial dimension through the first 3×1×1 convolutional layer, and then transmitted to the first BatchNorm3d layer; The output of the first 3×1×1 convolutional layer is batch normalized in time and space dimensions by the first BatchNorm3d layer and then transmitted to the third 1×3×3 convolutional layer; The output of the first BatchNorm3d layer is subjected to standard convolution processing in the spatial dimension and information interaction between channels in the temporal dimension through the third 1×3×3 convolutional layer, and then transmitted to the third LeakReLU activation layer; After performing nonlinear activation processing on the output of the third 1×3×3 convolutional layer through the third LeakReLU activation layer, the output is transmitted to the first 3×3×3 convolutional layer; The output of the third LeakReLU activation layer is subjected to standard convolution processing in time and space by the first 3×3×3 convolutional layer, and then transmitted to the fourth LeakReLU activation layer; After performing nonlinear activation processing on the output of the first 3×3×3 convolutional layer through the fourth LeakReLU activation layer, the output is transmitted to the second BatchNorm3d layer; The output of the fourth LeakReLU activation layer is batch normalized in time and space dimensions by the second BatchNorm3d layer; The output of the second BatchNorm3d layer is added to the output of the first 3×1×1 convolutional layer to obtain the corresponding spatiotemporal denoising feature map.

7. The multi-frame particle image denoising method based on spatiotemporal feature interaction according to claim 6, characterized in that: The fourth spatiotemporal feature encoding module further includes: a second 1×1×1 convolutional layer and a fifth LeakReLU activation layer; The process of the fourth spatiotemporal feature encoding module processing the input feature map further includes: The output of the first 3D convolution group in the fourth spatiotemporal feature encoding module is subjected to information exchange between feature channels in time and space by the second 1×1×1 convolution, and then transmitted to the fifth LeakReLU activation layer; The output of the second 1×1×1 convolution is subjected to nonlinear activation processing through the fifth LeakReLU activation layer to obtain the fourth spatiotemporal denoising feature map.

8. The multi-frame particle image denoising method based on spatiotemporal feature interaction according to claim 7, characterized in that: The first and second spatiotemporal feature decoding modules each include a second 3D convolution group, a third 1×1×1 convolution layer, and a sixth LeakReLU activation layer; The process of each spatiotemporal feature decoding module processing the input spatiotemporal denoising feature map includes: After processing the input spatiotemporal denoising feature map through the second 3D convolution group, it is transmitted to the third 1×1×1 convolution layer; After the output of the second 3D convolution group is subjected to information exchange between feature channels in time and space through the third 1×1×1 convolution layer, it is transmitted to the sixth LeakReLU activation layer; After the output of the third 1×1×1 convolutional layer is processed by the sixth LeakReLU activation layer with nonlinear activation, the corresponding spatiotemporal denoising feature map is obtained.

9. The multi-frame particle image denoising method based on spatiotemporal feature interaction according to claim 8, characterized in that: The third spatiotemporal feature decoding module includes a third 3D convolution group, a fourth 1×3×3 convolution layer, a seventh LeakReLU activation layer, a second 3×3×3 convolution layer and a third BatchNorm3d layer; The process of the third spatiotemporal feature decoding module processing the seventh spatiotemporal denoising feature map includes: After processing the seventh spatiotemporal denoising feature map through the third 3D convolution group, the map is transmitted to the fourth 1×3×3 convolution; The output of the third 3D convolution group is subjected to standard convolution processing in the spatial dimension and information interaction between channels in the temporal dimension by the fourth 1×3×3 convolution layer, and then transmitted to the seventh LeakReLU activation layer; After performing nonlinear activation processing on the output of the fourth 1×3×3 convolutional layer through the seventh LeakReLU activation layer, it is transmitted to the second 3×3×3 convolutional layer; The output of the seventh LeakReLU activation layer is subjected to standard convolution processing in both time and space by the second 3×3×3 convolutional layer, and then passed to the third BatchNorm3d layer; After batch normalization processing is performed on the output of the second 3×3×3 convolutional layer in the time and space dimensions through the third BatchNorm3d layer, the seventh spatiotemporal denoising feature map that is finally decoded is obtained.

Images