Ocean particle image velocity measurement method based on unsupervised learning and multi-scale deep convolution

Abstract

The invention belongs to the field of computer vision, and particularly relates to an ocean particle image velocity measurement method based on unsupervised learning and multi-scale deep convolution. Based on a classical optical flow network LiteFlowNet, in combination with a multi-scale Inception deep convolution module, the ability of the network to extract complex flow field features is enhanced. Meanwhile, a sub-pixel up-sampling layer is adopted to replace a traditional bilinear interpolation method, so that the reconstructed flow field is finer. Besides, the network is trained through unsupervised loss, dependence on a large-scale annotation data set is eliminated, structural similarity loss is fused in an unsupervised loss function, so that image structure information is more effectively reserved, and the precision of optical flow field estimation is further improved. The device can be used for detecting a seabed flow field, and plays a key role in ocean development and benthic organism research.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and particularly relates to an ocean particle image velocimetry method based on unsupervised learning and multi-scale deep convolution, which can be used for real-time detection of seabed flow fields and plays a key role in ocean development and seabed biology research. Background Art

[0002] Traditional PIV methods mainly achieve flow field reconstruction based on cross-correlation algorithms and optical flow algorithms. The cross-correlation algorithm obtains velocity vectors by calculating the correlation of query windows for particle image pairs at two moments. However, it assumes the consistency of particle motion within the query window, which often fails in flow fields with large velocity gradients, resulting in insufficient reconstruction accuracy. In addition, the calculation speed of the cross-correlation algorithm is relatively slow, restricting its application in real-time measurement. The optical flow algorithm estimates the velocity of each pixel point based on the assumptions of brightness consistency and spatial continuity, and can achieve the reconstruction of a dense flow field. However, the optical flow algorithm is vulnerable to imaging noise interference, and performs well under the assumption of small displacements, but has limited processing ability for large displacement scenarios.

[0003] In recent years, deep learning methods have brought new breakthroughs to the development of PIV technology. PIV algorithms based on deep learning significantly improve the accuracy and efficiency of velocity field reconstruction by automatically extracting image features. For example, Rabault et al. first proposed a PIV method based on a convolutional neural network (CNN) and verified its effectiveness in synthetic flow fields. Subsequently, Lee et al. designed the PIV-DCNN model, which further improved the accuracy and adaptability of velocity estimation through a four-layer convolutional network. The PIV-FlowNet model proposed by Cai et al., which combines the advantages of deep learning and optical flow algorithms, has made significant progress in dense flow field prediction. Lento et al. synthesized a training particle image dataset from a random displacement field and used a convolutional neural network to learn the kinematic relationship between position and velocity, solving the problem of limited flow field categories and outperforming the traditional cross-correlation method in terms of accuracy. Guo et al. designed the cascaded network CascLiteFlowNet-R-en, which combines time-resolved particle image velocimetry with deep learning, significantly improving the calculation accuracy and efficiency. Although these supervised learning methods perform excellently on synthetic datasets, they highly rely on synthetic data with ground truth, and the acquisition and annotation of real experimental data are difficult, resulting in limited generalization ability of the model in practical applications.

[0004] To overcome the above problems, unsupervised deep learning methods have gradually attracted attention. Different from supervised learning that relies on ground truth, unsupervised methods construct a loss function based on self-driven regularization, enabling the model to directly learn flow field information from unlabeled data. For example, Zhang et al. proposed the UnLiteFlowNet-PIV model, which applied LiteFlowNet to PIV flow field reconstruction and combined an unsupervised learning strategy to improve the adaptability to real flow fields. Yu et al. proposed an unsupervised learning-based method to solve the problem of difficult access to standard data in real scenarios by training the FlowNet neural network. This method uses photometric loss (based on temporal photometric constancy) and smooth loss to replace the traditional supervised learning loss function. Although unsupervised methods reduce the dependence on synthetic data, the consistency assumption in their training process limits the performance of the model in complex flow fields. Especially in cases with a lot of noise or severe particle occlusion, the improvement of the model's accuracy and robustness remains a major problem. The unsupervised method in the patent "An Unsupervised Particle Image Velocimetry Model Based on Unsupervised Learning and Attention Mechanism" does not consider the structural similarity loss, and the backbone network of the model is FlowNet2SD, which is not lightweight, making it difficult to transplant the algorithm to the final application hardware system. Summary of the Invention

[0005] Although existing unsupervised particle image velocimetry (PIV) methods overcome the deficiency of supervised learning methods that rely on large-scale real flow field data, the reconstruction accuracy is poor. To solve this problem, the present invention proposes a novel unsupervised particle image velocimetry model based on deep convolution and sub-pixel upsampling. This model is based on the classic lightweight optical flow network LiteFlowNet and combines the multi-scale Inception deep convolution module we proposed to enhance the network's ability to extract complex flow field features. At the same time, a sub-pixel upsampling layer is used to replace the traditional bilinear interpolation method, making the reconstructed flow field more refined. In addition, the network is trained with an unsupervised loss, getting rid of the dependence on large-scale labeled datasets, and fusing the structural similarity loss in the unsupervised loss function to more effectively retain image structure information, thereby further improving the accuracy of optical flow field estimation. It can be used for the detection of submarine flow fields, which plays a key role in ocean development and submarine biological research.

[0006] Aiming at the deficiencies of the existing technology, the present invention proposes an ocean particle image velocimetry method based on unsupervised learning and multi-scale deep convolution, including the following steps:

[0007] S1: Obtain a dataset, where the dataset contains two adjacent frames of particle image pairs I1, I2 under flow field conditions;

[0008] S2: Construct the UnLiteFlowNet-MSI-SU neural network with LiteFlowNet as the backbone network, including an encoder network and a decoder network, denoted as NetC and NetE respectively;

[0009] NetC is composed of an improved depth convolution module and six convolution modules. They are named the first to the sixth convolution modules in the order from front to back. The first, fifth, and sixth convolution modules have the same structure, each containing a convolution layer and an activation function layer, where the activation function layer follows immediately after the convolution layer; the second convolution module contains three convolution layers and three activation function layers, and the convolution layers and activation function layers are connected in sequence, with an activation function layer following each convolution layer; the third and fourth convolution modules have the same structure, each containing two convolution layers and two activation function layers, and an activation function layer is connected after each convolution layer; between the first and the second convolution modules, the improved depth convolution module is set; the second, third, fourth, fifth, and sixth convolution modules are connected in sequence;

[0010] The improved depth convolution module consists of six branches. The first three branches respectively use depth convolution kernels of 1×1, 3×3, and 5×5 to extract local features; the fourth and fifth branches reduce the computational cost and accelerate training by decomposing the large kernel 11×11 into two convolution operations of 1×11 and 11×1; the sixth branch is an identity mapping branch for retaining the global information of the input; by concatenating the outputs of these branches in the channel dimension, the module realizes efficient multi-scale feature fusion; NetC is a flow convolutional neural network with shared weights, used to extract the features of two images; NetC has a pyramid structure;

[0011] Set NetE after NetC, which is used to further process the features of the two images extracted by NetC;

[0012] NetE consists of 7 layers. In the first 6 layers, each layer performs an upsampling operation and flow field regularization on the predicted flow field result of the previous layer, and realizes the step-by-step inference and refinement of the optical flow at each layer; the last layer introduces a sub-pixel upsampling convolutional layer, which is used to upsample the final optical flow estimation result to the original image resolution.

[0013] The input for the flow field velocity estimation from I1 to I2 is (I1, I2), and the input for the flow field velocity estimation from I2 to I1 is (I2, I1).

[0014] S3: Construct an unsupervised loss function, and train the UnLiteFlowNet-MSI-SU neural network in S2 with the dataset in S1.

[0015] Preferably, step S3 specifically includes the following steps:

[0016] Construct the unsupervised loss function L(I1, I2, F f , F b ) = λ P L P + λ S L S + λ SSIM L SSIM + λ C L C ;

[0017] Among them, L P represents the photometric loss, L S represents the smoothness loss, L SSIM represents the structural similarity loss, L C represents the consistency loss;

[0018] Take two adjacent frame particle image pairs I1, I2 in the dataset as the input of the UnLiteFlowNet-MSI-SU neural network, and then predict the forward flow field velocity estimation F f = (u f , v f ) T , where u f , v f are the velocity field components in the horizontal and vertical directions of the forward flow field respectively;

[0019] Exchange the order of the image pairs input to the UnLiteFlowNet-MSI-SU neural network to obtain the reverse flow field velocity estimation F b = (u b , v b ) T , where u b , v b are the velocity field components in the horizontal and vertical directions of the reverse flow field respectively;

[0020] The photometric loss is composed of the loss of forward estimation and the loss of reverse estimation. The specific formula is as follows:

[0021] L p (I1, I2, F f , F b ) = ∑ X∈P ρ(I1(X) - I2(X + F f (X))) + ρ(I2 - I1(X + F b (X)))

[0022] where P is a function that maps each pixel in the image to its grayscale value, ρ() is the generalized Charbonnier penalty function, ρ = (x 2 +ε 2 ) γ , and ρ() is a differentiable convex function;

[0023] The smoothing loss uses a second-order smoothing constraint, and the specific formula is as follows:

[0024]

[0025] where N represents four-channel filters, which are in the x direction, y direction, and two diagonal directions respectively. The convolution of two flow components, namely u in the x direction and v in the y direction, is calculated using the four-channel filters respectively, and then the Charbonnier loss is calculated;

[0026] The structural similarity loss function is defined as:

[0027]

[0028] where P is the set of pixels in the image, SSIM(I1, I2) represents the structural similarity index, which is used to measure the structural similarity between two images, F f and F b are the forward and backward optical flow fields respectively. The calculation of SSIM is based on three factors: the brightness, contrast, and structure of the image, and the formula is as follows:

[0029]

[0030] where and are the local averages of images I1 and I2 respectively, and are the local standard deviations of the images, is the covariance of the two images, and C1 and C2 are constants. Usually, C1 = 10 -4 and C2 = 10 -4 to avoid a zero denominator.

[0031] The consistency loss function is defined as:

[0032] L C (F f ,F b ) = ∑ X∈P ρ(F f +F b (X+F f ))+ρ(F b +F f (X+F b ))

[0033] Preferably, set γ to 0.45 and ε to 10 -3 .

[0034] Preferably, in step S1, the dataset contains pairs of particle images under flow field conditions, and the flow field conditions specifically include the following flow field types: backward flow, cylindrical flow, DNS turbulence, SQG ocean surface flow, JHTDB-channel channel flow;

[0035] Among them, DNS turbulence represents a uniform and isotropic flow; SQG ocean surface flow represents an ocean surface flow driven by the ocean surface quasi-geostrophic model; JHTDB-channel channel flow represents a channel flow in the Johns Hopkins Turbulence Database.

[0036] Preferably, in step S2, the upsampling operation uses the deconvolution upsampling method, and the flow field regularization corrects the flow field based on the difference between the two input images.

[0037] The beneficial effects of the present invention are as follows:

[0038] The present invention proposes an unsupervised particle image velocimetry model based on deep convolution and sub-pixel upsampling, strengthens multi-scale feature extraction, and finally applies it to the model, thus improving the model and enhancing the model's ability to handle complex flow field environments. The proposed unsupervised model optimized by multi-scale deep convolution has obvious competitiveness compared with both traditional PIV methods and recent supervised learning-based methods, and is even superior to them in some complex flow field environments, and has potential advantages and prospects in real environment operations.

[0039] This article uses LiteFlowNet as the basic network architecture of the model, uses an improved multi-scale deep convolution module to improve the feature extraction ability of the convolutional neural network, uses an unsupervised learning method to solve the problem that there is no standard answer in the dataset in the real scenario and the problem that the synthetic dataset cannot be well applied to the real scenario, and obtains a high accuracy rate. Description of the Drawings

[0040] Figure 1 is the structural diagram of the multi-scale deep convolution module used in the present invention;

[0041] Figure 2 is the unsupervised loss calculation process in the present invention; Detailed Embodiments

[0042] The present invention will be further described below with reference to the accompanying drawings.

[0043] The present invention adopts an unsupervised learning method to solve the problem of no real results, proposes an unsupervised particle image velocimetry model based on deep convolution and sub-pixel upsampling, and uses multi-scale deep convolution to improve the feature extraction ability of the model. It can be used for the detection of submarine flow fields and plays a key role in ocean development and submarine biological research. It includes the following steps:

[0044] Step 1: Prepare the dataset and perform preprocessing

[0045] The dataset used in this paper is provided in the paper by Cai et al. The dataset is a simulation dataset, which is a pair of particle images under various flow field conditions artificially generated according to computational fluid dynamics. The dataset contains pairs of particle images under 5 different types of flow field conditions, including backward flow, cylinder flow, etc. 90% of the data is used for training in this paper, and the other 10% is used for testing.

[0046] Step 2: Customize the unsupervised loss function

[0047] We use two adjacent frames of particle image pairs I1 and I2 as the input of the neural network, and then predict the forward flow field F from I1 to I2 through the neural network f =(u f ,v f ) T , where u f ,v f are the velocity field components in the horizontal and vertical directions respectively.

[0048] In our unsupervised loss, the reverse prediction from the second image to the first image will be used, which we define as: F b =(u b ,v b ) T .

[0049] (2.1) Photometric loss

[0050] The photometric loss is related to the difference between the first image of the forward estimated flow field and the warped second image and the difference between the second image of the reverse estimated flow field and the warped first image. Therefore, the bidirectional photometric loss is defined as the sum of the two:

[0051] L p (I1, I2, F f , F b ) = ∑ x∈P ρ(I1(X) - I2(X + F f (X))) + ρ(I2 - I1(X + F b (X)))

[0052] where \(P\) is a function that maps each pixel in the image to its grayscale value, \(\rho(\cdot)\) is the generalized Charbonnier penalty function, \(\rho=(x 2 +\varepsilon 2 ) γ , which is a differentiable convex function with good robustness. In this paper, we set \(\gamma = 0.45\) and \(\varepsilon = 10 -3 .

[0053] (2.2) Smooth Loss

[0054] Some particles in the image are incomplete. Some particles are at the edge of the first-frame image but not in the second-frame image. There are also some particles that are not in the first-frame image but are at the edge of the second-frame image. To solve this problem, we introduce the smooth loss. To enhance the regularization effect, we use the second-order smooth constraint:

[0055]

[0056] where \(N\) represents four-channel filters in the \(x\)-direction, \(y\)-direction, and two diagonal directions. Therefore, the process here first calculates the convolutions of the two flow components (\(u\) in the \(x\)-direction and \(v\) in the \(y\)-direction) using the four-channel filters respectively, and then calculates their Charbonnier losses.

[0057] (2.3) Structural Similarity Loss

[0058] The structural similarity loss is used to measure the structural similarity between the remapped image and the original image, aiming to enhance the image reconstruction quality, especially to retain the structural features of the image. The SSIM loss can be defined as:

[0059]

[0060] where \(P\) is the set of pixels in the image, \(SSIM(I_1, I_2)\) represents the structural similarity index, used to measure the structural similarity between two images, \(F f and \(F b are the forward and backward optical flow fields respectively. The calculation of SSIM is based on three factors: the brightness, contrast, and structure of the image, and the formula is as follows:

[0061]

[0062] where, and are the local averages of images \(I_1\) and \(I_2\) respectively, and are the local standard deviations of the images, is the covariance of the two images, and \(C_1\) and \(C_2\) are constants, usually taking \(C_1 = 10 -4and C2 = 10 -4 To avoid a zero denominator.

[0063] This structural similarity loss evaluates the structural differences between the remapped image and the original image by calculating the SSIM value in each local region of the image. By maximizing the structural similarity, the model can preserve the visual structure of the original image when reconstructing the image.

[0064] (2.4) Consistency loss

[0065] In the case of absolute correctness, the forward estimate and the backward estimate should be consistent, i.e., F f and F b are numbers with the same absolute value but opposite signs. Therefore, the sum of the two should be zero. Thus, the consistency loss function can be defined as:

[0066] L C (F f , F b ) = ∑ X∈P ρ(F f + F b (X + F f )) + ρ(F b + F f (X + F b ))

[0067] (2.5) Final loss function

[0068] The final loss function is the sum of the above three loss functions multiplied by their corresponding weights, and the function is defined as:

[0069] L(I1, I2, F f , F b ) = λ P L P + λ S L S + λ SSIM λ SSIM + λ C L C

[0070] Step 3: The UnLiteFlowNet-MSI-SU neural network performs image feature extraction

[0071] (3.1) Improved multi-scale depth convolution

[0072] The improved deep convolutional module consists of five branches. The first three branches respectively adopt depthwise convolutional kernels of 1×1, 3×3, and 5×5 to extract local features. Another branch reduces the computational cost and accelerates training by decomposing the large kernel of 11×11 into two convolutional operations of 1×11 and 11×1. The last branch is the identity mapping branch, which retains the global information of the input. By concatenating the outputs of these branches in the channel dimension, the module achieves efficient multi-scale feature fusion. The specific structure is as Figure 1 shown.

[0073] (3.2) Internal Structure of Unsupervised Loss

[0074] Different from the previous losses based on supervised learning, the model proposed in this paper (UnLiteFlowNet-MSI-SU) not only predicts the velocity from the first frame to the second frame for a pair of adjacent particle image frames as input, but also predicts the velocity from the second frame to the first frame, that is, forward prediction and backward prediction. As Figure 2 shown, the red part represents forward prediction, and the blue part performs backward prediction. Among them, we use a backbone network with shared weights, and we finally obtain the unsupervised loss using the results of forward prediction and backward prediction.

[0075] (3.4) Neural Network Structure

[0076] The model in this paper is named UnLiteFlowNet-MSI-SU, and the backbone network used is LiteFlowNet. It uses a flow convolutional neural network with shared weights (NetC) to extract the features of two images. NetC has a pyramid structure. Then the decoder (NetE) performs cascaded flow inference with flow regularization, uses the deconvolution upsampling method to refine the flow field, and then regularizes the flow field based on the image difference. In the last layer, a sub-pixel upsampling layer is used to upsample the final flow estimate to the original resolution. The input for forward flow estimation is (I1, I2), and the backward flow is (I2, I1).

[0077] Step 4: Comparative Analysis of Experimental Results

[0078] (4.1) Evaluation Metrics

[0079] This paper uses the average endpoint error to measure the accuracy of the model's flow field prediction. To make the comparison results clearer, this paper sets the error unit of AEE to pixels of 100 pixels. AEE can be described as the L2 norm of the difference between the flow field prediction result Fe and the standard result Fg. The formula is as follows:

[0080] AEE = ||F e - F g ||2

[0081] (4.2) Comparison between the model of the present invention and the traditional PIV method

[0082] Table 1: Experimental results compared with the traditional PIV method

[0083]

[0084] The experimental results are shown in Table 1. Compared with the traditional correlation-based PIV method (such as WIDIM), the UnLiteFlowNet-MSI-SU network proposed by us shows higher accuracy in almost all flow scenarios, especially in the more challenging DNS turbulence and SQG flow scenarios. Although it does not exceed the Horn-Schunck (HS) optical flow method in some scenarios, the error gap is relatively small.

[0085] It should be noted that the HS optical flow method requires a large amount of computational time during the optimization process, especially when dealing with multiple sets of images, and its efficiency is relatively low. Our network only takes 130 seconds to process 500 sets of images with a size of 256×256 on an Nvidia RTX A4000 GPU. In contrast, the HS optical flow method takes about 600 seconds, and the WIDIM method (window size of 29×29) takes about 212 seconds on an Intel Core i7-7700 CPU. Therefore, compared with traditional methods, the deep learning-based PIV method shows significant advantages in terms of efficiency.

[0086] (4.3) Comparison between the model of the present invention and other supervised deep learning methods

[0087] Table 2: Experimental results compared with supervised deep learning methods

[0088]

[0089] As shown in Table 2, UnLiteFlowNet-MSI-SU exhibits a lower average absolute error (AEE) on multiple datasets. Especially on the Cylinder, DNS_turbulence, and SQG datasets, its performance is better than most supervised learning methods. On the Cylinder dataset, the AEE of UnLiteFlowNet-MSI-SU is 5.63, significantly lower than that of PIV-FlowNetS-en (15.58); on the DNS_turbulence dataset, the AEE of UnLiteFlowNet-MSI-SU is 14.21, significantly lower than that of PIV-LiteFlowNet (19.73) and PIV-FlowNetS-en (28.22). On the SQG dataset, the AEE of UnLiteFlowNet-MSI-SU is 16.42, significantly better than most supervised learning methods.

[0090] On the JHTDB_channel dataset, the AEE of UnLiteFlowNet-MSI-SU is 11.78, lower than that of PIV-FlowNetS-en (15.51) and PIV-NetS (15.63). This result indicates that UnLiteFlowNet-MSI-SU can maintain high precision when dealing with scenarios with complex flow features and outperforms supervised methods in some cases.

[0091] However, on the Back-step dataset, the AEE of UnLiteFlowNet-MSI-SU is 7.97, slightly inferior compared to most supervised learning methods. The main reason for this gap may be that the model relies on an unsupervised learning framework for training, which makes it unable to accurately capture the physically correct displacements when dealing with some flow scenarios with strong gradient changes and detailed features, especially on datasets like Back-step.

[0092] Although UnLiteFlowNet-MSI-SU outperforms supervised learning methods on most datasets, there is still a certain gap compared with the more advanced supervised network PIV-LiteFlowNet-en. This gap mainly stems from the limitations of unsupervised learning. The training of UnLiteFlowNet-MSI-SU relies on unsupervised loss functions such as photometric loss, consistency loss, and structural similarity loss. Although these loss functions can effectively reduce the inconsistency of the optical flow field, in some regions with strong physical gradient changes, they may lead to an underestimation of the optical flow displacement. In contrast, as a supervised method, PIV-LiteFlowNet-en can more accurately approximate the actual physical displacement by directly optimizing the L1 norm between the ground truth optical flow and the predicted optical flow, especially performing more excellently in flow fields with stronger gradient changes. Unsupervised methods mainly rely on the matching of image features to predict optical flow, and the particle features in PIV images have high similarity, which easily leads to uncertainties in local matching. In this case, the supervised learning method of PIV-LiteFlowNet-en can better overcome these ambiguities and provide more accurate displacement predictions.

[0093] Overall, UnLiteFlowNet-MSI-SU demonstrates great potential in the unsupervised learning framework, especially showing significant advantages in complex flow and turbulent scenarios.

[0094] (4.4) Comparison between the model of the present invention and other unsupervised deep learning methods

[0095] Table 3: Experimental results of comparison with unsupervised deep learning methods

[0096]

[0097]

[0098] We comprehensively compared the proposed network with other PIV neural networks based on unsupervised learning, and the results are shown in Table 3. It can be seen from the table that the average absolute error (AEE) of the proposed UnLiteFlowNet-MSI-SU network on all five test datasets is significantly lower than that of other methods, showing superior performance.

[0099] On the Cylinder dataset, the AEE of UnLiteFlowNet-MSI-SU is 5.63, significantly better than 7.33 of LiteFlowNet3 and 7.16 of UnLiteFlowNet. Compared with the relatively better-performing UnLiteFlowNet-en (6.57), our network still shows excellent performance. This indicates that our network has higher accuracy and robustness in processing complex flow features. On the Back-step dataset, the AEE of UnLiteFlowNet-MSI-SU is 7.97, also lower than all other unsupervised methods. In particular, compared with LiteFlowNet3 (9.53) and PIV-LiteFlowNet3-AL (9.32), the errors are reduced by 16.4% and 14.5% respectively. This improvement reflects the advantage of the network in capturing complex boundary layer flow characteristics. For the JHTDB channel dataset, the AEE of UnLiteFlowNet-MSI-SU is 11.78, while the errors of other methods are all greater than 13. This further shows that our network has significant improvements in the accuracy of flow field reconstruction and the capture of details, especially in highly regular scenarios such as channel flows. On the DNS turbulence dataset, UnLiteFlowNet-MSI-SU performs excellently again, with an AEE of 14.21, significantly better than 20.88 of LiteFlowNet3. This shows that in flow fields with higher complexity and turbulence characteristics, UnLiteFlowNet-MSI-SU can more accurately recover particle motion characteristics, improving the quality of turbulence simulation. Finally, on the SQG dataset, the AEE of UnLiteFlowNet-MSI-SU is 16.42, lower than all other comparison methods. Compared with UnLiteFlowNet-en (18.45), it is reduced by 11%. This indicates that our method has higher accuracy in simulating the particle motion characteristics of the quasi-geostrophic vorticity field.

[0100] Overall, UnLiteFlowNet-MSI-SU shows the lowest errors on all five test datasets, significantly outperforming other unsupervised PIV networks. This excellent performance benefits from the design of the multi-scale Inception deep convolutional block and the sub-pixel upsampling layer in our network. These modules enhance the network's ability to perceive multi-scale features and improve the resolution of the output flow field. In addition, the network can more effectively adapt to various flow field characteristics, including simple regular flows, complex turbulences, and boundary layer effects, etc., thus achieving a comprehensively leading performance among unsupervised networks. These results indicate that UnLiteFlowNet-MSI-SU has broad application prospects in actual flow field measurement and simulation.

Claims

1. Ocean particle image velocimetry method based on unsupervised learning and multi-scale deep convolution, characterized in that: The following steps are involved: S1: Acquire a data set, wherein the data set includes two adjacent frames of particle image pairs I1 and I2 under flow field conditions; S2: Construct the UnLiteFlowNet-MSI-SU neural network with LiteFlowNet as the backbone network, including the encoder network and the decoder network, denoted as NetC and NetE respectively; NetC is composed of an improved deep convolution module and 6 convolution modules. The first, fifth and sixth convolution modules have the same structure, each including a convolution layer and an activation function layer, wherein the activation function layer is followed by the convolution layer; the second convolution module includes three convolution layers and three activation function layers, the convolution layers and the activation function layers are connected in sequence, and each convolution layer is followed by an activation function layer; the third and fourth convolution modules have the same structure, each including two convolution layers and two activation function layers, and each convolution layer is followed by an activation function layer; the improved deep convolution module is arranged between the first and second convolution modules; The second, third, fourth, fifth, and sixth convolution modules are connected in sequence; The improved deep convolution module consists of six branches, and the first three branches use 1×1, 3×3, and 5×5 deep convolution kernels to extract local features. The fourth and fifth branches reduce computational cost and accelerate training by decomposing the large kernel 11×11 into two convolution operations of 1×11 and 11×1. The sixth branch is the identity mapping branch, which is used to preserve the global information of the input. By concatenating the outputs of these branches in the channel dimension, the module achieves efficient multi-scale feature fusion. The NetE is arranged after the NetC; The NetE consists of 7 layers. In the first 6 layers, each layer performs an upsampling operation and flow field regularization on the predicted flow field results of the previous layer; the last layer introduces a sub-pixel upsampling convolution layer; S3: Construct an unsupervised loss function and train the UnLiteFlowNet-MSI-SU neural network described in S2 using the dataset described in S1.

2. The method for measuring the speed of ocean particles using images based on unsupervised learning and multi-scale deep convolution as claimed in claim 1, characterized in that: Step S3 specifically includes the following steps: Construct the unsupervised loss function L(I1,I2,F f ,F b )=λ P L P +λ S L S +λ SSIM L SSIM +λ C L C ; Among them, L P Represents the luminosity loss, L S represents the smoothing loss, L SSIM represents the structural similarity loss, L C represents the consistency loss; The two adjacent particle image pairs I1 and I2 in the data set are used as the input of the UnLiteFlowNet-MSI-SU neural network, and then the forward flow field velocity estimate F from I1 to I2 is predicted by the UnLiteFlowNet-MSI-SU neural network. f =(u f ,v f ) T , where u f ,v f are the velocity field components in the horizontal and vertical directions in the forward flow field respectively; Swap the order of the image pairs input into the UnLiteFlowNet-MSI-SU neural network to obtain the reverse flow field velocity estimate F from I2 to I1 b =(u b ,v b ) T , where u b ,v b are the velocity field components in the horizontal and vertical directions in the reverse flow field respectively; The photometric loss consists of the forward estimated loss and the reverse estimated loss, and the specific formula is as follows: Where P is the function that maps each pixel in the image to its grayscale value, ρ() is the generalized Charbonnier penalty function, ρ=(x 2 +ε 2 ) γ , ρ() is a differentiable convex function; The smooth loss uses a second-order smooth constraint, and the specific formula is as follows: Among them, N represents 4 channel filters, namely the x direction, the y direction and the two diagonal directions. Four channel filters are used to calculate the convolution of the two flow components, namely u in the x direction and v in the y direction, and then the Charbonnier loss is calculated; The structural similarity loss function is defined as: Where P is the set of pixels in the image, SSIM(I1,I2) represents the structural similarity index, which is used to measure the structural similarity of two images, and F f and F b They are forward and reverse optical flow fields respectively; the calculation of SSIM is based on three factors: image brightness, contrast and structure. The formula is as follows: in, and are the local averages of images I1 and I2, and is the local standard deviation of the image, is the covariance of the two images, C1 and C2 are constants, usually C1 = 10 -4 and C2 = 10 -4 To avoid zero denominator; The consistency loss function is defined as: L C (F f ,F b )=∑ X∈P ρ(F f +F b (X+F f ))+ρ(F b +F f (X+F b ))。 3. The method for measuring ocean particle image velocimetry based on unsupervised learning and multi-scale deep convolution as claimed in claim 2, characterized in that: Set γ to 0.45 and ε to 10 -3 .

4. The method for measuring ocean particle image velocimetry based on unsupervised learning and multi-scale deep convolution as claimed in claim 2, characterized in that: In step S1, the data set includes particle image pairs under flow field conditions, and the flow field conditions specifically include the following flow field types: receding flow, cylindrical flow, DNS turbulence, SQG sea surface flow, and JHTDB-channel flow; Among them, DNS turbulence represents uniform and isotropic flow; SQG sea surface flow represents the sea surface flow driven by the quasi-geostrophic model; JHTDB-channel channel flow represents the channel flow in the Johns Hopkins Turbulence Database.

5. The method for measuring ocean particle image velocimetry based on unsupervised learning and multi-scale deep convolution as claimed in claim 2, characterized in that: In step S2, the upsampling operation uses a deconvolution upsampling method, and the flow field regularization corrects the flow field based on the difference between two input images.

Images