River surface flow velocity estimation method and device fusing multi-mode optical flow estimation and PINN

Abstract

The invention provides a river surface flow velocity estimation method and device fusing multi-mode optical flow estimation and PINN, and relates to the technical field of hydrological monitoring, and the method comprises the steps: obtaining meteorological observation data and continuous multi-frame river surface images; performing multi-scale feature processing on the plurality of frames of river surface images to obtain a plurality of related pyramid feature data; carrying out coding processing on the meteorological observation data and at least one frame of image in the continuous multiple frames of river surface images to obtain context mixed feature data; and obtaining an optical flow field according to the multiple pieces of related pyramid feature data and the context mixed feature data, and correcting the optical flow field to obtain a target optical flow field. According to the scheme, high-precision prediction of the river surface flow velocity is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of hydrological monitoring, and particularly relates to a river surface flow velocity estimation method and device fusing multi-modal optical flow estimation and PINN. BACKGROUND

[0002] Accurate monitoring of river flow is an important basis for hydrological analysis, flood warning and water resource management, and the key to monitoring river flow is river flow velocity. In recent years, with the rapid development of computer vision technology, non-contact flow velocity measurement based on video images has become a research hotspot, providing a new technical path for real-time online monitoring of river flow.

[0003] At present, the optical flow method is applied to river surface flow velocity estimation because it can capture pixel-level motion information, but traditional optical flow algorithms face many challenges in complex hydrological scenes, such as water surface reflection, water surface wave texture changes, etc. which will introduce noise, resulting in errors in optical flow estimation. SUMMARY

[0004] The technical problem to be solved by the present application is to provide a river surface flow velocity estimation method and device fusing multi-modal optical flow estimation and PINN. By fusing physical constraints and multi-modal data, high-precision and low-robustness river surface flow velocity monitoring is achieved.

[0005] To solve the above technical problems, the technical solutions of the present application are as follows:

[0006] In a first aspect, the embodiments of the present application provide a river surface flow velocity estimation method fusing multi-modal optical flow estimation and PINN, comprising:

[0007] Obtaining meteorological observation data and continuous multiple frames of river surface images;

[0008] Performing multi-scale feature processing on the continuous multiple frames of river surface images to obtain first related pyramid feature data and second related pyramid feature data;

[0009] Encoding at least one frame of image in the continuous multiple frames of river surface images and the meteorological observation data to obtain context mixed feature data;

[0010] According to the first related pyramid feature data, the second related pyramid feature data and the context mixed feature data, optical flow prediction is performed to obtain an optical flow field, and the optical flow field includes a flow velocity field vector reflecting the river flow velocity;

[0011] The optical flow field is corrected to obtain a target optical flow field.

[0012] Furthermore, multi-scale feature processing is performed on the continuous multiple-frame river surface images to obtain first related pyramid feature data and second related pyramid feature data, including:

[0013] Extracting features from a first frame of the continuous multiple-frame river surface images to obtain a first feature image;

[0014] performing feature extraction on a second frame image from the continuous multiple frames of river surface images to obtain a second feature image;

[0015] performing feature extraction on a third frame image from the continuous multiple frames of river surface images to obtain a third feature image;

[0016] Determining a first correlation pyramid feature according to a correlation between the first feature image and the second feature image;

[0017] A second correlation pyramid feature is determined according to the correlation between the second feature image and the third feature image.

[0018] Furthermore, the meteorological observation data and at least one frame of the continuous multiple-frame river surface images are coded to obtain context mixed feature data, including:

[0019] Physically encoding the meteorological observation data to generate physical feature tensor data having the same resolution as at least one frame of the continuous multiple-frame river surface images;

[0020] The physical feature tensor data is mixed and coded with at least one frame of a continuous multi-frame river surface image to obtain context mixed feature data.

[0021] Furthermore, optical flow prediction is performed based on the first related pyramid feature data, the second related pyramid feature data, and the context mixed feature data to obtain an optical flow field, including:

[0022] Obtaining a first branch vector according to the first correlation pyramid feature data and the context mixed feature data;

[0023] Obtaining a second branch vector according to the second correlation pyramid feature data and the context mixed feature data;

[0024] Obtaining a first target hidden state according to the first branch vector and a first hidden state of a preset network;

[0025] Obtaining a second target hidden state according to the second branch vector and a second hidden state of the preset network;

[0026] Performing optical flow prediction on the first target hidden state to obtain a first optical flow field;

[0027] Perform optical flow prediction on the second target hidden state to obtain a second optical flow field.

[0028] Furthermore, a first branch vector is obtained according to the first related pyramid feature data and the context mixed feature data, including:

[0029] Perform vector concatenation on the first related pyramid feature data and the context mixed feature data to obtain a first branch vector;

[0030] According to the second related pyramid feature data and the context mixed feature data, a second branch vector is obtained, including:

[0031] The second related pyramid feature data and the context mixed feature data are vector-concatenated to obtain a second branch vector.

[0032] Further, obtaining a first target hidden state according to the first branch vector and the first hidden state of the preset network includes:

[0033] Obtaining a first hidden state, a first update parameter, a second update parameter, and a third update parameter of a preset network;

[0034] Performing a convolution operation on the first branch vector, a first hidden state of the preset network, and a first update parameter to obtain a first update value;

[0035] Performing a convolution operation on the first branch vector, the first hidden state of the preset network, and the second update parameter to obtain a second update value;

[0036] Performing a convolution operation on the first branch vector, the first hidden state of the preset network, the second update value, and the third update parameter to obtain a third update value;

[0037] The first hidden state, the first update value, and the third update value of the preset network are operated to obtain a first target hidden state.

[0038] Further, obtaining a second target hidden state according to the second branch vector and the second hidden state of the preset network includes:

[0039] Obtaining a second hidden state, a first update parameter, a second update parameter, and a third update parameter of a preset network;

[0040] performing a convolution operation on the second branch vector, the second hidden state of the preset network, and the first update parameter to obtain a fourth update value;

[0041] performing a convolution operation on the second branch vector, the second hidden state of the preset network, and the second update parameter to obtain a fifth update value;

[0042] performing a convolution operation on the second branch vector, the second hidden state of the preset network, the fifth update value, and the third update parameter to obtain a sixth update value;

[0043] The second hidden state, the fourth update value, and the sixth update value of the preset network are operated to obtain a second target hidden state.

[0044] Furthermore, optical flow prediction is performed on the first target hidden state to obtain a first optical flow field, including:

[0045] Perform multi-layer convolution and upsampling operations on the first target hidden state to obtain a first optical flow field;

[0046] Perform optical flow prediction on the second target hidden state to obtain a second optical flow field, including:

[0047] Perform multi-layer convolution and upsampling operations on the second target hidden state to obtain the second optical flow field.

[0048] Furthermore, the optical flow field is corrected to obtain the target optical flow field, including:

[0049] Get the vorticity ω and stream function;

[0050] According to the vorticity ω and the stream function, the first vorticity and the second vorticity are obtained;

[0051] Correcting the first optical flow field according to the first vorticity and the temperature scalar to obtain a first target optical flow field;

[0052] The second optical flow field is corrected according to the second vorticity and the temperature scalar to obtain the second target optical flow field.

[0053] In a second aspect, an embodiment of the present invention further provides a river surface velocity estimation device integrating multimodal optical flow estimation and PINN, comprising: an acquisition module for acquiring meteorological observation data and continuous multi-frame river surface images;

[0054] A first processing module is used to perform multi-scale feature processing on a plurality of consecutive river surface image frames to obtain first related pyramid feature data and second related pyramid feature data;

[0055] A second processing module is configured to encode the meteorological observation data and at least one frame of the continuous multiple-frame river surface images to obtain context mixed feature data;

[0056] A third processing module is configured to perform optical flow prediction based on the first related pyramid feature data, the second related pyramid feature data, and the context mixed feature data to obtain an optical flow field, where the optical flow field includes a velocity field vector reflecting a river flow velocity;

[0057] The fourth processing module is used to correct the optical flow field to obtain the target optical flow field. The above solution of the present invention has at least the following beneficial effects:

[0058] The above-mentioned scheme of the present invention obtains contextual mixed feature data by encoding meteorological observation data and at least one frame of river surface image, thereby realizing the prediction of river flow field by video image recognition guided by meteorological elements, and ensuring the accuracy and physical consistency of flow field prediction; optical flow prediction is performed based on the first relevant pyramid feature data, the second relevant pyramid feature data and the contextual mixed feature data to obtain the optical flow field, and the optical flow field is corrected, thereby realizing the determination of the optical flow field by cross-modal feature fusion, and realizing high-precision and low-robustness monitoring of river surface flow velocity. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 4 is a flow chart of an embodiment of a method for estimating river surface velocity by fusing multimodal optical flow estimation and PINN according to the present invention;

[0060] Figure 2 is a flow chart of another embodiment of the river surface velocity estimation method integrating multimodal optical flow estimation and PINN of the present invention;

[0061] Figure 3 Schematic diagram of the cross-modal fusion module;

[0062] Figure 4 It is the overall PINN network diagram;

[0063] Figure 5 Schematic diagram of the module of the river surface velocity estimation device integrating multimodal optical flow estimation and PINN of the present invention. DETAILED DESCRIPTION

[0064] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0065] Figure 1 FIG. 1 is a flow chart of an embodiment of a method for estimating river surface velocity by integrating multimodal optical flow estimation and PINN in the present invention. Figure 1 As shown, the method of the embodiment of the present invention includes:

[0066] S101, obtaining meteorological observation data and continuous multi-frame river surface images.

[0067] In some embodiments, continuous multi-frame river surface images can be obtained by monitoring the video data of the river surface. The continuous multi-frame river surface images can be images of all frames in the river surface video data, or can be selected as needed in the river surface video data. The river surface video data can be monitored by shooting the river surface of a fixed area with a fixed shooting angle, shooting distance, and camera parameters (including focal length, exposure time, frame rate, picture resolution, etc.). As an example, the river surface video data can be obtained by a drone, a camera fixedly installed along the river bank or on a bridge. When monitoring the river surface, it is necessary to keep the video image of the river surface area from shaking to avoid affecting the optical flow estimation. As an example, the river surface video data can be high-resolution (such as 1080p and above) and high-frame rate (such as 30 frames per second or above) to capture the dynamic changes of the variable speed water flow and clearly distinguish the river ripples.

[0068] In some embodiments, after acquiring multiple consecutive frames of river surface images, the multiple consecutive frames of river surface images may be preprocessed in sequence:

[0069] (I) Region Cropping: Image cropping is performed to extract the region of interest (ROI). Image cropping is performed to exclude non-water areas such as vegetation and sky, focusing on the river surface area with water surface features related to flow velocity. It is important to note that a smaller ROI facilitates input into the image feature extraction network, reduces the number of pixels fed into the network, and reduces the complexity of the convolution operation.

[0070] (II) Geometric correction: For video data shot from a non-orthogonal view of the river surface, such as a camera installed on the river bank, it is necessary to correct the camera's viewing angle distortion (such as perspective distortion) and project the video onto a uniform plane to facilitate optical flow extraction of the image.

[0071] (III) Image data standardization: facilitates image training convergence and generalization.

[0072] In some embodiments, meteorological observation data is collected at the same time as continuous multi-frame river surface images. The meteorological observation data may include meteorological element data such as temperature, wind speed and direction, humidity, air pressure, light intensity, visibility, etc. These meteorological observation data will directly or indirectly affect the hydrodynamic process of river surface flow velocity and the quality of river surface images.

[0073] In some embodiments, after obtaining the meteorological observation data, the meteorological observation data may be preprocessed:

[0074] (I) Check the data integrity of the collected meteorological observation data and use linear interpolation or spline interpolation to handle missing values ​​and outliers.

[0075] (II) Aggregation or interpolation methods are used to downsample or upsample the temporal resolution of meteorological observation data to align it with the temporal resolution of the continuous multi-frame river surface images, that is, to ensure that the timestamps of meteorological observation data and the continuous multi-frame river surface images are aligned to facilitate subsequent modeling.

[0076] (III) Standardization or normalization preprocessing of variables of different dimensions (such as temperature, wind speed, and humidity) is performed to facilitate multimodal modeling and analysis.

[0077] S102 , performing multi-scale feature processing on a plurality of consecutive frames of river surface images to obtain first related pyramid feature data and second related pyramid feature data.

[0078] In some embodiments, feature extraction can be performed on multiple consecutive frames of river surface images to obtain multiple feature maps. Then, by calculating the inner product of the feature map and performing a pooling operation on the inner product value, first related pyramid feature data and second related pyramid feature data are constructed. As an example, a suitable network model can be selected as needed for feature extraction, such as ResNet, EfficientNet, Recurrent All-Pairs Field Transforms (RAFT), etc. Among them, the RAFT network is a commonly used deep learning architecture for optical flow estimation, which maps the input high-resolution image (i.e., multiple consecutive frames of river surface images) to a low-resolution, high-dimensional feature space; if lightweight deployment is required, it can be considered to reduce the number of backbone layers in the ResNet network or use the EfficientNet network; in an environment with high precision requirements, the number of backbone layers can be enhanced or complex neural networks such as ViT (Vision Transformer) can be used. As an example, the computational complexity can be reduced by constructing pyramid feature data through pooling operations.

[0079] In some embodiments, if an odd number of feature maps are obtained, first correlation pyramid feature data is constructed based on the feature maps from the first to the middle, and second correlation pyramid feature data is constructed based on the feature maps from the middle to the last. If an even number of feature maps are obtained, the feature maps are sequentially divided into two parts, and first correlation pyramid feature data is constructed based on the feature maps of the first part and the first part of the second part, and second correlation pyramid feature data is constructed based on the feature maps of the last part of the first part and the second part.

[0080] As an example, feature extraction is performed on two consecutive river surface images, resulting in two feature maps, denoted as F1 and F2. The inner product of the first feature map, F1, is calculated and pooled to obtain the first correlation pyramid feature data. The inner product of the second feature map, F2, is calculated and pooled to obtain the second correlation pyramid feature data.

[0081] As an example, feature extraction is performed on three consecutive river surface images to obtain three feature maps, denoted as F1, F2, and F3. The inner product of feature map F1 and feature map F2 is calculated and pooled to obtain the first correlation pyramid feature data. The inner product of feature map F3 and feature map F2 is calculated and pooled to obtain the second correlation pyramid feature data.

[0082] As an example, feature extraction is performed on four consecutive river surface images to obtain four feature maps, denoted as F1, F2, F3, and F4. A similarity measure is determined for feature maps F1, F2, and F3, and a pooling operation is performed on the results of the similarity measure to obtain first correlation pyramid feature data. A similarity measure is determined for feature maps F2, F3, and F4, and a pooling operation is performed on the results of the similarity measure to obtain second correlation pyramid feature data.

[0083] As an example, feature extraction is performed on five consecutive river surface images to obtain five feature maps, denoted as F1, F2, F3, F4, and F5. A similarity measure is determined for feature maps F1, F2, and F3, and a pooling operation is performed on the results of the similarity measures to obtain first correlation pyramid feature data. A similarity measure is determined for feature maps F5, F3, and F4, and a pooling operation is performed on the results of the similarity measures to obtain second correlation pyramid feature data.

[0084] In some embodiments, multi-scale feature processing can be performed on multiple consecutive river surface image frames as needed to obtain multiple sets of related pyramid feature data. For example, after extracting feature maps from multiple consecutive river surface image frames, the inner product of the feature maps of two adjacent frames is calculated, and related pyramid features are constructed based on the inner product value.

[0085] S103 , encoding the meteorological observation data and at least one frame of the continuous multiple-frame river surface images to obtain context mixed feature data.

[0086] In some embodiments, if there is one intermediate frame, the image of the intermediate frame and the meteorological observation data are selected for encoding processing; if there are two intermediate frames, the inner product value (or weighted average value) of the two frames is calculated, and the inner product value and the meteorological observation data are encoded and processed.

[0087] In some embodiments, the encoding process is to perform feature processing on the meteorological observation data and at least one frame of the continuous multiple-frame river surface images to obtain visual features and meteorological features, perform normalization processing based on the visual features and meteorological features, and perform a convolution operation on the normalized processing results to obtain context mixed feature data.

[0088] S104 , performing optical flow prediction based on the first correlation pyramid feature data, the second correlation pyramid feature data, and the context mixed feature data to obtain an optical flow field, where the optical flow field includes a velocity field vector reflecting the flow velocity of the river.

[0089] In some embodiments, the first relevant pyramid feature data and the context mixed feature data are spliced, and the second relevant pyramid feature data and the context mixed feature data are spliced, and the two splicing results are respectively input into the ConvGRU unit (RAFT network includes ConvGRU unit) and the optical flow estimation model (such as FlowNet, PWC-Net, etc.) to obtain the optical flow field. Among them, the ConvGRU unit is a hybrid model that combines convolutional neural network (CNN) and gated recurrent unit (GRU), which is mainly used to process spatiotemporal sequence data (such as video prediction, optical flow estimation, weather forecast, etc.). The ConvGRU unit can effectively capture spatial local features and temporal dependencies. The optical flow estimation model is responsible for converting the features extracted by the network into the final optical flow field.

[0090] S105: Correct the optical flow field to obtain a target optical flow field.

[0091] In some embodiments, the optical flow field can be corrected by a trained physical information neural network (Physics Informed Neural Network, referred to as PINN). Among them, the physical information neural network (PINN) is a framework that embeds physical laws into neural network training, and solves the forward / inverse problems in scientific computing by integrating data-driven methods and physical equation constraints. The core idea is to use neural networks to approximate the solutions of differential equations, and at the same time provide unsupervised training signals through the physical equation residuals. As an example, the physical law can be the vortex transport equation that takes into account the change of viscosity with temperature. The physical equation residual constructed by this equation is used to train the physical information neural network to obtain a trained physical information neural network.

[0092] An embodiment of the present invention provides a river surface velocity estimation method that integrates multimodal optical flow estimation and PINN. By encoding meteorological observation data and at least one frame of river surface image, contextual mixed feature data is obtained, thereby realizing meteorological element-guided video image recognition and river flow field prediction, and ensuring the accuracy and physical consistency of flow field prediction; optical flow prediction is performed based on first related pyramid feature data, second related pyramid feature data and contextual mixed feature data to obtain an optical flow field, and the optical flow field is corrected, thereby realizing the determination of the optical flow field by cross-modal feature fusion and realizing high-precision and low-robustness monitoring of river surface velocity.

[0093] Figure 2FIG1 is a flow chart of another embodiment of a method for estimating river surface velocity by integrating multimodal optical flow estimation and PINN in the present invention. Figure 2 As shown, the method of the embodiment of the present invention includes:

[0094] S201, obtaining meteorological observation data and three consecutive frames of river surface images.

[0095] As an example, three consecutive frames of river surface images may be images taken by a camera at consecutive moments.

[0096] The meteorological observation data and three consecutive frames of river surface images are preprocessed, and their time axes are aligned. For specific operations, please refer to the above S101.

[0097] In S102, multi-scale feature processing is performed on the continuous multiple-frame river surface images to obtain first related pyramid feature data and second related pyramid feature data, which can refer to S202 to S204, including:

[0098] S202 , performing feature extraction on three consecutive frames of river surface images to obtain corresponding first feature image, second feature image, and third feature image.

[0099] RAFT's architecture consists of three main modules: the Feature Encoder, the Correlation Pyramid, and the Recurrent Update Operator. The Feature Encoder extracts multi-scale features using a weight-shared CNN (such as ResNet). The Correlation Pyramid constructs a multi-scale pyramid. The Recurrent Update Operator samples local correlation features from the pyramid based on the current optical flow estimate. The concatenation of the optical flow, correlation features, and contextual features is then fed into a ConvGRU. The ConvGRU then outputs delta flow, gradually optimizing the optical flow.

[0100] As an example, three consecutive river surface images can be input into the RAFT network for feature extraction. That is, the three consecutive river surface images are recorded as I1, I2, and I3, and the shape is [B, 3, H, W], where B is the batch size, H and W are the image height and width, and 3 represents the RGB channel. Assume that f(I i ; θ) represents the feature encoder of the feature extraction network RAFT, and the feature extraction process can be written as:

[0101] F i =f(I i ;θ) Formula (1)

[0102] Where: i = 1, 2, 3; represents the feature map obtained after feature extraction for the i-th image. The specific feature extraction network can be determined based on the task requirements and equipment conditions. The feature extraction of three consecutive frames of river surface images is recorded as F1, F2, and F3.

[0103] S203 : Determine a first correlation pyramid feature according to the correlation between the first feature image and the second feature image.

[0104] Since the correlation pyramid module in the RAFT network can only receive two feature maps at a time, two independent correlation pyramid modules can be constructed (the structure of the correlation pyramid module can be referenced) to process the first feature image and the second feature image respectively. As an example, for three frames of input, two multi-scale correlation pyramids are generated using two branches. That is, the correlation between the first feature image and the second feature image is first determined by the first correlation pyramid module. The correlation is calculated as follows:

[0105]

[0106] Where: (i, j) and (k, l) are the spatial coordinates of F1 and F2 respectively. c is the channel index of F1 and F2, Corr 1,2 (i, j, k, l) represents the correlation diagram between the first feature image and the second feature image.

[0107] Then, to reduce computational complexity, reduce memory usage, and capture multi-scale matching information, RAFT usually uses pooling operations to construct a multi-scale correlation pyramid. That is, the last two dimensions of the correlation graph are pooled, with a pooling kernel size of s×s and a step size of s. The calculation formula for the first correlation pyramid feature is as follows:

[0108] CorrPyramid 1,2 ={Pool s (Corr 1,2 )∣s∈S} Formula (3)

[0109] Among them, S is a set of pooling scales, including pooling scales s=1 (no pooling), 2, 4, etc.

[0110] Pool s (·) represents the correlation graph Corr 1,2 A pooling operation is performed on the last two dimensions (corresponding to the spatial dimensions of F2), which reduces the spatial resolution.

[0111] S204 : Determine a second correlation pyramid feature according to the correlation between the second feature image and the third feature image.

[0112] Referring to S203, similarly, the correlation diagram calculation formula of F2 and F3 (i.e., the second feature image and the third feature image) is as follows:

[0113]

[0114] Where: (i, j) and (k, l) are the spatial coordinates of F3 and F2 respectively. c is the channel index of F3 and F2, Corr 2,3 (i, j, k, l) represents the correlation diagram between the first feature image and the second feature image.

[0115] Referring to S203, similarly, according to the correlation between the second feature image and the third feature image, a second correlation pyramid feature is determined, and the calculation formula is as follows:

[0116] CorrPyramid 2,3 ={Pool s (Corr 2,3 )∣s∈s} Formula (5)

[0117] Where S is a set of pooling scales, including pooling scales s=1 (no pooling), 2, 4, etc. Pool s (·) represents the correlation graph Corr 2,3 A pooling operation is performed on the last two dimensions (corresponding to the spatial dimensions of F2), which reduces the spatial resolution.

[0118] S205 , encoding the meteorological observation data and at least one frame of the continuous multiple-frame river surface images to obtain context mixed feature data.

[0119] In some optional implementations, the meteorological observation data and at least one frame of the continuous multiple-frame river surface images may be encoded and input into a cyclic update module of the RAFT network to obtain contextual mixed feature data.

[0120] In some optional implementations, in step S103 or S205, encoding the meteorological observation data and at least one frame of the continuous multiple-frame river surface images to obtain context mixed feature data may include S2051 and S2052:

[0121] S2051, physically encoding the meteorological observation data to generate physical feature tensor data having the same resolution as at least one frame of the continuous multiple-frame river surface image;

[0122] S2052: Perform mixed encoding processing on the physical feature tensor data and at least one frame of the continuous multiple-frame river surface images to obtain context mixed feature data.

[0123] As an example, the meteorological observation data is physically encoded through the physical encoder process to obtain a feature tensor P that is consistent with the image feature resolution:

[0124] P=Encoder Physical (M) Formula (6)

[0125] in, For meteorological observation data, encoder Physical (·) includes fully connected layers, feature reshaping layers, and upsampling layers. To avoid the fusion dominated by meteorological features, the number of channels n is not expanded to 256, but the original dimension n is retained. The obtained meteorological feature tensor Where B is the batch size, H and W are the image height and width.

[0126] As an example, the second feature image F2 can be selected from a continuous multi-frame river surface image, and the physical feature tensor data p and the second feature image F2 are mixed and encoded to obtain context mixed feature data. That is, the intermediate frame feature F2 (i.e., the second feature image) and the physical feature p (i.e., the physical feature tensor data) are combined through the cross-modal fusion module MultiModalFusion(·) to generate the context feature (i.e., context mixed feature data), as shown in formula (7):

[0127] C=MultiModalFusion(F2,P) Formula (7)

[0128] Specifically, the processing flow of the MultiModalFusion(·) module is as follows Figure 3 :

[0129] First, the flattened visual features and meteorological features are projected using the linear layer, and the visual features are mapped into the query matrix Query (i.e., Q projection), and the meteorological features are decomposed into key-value pairs Key and Value (i.e., K projection and V projection). The specific calculation process refers to formulas (8) to (12):

[0130] Q=W q ·Flatten(F2) formula (8)

[0131] {K,V}=W k Flatten(P),W v ·Flatten(P) formula (9)

[0132] in, Denotes the projection parameters, which are the learnable query projection weight matrix, key projection weight matrix, and value projection weight matrix. Q, K, V are the generated query, key, and value matrices, respectively. k=64 is the projection dimension of the query / key, F2 is the second feature image, and P is the physical feature tensor data.

[0133] Then, the cross-modal key information is filtered by temperature scaling attention to calculate the fusion attention weight, that is, the dot product attention of Query and Key is calculated, and the weighted sum of Value is performed after scaling, as shown in formula (10):

[0134]

[0135] Among them, Softmax represents the normalized exponential function. is the attention weight matrix, which represents the weighted combination distribution of meteorological features obtained at each visual feature position, d k =64 is the query / key projection dimension, Q, K, V are the generated query, key and value matrices respectively.

[0136] Then, after feature reconstruction, the original visual features are fused, the original visual feature information is retained through identity mapping, and the gradient disappearance is alleviated, as shown in formula (11):

[0137]

[0138] Where ⊕ represents element-by-element addition. C is the visual feature enhanced by weighted fusion of meteorological information, and F2 is the second feature image.

[0139] Finally, the multimodal fused feature C is input into the Context encoder (the encoder is usually composed of several convolutional layers, keeping the tensor shape unchanged) to further refine the information and generate the final Context feature (i.e., context mixed feature data), as shown in formula (12):

[0140]

[0141] Contextual mixed feature data C final Joint contextual information of imagery and meteorological elements will be provided.

[0142] S206 , performing optical flow prediction based on the first correlation pyramid feature data, the second correlation pyramid feature data, and the context mixed feature data to obtain an optical flow field, where the optical flow field includes a velocity field vector reflecting the flow velocity of the river.

[0143] As an example, optical flow prediction can be performed through a ConvGRU unit (the ConvGRU unit is in the cyclic update module of the RAFT network). Specifically, the first relevant pyramid feature data, the second relevant pyramid feature data, and the context mixed feature data can be input into the ConvGRU unit to obtain a first target hidden state and a second target hidden state, and optical flow prediction is performed based on the first target hidden state and the second target hidden state to obtain a first optical flow field; optical flow prediction is performed on the second target hidden state to obtain a second optical flow field.

[0144] In some optional implementations, in S104 or S206, performing optical flow prediction based on the first relevant pyramid feature data, the second relevant pyramid feature data, and the context mixed feature data to obtain an optical flow field may include S2061-S2066:

[0145] S2061, obtaining a first branch vector according to the first relevant pyramid feature data and the context mixed feature data;

[0146] S2062, obtaining a second branch vector according to the second related pyramid feature data and the context mixed feature data;

[0147] S2063, obtaining a first target hidden state according to the first branch vector and a first hidden state of a preset network;

[0148] S2064, obtaining a second target hidden state according to the second branch vector and a second hidden state of the preset network;

[0149] S2065, performing optical flow prediction on the hidden state of the first target to obtain a first optical flow field;

[0150] S2066: Perform optical flow prediction on the second target hidden state to obtain a second optical flow field.

[0151] As an example, the first hidden state of the preset network It can be the difference between the second feature map and the first feature map, and the second hidden state of the preset network It can be the difference between the third feature map and the second feature map.

[0152] As an example, the first branch vector and the first hidden state of the preset network can be input into a first ConvGRU unit for feature extraction to obtain a first target hidden state; and the second branch vector and the second hidden state of the preset network can be input into a second ConvGRU unit for feature extraction to obtain a second target hidden state. The first ConvGRU unit and the second ConvGRU unit are independent of each other.

[0153] In some optional implementation, S2061 can include: vector splicing the first related pyramid feature data and the context mixed feature data to obtain a first branch vector.

[0154] S2062 can include: vector splicing the second related pyramid feature data and the context mixed feature data to obtain a second branch vector.

[0155] As an example, the first branch vector can be denoted as The second branch vector is denoted as The first branch vector and the second branch vector are input respectively into two independent ConvGRU units after dimension reduction.

[0156] In some optional implementation, S2063 can include:

[0157] S20631, obtaining a first hidden state, a first update parameter, a second update parameter and a third update parameter of a preset network;

[0158] S20632, performing convolution operation on the first branch vector, the first hidden state and the first update parameter of the preset network to obtain a first update value;

[0159] S20633, performing convolution operation on the first branch vector, the first hidden state and the second update parameter of the preset network to obtain a second update value;

[0160] S20634, performing convolution operation on the first branch vector, the first hidden state, the second update value and the third update parameter of the preset network to obtain a third update value;

[0161] S20635, performing operation on the first hidden state, the first update value and the third update value of the preset network to obtain a first target hidden state.

[0162] As an example, the preset network is a ConvGRU unit, and the ConvGRU unit includes four processing steps: update gate, reset gate, candidate hidden state and hidden state update. The first update parameter, the second update parameter and the third update parameter respectively represent the convolution kernel weight of the update gate, the reset gate and the candidate hidden state.

[0163] As an example, the first branch vector, the first hidden state and the first update parameter of the preset network are operated through the update gate to obtain the first update value, that is, the first branch vector, the first hidden state and the first update parameter of the preset network are convolved, and then operated through the Sigmoid activation function to obtain the first update value.

[0164] Taking the above as an example, a reset gate operation is performed on the first branch vector, the first hidden state of the preset network, and the second update parameter to obtain a second update value, that is, a convolution operation is performed on the first branch vector, the first hidden state of the preset network, and the second update parameter, and then a Sigmoid activation function operation is performed to obtain the second update value.

[0165] Taking the above as an example, a candidate hidden state operation is performed on the first branch vector, the first hidden state of the preset network, the second update value and the third update parameter to obtain the third update value, that is, a convolution operation is performed on the first branch vector, the first hidden state of the preset network, the second update value and the third update parameter, and then the third update value is obtained through the tanh activation function operation.

[0166] Taking the above as an example, the first hidden state, the first update value, and the third update value of the preset network are operated to obtain the first target hidden state, that is, the first hidden state, the first update value, and the third update value of the preset network are element-wise multiplied to obtain the first target hidden state.

[0167] It should be noted that after obtaining the first target hidden, the first target hidden can be input into the ConvGRU unit again as the first hidden state, and iterated to a preset number of times to obtain the final first target hidden.

[0168] In some optional implementations, S2064 may include:

[0169] S20641, obtaining a second hidden state, a first update parameter, a second update parameter, and a third update parameter of a preset network;

[0170] S20642, performing a convolution operation on the second branch vector, the second hidden state of the preset network, and the first update parameter to obtain a fourth update value;

[0171] S20643, performing a convolution operation on the second branch vector, the second hidden state of the preset network, and the second update parameter to obtain a fifth update value;

[0172] S20644, performing a convolution operation on the second branch vector, the second hidden state of the preset network, the fifth update value, and the third update parameter to obtain a sixth update value;

[0173] S20645: Calculate the second hidden state, the fourth update value, and the sixth update value of the preset network to obtain a second target hidden state.

[0174] The calculation process of the second target hidden state can refer to the above-mentioned first target hidden state.

[0175] In some optional implementations, S2065 may include: performing multi-layer convolution operations and upsampling operations on the first target hidden state to obtain a first optical flow field; S2066 may include: performing multi-layer convolution operations and upsampling operations on the second target hidden state to obtain a second optical flow field.

[0176] As an example, the first / second target hidden state can be predicted by the optical flow prediction model. The optical flow prediction model includes three convolutional layers (Conv2d, with a kernel size of 3 and a stride of 1) and ReLU activation functions, which are stacked alternately to extract and integrate higher-level feature information while maintaining the spatial structure. Subsequently, the network gradually upsamples the low-resolution hidden state to the original resolution through three transposed convolutional layers (ConvTranspose2d, with a kernel size of 3 and a stride of 2) and ReLU activation functions to generate the corresponding optical flow field.

[0177] As an example, the first target hidden state is sequentially convolved through three convolutional layers and activated by the ReLU activation function to obtain higher-level feature information, and then the higher-level feature information is sequentially convolved through three transposed convolutional layers and activated by the ReLU activation function to obtain the first optical flow field. The input of the second optical flow field is the second target hidden state, and the calculation process can refer to the process of the first optical flow field.

[0178] As an example, a dual-branch cyclic optimization network can be constructed by using ConvGRU units to iteratively update the first / second optical flow field for progressive optimization of motion features. The dual-branch cyclic optimization network processes the motion information of the intermediate frame I2, the previous frame I1, and the next frame I3 respectively. Specifically, each independent branch includes a ConvGRU unit, each branch receives corresponding related pyramid features and contextual mixed feature data, and the input of each branch is generated by splicing the related pyramid features and contextual mixed feature data. The inputs of the two independent branches are respectively expressed as:

[0179]

[0180] in: The input tensors of the two branches, where Represents the tensor of the concatenation of the first related pyramid feature and the contextual mixed feature data, A tensor representing the concatenation of the second correlation pyramid feature and the contextual mixed feature data.

[0181] In order to unify the number of channels, the concatenated tensor can be reduced in dimension by 1x1 convolution:

[0182]

[0183] Where: Cx is the number of channels after dimensionality reduction (usually 128 or 256, depending on the network design).

[0184] Because each branch has an independent ConvGRU unit, it needs to be processed separately and And update the corresponding hidden state. The initial hidden states of the two branches are The initial hidden state is obtained using the frame difference information. The calculation formula is as follows:

[0185]

[0186] The update rule of ConvGRU is the same for each branch. Taking branch 1 as an example, the update formula of ConvGRU is as shown below:

[0187]

[0188] in, is the update gate, reset gate and candidate hidden state of branch 1; W z ,W r ,W h are the update gate, reset gate, and convolution kernel weights of the candidate hidden state, which are all learnable parameters; σ, tanh, ⊙ represent the Sigmoid activation function, tanh activation function, and element-wise multiplication, respectively. represents the hidden state of the t-1th update, represents the hidden state of the tth update.

[0189] After a certain number of update iterations, the final hidden state ( and That is, the first target hidden state and the second target hidden state are independently passed to the optical flow estimation head FlowPredictor (·), which first passes through three convolutional layers (Conv2d, convolution kernel size is 3, stride is 1) and ReLU activation functions alternately stacked to extract and integrate higher-level feature information while maintaining the spatial structure. Subsequently, the network successively passes through three transposed convolutional layers (ConvTranspose2d, convolution kernel size is 3, stride 2) and ReLU activation functions to gradually achieve upsampling, restore the low-resolution hidden state to the original resolution, and generate the corresponding optical flow field. That is, the first optical flow field and the second optical flow field. This process is shown in formula (19):

[0190]

[0191] Each optical flow field contains two components (u, v) in orthogonal directions, namely the first optical flow field Flow1 = (u1, v1), and the second optical flow field Flow2 = (u2, v2). These two optical flow fields are input into the subsequent network (PINN network) as initial flow field information for physical correction.

[0192] S207: Correct the optical flow field to obtain a target optical flow field.

[0193] Since the original RAFT optical flow network does not consider physical information, the output optical flow field depends on the optical flow field annotation information. However, the optical flow field information of the river surface is difficult to annotate, especially the optical flow annotation of natural river channels. Therefore, it is necessary to correct the optical flow field through a trained physical information neural network (Physics Informed Neural Network, referred to as PINN). As an example, the structure of the PINN network can be based on a fully connected network or other networks. Here is a network based on the combination of Unet and Resnet. The overall PINN network diagram is as follows Figure 4 As shown in Figure 1. The network input layer receives data with a resolution of H×W and generates a 64-channel feature map through a 7x7 convolution (stride 1). The resolution is then reduced to H / 2×W / 2 through max pooling with a stride of 2 before entering the encoder. The encoder consists of three stages (Encoder 1 to Encoder 3), each consisting of two residual blocks. Each residual block consists of two 3x3 convolution blocks and a residual connection. The first 3x3 convolution has a stride of 2, halving the feature map resolution, while the first 3x3 convolution uses a stride of 1, maintaining the resolution. Therefore, over the four encoder stages, the feature map resolution is gradually reduced to H / 4×W / 4, H / 8×W / 8, and H / 16×W / 16, while the number of channels is gradually increased to 128, 256, and 512. The output of each stage is passed to the decoder via skip connections, preserving low-level spatial information to enhance detail recovery. The feature map after the third encoder stage is input into the bottleneck layer (Bottleneck), which contains multiple residual blocks (the number can be adjusted according to task requirements). It integrates high-level semantic information through deep feature extraction, enhances the network's modeling ability of complex patterns (such as object shape and texture), while keeping the resolution and number of channels unchanged, and provides a compact and rich semantic representation for the decoder.

[0194] Symmetric to the encoder, the decoder also consists of three stages, including three upsampling decoder stages (decoder 3 to decoder 1). Each stage fuses the features of the corresponding encoder stage (the number of channels is 512, 256, and 128, respectively) through jump connections to integrate low-level spatial details and high-level semantic information. Each upsampling decoder contains a transposed convolution block (TransConv Block), which in turn includes a 1x1 convolution block, a 3x3 transposed convolution block, and a 1x1 convolution block. These two 1x1 convolution blocks are used to adjust the number of channels, and the 3x3 transposed convolution block is used to improve the feature map resolution with a stride of 2. Taking Dec3 as an example, the input feature map H / 16×W / 16×512 is transformed through the modified transposed convolution block (1×1 convolution reduces the channel to a quarter of the original to reduce the subsequent transposed convolution calculation amount, i.e. The transposed convolution upsamples to (H / 8)×(W / 8)×128→1×1 convolution, and adjusts the channels to be consistent with Enc3 (H / 8×W / 8×256), doubling the resolution and halving the number of channels. The three encoders sequentially restore the resolution to H / 8×W / 8×256, H / 4×W / 4×128, and H / 2×W / 2×64.

[0195] The final upsampling decoder stage Dec1 is followed by a 4x4 transposed convolution block with a stride of 2 to restore the feature map to its original resolution and a 1x1 convolution block to adjust the number of channels to 4, corresponding to the output

[0196] As an example, if multiple optical flow fields need to be corrected, the corrections can be made sequentially or in parallel as needed. According to the trained PINN network, the corrections to the optical flow fields may include:

[0197] The encoder extracts high-level semantic information of the optical flow field to obtain encoding features;

[0198] The decoder extracts the low-level semantic information of the encoded features to obtain the decoded features;

[0199] The encoded features and decoded features are integrated and then up-sampled to obtain the corrected optical flow field.

[0200] In some optional implementations, S2065 may include:

[0201] S20651, obtaining vorticity ω and stream function; obtaining a first vorticity according to the vorticity ω and the stream function;

[0202] S20652: Correct the first optical flow field according to the first vorticity and the temperature scalar to obtain a first target optical flow field.

[0203] In some optional implementations, S2066 may include:

[0204] S20661, obtaining vorticity ω and stream function; obtaining a second vorticity according to the vorticity ω and stream function;

[0205] S20662: Correct the second optical flow field according to the second vorticity and the temperature scalar to obtain a second target optical flow field.

[0206] The PINN network training process uses deep learning to solve partial differential equations (PDEs). It utilizes a small amount of sparse data combined with physical constraints (equation residuals) to generate a flow field solution that conforms to physical laws. Using the PINN network to modify the flow field output by the original RAFT optical flow network has three main benefits: (I) it requires only a small amount of sparse data, reducing reliance on annotation information; (II) it can effectively filter out noise contained in the RAFT optical flow estimate; and (III) by fully incorporating PDE information, it can eliminate non-physical solutions, making them more consistent with fluid physics and improving the physical consistency of the flow field calculation.

[0207] As an example, the following is the loss function construction process of the PINN network combined with the vorticity transport equation:

[0208] First, consider the PDE form of the river surface flow field motion. To simplify the analysis, the river is regarded as an incompressible fluid and the external force terms are not considered. The mathematical form of its Navier-Stokes equation is:

[0209]

[0210] Where u = (u, v) is the velocity field (u is the x-direction and v is the y-direction); p is the pressure; ρ is the constant density; and ν is the kinematic viscosity.

[0211] Introduce the vorticity ω, which is defined as:

[0212]

[0213] Then the stream function ψ is introduced to define the two-dimensional velocity components,

[0214]

[0215] Poisson's equation is obtained, and the vorticity can be expressed as

[0216] Taking the curl of the momentum equation and eliminating the pressure term, we get the vortex transport equation, which has the mathematical form:

[0217]

[0218] Here, considering the function ν(T) of kinematic viscosity ν varying with temperature T, the vortex transport equation considering the viscosity varying with temperature is obtained as follows:

[0219]

[0220] Based on the above vortex transport equation considering the change of viscosity with temperature, that is, formula (24), a PINN network containing multiple convolutional layers and nonlinear activation functions is constructed. The goal is to use the PINN network to solve the approximate vortex transport equation and correct the distribution of the velocity field by optimizing the spatiotemporal evolution of the vortex. The input data of the PINN network are the flow field data u1 = (u1, v1) and u2 = (u2, v2) of two consecutive frames, with a tensor shape of [B, 4, H, W], and the temperature scalar Output the corrected flow field and The tensor shape is also [B,4,H,W].

[0221] The total loss of the established PINN network for:

[0222]

[0223] Among them, λ1, λ2, λ3 are hyperparameters;

[0224] The first term is the residual loss of the vorticity transport equation

[0225]

[0226] Where N is the total number of grid points; is the residual of the vorticity transport equation; Indicates the root mean square; u pred and ω pred are the predicted flow field and vorticity, respectively;

[0227] The second term is the incompressibility constraint loss

[0228]

[0229] The third item is data loss Consider the supervised endpoint error:

[0230]

[0231] Among them, N i is the number of true label samples; M(x i ,y i ) is a mask function, when M(x i ,yi ) = 1, it means there is real data at position (x i ,y i ), and when M(x i ,y i ) = 0, it means there is no real data at this point. EPE1 and EPE2 are the end point errors of the flow field at two time points respectively.

[0232] The process of solving the PINN network parameters using the above loss function, i.e. formula (26) to formula (28), can refer to (i.e. the process of training PINN network using loss function combined with physical constraints):

[0233] Considering that the input is two adjacent time data, the Crank-Nicolson discrete method is used to discretize the time derivative of the vorticity transport equation formula (24), i.e. the time change rate of vorticity uses time center difference, and the velocity field and vorticity field both use two frame average values, to obtain the Crank-Nicolson discrete format of the vorticity transport equation:

[0234]

[0235] This is a second-order accuracy implicit time discrete method, which fully couples the vorticity information of two frames

[0236] For the flow field output by the PINN network after modification And The automatic differentiation of PyTorch can be used to calculate the spatial partial derivatives of each component in the x and y directions, and the calculation formula is shown in formula (30):

[0237]

[0238] First, the vorticity of two frames of flow field can be calculated using formula (21) and formula (30):

[0239]

[0240] Thus, the first-order partial derivative and the second-order partial derivative of vorticity in space can be calculated using automatic differentiation, as shown in formula (32):

[0241]

[0242] First, calculate the vorticity transport equation residual loss, the time derivative term In formula (29), it is the difference approximation of the vorticity of two frames of flow field, as shown in formula (33):

[0243]

[0244] Wherein, Δt is the frame difference time, which is determined according to the video frame rate. For example, if the video is set to 25 frames / s, then Δt=0.04s.

[0245] Convection term In formula (29), it is the average value of the convection term of the two-frame flow field, as shown in formula (34):

[0246]

[0247] Substituting the relevant variables in formula (30) and formula (32) into formula (34), the calculation of the convection term is shown in formula (35):

[0248]

[0249] Diffusion term In formula (29), it is the average value of the diffusion term of the flow field in two frames, as shown in formula (36):

[0250]

[0251] Substituting the relevant variables in formula (30) and formula (32) into formula (36), the calculation of the diffusion term is shown in formula (37):

[0252]

[0253] Considering the viscosity ν as a function of temperature T, the Stokes-Einstein empirical formula (38) is used as follows:

[0254] ν(T)=β·e α·T Formula (38)

[0255] Among them, α and β are constants. The viscosity value at a specific temperature can be selected as a reference. For example, the viscosity value ν0 is set to 1.007×10 at T0 = 20°C. -6 m 2 / s, the viscosity at any temperature can be calculated based on the reference viscosity value, as shown in formula (39):

[0256]

[0257] Substituting formula (39) into formula (37), the diffusion term considering the change of viscosity with temperature can be calculated as shown in formula (40):

[0258]

[0259] Combining formulas (33), (35) and (40), the residual of the vortex transport equation can be obtained As shown in formula (41):

[0260]

[0261] The residual Substituting into formula (26) and calculating the root mean square error, we can get the residual loss of the vortex transport equation: As shown in formula (42):

[0262]

[0263] Next, we calculate the incompressibility loss. Incompressibility requires that the divergence of the velocity field is zero. First, we use the partial derivative values ​​of each flow field obtained by automatic differentiation of formula (30) to calculate the divergence of the flow field of the two frames. and As shown in formula (43):

[0264]

[0265] Therefore, the incompressibility loss is the sum of the mean square error of the flow field divergence of the two frames, and the calculation formula is shown in (44):

[0266]

[0267] Finally, the data loss is calculated. It is assumed that there are surface velocity monitoring values ​​in the river area monitored by the video, which can be used as the true label constraint network prediction value. However, the actual flow field data is usually sparse, and the surface flow field of the river is often difficult to monitor. In reality, the velocity values ​​of some river points are generally monitored, or characteristic river sections are selected to set up monitoring sections for velocity testing. Therefore, only some locations have monitoring values ​​as supervision constraints. The spatial location distribution of the true value can be expressed by the mask function in formula (28). For the spatial location (x i ,y i ), the real monitoring values ​​corresponding to adjacent time frames are and The PINN network output prediction value for this spatial position is and

[0268] Consider using endpoint error loss for supervision constraint, which measures the estimation accuracy by calculating the average Euclidean distance between the predicted flow field and the true flow field. It comprehensively considers the direction and size of the flow field and has a clear physical meaning. The endpoint errors EPE1 and EPE2 of the flow field at two moments are shown in formulas (45) and (46):

[0269]

[0270] Substituting formulas (45) and (46) into formula (28), the calculation formula for data loss is shown in (47):

[0271]

[0272] The total loss of the PINN network can be obtained by combining formulas (42), (44), and (47).

[0273] The training process uses an initial learning rate of 0.0001, cosine annealing is used for learning rate scheduling, and the Adam optimizer is used to update network parameters to minimize the total loss function. The batch size is determined by the hardware selection, and the training batch is determined according to the actual training effect. The hyperparameters λ1, λ2, and λ3 are dynamically adjusted. Initially, λ1 and λ2 can be small (such as 0.01) and then increased later to enhance physical constraints.

[0274] An embodiment of the present invention provides a river surface velocity estimation method that integrates multimodal optical flow estimation and PINN, constructs a multimodal coupling model of "visual features-meteorological factors-fluid mechanics", combines deep learning and physical constraints, and makes full use of image and meteorological information to ensure the accuracy and physical consistency of flow field prediction. Compared with existing methods, the flow field prediction of video image recognition guided by meteorological elements is achieved through the combination of multi-dimensional methods such as cross-modal feature fusion, bidirectional multi-layer pyramid motion modeling, and ConvGRU iterative update. In addition, the physical control equations are embedded in the neural network architecture as differentiable constraints to solve the problem that pure image recognition models are prone to violate physical laws. It can achieve highly robust, physically interpretable, and spatiotemporally continuous river surface velocity field inversion in complex interference environments, which can provide new ideas and approaches for smart hydrological monitoring.

[0275] Figure 5 Schematic diagram of a module of a river surface velocity estimation device that integrates multimodal optical flow estimation and PINN in the present invention. Figure 5 As shown, the device comprises:

[0276] An acquisition module 501 is configured to acquire meteorological observation data and a plurality of continuous river surface images; a first processing module 502 is configured to perform multi-scale feature processing on the plurality of continuous river surface images to obtain first related pyramid feature data and second related pyramid feature data;

[0277] The second processing module 503 is configured to encode the meteorological observation data and at least one frame of the continuous multiple-frame river surface images to obtain context mixed feature data;

[0278] The third processing module 504 is configured to perform optical flow prediction based on the first related pyramid feature data, the second related pyramid feature data, and the context mixed feature data to obtain an optical flow field, where the optical flow field includes a velocity field vector reflecting the flow velocity of the river;

[0279] The fourth processing module 505 is used to correct the optical flow field to obtain a target optical flow field.

[0280] Optionally, the first processing module 502 is further configured to perform multi-scale feature processing on the continuous multiple frames of river surface images to obtain first related pyramid feature data and second related pyramid feature data, including:

[0281] Extracting features from a first frame of the continuous multiple-frame river surface images to obtain a first feature image;

[0282] performing feature extraction on a second frame image from the continuous multiple frames of river surface images to obtain a second feature image;

[0283] performing feature extraction on a third frame image from the continuous multiple frames of river surface images to obtain a third feature image;

[0284] Determining a first correlation pyramid feature according to a correlation between the first feature image and the second feature image;

[0285] A second correlation pyramid feature is determined according to the correlation between the second feature image and the third feature image.

[0286] Optionally, the second processing module 503 is also used to physically encode the meteorological observation data to generate physical feature tensor data with the same resolution as at least one frame of the continuous multi-frame river surface images; and perform mixed encoding processing on the physical feature tensor data and at least one frame of the continuous multi-frame river surface images to obtain context mixed feature data.

[0287] Optionally, the third processing module 504 is also used to obtain a first branch vector based on the first relevant pyramid feature data and the context mixed feature data; obtain a second branch vector based on the second relevant pyramid feature data and the context mixed feature data; obtain a first target hidden state based on the first branch vector and the first hidden state of the preset network; obtain a second target hidden state based on the second branch vector and the second hidden state of the preset network; perform optical flow prediction on the first target hidden state to obtain a first optical flow field; perform optical flow prediction on the second target hidden state to obtain a second optical flow field.

[0288] Optionally, the third processing module 504 is further used to perform vector splicing on the first relevant pyramid feature data and the context mixed feature data to obtain a first branch vector; and obtain a second branch vector based on the second relevant pyramid feature data and the context mixed feature data, including: performing vector splicing on the second relevant pyramid feature data and the context mixed feature data to obtain a second branch vector.

[0289] Optionally, the third processing module 504 is also used to obtain the first hidden state, first update parameter, second update parameter and third update parameter of the preset network; perform a convolution operation on the first branch vector, the first hidden state of the preset network and the first update parameter to obtain a first update value; perform a convolution operation on the first branch vector, the first hidden state of the preset network and the second update parameter to obtain a second update value; perform a convolution operation on the first branch vector, the first hidden state of the preset network, the second update value and the third update parameter to obtain a third update value; perform an operation on the first hidden state, the first update value and the third update value of the preset network to obtain a first target hidden state.

[0290] Optionally, the third processing module 504 is also used to obtain the second hidden state, the first update parameter, the second update parameter and the third update parameter of the preset network; perform a convolution operation on the second branch vector, the second hidden state of the preset network and the first update parameter to obtain a fourth update value; perform a convolution operation on the second branch vector, the second hidden state of the preset network and the second update parameter to obtain a fifth update value; perform a convolution operation on the second branch vector, the second hidden state of the preset network, the fifth update value and the third update parameter to obtain a sixth update value; perform an operation on the second hidden state of the preset network, the fourth update value and the sixth update value to obtain a second target hidden state.

[0291] Optionally, the third processing module 504 is further used to perform multi-layer convolution operations and upsampling operations on the first target hidden state to obtain a first optical flow field; perform optical flow prediction on the second target hidden state to obtain a second optical flow field, including: performing multi-layer convolution operations and upsampling operations on the second target hidden state to obtain a second optical flow field.

[0292] Optionally, the fourth processing module 505 is also used to obtain the vorticity ω and the stream function; obtain the first vorticity and the second vorticity based on the vorticity ω and the stream function; correct the first optical flow field based on the first vorticity and the temperature scalar to obtain the first target optical flow field; correct the second optical flow field based on the second vorticity and the temperature scalar to obtain the second target optical flow field.

[0293] The device embodiments described above are merely illustrative. Modules described as separate components may or may not be physically separate, and components shown as modules may or may not be physical modules, i.e., they may be located in one place or distributed across multiple network modules. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0294] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A river surface velocity estimation method integrating multimodal optical flow estimation and PINN, characterized in that: include: Acquire meteorological observation data and continuous multi-frame river surface images; Performing multi-scale feature processing on the continuous multi-frame river surface images to obtain first related pyramid feature data and second related pyramid feature data; encoding the meteorological observation data and at least one frame of the continuous multiple-frame river surface images to obtain contextual mixed feature data; performing optical flow prediction based on the first relevant pyramid feature data, the second relevant pyramid feature data, and the contextual mixed feature data to obtain an optical flow field, wherein the optical flow field includes a velocity field vector reflecting a flow velocity of a river; The optical flow field is corrected to obtain a target optical flow field.

2. The method for estimating river surface velocity by integrating multimodal optical flow estimation and PINN according to claim 1 is characterized in that: Performing multi-scale feature processing on the continuous multi-frame river surface images to obtain first related pyramid feature data and second related pyramid feature data, including: Performing feature extraction on a first frame of the continuous multiple frames of river surface images to obtain a first feature image; performing feature extraction on a second frame image among the plurality of continuous frames of river surface images to obtain a second feature image; performing feature extraction on a third frame of the continuous plurality of frames of river surface images to obtain a third feature image; determining the first correlation pyramid feature according to the correlation between the first feature image and the second feature image; The second correlation pyramid feature is determined according to the correlation between the second feature image and the third feature image.

3. The river surface velocity estimation method integrating multimodal optical flow estimation and PINN according to claim 1 is characterized in that: The meteorological observation data and at least one frame of the continuous multiple frames of river surface images are encoded to obtain context mixed feature data, including: Physically encoding the meteorological observation data to generate physical feature tensor data having the same resolution as at least one frame of the plurality of consecutive river surface images; The physical feature tensor data is mixed and coded with at least one frame of the continuous multiple frames of river surface images to obtain context mixed feature data.

4. The method for estimating river surface velocity by integrating multimodal optical flow estimation and PINN according to claim 1, characterized in that: Performing optical flow prediction based on the first relevant pyramid feature data, the second relevant pyramid feature data, and the context mixed feature data to obtain an optical flow field includes: Obtaining a first branch vector according to the first relevant pyramid feature data and the context mixed feature data; Obtaining a second branch vector according to the second relevant pyramid feature data and the context mixed feature data; Obtaining a first target hidden state according to the first branch vector and a first hidden state of a preset network; Obtaining a second target hidden state according to the second branch vector and a second hidden state of a preset network; Performing optical flow prediction on the first target hidden state to obtain a first optical flow field; Perform optical flow prediction on the second target hidden state to obtain a second optical flow field.

5. The method for estimating river surface velocity by integrating multimodal optical flow estimation and PINN according to claim 4 is characterized in that: Obtaining a first branch vector according to the first relevant pyramid feature data and the context mixed feature data includes: Performing vector concatenation on the first related pyramid feature data and the context mixed feature data to obtain a first branch vector; Obtaining a second branch vector according to the second related pyramid feature data and the context mixed feature data, including: Perform vector concatenation on the second related pyramid feature data and the context mixed feature data to obtain a second branch vector.

6. The method for estimating river surface velocity by integrating multimodal optical flow estimation and PINN according to claim 4, characterized in that: Obtaining a first target hidden state according to the first branch vector and a first hidden state of a preset network includes: Obtaining a first hidden state, a first update parameter, a second update parameter, and a third update parameter of a preset network; Performing a convolution operation on the first branch vector, a first hidden state of a preset network, and a first update parameter to obtain a first update value; Performing a convolution operation on the first branch vector, a first hidden state of a preset network, and a second update parameter to obtain a second update value; Performing a convolution operation on the first branch vector, a first hidden state of the preset network, the second update value, and a third update parameter to obtain a third update value; The first hidden state, the first update value, and the third update value of the preset network are operated to obtain a first target hidden state.

7. The method for estimating river surface velocity by integrating multimodal optical flow estimation and PINN according to claim 4, characterized in that: Obtaining a second target hidden state according to the second branch vector and a second hidden state of a preset network includes: Obtaining a second hidden state, a first update parameter, a second update parameter, and a third update parameter of a preset network; performing a convolution operation on the second branch vector, a second hidden state of the preset network, and the first update parameter to obtain a fourth update value; performing a convolution operation on the second branch vector, a second hidden state of the preset network, and a second update parameter to obtain a fifth update value; performing a convolution operation on the second branch vector, the second hidden state of the preset network, the fifth update value, and the third update parameter to obtain a sixth update value; The second hidden state, the fourth update value, and the sixth update value of the preset network are operated to obtain a second target hidden state.

8. The method for estimating river surface velocity by integrating multimodal optical flow estimation and PINN according to claim 4, characterized in that: Performing optical flow prediction on the first target hidden state to obtain a first optical flow field includes: Performing multi-layer convolution operations and upsampling operations on the first target hidden state to obtain a first optical flow field; Performing optical flow prediction on the second target hidden state to obtain a second optical flow field includes: Perform multi-layer convolution operations and upsampling operations on the second target hidden state to obtain a second optical flow field.

9. The method for estimating river surface velocity by integrating multimodal optical flow estimation and PINN according to claim 4, characterized in that: Correcting the optical flow field to obtain a target optical flow field includes: Get the vorticity ω and stream function; Obtaining a first vorticity and a second vorticity according to the vorticity ω and the stream function; Correcting the first optical flow field according to the first vorticity and the temperature scalar to obtain a first target optical flow field; The second optical flow field is corrected according to the second vorticity and the temperature scalar to obtain a second target optical flow field.

10. A river surface velocity estimation device integrating multimodal optical flow estimation and PINN, characterized in that: include: An acquisition module, used to acquire meteorological observation data and continuous multi-frame river surface images; A first processing module is configured to perform multi-scale feature processing on the continuous multi-frame river surface images to obtain first related pyramid feature data and second related pyramid feature data; A second processing module is configured to perform encoding processing on the meteorological observation data and at least one frame of the continuous multiple frames of river surface images to obtain context mixed feature data; a third processing module, configured to perform optical flow prediction based on the first relevant pyramid feature data, the second relevant pyramid feature data, and the context mixed feature data to obtain an optical flow field, wherein the optical flow field includes a velocity field vector reflecting a river flow velocity; The fourth processing module is used to correct the optical flow field to obtain a target optical flow field.

Images