River surface flow velocity measurement method and system based on physical constraint and multi-modal fusion

Abstract

The application discloses a river surface flow velocity measurement method and system based on physical constraints and multi-modal fusion, relates to the technical field of multi-modal data processing, determines a target region, and acquires river surface image sequence data, depth information and environmental data of the target region; the collected data is preprocessed, and a redundancy code is embedded in data representation to comprehensively form a basic data set; based on the basic data set, optical flow estimation is performed on the river surface image sequence data by using an optical flow method. The application realizes comprehensive utilization of multi-source data by fusing optical flow information, depth information and environmental data, and in the data fusion process, error detection and correction are performed in combination with the redundancy code in the data representation, error points in the data are effectively identified and corrected, the reliability and accuracy of the data are improved, the robustness of the system is significantly enhanced by using the multi-modal data fusion and error correction mechanism, and the system can stably operate in a complex and changeable river environment.

Description

Technical Field

[0001] This invention relates to the field of multimodal data processing technology, specifically to a method and system for measuring river surface velocity based on physical constraints and multimodal fusion. Background Technology

[0002] Accurate measurement of river surface velocity is of great significance for hydrological monitoring, flood warning, and ecological environmental protection. Traditional methods for measuring river velocity mainly include the current meter method and the tracer method. These methods usually require on-site operation by professionals, are costly and inefficient, and are difficult to achieve large-scale, continuous monitoring.

[0003] With the development of computer vision technology, optical flow methods based on image sequences have been gradually applied to the measurement of river surface velocity. However, in complex water flow scenarios, traditional optical flow methods face problems such as low accuracy and poor anti-interference ability. Therefore, how to embed shallow water equation constraints into optical flow estimation and fuse multimodal data to achieve high-precision and robust measurement of river surface velocity is the technical problem to be solved by this invention. To this end, a method and system for measuring river surface velocity based on physical constraints and multimodal fusion is proposed. Summary of the Invention

[0004] The purpose of this invention is to provide a method and system for measuring river surface velocity based on physical constraints and multimodal fusion, so as to solve the problems mentioned in the background art.

[0005] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0006] The first aspect is a method for measuring river surface velocity based on physical constraints and multimodal fusion, which includes the following steps:

[0007] S1. Determine the target area and acquire the river surface image sequence data, depth information, and environmental data of the target area;

[0008] S2. Preprocess the collected data and embed redundant codes in the data representation for subsequent error detection, and combine them to form a basic dataset;

[0009] S3. Based on the basic dataset, preliminary optical flow estimation is performed on the river surface image sequence data using the optical flow method;

[0010] S4. Based on the preliminary optical flow estimation results, the shallow water equation is introduced to constrain the optical flow estimation model to a physical constraint, thereby obtaining optical flow information, making the optical flow estimation results conform to physical laws, and improving accuracy.

[0011] S5. Fuse optical flow information, depth information and environmental data to obtain fused flow velocity data. Then, combine the redundancy code in the data representation to perform error detection on the fused flow velocity data, identify and mark the existing erroneous data points.

[0012] S6. Based on the redundancy code detection results, the erroneous data is corrected. Using the fused flow velocity data and the corrected results, the final river surface flow velocity is calculated, and a high-precision, robust flow velocity measurement value is output.

[0013] A further improvement of the technical solution of the present invention is that: by using geographic information system software and field survey maps, combined with hydrological data and topographic features, the river section of the target area is initially delineated. At the same time, according to the size of the target area and the complexity of the terrain, measuring equipment is selected, and the measuring equipment is calibrated and its parameters are set to ensure that it can operate stably in the river field.

[0014] Deploy measurement equipment in the target area and acquire river surface image sequence data, depth information, and environmental data in the target area according to the set acquisition frequency;

[0015] After data acquisition, the acquired river surface image sequence data, depth information, and environmental data are transferred to the storage device. The data is then preliminarily organized, arranged in chronological order, and a unified file naming rule and directory structure are established.

[0016] A further improvement to the technical solution of the present invention is that: S2 specifically includes:

[0017] The collected river surface image sequence data, depth information and environmental data are preprocessed. At the same time, the timestamps of all data are checked to ensure that the river surface image sequence data, depth information and environmental data are completely synchronized in time.

[0018] Redundancy codes are embedded in the preprocessed data representation for subsequent error detection. Specifically, for river surface image sequence data, cyclic redundancy check codes are embedded in the metadata part of the file. For depth data and environmental data, CRC codes are added to the end of the data frame. When embedding redundancy codes, the generation algorithm and embedding position of the redundancy codes are recorded so that the redundancy codes can be accurately extracted and verified during subsequent data transmission and processing.

[0019] The preprocessed river surface image sequence data, depth information, and environmental data with embedded redundancy codes are integrated to form a basic dataset. The river surface image sequence data, depth information, and environmental data at the same time are linked together by timestamps, and the basic dataset is then validated.

[0020] A further improvement to the technical solution of the present invention is that: S3 specifically includes:

[0021] Based on the characteristics of the river surface image sequence data and the measurement accuracy requirements, the Horn-Schunck optical flow algorithm was selected, and its parameters were initialized, including setting the time window, maximum number of iterations and convergence threshold for optical flow calculation. At the same time, the weight of the smoothing term was adjusted.

[0022] The preprocessed river surface image sequence data is input into the selected optical flow algorithm to start the preliminary optical flow estimation calculation. The optical flow vector between two adjacent frames is calculated sequentially according to the order of the image sequence. At the same time, the calculation process is monitored in real time, the calculation time and intermediate results are recorded, and the calculated optical flow vector results are visualized.

[0023] The accuracy of preliminary optical flow estimation is measured by the evaluation metric of average endpoint error (AEE). The average endpoint error is calculated to determine the average Euclidean distance between the estimated optical flow vector and the true optical flow vector. Manually labeled data is used as a reference. The error value of the preliminary optical flow estimation result is calculated according to the evaluation metric to evaluate its accuracy and reliability. Based on the evaluation results, the problems in the preliminary optical flow estimation are analyzed. For the problems found, the optical flow algorithm or parameters are adjusted and optimized. Through multiple iterations, the preliminary optical flow estimation result meets the accuracy requirements.

[0024] A further improvement to the technical solution of this invention lies in the fact that the specific process of the preliminary optical flow estimation calculation includes:

[0025] The preprocessed river surface image sequence data is arranged in order to ensure that the image format is uniform and the timestamps are accurate. The image sequence is loaded into memory frame by frame using OpenCV's video reading function to build an image frame list. During the data input process, the integrity of the image is checked to avoid data loss or corruption.

[0026] Based on the selected Horn-Schunck optical flow algorithm, the initial configuration is performed, setting the time window, maximum number of iterations, and convergence threshold for optical flow calculation. At the same time, the weight of the smoothing term is adjusted, and an initial value is determined. After the initial configuration is completed, the optical flow algorithm is associated with the river surface image sequence data.

[0027] Following the order of the image sequence, two adjacent frames are selected as inputs, and the initialized Horn-Schunck optical flow algorithm is called to calculate the optical flow vector between the two adjacent frames. Specifically, the partial derivatives of the two adjacent frames in the x, y and time directions are calculated, and the optical flow vector is initialized to 0. The optical flow vector is iteratively solved until the maximum number of iterations or the error is less than the convergence threshold is met.

[0028] The calculated optical flow vector results are visualized to intuitively observe the distribution and motion characteristics of the optical flow field. Arrow diagrams are used to represent the direction and magnitude of the optical flow vector. The starting point of the arrow is the pixel position, the direction of the arrow indicates the motion direction of the pixel, which is determined by the optical flow vector, and the length of the arrow indicates the magnitude of the motion speed.

[0029] A further improvement to the technical solution of the present invention is that: S4 specifically includes:

[0030] Based on the preliminary optical flow estimation results, and using the shallow water equation as a foundation, the physical laws of the flow velocity field are embedded into the optical flow estimation to construct a physically constrained optical flow estimation model. In this model, the constraint terms of the shallow water equation are added as regularization terms to the optimization objective function, forming an optical flow estimation model that combines physical constraints.

[0031] The continuity and momentum equations of the shallow water equations are introduced into the physically constrained optical flow estimation model. The constraint terms of the shallow water equations are expressed as a penalty term and added to the optimization objective function. The optical flow estimation results are gradually adjusted by the gradient descent method to satisfy the physical constraints.

[0032] An optimization algorithm is used to solve the physical constraint optical flow estimation model. The optical flow estimation results are iteratively adjusted to gradually satisfy the constraints of the shallow water equation. During the optimization process, the convergence of the model is monitored in real time, the optical flow estimation results after each iteration are recorded, and the difference between them and the initial optical flow estimation results is calculated. When the model converges or reaches the predetermined number of iterations, it is verified whether the optimized optical flow estimation results conform to physical laws.

[0033] A further improvement to the technical solution of this invention lies in the following: the specific process of gradually adjusting the optical flow estimation result using the iterative optimization algorithm of gradient descent includes:

[0034] To address the discrete grid characteristics of river surface images, the continuity and momentum equations of the shallow water equations are discretized. Constraints related to the velocity field (corresponding to the optical flow vector) are extracted from the discretized equations and transformed into a form suitable for embedding the optical flow estimation model.

[0035] Based on the preliminary optical flow estimation results, an optimization objective function for optical flow estimation is constructed. The extracted shallow water equation constraint terms are added as penalty terms to the optimization objective function to form an optical flow estimation model that combines physical constraints.

[0036] The optical flow estimation result is gradually adjusted using an iterative optimization algorithm based on gradient descent to satisfy the constraints of the shallow water equation. In each iteration, the gradient of the objective function with respect to the optical flow vector is calculated, and the initial iteration step size is determined based on the gradient information. Then, iterative optimization is performed according to the rules of gradient descent to update the optical flow vector. At the same time, the optical flow estimation result after each iteration is recorded. In each iteration, the value of the optical flow vector is adjusted according to the current gradient and step size to obtain a new optical flow estimation result. The new objective function value is calculated and compared with the objective function value of the previous iteration to determine whether the objective function value has decreased. If it has decreased, the iteration continues; if it has increased, the step size is adjusted and the iteration is restarted.

[0037] A further improvement to the technical solution of the present invention is that S5 specifically includes:

[0038] The preliminarily optimized optical flow information, depth information, and environmental data are fused. Specifically, the optical flow information is converted into velocity field data, and the velocity is normalized by combining it with depth information to eliminate the influence of different depths on the velocity. At the same time, the velocity field is corrected according to the environmental data. The weighted average method is used to fuse the multimodal data to obtain the fused velocity data. During the fusion process, the temporal and spatial alignment of each data mode is ensured.

[0039] Redundancy codes are embedded in the fused flow rate data representation. According to the redundancy code generation algorithm, the check code of each data block is calculated and compared with the stored redundancy code. If the check codes do not match, the data block is marked as a data point with errors. The redundancy code adopts cyclic redundancy check code and is embedded in the storage format of the flow rate data. The redundancy code verification mechanism is used to detect errors in the fused flow rate data.

[0040] The marked erroneous data points are analyzed, and their locations, timestamps, and error types are recorded. The causes of the errors are also analyzed. For marked erroneous data points, they are temporarily excluded from subsequent calculations. At the same time, the proportion and distribution of erroneous data points are statistically analyzed to evaluate the overall quality of the data. Based on the analysis results, the data fusion algorithm is further optimized or the embedding strategy of redundant codes is adjusted.

[0041] A further improvement to the technical solution of the present invention is that: S6 specifically includes:

[0042] Based on the redundancy code detection results, the data points marked as errors are corrected. A statistical outlier detection method is used to repair the erroneous data points. For a single erroneous data point, interpolation correction is performed using valid data in its neighborhood. For consecutive erroneous data blocks, a moving average method is used for smoothing to restore the continuity of the data. The corrected data points replace the original erroneous data points to form the corrected flow rate dataset.

[0043] The corrected flow velocity data is reintegrated into the fused flow velocity dataset, replacing the original erroneous data points. Based on the updated fused data, the final river surface flow velocity is calculated. The flow velocity is weighted using depth information, and the flow velocity at each location is calculated using a weighted average method.

[0044] The calculated river surface velocity is verified by comparing the results with known velocity field data (obtained through traditional measurement methods) to assess its accuracy and reliability. The average endpoint error is calculated to ensure the accuracy of the final velocity measurement. If the verification results meet the accuracy requirements, the final velocity measurement value, including the magnitude and direction of the velocity, is output. If the verification results still have large errors, the data correction algorithm is further optimized or the fusion strategy is readjusted until a high-precision and robust velocity measurement value is obtained.

[0045] Secondly, a river surface velocity measurement system based on physical constraints and multimodal fusion is provided to implement the aforementioned river surface velocity measurement method based on physical constraints and multimodal fusion. The system includes a velocity measurement platform, which is communicatively connected to the following modules:

[0046] The data acquisition module is used to acquire sequence data of river surface images, depth information, and environmental data to ensure the integrity and diversity of the data;

[0047] The data preprocessing module is used to denoise, smooth, unify units, and embed redundant codes into the acquired data to form a basic dataset, improve data quality, and reduce noise interference.

[0048] The optical flow estimation module is used to perform preliminary optical flow estimation on the preprocessed river surface image sequence data using the optical flow method, and to preliminarily determine the flow velocity distribution on the river surface.

[0049] The physical constraint fusion module is used to combine shallow water equation constraints to optimize optical flow estimation results, improve the accuracy and physical rationality of flow velocity estimation, make the results conform to the laws of fluid dynamics, and fuse optical flow information, depth information and environmental data to perform error detection and correction, obtain fused flow velocity data, realize the comprehensive utilization of multi-source data, and improve the accuracy and robustness of flow velocity measurement.

[0050] The velocity calculation and verification module calculates the surface velocity of the river based on the fused velocity data, verifies its accuracy, and outputs the final velocity measurement value.

[0051] Due to the adoption of the above technical solution, the technical progress achieved by this invention compared to the prior art is as follows:

[0052] 1. This invention provides a method and system for measuring river surface velocity based on physical constraints and multimodal fusion. By fusing optical flow information, depth information and environmental data, it achieves comprehensive utilization of multi-source data. During the data fusion process, it combines redundancy codes in the data representation for error detection and correction, effectively identifying and correcting errors in the data, improving the reliability and accuracy of the data. By utilizing the multimodal data fusion and error correction mechanism, it significantly enhances the robustness of the system, enabling it to operate stably in complex and ever-changing river environments.

[0053] 2. This invention provides a method and system for measuring river surface velocity based on physical constraints and multimodal fusion. By embedding shallow water equation constraints into optical flow estimation, the velocity estimation results not only conform to image change characteristics but also satisfy fluid dynamics laws. By introducing physical constraints, the problem of low accuracy of traditional optical flow methods in complex water flow scenarios is effectively overcome, and the measurement accuracy of river surface velocity is significantly improved. Attached Figure Description

[0054] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0055] Figure 1 This is a schematic diagram of the workflow of the present invention;

[0056] Figure 2 This is a schematic diagram of the method flow of the present invention. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0058] Example 1, as Figure 1 , Figure 2As shown, this invention provides a method for measuring river surface velocity based on physical constraints and multimodal fusion, comprising the following steps:

[0059] S1. Determine the target area and acquire river surface image sequence data, depth information, and environmental data for the target area. Using Geographic Information System (GIS) software and on-site survey maps, combined with hydrological data and topographic features, preliminarily delineate the river sections within the target area. Simultaneously, select surveying equipment based on the size and topographic complexity of the target area. For image acquisition, prepare a high-resolution, high-frame-rate UAV-mounted camera, ensuring sufficient zoom capability to adapt to shooting needs at different distances. For depth information acquisition, use a high-precision LiDAR with a stable support. For environmental data acquisition, configure a meteorological sensor. Calibrate and set the parameters of the surveying equipment to ensure stable operation on the river. Deploy the surveying equipment in the target area and collect data at the set frequency. The system acquires sequence data of river surface images, depth information, and environmental data for the target area. Specifically, a drone camera is used to capture detailed features of the river surface by aiming at a predetermined shooting position, adjusting the angle and focal length, and taking sequence images of the river surface at set time intervals. Simultaneously, a lidar is operated to acquire depth information of the river surface according to a preset scanning path. During the acquisition of images and depth data, meteorological sensors are activated to record environmental parameters such as wind speed, wind direction, air temperature, and water temperature, covering different water flow conditions and changes in light intensity. After data acquisition is completed, the acquired sequence data of river surface images, depth information, and environmental data are transferred to a storage device, and the data is initially organized, arranged in chronological order, and a unified file naming rule and directory structure are established.

[0060] S2. Preprocess the collected data and embed redundant codes in the data representation for subsequent error detection. This forms a basic dataset. The preprocessing involves the collected river surface image sequence data, depth information, and environmental data. Specifically, for the river surface image sequence data, denoising is performed to remove image noise caused by factors such as illumination changes and sensor noise, and contrast enhancement is applied to highlight detailed features. Depth data undergoes smoothing and outlier removal, eliminating points that clearly do not conform to the water depth range. Environmental data undergoes unit unification and dimension normalization to ensure data consistency between different environmental parameters. Simultaneously, the timestamps of all data are checked to ensure complete temporal synchronization of the river surface image sequence data, depth information, and environmental data. Redundant codes are embedded in the preprocessed data representation for subsequent error detection. Specifically, for the river surface image sequence data... The metadata portion of the file embeds a Cyclic Redundancy Check (CRC) code. For depth and environmental data, a CRC code is added to the end of the data frame. When embedding the redundancy code, the generation algorithm and embedding position of the redundancy code are recorded so that the redundancy code can be accurately extracted and verified during subsequent data transmission and processing. The preprocessed river surface image sequence data, depth information, and environmental data with embedded redundancy codes are integrated to form a basic dataset. The river surface image sequence data, depth information, and environmental data at the same time are linked together by timestamps, and the basic dataset is verified to check the integrity and consistency of the data. The verification includes: checking whether all data have successfully embedded redundancy codes, whether the redundancy codes can be correctly extracted and verified, whether the time series of the data is continuous, whether there is any data loss or disordered time order, and whether the data of different modalities can correspond accurately.

[0061] S3. Based on the basic dataset, preliminary optical flow estimation is performed on the river surface image sequence data using the optical flow method. According to the characteristics of the river surface image sequence data and the measurement accuracy requirements, the Horn-Schunck optical flow algorithm is selected. Its parameters are initialized, including setting the time window for optical flow calculation, the maximum number of iterations, and the convergence threshold. Simultaneously, the weight of the smoothing term is adjusted; too much weight will make the optical flow field too smooth, losing local features, while too little weight may introduce too much noise. Through multiple experiments and comparisons, the optimal parameter combination is determined to improve the accuracy of the preliminary optical flow estimation. The preprocessed river surface image sequence data is input into the selected optical flow algorithm to begin the preliminary optical flow estimation calculation. Following the order of the image sequence, the optical flow vector between adjacent frames is calculated sequentially. Simultaneously, the calculation process is monitored in real time, recording the calculation time and intermediate results. The obtained optical flow vector results are visualized, using arrows to represent the direction and magnitude of the optical flow vector. The starting point of the arrow is the pixel position, the direction indicates the movement direction of the pixel, and the length represents the movement speed. The accuracy of the preliminary optical flow estimation is measured based on the average endpoint error (AEE) evaluation metric. The AEE calculates the average Euclidean distance between the estimated optical flow vector and the true optical flow vector, using manually labeled data as a reference. The error value of the preliminary optical flow estimation result is calculated according to the evaluation metric to evaluate its accuracy and reliability. Based on the evaluation results, the problems existing in the preliminary optical flow estimation are analyzed. If the error is large, the appropriateness of the optical flow algorithm selection, the rationality of the parameter settings, and the adequacy of image preprocessing are checked. For the problems found, the optical flow algorithm or parameters are adjusted and optimized. Through multiple iterations, the preliminary optical flow estimation result meets the accuracy requirements.

[0062] In addition, the specific process of preliminary optical flow estimation calculation includes:

[0063] The preprocessed river surface image sequence data was arranged sequentially to ensure uniform image format and accurate timestamps. OpenCV's video loading functions were used to load the image sequence frame by frame into memory, constructing an image frame list. During data input, image integrity was checked to prevent data loss or corruption. Based on the selected Horn-Schunck optical flow algorithm, initial configuration was performed, setting the time window, maximum number of iterations, and convergence threshold for optical flow calculation. Simultaneously, the weight of the smoothing term was adjusted, and an initial value was determined. After initialization, the optical flow algorithm was associated with the river surface image sequence data. Adjacent images were selected sequentially according to the order of the image sequence. The frame image is used as input, and the initialized Horn-Schunck optical flow algorithm is called to calculate the optical flow vector between two adjacent frames. The partial derivatives of the two adjacent frames in the x, y and time directions are calculated, and the optical flow vector is initialized to 0. The optical flow vector is solved iteratively until the maximum number of iterations or the error is less than the convergence threshold is met. The calculated optical flow vector results are visualized to intuitively observe the distribution and motion characteristics of the optical flow field. The direction and magnitude of the optical flow vector are represented by an arrow diagram. The starting point of the arrow is the pixel position, the direction of the arrow indicates the motion direction of the pixel, which is determined by the optical flow vector, and the length of the arrow indicates the magnitude of the motion speed.

[0064] The Horn-Schunck optical flow method estimates the optical flow field by minimizing the following energy function:

[0065] ;

[0066] In the formula, , and These represent the partial derivatives of the image in the x, y, and time directions, respectively. and These represent the velocity components in the x and y directions, respectively. For the weights of the smoothing term, For the image domain, the Horn-Schunck optical flow method is a classic optical flow estimation method. It is based on the assumption of the continuity of pixel intensity in an image sequence, that is, the intensity value of a pixel remains unchanged between two consecutive frames. Based on this assumption, Horn and Schunk derived the energy function for optical flow estimation, assuming that there are two consecutive frames in the image sequence. and ,in and These are spatial coordinates. Given time, the optical flow constraint equation can be expressed as: , and They are optical flow in direction and The directional component, the Horn-Schunck method estimates the optical flow field by minimizing an energy function. This energy function consists of two parts: a data term and a regularization term. The data term ensures that the optical flow constraint equations hold at every pixel, while the regularization term smooths the optical flow field, making it spatially smooth. Ensure that the optical flow field is spatially smooth. and They represent and The sum of squares of the gradients, i.e.: , ;in, , , and They are and exist and By minimizing the energy function using the partial derivatives in the direction of light, the optical flow field can be obtained. and ;

[0067] The expression for iteratively solving the optical flow vector is:

[0068] ;

[0069] ;

[0070] In the formula, For the first The optical flow vector components of the next iteration For the number of iterations, These are the components of the optical flow vector in the x and y directions;

[0071] S4. Based on the preliminary optical flow estimation results, the shallow water equation is introduced to constrain the optical flow estimation model to a physical constraint, thereby obtaining optical flow information, making the optical flow estimation results conform to physical laws, and improving accuracy.

[0072] S5. Fuse optical flow information, depth information and environmental data to obtain fused flow velocity data. Then, combine the redundancy code in the data representation to perform error detection on the fused flow velocity data, identify and mark the existing erroneous data points.

[0073] S6. Based on the redundancy code detection results, the erroneous data is corrected. Using the fused flow velocity data and the corrected results, the final river surface flow velocity is calculated, and a high-precision, robust flow velocity measurement value is output.

[0074] Example 2, as Figure 1 , Figure 2As shown, based on Embodiment 1, the present invention provides a technical solution: preferably, S4 specifically includes:

[0075] Based on preliminary optical flow estimation results, and using the shallow water equations as a foundation, the physical laws of the velocity field are embedded into the optical flow estimation to construct a physically constrained optical flow estimation model. An optimization algorithm is then used to adjust the optical flow estimation results to satisfy the constraints of the shallow water equations, thereby improving the accuracy and physical plausibility of the velocity estimation. Specifically, the constraint terms of the shallow water equations are added as regularization terms to the optimization objective function, forming a physically constrained optical flow estimation model. The continuity and momentum equations of the shallow water equations are introduced into the physically constrained optical flow estimation model. The constraint terms of the shallow water equations are represented as penalty terms and added to the optimization objective function. An iterative optimization algorithm using gradient descent is then used to gradually adjust the optical flow estimation results to satisfy the physical constraints. The conditions are as follows: the continuity equation ensures the mass conservation of the velocity field, and the momentum equation describes the motion of the fluid under the action of gravity and pressure gradient. By embedding physical constraints into the optical flow estimation process, the optical flow estimation results not only conform to the motion characteristics of the image sequence, but also satisfy the basic laws of fluid dynamics. The physical constraint optical flow estimation model is solved using an optimization algorithm. The optical flow estimation results are iteratively adjusted to gradually satisfy the constraint conditions of the shallow water equation. During the optimization process, the convergence of the model is monitored in real time, the optical flow estimation results after each iteration are recorded, and the difference between them and the initial optical flow estimation results is calculated. When the model converges or reaches the predetermined number of iterations, it is verified whether the optimized optical flow estimation results conform to physical laws.

[0076] Furthermore, the specific process of gradually adjusting the optical flow estimation results using the iterative optimization algorithm of gradient descent includes:

[0077] To address the discrete grid characteristics of river surface images, the continuity and momentum equations of the shallow water equations are discretized. Constraints related to the velocity field (corresponding to optical flow vectors) are extracted from the discretized equations and transformed into a form suitable for embedding in the optical flow estimation model. Based on the preliminary optical flow estimation results, an optimization objective function for optical flow estimation is constructed. The extracted shallow water equation constraint terms are added as penalty terms to the optimization objective function, forming an optical flow estimation model incorporating physical constraints. An iterative optimization algorithm using gradient descent is used to progressively adjust the optical flow estimation results to satisfy the constraints of the shallow water equations. In each iteration, the gradient of the objective function with respect to the optical flow vector is calculated, and the initial iteration step size is determined based on the gradient information. Then, iterative optimization is performed according to the rules of gradient descent, and the optical flow vector is updated. At the same time, the optical flow estimation result after each iteration is recorded. In each iteration, the value of the optical flow vector is adjusted according to the current gradient and step size to obtain a new optical flow estimation result. The new objective function value is calculated and compared with the objective function value of the previous iteration to determine whether the objective function value has decreased. If it has decreased, the iteration continues; if it has increased, the step size is adjusted and the iteration is repeated.

[0078] The expression for the objective function to be optimized is:

[0079] ;

[0080] In the formula, , and These represent the partial derivatives of the image in the x, y, and time directions, respectively. and These represent the velocity components in the x and y directions, respectively. For the weights of the smoothing term, For the image domain;

[0081] The expression for the constraint terms of the shallow water equation is:

[0082] ;

[0083] In the formula, and This is a regularization parameter used to control the weight of the shallow water equation constraint terms in the optimization objective function. Because of the water depth, For the velocity vector, Represented as a vector, and It consists of two components. It is the acceleration due to gravity. This refers to the water surface elevation. For time, For gradient operators, This represents the cross product of the velocity vector and itself. Indicates water depth The divergence of the cross product of the velocity vector and itself is a scalar value used to describe the motion of the fluid under the influence of gravity and pressure gradient;

[0084] The expression for the continuity equation is:

[0085] ;

[0086] The expression for the momentum equation is:

[0087] ;

[0088] The expression for updating the optical flow vector is:

[0089] ;

[0090] ;

[0091] In the formula, For learning rate, To optimize the objective function;

[0092] S5 specifically includes:

[0093] The preliminarily optimized optical flow information, depth information, and environmental data are fused. Specifically, the optical flow information is converted into velocity field data, and the velocity is normalized by combining it with depth information to eliminate the influence of different depths on the velocity. Simultaneously, the velocity field is corrected based on environmental data. A weighted average method is used to fuse the multimodal data to obtain the fused velocity data. During the fusion process, temporal and spatial alignment of each data mode is ensured. Redundancy codes are embedded in the fused velocity data representation. A checksum for each data block is calculated according to the redundancy code generation algorithm and compared with the stored redundancy codes. If the checksums do not match... If a data block is found to contain an error, it is marked as an erroneous data point. The redundancy code uses a cyclic redundancy check code, which is embedded in the storage format of the flow rate data. The redundancy code's verification mechanism is used to detect errors in the fused flow rate data. The marked erroneous data points are analyzed, and their locations, timestamps, and error types are recorded. The causes of the errors are also analyzed. For marked erroneous data points, they are temporarily excluded from subsequent calculations. At the same time, the proportion and distribution of erroneous data points are statistically analyzed to assess the overall quality of the data. Based on the analysis results, the data fusion algorithm is further optimized or the redundancy code embedding strategy is adjusted.

[0094] S6 specifically includes:

[0095] Based on the redundancy code detection results, data points marked as erroneous are corrected. A statistical outlier detection method is used to repair these erroneous data points. For a single erroneous data point, interpolation correction is performed using valid data within its neighborhood. For consecutive erroneous data blocks, a moving average method is used for smoothing to restore data continuity. The corrected data points replace the original erroneous data points, forming a corrected flow velocity dataset. The corrected flow velocity data is then reintegrated into the fused flow velocity dataset, replacing the original erroneous data points. Based on the updated fused data, the final river surface flow velocity is calculated. The flow velocity is weighted using depth information, and the flow velocity at each location is calculated using a weighted average method. The final calculated river surface flow velocity is then verified by comparing the results with known flow velocity field data (data obtained through traditional measurement methods) to evaluate its accuracy and reliability. The average endpoint error is calculated to ensure the accuracy of the final flow velocity measurement. If the verification results meet the accuracy requirements, the final flow velocity measurement value, including the magnitude and direction of the flow velocity, is output. If the verification results still have large errors, the data correction algorithm is further optimized or the fusion strategy is readjusted until a high-precision and robust flow velocity measurement value is obtained.

[0096] The expression for calculating the flow velocity at each location using the weighted average method is as follows:

[0097] ;

[0098] In the formula, For the fused flow velocity data, This is the depth information for the corresponding location. This refers to the final calculated surface velocity of the river.

[0099] The expression for calculating the average endpoint error is:

[0100] ;

[0101] In the formula, The average endpoint error, Given the known actual flow velocity value, This represents the total number of data points.

[0102] Example 3, as Figure 1 , Figure 2 As shown, based on Examples 1-2, this invention also provides a river surface velocity measurement system based on physical constraints and multimodal fusion, used to implement the above-mentioned river surface velocity measurement method based on physical constraints and multimodal fusion. The system includes a velocity measurement platform, which is communicatively connected to the following modules:

[0103] The data acquisition module is used to acquire sequence data of river surface images, depth information, and environmental data to ensure the integrity and diversity of the data;

[0104] The data preprocessing module is used to denoise, smooth, unify units, and embed redundant codes into the acquired data to form a basic dataset, improve data quality, and reduce noise interference.

[0105] The optical flow estimation module is used to perform preliminary optical flow estimation on the preprocessed river surface image sequence data using the optical flow method, and to preliminarily determine the flow velocity distribution on the river surface.

[0106] The physical constraint fusion module is used to combine shallow water equation constraints to optimize optical flow estimation results, improve the accuracy and physical rationality of flow velocity estimation, make the results conform to the laws of fluid dynamics, and fuse optical flow information, depth information and environmental data to perform error detection and correction, obtain fused flow velocity data, realize the comprehensive utilization of multi-source data, and improve the accuracy and robustness of flow velocity measurement.

[0107] The velocity calculation and verification module calculates the surface velocity of the river based on the fused velocity data, verifies its accuracy, and outputs the final velocity measurement value.

[0108] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for measuring river surface velocity based on physical constraints and multimodal fusion, characterized in that, Includes the following steps: S1. Determine the target area and acquire the river surface image sequence data, depth information, and environmental data of the target area; S2. Preprocess the collected data and embed redundant codes in the data representation to form a basic dataset, specifically including: The collected river surface image sequence data, depth information and environmental data are preprocessed. At the same time, the timestamps of all data are checked to synchronize the river surface image sequence data, depth information and environmental data in time. Redundancy codes are embedded in the preprocessed data representation. For river surface image sequence data, cyclic redundancy check codes are embedded in the metadata part of the file. For depth data and environmental data, CRC codes are added to the end of the data frame. When embedding redundancy codes, the generation algorithm and embedding position of the redundancy codes are recorded. The preprocessed river surface image sequence data, depth information, and environmental data with embedded redundancy codes are integrated to form a basic dataset. The river surface image sequence data, depth information, and environmental data at the same time are linked together by timestamps, and the basic dataset is verified. S3. Based on the basic dataset, preliminary optical flow estimation is performed on the river surface image sequence data using the optical flow method; S4. Based on the preliminary optical flow estimation results, a shallow water equation-constrained to physically constrained optical flow estimation model is introduced to obtain optical flow information, specifically including: Based on the preliminary optical flow estimation results, and using the shallow water equation as a foundation, the physical laws of the flow velocity field are embedded into the optical flow estimation to construct a physically constrained optical flow estimation model. In this model, the constraint terms of the shallow water equation are added as regularization terms to the optimization objective function, forming an optical flow estimation model that combines physical constraints. The continuity and momentum equations of the shallow water equations are introduced into the physically constrained optical flow estimation model. The constraint terms of the shallow water equations are expressed as a penalty term and added to the optimization objective function. The optical flow estimation results are gradually adjusted by the gradient descent method to satisfy the physical constraints. An optimization algorithm is used to solve the physical constraint optical flow estimation model. The optical flow estimation results are iteratively adjusted to gradually satisfy the constraint conditions of the shallow water equation. During the optimization process, the convergence of the model is monitored in real time, the optical flow estimation results after each iteration are recorded, and the difference between them and the initial optical flow estimation results is calculated. When the model converges or reaches the predetermined number of iterations, it is verified whether the optimized optical flow estimation results conform to physical laws. S5. Fuse optical flow information, depth information and environmental data to obtain fused flow velocity data. Then, combine the redundancy code in the data representation to perform error detection on the fused flow velocity data, identify and mark the existing erroneous data points. S6. Based on the redundancy code detection results, the erroneous data is corrected, and the final river surface velocity is calculated using the fused flow velocity data and the corrected results.

2. The method for measuring river surface velocity based on physical constraints and multimodal fusion according to claim 1, characterized in that: S1 specifically includes: Using geographic information system software and field survey maps, combined with hydrological data and topographic features, the river sections in the target area were initially delineated. At the same time, based on the size of the target area and the complexity of the terrain, surveying equipment was selected, and the equipment was calibrated and its parameters were set. Deploy measurement equipment in the target area and acquire river surface image sequence data, depth information, and environmental data in the target area according to the set acquisition frequency; After data acquisition, the acquired river surface image sequence data, depth information, and environmental data are transferred to the storage device. The data is then preliminarily organized, arranged in chronological order, and a unified file naming rule and directory structure are established.

3. The method for measuring river surface velocity based on physical constraints and multimodal fusion according to claim 1, characterized in that: S3 specifically includes: Based on the characteristics of the river surface image sequence data and the measurement accuracy requirements, the Horn-Schunck optical flow algorithm was selected, and its parameters were initialized, including setting the time window, maximum number of iterations and convergence threshold for optical flow calculation. At the same time, the weight of the smoothing term was adjusted. The preprocessed river surface image sequence data is input into the selected optical flow algorithm to start the preliminary optical flow estimation calculation. The optical flow vector between two adjacent frames is calculated sequentially according to the order of the image sequence. At the same time, the calculation process is monitored in real time, the calculation time and intermediate results are recorded, and the calculated optical flow vector results are visualized. The evaluation index based on average endpoint error measures the accuracy of preliminary optical flow estimation. The average endpoint error is calculated to determine the average Euclidean distance between the estimated optical flow vector and the true optical flow vector. Manually labeled data is used as a reference. The error value of the preliminary optical flow estimation result is calculated according to the evaluation index to evaluate its accuracy and reliability. Based on the evaluation results, the problems existing in the preliminary optical flow estimation are analyzed. For the problems found, the optical flow algorithm or parameters are adjusted and optimized. Through multiple iterations, the preliminary optical flow estimation result meets the accuracy requirements.

4. The method for measuring river surface velocity based on physical constraints and multimodal fusion according to claim 3, characterized in that: The specific process of the preliminary optical flow estimation calculation includes: The preprocessed river surface image sequence data is arranged in order, and the image sequence is loaded into memory frame by frame using OpenCV's video reading function to build an image frame list during the data input process; Based on the selected Horn-Schunck optical flow algorithm, the initial configuration is performed, setting the time window, maximum number of iterations, and convergence threshold for optical flow calculation. At the same time, the weight of the smoothing term is adjusted, and an initial value is determined. After the initial configuration is completed, the optical flow algorithm is associated with the river surface image sequence data. Following the order of the image sequence, two adjacent frames are selected as inputs, and the initialized Horn-Schunck optical flow algorithm is called to calculate the optical flow vector between the two adjacent frames. Specifically, the partial derivatives of the two adjacent frames in the x, y and time directions are calculated, and the optical flow vector is initialized to 0. The optical flow vector is iteratively solved until the maximum number of iterations or the error is less than the convergence threshold is met. The calculated optical flow vector results are visualized using arrows to represent the direction and magnitude of the optical flow vector. The starting point of the arrow is the pixel position, the direction of the arrow indicates the direction of movement of the pixel, which is determined by the optical flow vector, and the length of the arrow indicates the magnitude of the movement speed.

5. The method for measuring river surface velocity based on physical constraints and multimodal fusion according to claim 1, characterized in that: The specific process of gradually adjusting the optical flow estimation results using the iterative optimization algorithm of gradient descent includes: To address the discrete grid characteristics of river surface images, the continuity and momentum equations of the shallow water equations are discretized, and constraints related to the flow velocity field are extracted from the discretized equations. Based on the preliminary optical flow estimation results, an optimization objective function for optical flow estimation is constructed. The extracted shallow water equation constraint terms are added as penalty terms to the optimization objective function to form an optical flow estimation model that combines physical constraints. The optical flow estimation result is gradually adjusted using an iterative optimization algorithm based on gradient descent to satisfy the constraints of the shallow water equation. In each iteration, the gradient of the objective function with respect to the optical flow vector is calculated, and the initial iteration step size is determined based on the gradient information. Then, iterative optimization is performed according to the rules of gradient descent to update the optical flow vector. At the same time, the optical flow estimation result after each iteration is recorded. In each iteration, the value of the optical flow vector is adjusted according to the current gradient and step size to obtain a new optical flow estimation result. The new objective function value is calculated and compared with the objective function value of the previous iteration to determine whether the objective function value has decreased. If it has decreased, the iteration continues; if it has increased, the step size is adjusted and the iteration is restarted.

6. The method for measuring river surface velocity based on physical constraints and multimodal fusion according to claim 1, characterized in that: S5 specifically includes: The preliminarily optimized optical flow information, depth information, and environmental data are fused. Specifically, the optical flow information is converted into velocity field data, and the velocity is normalized by combining the depth information. At the same time, the velocity field is corrected according to the environmental data. The weighted average method is used to fuse the multimodal data to obtain the fused velocity data. Redundancy codes are embedded in the fused flow rate data representation. According to the redundancy code generation algorithm, the check code of each data block is calculated and compared with the stored redundancy code. If the check codes do not match, the data block is marked as a data point with errors. The redundancy code adopts cyclic redundancy check code and is embedded in the storage format of the flow rate data. The redundancy code verification mechanism is used to detect errors in the fused flow rate data. The marked erroneous data points are analyzed, and their locations, timestamps, and error types are recorded. The causes of the errors are also analyzed. For marked erroneous data points, they are temporarily excluded from subsequent calculations. At the same time, the proportion and distribution of erroneous data points are statistically analyzed to evaluate the overall quality of the data. Based on the analysis results, the data fusion algorithm is further optimized or the embedding strategy of redundant codes is adjusted.

7. The method for measuring river surface velocity based on physical constraints and multimodal fusion according to claim 6, characterized in that: S6 specifically includes: Based on the redundancy code detection results, the data points marked as errors are corrected. A statistical outlier detection method is used to repair the erroneous data points. For a single erroneous data point, interpolation correction is performed using valid data in its neighborhood. For consecutive erroneous data blocks, a moving average method is used for smoothing to restore the continuity of the data. The corrected data points replace the original erroneous data points to form the corrected flow rate dataset. The corrected flow velocity data is reintegrated into the fused flow velocity dataset, replacing the original erroneous data points. Based on the updated fused data, the final river surface flow velocity is calculated. The flow velocity is weighted using depth information, and the flow velocity at each location is calculated using a weighted average method. The final calculated river surface velocity is verified by comparing the calculation results with known velocity field data to evaluate its accuracy and reliability. The average endpoint error is calculated. If the verification results meet the accuracy requirements, the final velocity measurement value, including the magnitude and direction of the velocity, is output. If the verification results still have large errors, the data correction algorithm is further optimized or the fusion strategy is readjusted.

8. A river surface velocity measurement system based on physical constraints and multimodal fusion, used to implement the river surface velocity measurement method based on physical constraints and multimodal fusion as described in any one of claims 1-7, comprising a velocity measurement platform, characterized in that, The flow velocity measurement platform has the following communication connection modules, wherein: The data acquisition module is used to acquire sequence data of river surface images, depth information, and environmental data; The data preprocessing module is used to denoise, smooth, unify units, and embed redundant codes into the acquired data to form a basic dataset. The optical flow estimation module is used to perform preliminary optical flow estimation on the preprocessed river surface image sequence data using the optical flow method, and to preliminarily determine the flow velocity distribution on the river surface. The physical constraint fusion module is used to combine the shallow water equation constraint optimization of optical flow estimation results, and fuse optical flow information, depth information and environmental data to perform error detection and correction, and obtain fused flow velocity data; The velocity calculation and verification module calculates the surface velocity of the river based on the fused velocity data, verifies its accuracy, and outputs the final velocity measurement value.

Images