River surface flow velocity measurement method and system based on physical constraint and multi-modal fusion

Abstract

The invention discloses a river surface flow velocity measurement method and system based on physical constraint and multi-modal fusion, and relates to the technical field of multi-modal data processing, a target area is determined, and river surface image sequence data, depth information and environment data of the target area are acquired; preprocessing the collected data, embedding redundant codes in data representation, and synthesizing to form a basic data set; and based on the basic data set, performing preliminary optical flow estimation on the river surface image sequence data by using an optical flow method. According to the method, optical flow information, depth information and environment data are fused, comprehensive utilization of multi-source data is achieved, in the data fusion process, redundant codes in data representation are combined for error detection and correction, error points in the data are effectively recognized and corrected, the reliability and accuracy of the data are improved, and the data fusion efficiency is improved. And by utilizing a multi-modal data fusion and error correction mechanism, the robustness of the system is remarkably enhanced, so that the system can stably operate in a complex and changeable river environment.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal data processing, and in particular to a river surface velocity measurement method and system based on physical constraints and multimodal fusion. Background Art

[0002] Accurately measuring river surface velocity is crucial for hydrological monitoring, flood prevention and early warning, and ecological and environmental protection. Traditional methods for measuring river velocity, such as current meters and tracer methods, typically require on-site expertise, resulting in high costs, low efficiency, and difficulty achieving large-scale, continuous monitoring.

[0003] With the development of computer vision technology, the optical flow method based on image sequences has gradually been applied to river surface velocity measurement. However, in complex water flow scenarios, the traditional optical flow method faces problems such as low precision and poor anti-interference ability. Therefore, how to embed shallow water equation constraints in optical flow estimation and fuse multimodal data to achieve high-precision and robust measurement of river surface velocity is the technical problem to be solved by this invention. To this end, a river surface velocity measurement method and system based on physical constraints and multimodal fusion are proposed. Summary of the Invention

[0004] The purpose of the present invention is to provide a river surface velocity measurement method and system based on physical constraints and multimodal fusion to solve the problems raised in the above background technology.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is: First, a river surface velocity measurement method based on physical constraints and multimodal fusion includes the following steps: S1. Determine the target area and obtain river surface image sequence data, depth information and environmental data of the target area; S2, pre-processing the collected data and embedding redundant codes in the data representation for subsequent error detection to form a basic data set; S3. Based on the basic data set, the optical flow method is used to perform preliminary optical flow estimation on the river surface image sequence data; S4. Combined with the preliminary optical flow estimation results, the shallow water equation constraint is introduced into the physical constraint optical flow estimation model to obtain optical flow information, so that the optical flow estimation results conform to physical laws and improve accuracy. S5. Fusing the optical flow information, depth information, and environmental data to obtain fused velocity data, and then performing error detection on the fused velocity data in combination with redundant codes in the data representation, identifying and marking any erroneous data points. S6. Based on the redundant code detection results, the erroneous data is corrected, and the final river surface velocity is calculated using the fused velocity data and the corrected results, outputting a high-precision and robust velocity measurement value.

[0006] A further improvement of the technical solution of the present invention is to utilize geographic information system software and on-site survey maps, combined with hydrological data and terrain features, to preliminarily delineate the river sections in the target area. Simultaneously, based on the size of the target area and the complexity of the terrain, measurement equipment is selected, calibrated, and parameterized to ensure stable operation at the river site. Deploy measurement equipment in the target area and obtain river surface image sequence data, depth information and environmental data of the target area according to the set acquisition frequency; After data collection is completed, the acquired river surface image sequence data, depth information and environmental data are transferred to the storage device, and the data are preliminarily sorted, arranged in chronological order, and a unified file naming rule and directory structure are established.

[0007] A further improvement of the technical solution of the present invention is that: S2 specifically includes: Preprocess the collected river surface image sequence data, depth information and environmental data. At the same time, check the timestamps of all data to ensure that the river surface image sequence data, depth information and environmental data are fully synchronized in time. Redundant codes are embedded in the preprocessed data representation for subsequent error detection. For river surface image sequence data, a cyclic redundancy check code is embedded in the metadata portion of the file. For depth data and environmental data, a CRC code is added at the end of the data frame. When embedding redundant codes, the generation algorithm and embedding position of the redundant codes are recorded so that the redundant codes can be accurately extracted and verified during subsequent data transmission and processing. The river surface image sequence data, depth information and environmental data that have been preprocessed and embedded with redundant codes are integrated to form a basic data set, in which the river surface image sequence data, depth information and environmental data at the same moment are linked together through timestamps, and the basic data set is verified.

[0008] A further improvement of the technical solution of the present invention is that: S3 specifically includes: According to the characteristics of river surface image sequence data and the measurement accuracy requirements, the Horn-Schunck optical flow algorithm is selected and its parameters are initialized, including setting the time window of optical flow calculation, the maximum number of iterations and the convergence threshold. At the same time, the weight of the smoothing term is adjusted. The pre-processed river surface image sequence data is input into the selected optical flow algorithm to start the preliminary optical flow estimation calculation. The optical flow vector between two adjacent frames of the image is calculated in sequence according to the order of the image sequence. At the same time, the calculation process is monitored in real time, the calculation time and intermediate results are recorded, and the calculated optical flow vector results are visualized. The accuracy of the preliminary optical flow estimation is measured based on the average endpoint error (AEE) evaluation indicator, where the average endpoint error calculates the average Euclidean distance between the estimated optical flow vector and the true optical flow vector. Using some manually annotated data as a reference, the error value of the preliminary optical flow estimation result is calculated according to the evaluation indicator to evaluate its accuracy and reliability. Based on the evaluation results, the problems existing in the preliminary optical flow estimation are analyzed. According to the problems found, the optical flow algorithm or parameters are adjusted and optimized, and multiple iterative adjustments are made until the preliminary optical flow estimation result meets the accuracy requirements.

[0009] A further improvement of the technical solution of the present invention is that the specific process of the preliminary optical flow estimation calculation includes: The pre-processed river surface image sequence data was sorted in order to ensure uniform image format and accurate timestamps. The image sequence was loaded into memory frame by frame using OpenCV's video reading function. An image frame list was constructed. During the data input process, the image integrity was checked to avoid data loss or corruption. Based on the selected Horn-Schunck optical flow algorithm, the initial configuration is performed, and the time window, maximum number of iterations, and convergence threshold of the optical flow calculation are set. At the same time, the weight of the smoothing term is adjusted and an initial value is determined. After completing the initial configuration, the optical flow algorithm is associated with the river surface image sequence data; In the order of the image sequence, two adjacent frames are selected as input, and the initialized Horn-Schunck optical flow algorithm is called to calculate the optical flow vector between the two adjacent frames. The partial derivatives of the two adjacent frames in the x-direction, y-direction, and time direction are calculated, and the optical flow vector is initialized to 0. The optical flow vector is solved iteratively until the maximum number of iterations is met or the error is less than the convergence threshold. The calculated optical flow vector results are visualized to intuitively observe the distribution and motion characteristics of the optical flow field. Arrow diagrams are used to represent the direction and magnitude of the optical flow vector. The starting point of the arrow is the pixel position, and the direction of the arrow represents the direction of movement of the pixel, which is determined by the optical flow vector. The length of the arrow represents the speed of movement.

[0010] A further improvement of the technical solution of the present invention is that: S4 specifically includes: Based on the preliminary optical flow estimation results and the shallow water equation, the physical laws of the velocity field are embedded into the optical flow estimation, and a physically constrained optical flow estimation model is constructed. The constraint term of the shallow water equation is added as a regularization term to the optimization objective function, forming an optical flow estimation model that combines physical constraints. The continuity equation and momentum equation of the shallow water equation are introduced into the physical constraint optical flow estimation model. The constraint term of the shallow water equation is expressed as a penalty term and added to the optimization objective function. The optical flow estimation result is gradually adjusted through the iterative optimization algorithm of the gradient descent method to make it meet the physical constraint conditions. The physical constraint optical flow estimation model is solved using an optimization algorithm. The optical flow estimation results are iteratively adjusted to gradually satisfy the constraints of the shallow water equation. During the optimization process, the convergence of the model is monitored in real time, the optical flow estimation results after each iteration are recorded, and the difference between them and the preliminary optical flow estimation results is calculated. When the model converges or reaches the predetermined number of iterations, it is verified whether the optimized optical flow estimation results conform to physical laws.

[0011] A further improvement of the technical solution of the present invention is that the specific process of gradually adjusting the optical flow estimation result by the iterative optimization algorithm of the gradient descent method includes: Based on the discrete grid characteristics of river surface images, the continuity equation and momentum equation of the shallow water equation are discretized. The constraints related to the velocity field (corresponding to the optical flow vector) are extracted from the discretized equations and converted into a form suitable for embedding in the optical flow estimation model. Based on the preliminary optical flow estimation results, an optimization objective function for optical flow estimation is constructed. The extracted shallow water equation constraint term is added as a penalty term to the optimization objective function to form an optical flow estimation model that combines physical constraints. The optical flow estimation result is gradually adjusted through the iterative optimization algorithm of the gradient descent method to satisfy the constraints of the shallow water equation. In each iteration, the gradient of the optimization objective function with respect to the optical flow vector is calculated, and the initial iteration step size is determined based on the gradient information. Then, iterative optimization is performed according to the rules of the gradient descent method to update the optical flow vector. At the same time, the optical flow estimation result after each iteration is recorded. In each iteration, the value of the optical flow vector is adjusted according to the gradient and step size of the current optical flow vector to obtain a new optical flow estimation result, and a new optimization objective function value is calculated. It is compared with the optimization objective function value of the previous iteration to determine whether the optimization objective function value decreases. If it decreases, the iteration continues. If it increases, the step size is adjusted and the iteration is performed again.

[0012] A further improvement of the technical solution of the present invention is that: S5 specifically includes: The optical flow information, depth information and environmental data after preliminary optimization are fused. The optical flow information is converted into velocity field data. The velocity is normalized in combination with the depth information to eliminate the influence of different depths on the velocity. At the same time, the velocity field is corrected according to the environmental data. The multimodal data is fused using the weighted average method to obtain the fused velocity data. During the fusion process, the time and space alignment of each data modality is ensured. Redundant codes are embedded in the fused velocity data representation. Based on the redundant code generation algorithm, a check code is calculated for each data block and compared with the stored redundant code. If the check code does not match, the data block is marked as an erroneous data point. The redundant code adopts a cyclic redundancy check code and is embedded in the storage format of the velocity data. The check mechanism of the redundant code is used to perform error detection on the fused velocity data. Analyze the marked erroneous data points, record their location, timestamp, and error type, and analyze the cause of the error. Temporarily exclude the marked erroneous data points from subsequent calculations. At the same time, calculate the proportion and distribution of erroneous data points and evaluate the overall quality of the data. Based on the analysis results, further optimize the data fusion algorithm or adjust the redundant code embedding strategy.

[0013] A further improvement of the technical solution of the present invention is that: S6 specifically includes: According to the redundant code detection results, the data points marked as errors are corrected and repaired using a statistical outlier detection method. For a single error data point, valid data in its neighborhood are used for interpolation correction. For continuous error data blocks, a moving average method is used for smoothing to restore data continuity. The corrected data points replace the original error data points to form a corrected velocity data set. The corrected velocity data is reintegrated into the fused velocity dataset to replace the original erroneous data points. The final river surface velocity is calculated based on the updated fused data. The velocity is weighted using depth information, and the velocity at each location is calculated using a weighted average method. The final calculated river surface velocity is verified, and the calculated results are compared with the known velocity field data (data obtained by traditional measurement methods) to evaluate their accuracy and reliability, calculate the average endpoint error, and ensure the accuracy of the final velocity measurement value. If the verification result meets the accuracy requirements, the final velocity measurement value is output, including the magnitude and direction of the velocity. If the verification result still has a large error, further optimize the data correction algorithm or readjust the fusion strategy until a high-precision and robust velocity measurement value is obtained.

[0014] In a second aspect, a river surface velocity measurement system based on physical constraints and multimodal fusion is used to implement the above-mentioned river surface velocity measurement method based on physical constraints and multimodal fusion, including a velocity measurement platform, wherein the velocity measurement platform is communicatively connected to the following modules, wherein: Data acquisition module, used to obtain river surface image sequence data, depth information and environmental data to ensure data integrity and diversity; The data preprocessing module is used to remove noise, smooth, unify units, and embed redundant codes on the acquired data to form a basic data set, improve data quality, and reduce noise interference; The optical flow estimation module is used to perform preliminary optical flow estimation on the preprocessed river surface image sequence data using the optical flow method to preliminarily determine the flow velocity distribution on the river surface; The physical constraint fusion module is used to optimize the optical flow estimation results by combining the shallow water equation constraints, improve the accuracy and physical rationality of the velocity estimation, make the results conform to the laws of fluid dynamics, and fuse the optical flow information, depth information and environmental data to perform error detection and correction to obtain the fused velocity data, realize the comprehensive utilization of multi-source data, and improve the accuracy and robustness of the velocity measurement; The velocity calculation and verification module calculates the river surface velocity based on the fused velocity data, verifies its accuracy, and outputs the final velocity measurement value.

[0015] Due to the adoption of the above technical solution, the present invention has the following technical advancements compared to the prior art: 1. The present invention provides a river surface velocity measurement method and system based on physical constraints and multimodal fusion. By fusing optical flow information, depth information and environmental data, the comprehensive utilization of multi-source data is achieved. In the data fusion process, error detection and correction are performed in combination with redundant codes in the data representation, which effectively identifies and corrects erroneous points in the data, improves the reliability and accuracy of the data, and utilizes multimodal data fusion and error correction mechanisms to significantly enhance the robustness of the system, enabling it to operate stably in complex and changeable river environments.

[0016] 2. The present invention provides a river surface velocity measurement method and system based on physical constraints and multimodal fusion. By embedding shallow water equation constraints in optical flow estimation, the velocity estimation results not only conform to the image change characteristics, but also meet the laws of fluid mechanics. By introducing physical constraints, the low accuracy of traditional optical flow methods in complex water flow scenes is effectively overcome, and the measurement accuracy of river surface velocity is significantly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments described in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0018] Figure 1 It is a schematic diagram of the workflow of the present invention; Figure 2 Schematic diagram of the method of the present invention. DETAILED DESCRIPTION

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0020] Example 1, as Figure 1 、 Figure 2 As shown, the present invention provides a river surface velocity measurement method based on physical constraints and multimodal fusion, comprising the following steps: S1. Determine the target area and obtain river surface image sequence data, depth information and environmental data in the target area. Use geographic information system software and on-site survey maps, combined with hydrological data and terrain features, to preliminarily delineate the river section in the target area. At the same time, select measurement equipment based on the size of the target area and the complexity of the terrain. For image acquisition, prepare a high-resolution, high-frame-rate drone-mounted camera to ensure that it has sufficient zoom capability to meet the shooting requirements of different distances. For depth information acquisition, use a high-precision lidar with a stable bracket. For environmental data acquisition, configure a meteorological sensor, and calibrate and set parameters for the measurement equipment to ensure that it can operate stably on the river site. Deploy measurement equipment in the target area according to the set acquisition frequency. Acquire river surface image sequence data, depth information, and environmental data in the target area, wherein the drone camera is aimed at a predetermined shooting position, the angle and focal length are adjusted to capture detailed features of the river surface, and a river surface image sequence is captured at a set time interval. Simultaneously, a lidar is operated according to a preset scanning path to acquire depth information of the river surface. During the process of acquiring image and depth data, a meteorological sensor is synchronously activated to record environmental parameters such as wind speed, wind direction, air temperature, and water temperature, covering different water flow conditions and lighting changes. After data acquisition is completed, the acquired river surface image sequence data, depth information, and environmental data are transferred to a storage device, and the data is preliminarily organized, arranged in chronological order, and a unified file naming convention and directory structure are established; S2. Preprocess the collected data and embed redundant codes in the data representation for subsequent error detection, comprehensively form a basic data set, and preprocess the collected river surface image sequence data, depth information and environmental data. Among them, for the river surface image sequence data, denoising is performed to remove image noise caused by factors such as illumination changes and sensor noise, and contrast enhancement is performed to highlight the detailed features in the image. The depth data is smoothed and outliers are removed to remove points that obviously do not conform to the water depth range. The environmental data is unified in units and dimensionally normalized to ensure data consistency between different environmental parameters. At the same time, the timestamps of all data are checked to make the river surface image sequence data, depth information and environmental data completely synchronized in time. Redundant codes are embedded in the preprocessed data representation for subsequent error detection. Among them, for the river surface image sequence data, in The metadata part of the file is embedded with a cyclic redundancy check code. For depth data and environmental data, a CRC code is added at the end of the data frame. When embedding the redundant code, the redundant code generation algorithm and embedding position are recorded so that the redundant code can be accurately extracted and verified in the subsequent data transmission and processing. The river surface image sequence data, depth information and environmental data that have been preprocessed and embedded with redundant codes are integrated to form a basic data set. The river surface image sequence data, depth information and environmental data at the same moment are linked together through timestamps, and the basic data set is verified to check the integrity and consistency of the data. The verification content includes: checking whether all data have been successfully embedded with redundant codes, whether the redundant codes can be correctly extracted and verified, verifying whether the time series of the data is continuous, whether there is any data loss or time sequence confusion, and checking whether the data of different modalities can accurately correspond to each other; S3. Based on the basic data set, the optical flow method is used to perform preliminary optical flow estimation on the river surface image sequence data. According to the characteristics of the river surface image sequence data and the measurement accuracy requirements, the Horn-Schunck optical flow algorithm is selected and its parameters are initialized, including setting the time window, maximum number of iterations and convergence threshold for optical flow calculation. At the same time, the weight of the smoothing term is adjusted. If the weight is too large, the optical flow field will be too smooth and lose local features. If the weight is too small, it may introduce too much noise. Through multiple experiments and comparisons, the optimal parameter combination is determined to improve the accuracy of the preliminary optical flow estimation. The preprocessed river surface image sequence data is input into the selected optical flow algorithm, and the preliminary optical flow estimation calculation is started. The optical flow vector between two adjacent frames of images is calculated in sequence according to the order of the image sequence. At the same time, the calculation process is monitored in real time, the calculation time and intermediate results are recorded, and the calculated results are displayed. The obtained optical flow vector results are visualized, where an arrow diagram is used to represent the direction and size of the optical flow vector, the starting point of the arrow is the pixel position, the direction represents the movement direction of the pixel, and the length represents the movement speed. The accuracy of the preliminary optical flow estimation is measured based on the evaluation index of the average endpoint error (AEE), where the average endpoint error calculates the average Euclidean distance between the estimated optical flow vector and the true optical flow vector. Some manually annotated data are used as a reference, and the error value of the preliminary optical flow estimation result is calculated according to the evaluation index to evaluate its accuracy and reliability. Based on the evaluation results, the problems existing in the preliminary optical flow estimation are analyzed. If the error is large, check whether the optical flow algorithm is selected appropriately, whether the parameter settings are reasonable, and whether the image preprocessing is sufficient. According to the problems found, the optical flow algorithm or parameters are adjusted and optimized, and adjusted through multiple iterative adjustments until the preliminary optical flow estimation result meets the accuracy requirements; In addition, the specific process of calculating the preliminary optical flow estimation includes: The pre-processed river surface image sequence data is sorted in order to ensure that the image format is unified and the timestamp is accurate. The image sequence is loaded into the memory frame by frame using the OpenCV video reading function, and an image frame list is constructed. During the data input process, the integrity of the image is checked to avoid data loss or damage. Based on the selected Horn-Schunck optical flow algorithm, the initialization configuration is performed, and the time window, maximum number of iterations and convergence threshold of the optical flow calculation are set. At the same time, the smoothing term weight is adjusted to determine an initial value. After completing the initialization configuration, the optical flow algorithm is associated with the river surface image sequence data. In the order of the image sequence, two adjacent frames are selected in turn. The frame image is used as input, and the initialized Horn-Schunck optical flow algorithm is called to calculate the optical flow vector between two adjacent frame images. The partial derivatives of the two adjacent frame images in the x direction, y direction and time direction are calculated, and the optical flow vector is initialized to 0. The optical flow vector is solved iteratively until the maximum number of iterations is met or the error is less than the convergence threshold. The calculated optical flow vector results are visualized to intuitively observe the distribution and motion characteristics of the optical flow field. The direction and size of the optical flow vector are represented by an arrow diagram. The starting point of the arrow is the pixel position, and the direction of the arrow represents the direction of movement of the pixel, which is determined by the optical flow vector. The length of the arrow represents the size of the movement speed. The Horn-Schunck optical flow method estimates the optical flow field by minimizing the following energy function: ; Where, 、 and Represent the partial derivatives of the image in the x-direction, y-direction and time direction respectively, and represent the flow velocity components in the x and y directions respectively, is the smoothing term weight, For image domain, Horn-Schunck optical flow method is a classic optical flow estimation method, which is based on the continuity assumption of pixel intensity in image sequence, that is, the intensity value of a pixel remains unchanged between two consecutive frames. Based on this assumption, Horn and Schunck derived the energy function of optical flow estimation. Assuming that there are two consecutive frames in the image sequence and ,in and are spatial coordinates, is time, and the optical flow constraint equation can be expressed as: , and The optical flow is Direction and The Horn-Schunck method estimates the optical flow field by minimizing the energy function. The energy function consists of two parts: the data term and the regularization term. The data term is used to ensure that the optical flow constraint equation is valid at each pixel point. The regularization term is used to smooth the optical flow field so that it changes smoothly in space. Ensure that the optical flow field is spatially smooth, and Respectively and The sum of the squares of the gradients, that is: , ;in, 、 、 and They are and exist and The partial derivative of the direction is minimized by the gradient descent method to obtain the optical flow field. and ; The expression for iteratively solving the optical flow vector is: ; ; Where, For the The optical flow vector component of the iteration, is the number of iterations, are the components of the optical flow vector in the x and y directions; S4. Combined with the preliminary optical flow estimation results, the shallow water equation constraint is introduced into the physical constraint optical flow estimation model to obtain optical flow information, so that the optical flow estimation results conform to physical laws and improve accuracy. S5. Fusing the optical flow information, depth information, and environmental data to obtain fused velocity data, and then performing error detection on the fused velocity data in combination with redundant codes in the data representation, identifying and marking any erroneous data points. S6. Based on the redundant code detection results, the erroneous data is corrected, and the final river surface velocity is calculated using the fused velocity data and the corrected results, outputting a high-precision and robust velocity measurement value.

[0021] Example 2, as Figure 1 、 Figure 2 As shown, based on Example 1, the present invention provides a technical solution: preferably, S4 specifically includes: Based on the preliminary optical flow estimation results and taking the shallow water equation as the basis, the physical laws of the flow field are embedded into the optical flow estimation, and a physical constraint optical flow estimation model is constructed. Through the optimization algorithm, the optical flow estimation results are adjusted to meet the constraints of the shallow water equation, thereby improving the accuracy and physical rationality of the flow velocity estimation. Among them, the constraint term of the shallow water equation is added as a regularization term to the optimization objective function to form an optical flow estimation model combined with physical constraints. The continuity equation and momentum equation of the shallow water equation are introduced into the physical constraint optical flow estimation model, and the constraint term of the shallow water equation is expressed as a penalty term, added to the optimization objective function, and the optical flow estimation results are gradually adjusted through the iterative optimization algorithm of the gradient descent method to meet the physical constraints. Conditions, among which the continuity equation ensures the conservation of mass in the velocity field, and the momentum equation describes the motion of the fluid under the action of gravity and pressure gradient. By embedding physical constraints into the optical flow estimation process, the optical flow estimation results not only conform to the motion characteristics of the image sequence, but also meet the basic laws of fluid dynamics. The physical constraint optical flow estimation model is solved by the optimization algorithm, and the optical flow estimation results are iteratively adjusted to gradually meet the constraints of the shallow water equation. During the optimization process, the convergence of the model is monitored in real time, the optical flow estimation results after each iteration are recorded, and the difference between them and the preliminary optical flow estimation results is calculated. When the model converges or reaches the predetermined number of iterations, it is verified whether the optimized optical flow estimation results conform to the physical laws. In addition, the specific process of gradually adjusting the optical flow estimation results through the iterative optimization algorithm of the gradient descent method includes: Aiming at the discrete grid characteristics of river surface images, the continuity equation and momentum equation of the shallow water equation are discretized, and the constraints related to the velocity field (corresponding to the optical flow vector) are extracted from the discretized equations. They are converted into a form suitable for embedding into the optical flow estimation model. Based on the preliminary optical flow estimation results, the optimization objective function of the optical flow estimation is constructed, and the extracted shallow water equation constraints are added as penalty terms to the optimization objective function to form an optical flow estimation model combined with physical constraints. The optical flow estimation results are gradually adjusted through the iterative optimization algorithm of the gradient descent method to make them meet the constraints of the shallow water equation. In the iteration, the gradient of the optimization objective function with respect to the optical flow vector is calculated, and the initial iteration step size is determined based on the gradient information, and then the iterative optimization is performed according to the rules of the gradient descent method to update the optical flow vector. At the same time, the optical flow estimation result after each iteration is recorded. In each iteration, the value of the optical flow vector is adjusted according to the gradient and step size of the current optical flow vector to obtain a new optical flow estimation result, and the new optimization objective function value is calculated and compared with the optimization objective function value of the previous iteration to determine whether the optimization objective function value decreases. If it decreases, the iteration is continued. If it increases, the step size is adjusted and the iteration is repeated. The expression of the optimization objective function is: ; Where, 、 and Represent the partial derivatives of the image in the x-direction, y-direction and time direction respectively, and represent the flow velocity components in the x and y directions respectively, is the smoothing term weight, is the image domain; The expression of the shallow water equation constraint is: ; Where, and is a regularization parameter used to control the weight of the shallow water equation constraint in the optimization objective function. For water depth, is the velocity vector, is a vector representation, and is two components, is the acceleration due to gravity, is the water surface elevation, For time, is the gradient operator, represents the outer product of the velocity vector and itself, Indicates water depth The divergence, multiplied by the outer product of the velocity vector and itself, is a scalar value that describes the motion of the fluid under the influence of gravity and pressure gradients; The continuity equation is expressed as: ; The momentum equation is expressed as: ; The expression for updating the optical flow vector is: ; ; Where, is the learning rate, To optimize the objective function; S5 specifically includes: The optical flow information, depth information and environmental data after preliminary optimization are fused. The optical flow information is converted into velocity field data, and the velocity is normalized in combination with the depth information to eliminate the influence of different depths on the velocity. At the same time, the velocity field is corrected according to the environmental data, and the multimodal data is fused using the weighted average method to obtain the fused velocity data. During the fusion process, the time and space alignment of each data mode is ensured, and redundant codes are embedded in the fused velocity data representation. According to the redundant code generation algorithm, the check code of each data block is calculated and compared with the stored redundant code. If the check code does not match , then mark the data block as an erroneous data point, wherein the redundant code adopts a cyclic redundancy check code, which is embedded in the storage format of the flow velocity data. The check mechanism of the redundant code is used to detect errors in the fused flow velocity data, analyze the marked erroneous data points, record the position, timestamp and error type of the erroneous data points, and analyze the cause of the error. For the marked erroneous data points, temporarily exclude them from subsequent calculations. At the same time, the proportion and distribution of the erroneous data points are counted to evaluate the overall quality of the data. According to the analysis results, further optimize the data fusion algorithm or adjust the embedding strategy of the redundant code; S6 specifically includes: According to the redundant code detection results, the data points marked as errors are corrected, and the erroneous data points are repaired using a statistical outlier detection method. For a single erroneous data point, the valid data in its neighborhood are used for interpolation correction. For continuous erroneous data blocks, the moving average method is used for smoothing to restore the continuity of the data. The corrected data points replace the original erroneous data points to form a corrected velocity data set. The corrected velocity data are reintegrated into the fused velocity data set to replace the original erroneous data points. Based on the updated fused data, the final river surface velocity is calculated. Use depth information to weight the flow velocity, calculate the flow velocity at each location using a weighted average method, verify the final calculated river surface flow velocity, compare the calculated result with known velocity field data (data obtained through traditional measurement methods), evaluate its accuracy and reliability, calculate the average endpoint error, and ensure the accuracy of the final flow velocity measurement value. If the verification result meets the accuracy requirements, the final flow velocity measurement value is output, including the magnitude and direction of the flow velocity. If the verification result still has a large error, further optimize the data correction algorithm or readjust the fusion strategy until a high-precision and robust flow velocity measurement value is obtained; The expression for calculating the flow velocity at each location using the weighted average method is: ; Where, is the fused velocity data, is the depth information of the corresponding position, is the final calculated river surface velocity; The calculation expression of the average endpoint error is: ; Where, is the average endpoint error, is the known true flow velocity value, is the total number of data points.

[0022] Example 3, as Figure 1 、 Figure 2 As shown, based on Examples 1-2, the present invention further provides a river surface flow velocity measurement system based on physical constraints and multimodal fusion, which is used to implement the above-mentioned river surface flow velocity measurement method based on physical constraints and multimodal fusion, including a flow velocity measurement platform, which is communicatively connected to the following modules, wherein: Data acquisition module, used to obtain river surface image sequence data, depth information and environmental data to ensure data integrity and diversity; The data preprocessing module is used to remove noise, smooth, unify units, and embed redundant codes on the acquired data to form a basic data set, improve data quality, and reduce noise interference; The optical flow estimation module is used to perform preliminary optical flow estimation on the preprocessed river surface image sequence data using the optical flow method to preliminarily determine the flow velocity distribution on the river surface; The physical constraint fusion module is used to optimize the optical flow estimation results by combining the shallow water equation constraints, improve the accuracy and physical rationality of the velocity estimation, make the results conform to the laws of fluid dynamics, and fuse the optical flow information, depth information and environmental data to perform error detection and correction to obtain the fused velocity data, realize the comprehensive utilization of multi-source data, and improve the accuracy and robustness of the velocity measurement; The velocity calculation and verification module calculates the river surface velocity based on the fused velocity data, verifies its accuracy, and outputs the final velocity measurement value.

[0023] The above are only specific embodiments of the present application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A river surface velocity measurement method based on physical constraints and multimodal fusion, characterized by: The following steps are involved: S1. Determine the target area and obtain river surface image sequence data, depth information and environmental data of the target area; S2, preprocessing the collected data and embedding redundant codes in the data representation to comprehensively form a basic data set; S3. Based on the basic data set, the optical flow method is used to perform preliminary optical flow estimation on the river surface image sequence data; S4. Combined with the preliminary optical flow estimation results, the shallow water equation constraint is introduced into the physical constraint optical flow estimation model to obtain the optical flow information; S5. Fusing the optical flow information, depth information, and environmental data to obtain fused velocity data, and then performing error detection on the fused velocity data in combination with redundant codes in the data representation, identifying and marking any erroneous data points. S6. Based on the redundant code detection result, the erroneous data is corrected, and the final river surface velocity is calculated using the fused velocity data and the corrected result.

2. The method for measuring river surface velocity based on physical constraints and multimodal fusion according to claim 1 is characterized in that: Said S1 specifically includes: Using geographic information system software and on-site survey maps, combined with hydrological data and terrain features, the river sections in the target area are preliminarily delineated. At the same time, based on the size of the target area and the complexity of the terrain, the surveying equipment is selected and calibrated, and its parameters are set; Deploy measurement equipment in the target area and obtain river surface image sequence data, depth information and environmental data of the target area according to the set acquisition frequency; After data collection is completed, the acquired river surface image sequence data, depth information and environmental data are transferred to the storage device, and the data are preliminarily sorted, arranged in chronological order, and a unified file naming rule and directory structure are established.

3. The method for measuring river surface velocity based on physical constraints and multimodal fusion according to claim 1 is characterized in that: The S2 specifically includes: Preprocessing the collected river surface image sequence data, depth information and environmental data, and checking the timestamps of all data to synchronize the river surface image sequence data, depth information and environmental data in time; Redundant codes are embedded in the preprocessed data representation. For river surface image sequence data, a cyclic redundancy check code is embedded in the metadata part of its file. For depth data and environmental data, a CRC code is added at the end of the data frame. When embedding the redundant code, the generation algorithm and embedding position of the redundant code are recorded. The river surface image sequence data, depth information and environmental data that have been preprocessed and embedded with redundant codes are integrated to form a basic data set, in which the river surface image sequence data, depth information and environmental data at the same moment are linked together through timestamps, and the basic data set is verified.

4. The method for measuring river surface velocity based on physical constraints and multimodal fusion according to claim 1, characterized in that: The S3 specifically includes: According to the characteristics of river surface image sequence data and the measurement accuracy requirements, the Horn-Schunck optical flow algorithm is selected and its parameters are initialized, including setting the time window of optical flow calculation, the maximum number of iterations and the convergence threshold. At the same time, the weight of the smoothing term is adjusted. The pre-processed river surface image sequence data is input into the selected optical flow algorithm to start the preliminary optical flow estimation calculation. The optical flow vector between two adjacent frames of the image is calculated in sequence according to the order of the image sequence. At the same time, the calculation process is monitored in real time, the calculation time and intermediate results are recorded, and the calculated optical flow vector results are visualized. The accuracy of the preliminary optical flow estimation is measured by the evaluation index based on the average endpoint error, where the average endpoint error calculates the average Euclidean distance between the estimated optical flow vector and the true optical flow vector. Some manually annotated data is used as a reference, and the error value of the preliminary optical flow estimation result is calculated according to the evaluation index to evaluate its accuracy and reliability. Based on the evaluation results, the problems existing in the preliminary optical flow estimation are analyzed. According to the problems found, the optical flow algorithm or parameters are adjusted and optimized, and multiple iterative adjustments are made until the preliminary optical flow estimation result meets the accuracy requirements.

5. The method for measuring river surface velocity based on physical constraints and multimodal fusion according to claim 4 is characterized in that: The specific process of calculating the preliminary optical flow estimation includes: Arrange the pre-processed river surface image sequence data in order, use OpenCV's video reading function to load the image sequence into the memory frame by frame, build an image frame list, and in the data input process; Based on the selected Horn-Schunck optical flow algorithm, the initial configuration is performed, and the time window, maximum number of iterations, and convergence threshold of the optical flow calculation are set. At the same time, the weight of the smoothing term is adjusted and an initial value is determined. After completing the initial configuration, the optical flow algorithm is associated with the river surface image sequence data; In the order of the image sequence, two adjacent frames are selected as input, and the initialized Horn-Schunck optical flow algorithm is called to calculate the optical flow vector between the two adjacent frames. The partial derivatives of the two adjacent frames in the x-direction, y-direction, and time direction are calculated, and the optical flow vector is initialized to 0. The optical flow vector is solved iteratively until the maximum number of iterations is met or the error is less than the convergence threshold. The calculated optical flow vector results are visualized, and an arrow diagram is used to represent the direction and magnitude of the optical flow vector. The starting point of the arrow is the pixel position, and the direction of the arrow represents the direction of movement of the pixel, which is determined by the optical flow vector. The length of the arrow represents the magnitude of the movement speed.

6. The method for measuring river surface velocity based on physical constraints and multimodal fusion according to claim 5, characterized in that: The S4 specifically includes: Based on the preliminary optical flow estimation results and the shallow water equation, the physical laws of the velocity field are embedded into the optical flow estimation, and a physically constrained optical flow estimation model is constructed. The constraint term of the shallow water equation is added as a regularization term to the optimization objective function, forming an optical flow estimation model that combines physical constraints. The continuity equation and momentum equation of the shallow water equation are introduced into the physical constraint optical flow estimation model. The constraint term of the shallow water equation is expressed as a penalty term and added to the optimization objective function. The optical flow estimation result is gradually adjusted through the iterative optimization algorithm of the gradient descent method to make it meet the physical constraint conditions. The physical constraint optical flow estimation model is solved using an optimization algorithm. The optical flow estimation results are iteratively adjusted to gradually satisfy the constraints of the shallow water equation. During the optimization process, the convergence of the model is monitored in real time, the optical flow estimation results after each iteration are recorded, and the difference between them and the preliminary optical flow estimation results is calculated. When the model converges or reaches the predetermined number of iterations, it is verified whether the optimized optical flow estimation results conform to physical laws.

7. The method for measuring river surface velocity based on physical constraints and multimodal fusion according to claim 6, characterized in that: The specific process of gradually adjusting the optical flow estimation result by the iterative optimization algorithm of the gradient descent method includes: According to the discrete grid characteristics of river surface images, the continuity equation and momentum equation of shallow water equation are discretized, and the constraints related to the velocity field are extracted from the discretized equations. Based on the preliminary optical flow estimation results, an optimization objective function for optical flow estimation is constructed. The extracted shallow water equation constraint term is added as a penalty term to the optimization objective function to form an optical flow estimation model that combines physical constraints. The optical flow estimation result is gradually adjusted through the iterative optimization algorithm of the gradient descent method to satisfy the constraints of the shallow water equation. In each iteration, the gradient of the optimization objective function with respect to the optical flow vector is calculated, and the initial iteration step size is determined based on the gradient information. Then, iterative optimization is performed according to the rules of the gradient descent method to update the optical flow vector. At the same time, the optical flow estimation result after each iteration is recorded. In each iteration, the value of the optical flow vector is adjusted according to the gradient and step size of the current optical flow vector to obtain a new optical flow estimation result, and a new optimization objective function value is calculated. It is compared with the optimization objective function value of the previous iteration to determine whether the optimization objective function value decreases. If it decreases, the iteration continues. If it increases, the step size is adjusted and the iteration is performed again.

8. The method for measuring river surface velocity based on physical constraints and multimodal fusion according to claim 1, characterized in that: The S5 specifically includes: The optical flow information, depth information and environmental data after preliminary optimization are fused. The optical flow information is converted into velocity field data, and the velocity is normalized in combination with the depth information. At the same time, the velocity field is corrected according to the environmental data, and the weighted average method is used to fuse the multimodal data to obtain the fused velocity data. Redundant codes are embedded in the fused velocity data representation. Based on the redundant code generation algorithm, a check code is calculated for each data block and compared with the stored redundant code. If the check code does not match, the data block is marked as an erroneous data point. The redundant code adopts a cyclic redundancy check code and is embedded in the storage format of the velocity data. The check mechanism of the redundant code is used to perform error detection on the fused velocity data. Analyze the marked erroneous data points, record their location, timestamp, and error type, and analyze the cause of the error. Temporarily exclude the marked erroneous data points from subsequent calculations. At the same time, calculate the proportion and distribution of erroneous data points and evaluate the overall quality of the data. Based on the analysis results, further optimize the data fusion algorithm or adjust the redundant code embedding strategy.

9. The method for measuring river surface velocity based on physical constraints and multimodal fusion according to claim 8, characterized in that: The S6 specifically includes: According to the redundant code detection results, the data points marked as errors are corrected and repaired using a statistical outlier detection method. For a single error data point, valid data in its neighborhood are used for interpolation correction. For continuous error data blocks, a moving average method is used for smoothing to restore data continuity. The corrected data points replace the original error data points to form a corrected velocity data set. The corrected velocity data is reintegrated into the fused velocity dataset to replace the original erroneous data points. The final river surface velocity is calculated based on the updated fused data. The velocity is weighted using depth information, and the velocity at each location is calculated using a weighted average method. The final calculated river surface velocity is verified, and the calculated results are compared with the known velocity field data to evaluate their accuracy and reliability, and the average endpoint error is calculated. If the verification result meets the accuracy requirements, the final velocity measurement value is output, including the magnitude and direction of the velocity. If the verification result still has a large error, the data correction algorithm is further optimized or the fusion strategy is readjusted.

10. A river surface velocity measurement system based on physical constraints and multimodal fusion, used to implement the river surface velocity measurement method based on physical constraints and multimodal fusion as described in any one of claims 1 to 9, comprising a velocity measurement platform, characterized in that: The flow rate measurement platform is communicatively connected to the following modules, wherein: Data acquisition module, used to obtain river surface image sequence data, depth information and environmental data; The data preprocessing module is used to remove noise, smooth, unify units and embed redundant codes on the acquired data to form a basic data set; The optical flow estimation module is used to perform preliminary optical flow estimation on the preprocessed river surface image sequence data using the optical flow method to preliminarily determine the flow velocity distribution on the river surface; The physical constraint fusion module is used to optimize the optical flow estimation results by combining the shallow water equation constraints, and fuse the optical flow information, depth information and environmental data to perform error detection and correction to obtain the fused flow velocity data; The velocity calculation and verification module calculates the river surface velocity based on the fused velocity data, verifies its accuracy, and outputs the final velocity measurement value.

Images