Amphibious unmanned aerial vehicle river sediment concentration real-time monitoring method based on underwater light field imaging and cross-modal fusion

Abstract

The invention discloses an amphibious unmanned aerial vehicle riverway sediment concentration real-time monitoring method based on underwater light field imaging and cross-modal fusion, and relates to the technical field of riverway sediment monitoring. Comprising the following steps: acquiring aerial multispectral data and preprocessing the aerial multispectral data; an integrated polarized light field camera is carried on an amphibious unmanned aerial vehicle to capture underwater light field data, and underwater three-dimensional light field information is reconstructed and sediment scattering characteristics are extracted through an optical model and a lightweight deep learning network; the air multispectral data and sediment scattering characteristics are fused through space-time alignment and an attention mechanism, a sediment concentration prediction model is generated, and then a spatial distribution diagram of the river sediment concentration is output; and carrying out real-time monitoring and early warning on the river sediment concentration according to the spatial distribution diagram of the river sediment concentration through edge calculation and cloud collaboration architecture. The method is combined with an advanced deep learning algorithm, high-resolution silt concentration inversion can be realized, and the method has real-time performance and adapts to a complex water body environment.

Description

Technical Field

[0001] The present invention relates to the technical field of river sediment monitoring, and more specifically to a real-time monitoring method for river sediment concentration using an amphibious unmanned aerial vehicle based on underwater light field imaging and cross-modal fusion. Background Art

[0002] River sediment monitoring is an important foundation for water conservancy projects, ecological protection and flood control, but traditional methods have long faced technical bottlenecks. The current mainstream methods include manual sampling and analysis and single sensor survey methods. Manual sampling requires ships or manpower to collect water samples at specific points, and its time period is relatively long (usually weekly / monthly), which makes it difficult to reflect the dynamic process of sediment transport during flood peaks (such as the sediment transport rate of the lower reaches of the Yellow River during the flood season can reach several tons per second). Although sensors such as sonar and optical turbidity meters can achieve continuous monitoring, they also have significant limitations:

[0003] Traditional sonar: Inverts particle concentration by analyzing echo signals, but its accuracy is seriously affected by the distribution of sediment particle size and water turbidity. Studies have shown that when the sediment particle size is large (exceeding the range of 0.1mm-2mm), the nonlinear response of the sonar scattering cross section will introduce a high concentration error. In addition, the multiple scattering effect of suspended particles in highly turbid water bodies will cause signal attenuation, and the actual effective detection depth may even be less than 1 meter, which is difficult to meet the monitoring needs of deep water areas.

[0004] Optical equipment: Optical equipment such as multispectral cameras and hyperspectral imagers can infer sediment concentration by analyzing the reflectivity of water bodies. However, such methods rely on the ability of light to penetrate, and their effective detection depth in turbid water bodies is greatly limited. They are also unable to distinguish between suspended sediment and interference signals from algae and plankton. It is still difficult to overcome complex multi-directional scattering noise by simply studying the forward scattering suppression scheme.

[0005] Economic and reliability issues of multi-device collaboration: To improve the monitoring dimension, existing technologies require the combination of drones (aerial remote sensing), underwater robots (sonar detection) and buoy stations (in-situ sensors), but the temporal and spatial registration errors between multiple platforms will cause data fusion failure. In addition, the maintenance cost of multiple systems is high.

[0006] Therefore, it is an urgent problem for technical personnel in this field to propose a real-time monitoring method for river sediment concentration using an amphibious UAV based on underwater light field imaging and cross-modal fusion to solve the difficulties existing in the existing technology. Summary of the invention

[0007] In view of this, the present invention provides a real-time monitoring method for the sediment concentration in a river channel by an amphibious unmanned aerial vehicle based on underwater light field imaging and cross-modal fusion. Through multi-dimensional data collaboration (aerial spectroscopy + underwater light field), it breaks through the limitations of the accuracy and efficiency of traditional methods, realizes real-time and high-resolution inversion of the sediment concentration in complex water bodies, and at the same time reduces the collaborative cost of multi-source devices, providing an innovative tool for intelligent water conservancy.

[0008] To achieve the above object, the present invention provides the following technical solutions:

[0009] A real-time monitoring method for the sediment concentration in a river channel by an amphibious unmanned aerial vehicle based on underwater light field imaging and cross-modal fusion, comprising the following steps:

[0010] S1. Obtain aerial multispectral data and preprocess the aerial multispectral data;

[0011] S2. Mount an integrated polarized light field camera on the amphibious unmanned aerial vehicle to capture underwater light field data, and reconstruct the underwater three-dimensional light field information and extract the sediment scattering characteristics through an optical model and a lightweight deep learning network;

[0012] S3. Through spatio-temporal alignment and an attention mechanism, fuse the aerial multispectral data and the sediment scattering characteristics to generate a sediment concentration prediction model, and then output the spatial distribution map of the sediment concentration in the river channel;

[0013] S4. According to the spatial distribution map of the sediment concentration in the river channel, through an edge computing and cloud collaboration architecture, conduct real-time monitoring and early warning of the sediment concentration in the river channel.

[0014] For the above method, optionally, the specific content of obtaining aerial multispectral data and preprocessing the aerial multispectral data in S1 is as follows:

[0015] S11. Multispectral data acquisition and radiometric correction;

[0016] S12. Geometric correction;

[0017] S13. Optical parameter inversion;

[0018] S14. Output and format.

[0019] For the above method, optionally, the specific content of the radiometric correction in S11 is as follows:

[0020] Radiometric calibration: Using the empirical linear method, establish a linear relationship between the digital number DN and the incident radiance L using the reflectance value of the ground calibration panel. The formula is as follows: λ The linear relationship is as follows:

[0021] L λ = a·DN λ + b

[0022] Among them, the coefficients a and b are calibrated by the least squares method and independently fitted band by band;

[0023] Atmospheric correction: Simulate the atmospheric path radiation L through the MODTRAN model path , combined with the solar irradiance E sun and the atmospheric transmittance T, to invert the water surface reflectance ρ λ , the formula is as follows:

[0024]

[0025] Among them, cosθ is the angle between the solar radiation and the ground normal.

[0026] For the above method, optionally, the specific content of the geometric correction in S12 is:

[0027] Spatial registration: Use RTK coordinates to target the hovering vibration of the UAV, and adopt Kalman filter to fuse GPS data. The formula is as follows:

[0028]

[0029] Among them, (X0, Y0, Z0) is the camera projection center coordinate, a i , b i , c i are the projection matrix parameters, f is the focal length, and (X, Y, Z) is the three-dimensional coordinate value of a point in space to be projected onto the image, with the unit of m;

[0030] Use the matching point coordinates to construct an affine transformation matrix to optimize the image geometric positioning error. The formula is as follows:

[0031]

[0032] Among them, m ij is the rotation and scaling parameter, t x , t y are the translation parameters, which are solved by the least squares method.

[0033] For the above method, optionally, the specific content of reconstructing the underwater three-dimensional light field information and extracting the sediment scattering characteristics through the optical model and the lightweight deep learning network in S2 is:

[0034] S21. Polarized light field imaging hardware configuration;

[0035] S22. Underwater light field reconstruction and scattering suppression;

[0036] S23. Multi-dimensional light field feature extraction;

[0037] S24. Cross-modal feature association;

[0038] S25. Output and format.

[0039] For the above method, optionally, the specific content of underwater light field reconstruction and scattering suppression in S22 is as follows:

[0040] Light field depth estimation: Generate a depth map using the consistency of the light field focal length, and solve the scene depth z through integral optimization:

[0041]

[0042] where is the gradient field of the i-th viewpoint, TV(z) is the total variation regularization term to suppress noise interference, λ is the regularization weight coefficient, and N is the number of viewpoints or images;

[0043] Polarization scattering suppression: Based on Mueller matrix decomposition of the water body scattering components, restore the target reflected light intensity I object :

[0044] S out = M water ·M object ·S in

[0045] where M water is the water body scattering matrix, M object is the target object scattering matrix, S in S out are the Stokes vectors of the incident light and the output light respectively, representing the polarization state of the light beam;

[0046] Model the attenuation coefficient through the Beer-Lambert law:

[0047]

[0048] where c is the water body extinction coefficient, which is fed back in real time by a pre-set turbidity sensor, z is the depth, and β(z) is the polarization coupling coefficient related to the incident / observation angle.

[0049] For the above method, optionally, the specific content of dimension light field feature extraction in S23 is as follows:

[0050] Sediment scattering element modeling: Based on Mie scattering theory, construct scattering phase functions of different particle sizes:

[0051]

[0052] where a n and b n are Mie coefficients, π n is the radiation mode or radiation directivity in the scattering process, τ nis a function related to the scattering intensity of particles and the particle size parameter, d is the characteristic size of the scattering particles, and n is the order index in the Mie scattering series expansion;

[0053] Feature Encoding Network: Design a lightweight two-stream CNN to process the optical flow and polarization components respectively;

[0054] Optical flow: The input is a 9×9 sub-aperture view with a viewing angle baseline Δ = 0.5°, and a 3D convolution kernel is used to extract the disparity implicit features; Polarization component: The input is the Stokes vector, i.e., S0, S1, S2, and the anisotropic scattering features are extracted through angle-sensitive convolution; The features of the two streams are fused through a cross-attention gating to output a multi-scale feature vector F ∈ R 512 .

[0055] For the above method, optionally, the specific content of fusing the aerial multi-spectral data and the sediment scattering features through spatio-temporal alignment and attention mechanism in S3 to generate a sediment concentration prediction model and then output the spatial distribution map of the river channel sediment concentration is as follows:

[0056] S31. Multimodal spatio-temporal registration and encoding;

[0057] S32. Hierarchical cross-modal attention mechanism;

[0058] S33. Physically constrained concentration inversion;

[0059] S34. Output and format.

[0060] For the above method, optionally, the specific content of the hierarchical cross-modal attention mechanism in S32 is as follows:

[0061] Pixel-level modulation: Modulate the underwater coarse-grained features with the highest-resolution features in the air to enhance the edge sensitivity. The formula is as follows:

[0062]

[0063] where σ is the Sigmoid function, ⊙ is the element-wise multiplication, and the modulation weight G l ∈[0,1] H×W , is the feature map extracted from the aerial branch at the l-th network level, is the feature map extracted from the underwater branch, Conv 1×1 is the convolution operation with a kernel size of k×1, G l is the modulation weight, is the improved underwater feature obtained after pixel-level modulation;

[0064] Semantic Cross-Attention: Construct a multi-head attention mechanism. Using underwater features as Query and aerial features as Key-Value, calculate cross-modal associations. The formula is as follows:

[0065]

[0066] Among them, W q ,W k ,W v are learnable weight matrices, and d k is the feature dimension or number of channels of the Key vector;

[0067] Residual fusion and upsampling: Stack the attention outputs layer by layer. The formula is as follows:

[0068]

[0069] Use transposed convolution to gradually upsample to the original resolution;

[0070] The specific content of the concentration inversion of the physical constraint in S33 is:

[0071] Coupled sediment transport equation: Introduce the sediment transport continuity equation as a physically aware loss and solve it by the finite difference method. The formula is as follows:

[0072]

[0073] Among them, c is the sediment concentration, D is the diffusion coefficient, q s is the flux, υ is the sediment sink term, representing the rate of sediment settlement, deposition or removal, μ is the sediment source term, representing the intensity of new or replenished sediment, and x, y, z are the three-dimensional spatial coordinates in the river channel or water area;

[0074] Regression network: Design a lightweight U-Net decoder. The input is the final fused feature Output the sediment concentration field C∈R H×W The formula is as follows:

[0075]

[0076] For the above method, optionally, the specific content of real-time monitoring and early warning of river channel sediment concentration in S4 is:

[0077] S41. Design the early warning system architecture;

[0078] S42. Formulate a multi-level intelligent early warning mechanism;

[0079] S43. Early warning verification and decision support;

[0080] S44. System Output and Interface.

[0081] As can be seen from the above technical solutions, compared with the prior art, the present invention discloses a real-time monitoring method for river sediment concentration of an amphibious unmanned aerial vehicle based on underwater light field imaging and cross-modal fusion, and its beneficial effects are as follows:

[0082] 1) It breaks through the accuracy and efficiency limitations of traditional monitoring methods. Through the in-depth fusion of aerial and underwater data and the combination of advanced deep learning algorithms, it can achieve high-resolution sediment concentration inversion, with real-time performance and high accuracy, and is suitable for complex water environments;

[0083] 2) It can not only improve the accuracy and efficiency of river sediment monitoring, but also provide innovative technical means for fields such as intelligent water conservancy, flood prevention, and environmental protection, with broad application prospects and significant social benefits. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only the embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained according to the provided drawings without creative efforts.

[0085] Figure 1 It is a flowchart of a real-time monitoring method for river sediment concentration of an amphibious unmanned aerial vehicle based on underwater light field imaging and cross-modal fusion provided by the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0086] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts belong to the scope of protection of the present invention.

[0087] See Figure 1 As shown, the present invention discloses a real-time monitoring method for river sediment concentration of an amphibious unmanned aerial vehicle based on underwater light field imaging and cross-modal fusion, including the following steps:

[0088] S1. Obtain aerial multispectral data and preprocess the aerial multispectral data;

[0089] S2. Mount an integrated polarized light field camera on the amphibious unmanned aerial vehicle to capture underwater light field data, and reconstruct underwater three-dimensional light field information and extract sediment scattering characteristics through an optical model and a lightweight deep learning network;

[0090] S3. Through spatio-temporal alignment and attention mechanism, fuse aerial multi-spectral data with sediment scattering characteristics to generate a sediment concentration prediction model, and then output the spatial distribution map of river channel sediment concentration;

[0091] S4. According to the spatial distribution map of river channel sediment concentration, through the edge computing and cloud collaborative architecture, conduct real-time monitoring and early warning of river channel sediment concentration.

[0092] Furthermore, the specific content of obtaining aerial multi-spectral data and preprocessing the aerial multi-spectral data in S1 is as follows:

[0093] S11. Multi-spectral data acquisition and radiometric correction;

[0094] S12. Geometric correction;

[0095] S13. Optical parameter inversion;

[0096] Specifically, the Normalized Difference Turbidity Index: Based on the reflectance difference ratio between the red band (650nm) and the near-infrared band (850nm), quantify the surface suspended sediment concentration. The formula is as follows:

[0097]

[0098] When NDTI > 0.4, it is determined as a high turbidity area (sediment concentration > 200mg / L).

[0099] Chlorophyll-a concentration (Chl-a): Use the three-band algorithm to eliminate the interference of water body absorption and suppress the false positive signal of algae:

[0100]

[0101] Among them, the coefficients in the formula are obtained through laboratory calibration of the actual river channel water body;

[0102] Denoising and outlier removal: Apply a guided filter to the reflectance image:

[0103]

[0104] Among them, I is the guidance image (near-infrared band), ω k is the local window, and retain the edge texture by minimizing ‖q - p‖ 2 (p is the input image).

[0105] S14. Output and format.

[0106] Specifically, the processed data is stored in a grid GeoTIFF format, including: surface reflectance of each band (ρ1-ρ5); derived parameters (NDTI, Chl-a); RGB true color image after spatial registration; metadata (timestamp, flight altitude, solar geometric parameters).

[0107] Specifically, the airborne multispectral data is collected by an unmanned aerial vehicle at a constant flight altitude (50 m) and speed (5 m / s). The multispectral camera is set with 5 narrow bands (central wavelengths: 450 nm, 550 nm, 650 nm, 750 nm, 850 nm, bandwidth ±10 nm), and the solar zenith angle θ and the sensor view angle φ are synchronously recorded. The RTK module provides centimeter-level positioning (horizontal ±1 cm, vertical ±2 cm).

[0108] Furthermore, the specific content of the radiometric correction in S11 is as follows:

[0109] Radiometric calibration: The empirical linear method is adopted, and the linear relationship between the digital number DN and the incident radiance L is established using the reflectance values (20%, 40%, 60%) of the ground calibration panel. The formula is as follows: λ as follows:

[0110] L λ = a·DN λ + b

[0111] where the coefficients a and b are calibrated by the least squares method and independently fitted for each band;

[0112] Atmospheric correction: The atmospheric path radiance L is simulated through the MODTRAN model, combined with the solar irradiance E path and the atmospheric transmittance T to invert the water surface reflectance ρ sun , and the formula is as follows: λ as follows:

[0113]

[0114] where cosθ is the angle between the solar radiation and the ground normal.

[0115] Specifically, the sky light reflection component (proportion of water surface ripples) needs to be deducted in the calculation.

[0116] Furthermore, the specific content of the geometric correction in S12 is as follows:

[0117] Spatial registration: Using the RTK coordinates to address the hovering vibration of the unmanned aerial vehicle (amplitude < 3 cm), the Kalman filter is used to fuse the GPS data. The formula is as follows:

[0118]

[0119] where (X0, Y0, Z0) is the camera projection center coordinate, ai , b i , c i are the projection matrix parameters (provided by the factory calibration parameters of the drone, and the default three-axis attitude is the horizontal aircraft state), f is the focal length, and (X, Y, Z) are the three-dimensional coordinate values of a point in space to be projected onto the image (such as a point in a river channel, a ground target, etc.), with the unit of m, and are consistent with the RTK / GPS coordinate system used by the drone;

[0120] Construct an affine transformation matrix using the matching point coordinates to optimize the image geometric positioning error. The formula is as follows:

[0121]

[0122] where m ij is the rotation and scaling parameter, t x , t y is the translation parameter, which is solved by the least squares method.

[0123] Specifically, to eliminate the residual distortion caused by the flight trajectory deviation, high-reflectivity targets (sub-meter GPS coordinates are known) with a size of 15 cm × 15 cm are arranged on both banks of the monitored river section, and are distributed along the river bank at intervals of 50 - 100 meters. SIFT features are extracted from the near-infrared band (850 nm) of the multi-spectral image and matched with the target template library, and the matching pairs with a confidence level > 0.9 are selected.

[0124] Furthermore, the specific content of reconstructing the underwater three-dimensional light field information and extracting the sediment scattering characteristics through the optical model and the lightweight deep learning network in S2 is as follows:

[0125] S21. Polarized light field imaging hardware configuration;

[0126] Specifically, lens group: An aperture polarization decomposition sensor array is adopted, and four-dimensional acquisition of vectors is realized through an optical path folding device. Each sub-aperture corresponds to 4 polarization directions (0°, 45°, 90°, 135°);

[0127] Sensor: Global shutter CMOS (resolution 4096 × 2160), dynamic range 14 bit, supporting multi-exposure fusion (short exposure 50 μs to capture the scattering main peak, long exposure 5 ms to extract deep details);

[0128] Light source compensation: Bilateral 450 nm blue LED arrays (peak power 80 W), triggered by a synchronous pulse (delay < 10 μs), and the beam angle is 20° to suppress lateral scattering.

[0129] S22. Underwater light field reconstruction and scattering suppression;

[0130] S23. Multi-dimensional light field feature extraction;

[0131] S24, Cross-modal Feature Association;

[0132] Specifically, air-water feature alignment: Construct a joint embedding space with the light field feature F underwater and the aerial multispectral data F air to maximize cross-modal correlation through canonical correlation analysis.

[0133] S25, Output and Format.

[0134] Specifically, the processed data is stored in the format of a point cloud matrix, including: 3D light field point cloud (XYZ coordinates + Stokes vector + scattering phase); light field feature tensor (512-dimensional feature vector + attention weight map); scattering parameter matrix (particle size distribution, attenuation coefficient, optical water depth); cross-modal correlation weight (CCA projection matrix W u , W a ).

[0135] Furthermore, the specific content of underwater light field reconstruction and scattering suppression in S22 is:

[0136] Light field depth estimation: Generate a depth map using the light field focal consistency (Focal Stack), and solve the scene depth z through integral optimization:

[0137]

[0138] where is the gradient field at the i-th viewpoint, TV(z) is the total variation regularization term to suppress noise interference, λ is the regularization weight coefficient, and N is the number of viewpoints or images;

[0139] Polarization scattering suppression: Based on Mueller matrix decomposition of the water body scattering component, restore the target reflected light intensity I object :

[0140] S out = M water · M object · S in

[0141] where M water is the water body scattering matrix, M object is the target object scattering matrix, and S in S out are the Stokes vectors of the incident light and the output light, respectively, representing the polarization state of the light beam;

[0142] Model the attenuation coefficient through the Beer-Lambert law:

[0143]

[0144] Among them, c is the water body extinction coefficient, which is real-time feedback by a preset turbidity sensor, z is the depth, and β(z) is the polarization coupling coefficient related to the incident / observation angle. The target reflected light intensity I is extracted by inversely solving the Mueller equation. object 。

[0145] Furthermore, the specific content of dimension light field feature extraction in S23 is as follows:

[0146] Sediment scattering element modeling: Based on the Mie scattering theory, scattering phase functions with different particle sizes (d = 0.05 - 2 mm) are constructed:

[0147]

[0148] Among them, a n 、b n are Mie coefficients, π n is the radiation pattern or radiation directivity during the scattering process, τ n is a function related to the scattering intensity of the particle and the particle size parameter (such as particle size), d is the characteristic size of the scattering particle, n is the order index in the Mie scattering series expansion, and the value of n > 1;

[0149] Specifically, π n is usually a function related to the scattering angle, which represents the radiation pattern or radiation directivity during the scattering process, especially related to the scattering characteristics of spherical particles; τ n is a function related to the scattering intensity of the particle and the particle size parameter (such as particle size), which helps to characterize the scattering effect of electromagnetic waves with different wavelengths on the particle. The measured polarized light field is matched with the theoretical phase function to generate a particle size probability distribution map.

[0150] Feature encoding network: Design a lightweight two-stream CNN (LFA-Net) to process the light field flow and polarization components respectively;

[0151] Light field flow: The input is a 9×9 sub-aperture view, the viewing angle baseline Δ = 0.5°, and a 3D convolution kernel (3×3×3) is used to extract the disparity implicit features; Polarization component: The input is the Stokes vector, namely S0, S1, S2, and the anisotropic scattering features are extracted through an angle-sensitive convolution (OrientedKernel); The features of the two streams are fused through a cross-attention gate (Cross-AttentionGate) to output a multi-scale feature vector F ∈ R 512 。

[0152] Furthermore, the specific content of generating a sediment concentration prediction model by fusing aerial multi-spectral data and sediment scattering features through spatio-temporal alignment and attention mechanism in S3, and then outputting the spatial distribution map of the channel sediment concentration is as follows:

[0153] S31. Multimodal Spatiotemporal Registration and Encoding;

[0154] Specifically, for spatiotemporal alignment: Using the PTP (Precision Time Protocol) protocol of the drone and the underwater camera, align the timestamps to the microsecond level (deviation < 1ms); Based on the GPS / INS integrated navigation data, map the underwater point cloud to the multispectral image coordinate system through the rigid body transformation matrix T water →air.

[0155] Feature Encoder:

[0156] Aerial Branch: The input is the multispectral reflectance (ρ 450 , ρ 550 , ρ 650 , ρ 750 , ρ 850 ) and NDTI. Extract multi-scale features F l air (hierarchy l = 1, 2, 3, 4) through ResNet-18, and the number of output channels is 64, 128, 256, 512.

[0157] Underwater Branch: The input is the light field feature tensor F underwater ∈R 512 and the scattering parameter matrix. Generate voxelized features F waterl ∈R H×W×C using PointNet++, and align with the aerial feature resolution.

[0158] S32. Hierarchical Cross-modal Attention Mechanism;

[0159] S33. Concentration Inversion with Physical Constraints;

[0160] S34. Output and Format.

[0161] Specifically, the fusion result is stored in the multi-layer HDF5 format, including: 3D sediment concentration field (XYZ grid + concentration value, resolution 0.1m × 0.1m × 0.05m); Attention weight distribution map (attention heat map for each level); Spatiotemporal alignment parameter set (rotation matrix, translation vector, projection error residual); Inversion uncertainty map (calculate the per-pixel standard deviation σ based on Monte Carlo Dropout).

[0162] Furthermore, the specific content of the hierarchical cross-modal attention mechanism in S32 is:

[0163] Pixel-level Modulation: Modulate the underwater coarse-grained features using the highest-resolution aerial features to enhance edge sensitivity. The formula is as follows:

[0164]

[0165] where σ is the Sigmoid function, ⊙ is the element-wise multiplication, and the modulation weight G l ∈[0,1] H×W , is the feature map extracted from the aerial branch at the l-th network layer, is the feature map extracted from the underwater branch, and Conv 1×1 is the convolution operation with a kernel size of k×1, and G l is the modulation weight, is the improved underwater feature obtained after pixel-level modulation;

[0166] Semantic Cross-Attention: Construct a multi-head attention mechanism (4 heads), use the underwater feature as the Query, the aerial feature as the Key-Value, and calculate the cross-modal correlation. The formula is as follows:

[0167]

[0168] where, W q , W k , W v are learnable weight matrices, and d k is the feature dimension or number of channels of the Key vector, and can take the value of 64;

[0169] Residual fusion and upsampling: Stack the attention outputs layer by layer. The formula is as follows:

[0170]

[0171] Use transposed convolution to gradually upsample to the original resolution (0.1m);

[0172] The specific content of the concentration inversion of the physical constraint in S33 is:

[0173] Coupled sediment transport equation: Introduce the sediment transport continuity equation as the physical perception loss and solve it by the finite difference method. The formula is as follows:

[0174]

[0175] where c is the sediment concentration, D is the diffusion coefficient, q s is the flux, υ is the sediment sink term, representing the rate of sediment settlement, deposition or removal, μ is the sediment source term, representing the intensity of new or replenished sediment, and x, y, z are the three-dimensional spatial coordinates in the river channel or water area;

[0176] Regression network: Design a lightweight U-Net decoder, with the input being the final fused feature and the output being the sediment concentration field C∈R H×W(Unit: mg / L), and the formula is as follows:

[0177]

[0178] Specifically, the maximum value is constrained to 1500 mg / L (corresponding to high sediment concentration scenarios such as the Yellow River).

[0179] Furthermore, the specific content of real-time monitoring and early warning of river channel sediment concentration in S4 is as follows:

[0180] S41. Design the early warning system architecture;

[0181] Specifically, for hardware deployment: Edge computing unit: Deploy embedded devices equipped with high-performance GPUs (such as NVIDIA Jetson), directly run lightweight AI models on drones and ground stations to complete data preprocessing (denoising, registration) and preliminary analysis, and control the response time within 50 milliseconds; Communication link: 5G dual-redundant network (main link: Sub-6GHz 100Mbps, backup link: satellite UHF band 10kbps), ensuring an online rate > 99.9%; Alarm trigger device: LoRa IoT acoustic-optic alarms distributed along the shore (working radius 3km, trigger response time < 0.5s).

[0182] Data processing flow: The three-dimensional concentration field data is transmitted after being compressed by Huffman coding (compression ratio 5:1); The edge node performs sliding window spatio-temporal filtering (window size 5×5×3, interval 0.1s); The cloud synchronously updates the parameters of the digital twin hydrodynamic model (time step Δt = 10s).

[0183] S42. Formulate a multi-level intelligent early warning mechanism;

[0184] Specifically, for dynamic threshold setting:

[0185] Risk classification: Combining historical data and real-time hydrological and meteorological information, divide into three levels of dynamic thresholds of yellow, orange, and red (for example: yellow warning is for concentration > 250 mg / L, red warning is for > 1000 mg / L and the range continues to expand);

[0186] Adaptive adjustment: When the upstream rainfall suddenly increases or the water body flow velocity fluctuates beyond the preset value, automatically raise the warning level and dynamically optimize the threshold using the exponential weighted algorithm.

[0187] Prediction of flood peak propagation:

[0188] Based on the hydrodynamic model, simulate the movement path and arrival time of the sediment front, and combine with real-time flow velocity data to estimate the risk time window of key downstream areas (such as sluice dams and bridges), with the error controlled within 15 minutes. If the prediction shows that important facilities will be impacted by high-concentration sediment within 2 hours, immediately initiate a warning above orange level.

[0189] Automated response strategy:

[0190] Primary response: Notify water conservancy management personnel through text messages and APP push, and mark high-risk areas; Intermediate response: Trigger the coastal broadcast system to play voice alerts, and drones automatically fly to the target area for verification by taking pictures; Advanced response: Link the water conservancy gate control system, adjust the opening degree according to the pre-programmed procedure, and guide the diversion of sediment.

[0191] S43. Early warning verification and decision-making support;

[0192] Specifically, digital twin verification:

[0193] Build a digital twin model in the cloud that is completely mirrored to the physical environment, compare the actual monitoring data with the simulation results in real time. If the deviation exceeds 20%, automatically trigger the calibration of model parameters; Use historical flood event inversion tests to ensure the accuracy of early warnings (false alarm rate < 5%).

[0194] Event backtracking and traceability:

[0195] Encrypt and store the whole-chain data (including original images, processing logs, and early warning instructions) based on blockchain technology to provide non-tamperable evidence traceability; Three-dimensional scene snapshots at any time node can be retrieved to analyze the sediment diffusion path and the effectiveness of prevention and control measures.

[0196] S44. System output and interfaces.

[0197] Specifically, real-time early warning signals: Push to the government emergency platform and mobile terminals in the standard JSON format, including risk levels, affected ranges, and recommended measures;

[0198] Visualization terminal: Support multi-platform access on the PC-side WebGL three-dimensional sand table, AR headset holographic interface, and lightweight mobile APP;

[0199] Emergency control interface: Open the API for the automatic control system of water conservancy facilities to call. For example, the instruction format for adjusting the gate opening degree is in the Modbus / TCP protocol.

[0200] Specifically, the technical solutions in the present invention can be applied to the following scenarios:

[0201] 1. Emergency command for flood control and rescue

[0202] In the scenario of flood control and emergency rescue command, the system needs to achieve dynamic tracking of flood peaks and prediction of dam break risks. Dynamic tracking of flood peaks can, in emergency situations such as heavy rain and dam breaks, predict in real time the diffusion path and arrival time of hyperconcentrated floods, providing a valuable time window for personnel evacuation and material dispatching. Prediction of dam break risks monitors the sediment deposition around key dams and evaluates the structural bearing capacity in combination with digital twin models to prevent chain collapses caused by local scour. The applications of the system include pushing a 3D risk map to the command headquarters 2 hours before the arrival of the flood front, automatically marking flooded areas (such as villages and roads); using an AR sand table to simulate the influence scope of different flood discharge schemes (such as the water flow trajectory after opening specific gates) to support rapid decision-making by commanders; and carrying out tasks such as breach blocking and material airdropping through the linkage of unmanned aerial vehicle clusters. Typical cases include the prevention and control of ice floods in the lower reaches of the Yellow River. By combining ice monitoring data, the formation location of ice jams is predicted, and blasting and dredging are carried out in advance to reduce the superimposed risks of ice impact and sediment deposition.

[0203] 2. Safe operation and maintenance of water conservancy projects

[0204] In the scenario of safe operation and maintenance of water conservancy projects, the system needs to achieve reservoir silt removal scheduling and protection of hydropower station equipment. Reservoir silt removal scheduling optimizes the silt removal cycle and mechanical deployment path by accurately monitoring the sediment settlement rate in the reservoir area to reduce operation and maintenance costs. Protection of hydropower station equipment extends the trouble-free operation time of turbine units by predicting the risk of sediment blockage at the water inlet. System applications include generating a monthly 3D thermal map of the silt deposition thickness in the reservoir area and recommending silt removal priorities based on historical data; superimposing the real-time status of underwater gates and sediment accumulation warning lines in the AR inspection interface to prompt maintenance personnel to focus on worn components; and automatically generating the optimal path planning for silt removal vehicles.

[0205] 3. Monitoring of river ecological protection

[0206] In the scenario of monitoring river ecological protection, the system needs to achieve protection of endangered biological habitats and supervision of illegal sand mining. Protection of endangered biological habitats prevents large-scale death of benthic organisms by identifying areas with deteriorated water quality caused by abnormal sediment concentrations; supervision of illegal sand mining locks in suspected sand mining points through detection of sudden changes in riverbed topography to assist law enforcement in obtaining evidence.

[0207] 4. Urban waterlogging early warning linkage

[0208] In the scenario of urban waterlogging early warning linkage, the system needs to achieve warning of pipeline blockages and traffic emergency management. Warning of pipeline blockages prevents the drainage system from collapsing by monitoring the sediment deposition volume at the water inlet during heavy rain; traffic emergency management avoids accidents caused by vehicles getting stuck in silt by predicting the sediment content in road waterlogging zones.

[0209] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. For the same or similar parts among the various embodiments, reference can be made to each other.

[0210] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be obvious to those skilled in the art. The general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the broadest scope consistent with the principles and novel features disclosed herein.

Claims

1. A real-time monitoring method for river sediment concentration using an amphibious UAV based on underwater light field imaging and cross-modal fusion, characterized in that: The following steps are involved: S1, acquiring aerial multispectral data and preprocessing the aerial multispectral data; S2, mount the integrated polarized light field camera on an amphibious drone to capture underwater light field data, reconstruct underwater three-dimensional light field information and extract sediment scattering characteristics through optical models and lightweight deep learning networks; S3, through the spatiotemporal alignment and attention mechanism, the aerial multispectral data and sediment scattering characteristics are integrated to generate a sediment concentration prediction model, and then output the spatial distribution map of river sediment concentration; S4. Based on the spatial distribution map of river sediment concentration, edge computing and cloud collaborative architecture are used to conduct real-time monitoring and early warning of river sediment concentration.

2. According to claim 1, a method for real-time monitoring of river sediment concentration by an amphibious unmanned aerial vehicle based on underwater light field imaging and cross-modal fusion is characterized in that: The specific contents of obtaining and preprocessing aerial multispectral data in S1 are as follows: S11, multispectral data acquisition and radiation correction; S12, geometric correction; S13, optical parameter inversion; S14. Output and format.

3. According to claim 2, a method for real-time monitoring of river sediment concentration by an amphibious unmanned aerial vehicle based on underwater light field imaging and cross-modal fusion is characterized in that: The specific contents of radiation correction in S11 are: Radiation calibration: The empirical linear method is used to establish the digital value DN and the incident radiance L using the reflectivity value of the ground calibration plate. λ The linear relationship is as follows: L λ =a·DN λ +b Among them, coefficients a and b are calibrated by the least square method and fitted independently for each band; Atmospheric correction: Simulate atmospheric path radiation L through MODTRAN model path , combined with the solar irradiance E sun With the atmospheric transmittance T, the water surface reflectivity ρ is inverted λ , the formula is as follows: Among them, cosθ is the angle between solar radiation and the ground normal.

4. According to claim 2, a method for real-time monitoring of river sediment concentration by an amphibious unmanned aerial vehicle based on underwater light field imaging and cross-modal fusion is characterized in that: The specific contents of geometric correction in S12 are: Spatial registration: Use RTK coordinates to target the hovering and shaking of the drone, and use Kalman filtering to fuse GPS data. The formula is as follows: Among them, (X0, Y0, Z0) is the camera projection center coordinates, a i , b i , c i is the projection matrix parameter, f is the focal length, (X, Y, Z) is the three-dimensional coordinate value of a point in space to be projected onto the image, the unit is m; The affine transformation matrix is ​​constructed using the coordinates of the matching points to optimize the image geometric positioning error. The formula is as follows: Among them, m ij is the rotation and scaling parameter, t x ,t y is the translation parameter, which is solved by the least squares method.

5. According to claim 1, a method for real-time monitoring of river sediment concentration by an amphibious unmanned aerial vehicle based on underwater light field imaging and cross-modal fusion is characterized in that: In S2, the optical model and lightweight deep learning network are used to reconstruct underwater three-dimensional light field information and extract sediment scattering characteristics. S21, polarized light field imaging hardware configuration; S22, underwater light field reconstruction and scattering suppression; S23, multi-dimensional light field feature extraction; S24, cross-modal feature association; S25. Output and format.

6. The method for real-time monitoring of river sediment concentration by an amphibious UAV based on underwater light field imaging and cross-modal fusion according to claim 5 is characterized in that: The specific contents of underwater light field reconstruction and scattering suppression in S22 are: Light field depth estimation: Use the light field focal length consistency to generate a depth map, and solve the scene depth z through integral optimization: in, is the gradient field of the i-th viewpoint, TV(z) is the total variation regularization term to suppress noise interference, λ is the regularization weight coefficient, and N is the number of viewpoints or images; Polarization scattering suppression: Decompose the water scattering component based on the Mueller matrix and restore the target reflected light intensity I object : S out =M water ·M object ·S in Among them, M water is the water scattering matrix, M object is the scattering matrix of the target object, S in S out are the Stokes vectors of the incident light and the output light, representing the polarization state of the light beam; The attenuation coefficient is modeled by the Beer-Lambert law: Among them, c is the water extinction coefficient, which is fed back in real time by the preset turbidity sensor, z is the depth, and β(z) is the polarization coupling coefficient related to the incident / observation angle.

7. The method for real-time monitoring of river sediment concentration by an amphibious UAV based on underwater light field imaging and cross-modal fusion according to claim 5 is characterized in that: The specific content of dimensional light field feature extraction in S23 is: Sediment scattering primitive modeling: Construct scattering phase functions of different particle sizes based on Mie scattering theory: Among them, a n , b n is the Mie coefficient, π n is the radiation pattern or radiation directivity in the scattering process, τ n is a function related to the scattering intensity and particle size parameters of the particle, d is the characteristic size of the scattering particle, and n is the order index in the Mie scattering series expansion; Feature encoding network: Design a lightweight two-stream CNN to process the light field flow and polarization components respectively; Light field stream: The input is a 9×9 sub-aperture view with a viewing angle baseline of Δ=0.5°. A 3D convolution kernel is used to extract the implicit features of the disparity. Polarization component: The input is a Stokes vector, i.e., S0, S1, S2. Anisotropic scattering features are extracted through angle-sensitive convolution. The two stream features are fused by cross-attention gating to output a multi-scale feature vector F∈R 512 .

8. The method for real-time monitoring of river sediment concentration by an amphibious UAV based on underwater light field imaging and cross-modal fusion according to claim 1 is characterized in that: In S3, the aerial multispectral data and sediment scattering characteristics are fused through spatiotemporal alignment and attention mechanism to generate a sediment concentration prediction model, and then the specific content of the spatial distribution map of river sediment concentration is output as follows: S31, Multimodal spatiotemporal registration and encoding; S32, hierarchical cross-modal attention mechanism; S33, physically constrained concentration inversion; S34. Output and format.

9. The method for real-time monitoring of river sediment concentration by an amphibious UAV based on underwater light field imaging and cross-modal fusion according to claim 8 is characterized in that: The specific content of the hierarchical cross-modal attention mechanism in S32 is: Pixel-level modulation: Use the highest resolution features in the air to modulate the coarse-grained features underwater to enhance edge sensitivity. The formula is as follows: Among them, σ is the Sigmoid function, ⊙ is the element-by-element multiplication, and the modulation weight G l ∈[0,1] H×W , is the feature map extracted from the aerial branch at the lth network level, is the feature map extracted from the underwater branch, Conv 1×1 is a convolution operation with a kernel size of k×1, G l is the modulation weight, The improved underwater features are obtained after pixel-level modulation; Semantic Cross-Attention: Construct a multi-head attention mechanism, use underwater features as query and aerial features as key-value, and calculate cross-modal associations. The formula is as follows: in, W q , W k , W v is the learnable weight matrix, d k is the feature dimension or number of channels of the Key vector; Residual fusion and upsampling: stack the attention output layer by layer, the formula is as follows: Use transposed convolution to gradually upsample to the original resolution; The specific contents of the concentration inversion of physical constraints in S33 are: Coupled sediment transport equation: The sediment transport continuity equation is introduced as the physical perception loss and solved by the finite difference method. The formula is as follows: Where c is the sediment concentration, D is the diffusion coefficient, and q s is the flux, υ is the sediment sink term, which indicates the sedimentation, deposition or removal rate of sediment, μ is the sediment source term, which indicates the intensity of new or replenished sediment, and x, y, z are the three-dimensional spatial coordinates in the river or water area; Regression network: Design a lightweight U-Net decoder with the final fusion feature as input Output sediment concentration field C∈R H×W , the formula is as follows:

10. The method for real-time monitoring of river sediment concentration by an amphibious UAV based on underwater light field imaging and cross-modal fusion according to claim 1 is characterized in that: The specific contents of real-time monitoring and early warning of river sediment concentration in S4 are as follows: S41. Design the early warning system architecture; S42. Develop a multi-level intelligent early warning mechanism; S43, early warning verification and decision support; S44, system output and interface.

Images