A multi-modal fusion water conservancy monitoring method and system

Abstract

The application discloses a kind of multi-modal fusion water conservancy monitoring methods, the method daytime through visible light camera image acquisition, night through infrared thermal imager image acquisition, after based on water body contour cross-correlation spatial registration, specular reflection and illumination normalization processing, denoising and thermal gradient tensor analysis, respectively extract visible light water quality features and infrared water quality features, input shared cross-modal feature alignment network obtains unified water quality embedding vector, generates water quality evaluation image using day and night unified pseudo-color mapping rule;The application also discloses a kind of multi-modal fusion water conservancy monitoring system.The application solves the problem that visible light and infrared image are different due to imaging mechanism, leading to day and night monitoring results are inconsistent in feature space and visual expression, realizes all-weather continuous unified water quality monitoring.

Description

Technical Field

[0001] This invention belongs to the field of water conservancy monitoring technology, and in particular relates to a multimodal fusion water conservancy monitoring method and system. Background Technology

[0002] Water quality monitoring is a fundamental task for water environment protection and water resource management. Traditional water quality monitoring mainly relies on in-situ physicochemical sensors and laboratory sampling and analysis, which has shortcomings such as low spatiotemporal resolution, high deployment costs, and difficulty in covering large areas of water. With the development of image processing technology, water quality monitoring methods based on visible light images have gradually been applied. By analyzing visual features such as water color, texture, and edges, these methods can non-contactly assess water turbidity, color, and algal bloom status.

[0003] However, visible light images rely on solar reflected radiation and are ineffective at night. To address the problem of nighttime monitoring, existing technologies have introduced infrared thermal imagers to acquire thermal radiation images of water bodies. Infrared images are not limited by lighting conditions and can acquire information on the surface temperature distribution of water bodies at night, thereby reflecting water quality conditions such as thermal pollution and pollution plume diffusion.

[0004] However, existing technologies have a key drawback: the imaging physical mechanisms of visible light images and infrared images are completely different. Visible light images reflect the reflectivity of water to solar radiation, with pixel values ​​corresponding to the reflectivity of the red, green, and blue bands; infrared images reflect the thermal radiation characteristics of water itself, with pixel values ​​corresponding to the surface temperature of the water. Therefore, water quality monitoring results based on visible light images during the day are usually RGB color images or color feature maps, while water quality monitoring results based on infrared images at night are usually grayscale heatmaps or temperature distribution maps. The two are completely inconsistent in feature space and visual representation, making it impossible to continuously stitch together, uniformly interpret, and perform long-term time-series image analysis of day and night monitoring data within the same visual semantic framework.

[0005] In existing technologies, some solutions attempt to input visible light and infrared images into the same deep learning network for feature concatenation. However, these fusion methods are mostly simple concatenation using fully connected layers, failing to consider the fundamental differences in the physical imaging mechanisms of the two modalities, and also failing to address the adaptive fusion weight problem caused by changes in modality effectiveness during day-night switching. Other solutions output visible light and infrared analysis results separately, but do not unify them into a single image semantic representation. Maintenance personnel still need to switch analysis frameworks to interpret the all-weather monitoring results. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide a multimodal fusion water conservancy monitoring method to address the shortcomings of the prior art.

[0007] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is: a multimodal fusion water conservancy monitoring method, comprising the following steps: S1: During the day, visible light images of the monitored water area are collected using a visible light camera, and at night, infrared images of the monitored water area are collected using an infrared thermal imager. S2: Perform spatial registration on the visible light image and the infrared image based on the cross-correlation of water body contours to align them pixel by pixel; S3: Perform specular destigmatization and illumination normalization on the registered visible light image to extract visible light water quality features; perform denoising and thermal gradient tensor analysis on the registered infrared image to extract infrared water quality features. S4: Input the visible light water quality features and the infrared water quality features into a shared cross-modal feature alignment network to obtain a unified water quality embedding vector; the cross-modal feature alignment network is trained with loss to map the same water quality state to the same embedding space in both the visible light and infrared modes; S5: The unified water quality embedding vector is mapped to color values ​​using a day-night unified pseudo-color mapping rule to generate a water quality assessment image, so that the water quality assessment images generated during the day and at night have the same color semantics.

[0008] Furthermore, in step S2, the spatial registration based on water body contour cross-correlation includes: Otsu adaptive threshold segmentation is performed on infrared images to extract the infrared water thermal radiation region mask; The visible light water mask is extracted by analyzing the normalized water index or color space of the visible light image. Calculate the cross-correlation function of the outlines of the infrared water thermal radiation region mask and the visible light water mask, and solve the translation and scaling parameters by maximizing the outline overlap. The infrared image is spatially transformed according to the translation parameters and the scaling parameters to align it pixel-by-pixel with the visible light image.

[0009] Furthermore, in step S3, the extraction of visible light water quality features includes: Detect and repair highly saturated pixels in visible light images to remove specular reflections from water surfaces; A multi-scale illumination normalization algorithm based on Retinex theory is used to decompose a visible light image into a reflection component and an illumination component, while retaining the reflection component. Color features, texture features, and edge features are extracted from the reflection component, wherein the color features include a hue distribution histogram and chromaticity difference factor in the HIS color space, the texture features include water surface texture energy distribution based on Gabor filtering, and the edge features include water surface ripple edge density and direction distribution.

[0010] Furthermore, in step S3, the extraction of infrared water quality features includes: Nonlocal mean denoising is performed on the registered infrared image; Calculate the local gray-level gradient tensor of the denoised infrared image; Principal component analysis was performed on the local gray-level gradient tensor, and the first three principal components were extracted as infrared water quality feature vectors. The first three principal components correspond to the thermal plume intensity, thermal plume direction, and thermal plume spatial scale, respectively.

[0011] Furthermore, in step S4, the cross-modal feature alignment network includes an input layer, a first fully connected hidden layer, a second fully connected hidden layer, and an output layer. The first fully connected hidden layer has a dimension of 64, the second fully connected hidden layer has a dimension of 32, and the output layer outputs a D-dimensional unified water quality embedding vector. The loss training includes classification loss and modality consistency loss.

[0012] Furthermore, the cross-modal feature alignment network includes a modality discriminator, which is used to determine whether the input uniform water quality embedding vector originates from the visible light modality or the infrared modality; through adversarial training, the modality discriminator is unable to distinguish the source modality of the uniform water quality embedding vector.

[0013] Furthermore, in step S5, the day-night unified pseudo-color mapping rule is a learnable color lookup table, which is optimized by minimizing the color difference between the visible light evaluation image and the infrared evaluation image under the same water quality label. The learnable color lookup table maps a uniform water quality embedding vector to RGB color values, where different color values ​​correspond to clean, slightly eutrophic, moderately polluted, and heavily polluted water quality states, respectively.

[0014] Furthermore, in step S4, the unified water quality embedding vector is calculated using the following formula:

[0015] In the formula, for The unified water quality embedding vector after time-lapse fusion For visible light embedding vectors, For infrared embedding vectors, for The calculation of the fusion weight for the solar altitude angle at any given time; in,

[0016] In the formula, For the slope parameter, for The solar altitude angle at that moment, For reference solar altitude angle.

[0017] Furthermore, after generating the water quality assessment image in step S5, the process also includes: The water quality assessment images generated over multiple consecutive days are arranged in chronological order to form a time-series water quality assessment image sequence. The time-series water quality assessment image sequence is subjected to inter-frame difference or background modeling to identify the gradual trend and abrupt events of water quality status.

[0018] Based on the same inventive concept, this invention also proposes a multimodal fusion water conservancy monitoring system, comprising: The image acquisition module is used to acquire visible light images of the monitored water area during the day using a visible light camera, and to acquire infrared images of the monitored water area at night using an infrared thermal imager. The registration module is used to perform spatial registration of the visible light image and the infrared image based on the cross-correlation of water body contours, so that the two are aligned pixel by pixel. The feature extraction module is used to perform specular reflection removal and illumination normalization processing on the registered visible light image to extract visible light water quality features, and to perform denoising and thermal gradient tensor analysis on the registered infrared image to extract infrared water quality features. A cross-modal alignment module is used to input the visible light water quality features and the infrared water quality features into a shared cross-modal feature alignment network to obtain a unified water quality embedding vector; the cross-modal feature alignment network is trained with loss so that the same water quality state is mapped to the same embedding space in both the visible light mode and the infrared mode; The unified mapping module is used to map the unified water quality embedding vector into color values ​​using a day-night unified pseudo-color mapping rule, thereby generating a water quality assessment image, so that the water quality assessment images generated during the day and at night have the same color semantics.

[0019] This invention provides a multimodal fusion method and system for water conservancy monitoring, which has the following advantages compared with the prior art: 1. This invention utilizes a spatial registration method based on the cross-correlation of water body contours. It takes the boundary contours of water bodies in both the visible light and thermal infrared bands as the registration reference. By maximizing the contour overlap between the visible light and infrared water body masks, the translation and scaling parameters are solved, achieving pixel-by-pixel alignment between the visible light and infrared images. This solves the problem that traditional feature point matching is prone to failure in the prior art due to the sparse water surface texture and its rapid changes over time, providing a reliable spatial reference for the unified processing of day and night images.

[0020] 2. This invention performs specular destigmatization and Retinex-based multi-scale illumination normalization on the registered visible light image, extracting color, texture, and edge features from the reflection component. Simultaneously, it performs nonlocal mean denoising and thermal gradient tensor principal component analysis on the registered infrared image to extract infrared features reflecting the intensity, direction, and spatial scale of the thermal plume. This achieves a physically interpretable representation of water quality features in both visible and infrared modes, eliminating the feature distribution differences caused by the different imaging mechanisms of the two modes, and laying a solid foundation for cross-modal feature alignment.

[0021] 3. This invention maps visible light and infrared features from different physical sources to the same water quality embedding space through a shared cross-modal feature alignment network and modal consistency loss training. It also constrains the minimization of the cosine distance between the visible light embedding vector and the infrared embedding vector of the same water body at the time of day-night switching, thus realizing the mapping of the same water quality state to the same embedding space under visible light and infrared modes. Furthermore, by introducing a modal discriminator and adversarial training, the embedding vector cannot be distinguished by its source modality, thereby truly unifying the day-night monitoring results at the feature level and overcoming the shortcomings of the simple fully connected layer splicing in the prior art that does not consider the physical differences of the modes.

[0022] 4. This invention maps a unified water quality embedding vector to RGB color values ​​through a uniform pseudo-color mapping rule for day and night. It also optimizes the process by using a learnable color lookup table to minimize the color difference between visible light assessment images and infrared assessment images under the same water quality label. This achieves color semantic consistency between daytime visible light assessment images and nighttime infrared assessment images, ensuring that the same water quality state presents the same color code in day and night images. Maintenance personnel can interpret all-weather monitoring results without switching analysis frameworks, and day and night monitoring data can be continuously spliced ​​and uniformly interpreted under the same visual semantic framework.

[0023] 5. This invention calculates the fusion weight based on the solar elevation angle and performs weighted fusion of the visible light embedding vector and the infrared embedding vector, thereby achieving continuous and uninterrupted monitoring during the day-night transition period before and after sunrise and before and after sunset. The fusion weight is adaptively adjusted as the solar elevation angle changes continuously, eliminating the monitoring blind spots and fusion discontinuity problems caused by mode switching. It solves the technical defects of existing technologies where the fusion weight is fixed or simply set and cannot adapt to changes in the effectiveness of day and night modes.

[0024] 6. This invention, by arranging water quality assessment images generated over multiple consecutive days in chronological order to form a time-series water quality assessment image sequence after generating the water quality assessment image, and performing inter-frame difference or background modeling on the sequence, achieves long-term water quality change detection unaffected by light switching. It can accurately identify the gradual trend and abrupt events of water quality status, avoid the interference of spurious change signals caused by day-night switching on the time-series analysis results, and improve the accuracy and reliability of long-term water quality change detection.

[0025] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0026] Figure 1 This is a flowchart of the method of the present invention.

[0027] Figure 2 This is a system flowchart of the present invention. Detailed Implementation

[0028] like Figure 1 The multimodal fusion water conservancy monitoring method shown includes the following steps: S1: During the day, visible light images of the monitored water area are collected using a visible light camera, and at night, infrared images of the monitored water area are collected using an infrared thermal imager. S2: Perform spatial registration on the visible light image and the infrared image based on the cross-correlation of water body contours to align them pixel by pixel; S3: Perform specular destigmatization and illumination normalization on the registered visible light image to extract visible light water quality features; perform denoising and thermal gradient tensor analysis on the registered infrared image to extract infrared water quality features. S4: Input the visible light water quality features and the infrared water quality features into a shared cross-modal feature alignment network to obtain a unified water quality embedding vector; the cross-modal feature alignment network is trained with loss to map the same water quality state to the same embedding space in both the visible light and infrared modes; S5: The unified water quality embedding vector is mapped to color values ​​using a day-night unified pseudo-color mapping rule to generate a water quality assessment image, so that the water quality assessment images generated during the day and at night have the same color semantics.

[0029] The aforementioned technical solution overcomes the inherent limitation of visible light images not working effectively at night by employing a time-sharing acquisition strategy using visible light cameras during the day and infrared thermal imagers at night, providing a data foundation for all-weather water quality monitoring. Spatial registration based on water body contour cross-correlation provides a reliable spatial benchmark for subsequent unified processing of day and night images. Physically guided enhancement and feature extraction eliminate the feature distribution differences caused by the different imaging mechanisms of the two modalities. Through a shared cross-modal feature alignment network and modal consistency loss training, visible light and infrared features from different physical sources are mapped to the same water quality embedding space, ensuring that the same water quality state produces the same or similar embedding vectors in both modalities. This truly unifies day and night monitoring results at the feature level, overcoming the shortcomings of existing technologies that use simple fully connected layer stitching without considering modal physical differences. Through a unified pseudo-color mapping rule for day and night, color semantic consistency is achieved between daytime visible light assessment images and nighttime infrared assessment images, ensuring that the same water quality state presents the same color encoding in day and night images. This allows day and night monitoring data to be continuously stitched and uniformly interpreted within the same visual semantic framework.

[0030] The principles, implementation methods, and technical effects of each step are explained one by one.

[0031] Step S1: Image acquisition.

[0032] In this step, visible light images of the monitored water area are collected during the day using a visible light camera, and infrared images of the monitored water area are collected at night using an infrared thermal imager.

[0033] It should be noted that the visible light camera and the infrared thermal imager can be deployed on the same monitoring platform, such as a shore-based fixed station, an airborne platform for unmanned aerial vehicles (UAVs), or an unmanned surface vessel (USV). To ensure the accuracy of subsequent day-night transition fusion processing, the visible light camera and the infrared thermal imager are preferably synchronized via hardware to ensure their timestamps are consistent. The visible light camera can be an RGB camera or a hyperspectral camera, and the infrared thermal imager can be a long-wave infrared thermal imager, preferably operating in the 8 to 14 micrometer band. In this band, water has a high emissivity, and the water surface area appears as a bright, connected region in the infrared image, which is beneficial for subsequent water mask extraction.

[0034] The technical advantage of this step is that by using a time-sharing acquisition strategy of using a visible light camera during the day and an infrared thermal imager at night, it overcomes the inherent limitation that visible light images cannot work effectively at night, thus providing a data foundation for all-weather water quality monitoring.

[0035] Step S2: Spatial registration based on cross-correlation of water body contours.

[0036] Because visible light cameras and infrared thermal imagers may differ in their field of view, resolution, and installation location, the images they acquire do not correspond directly in space. Therefore, this step performs spatial registration of the visible light and infrared images based on the cross-correlation of water body contours to align them pixel by pixel.

[0037] The core principle of this step is that water bodies exhibit characteristics distinct from shorelines and background features in both the visible light and thermal infrared bands. In visible light images, water areas can be extracted using normalized water index or color space analysis; in infrared images, water bodies appear as bright areas due to their high emissivity and can be extracted through threshold segmentation. The water body masks extracted in both modalities should have a high degree of spatial overlap, and their boundary contours should constitute registration constraints.

[0038] Specifically, in this step, the spatial registration based on water body contour cross-correlation includes: Otsu adaptive threshold segmentation is performed on infrared images to extract the infrared water thermal radiation region mask; The visible light water mask is extracted by analyzing the normalized water index or color space of the visible light image. Calculate the cross-correlation function of the outlines of the infrared water thermal radiation region mask and the visible light water mask, and solve the translation and scaling parameters by maximizing the outline overlap. The infrared image is spatially transformed according to the translation parameters and the scaling parameters to align it pixel-by-pixel with the visible light image.

[0039] The purpose of this technical solution is to provide a precise spatial reference for subsequent cross-modal feature alignment. Because the field of view, resolution, and installation location of visible light cameras and infrared thermal imagers may differ, the images acquired by the two do not directly correspond in space; therefore, spatial registration is required.

[0040] When extracting the infrared water thermal radiation region mask, the specific process of Otsu adaptive thresholding segmentation is as follows: The gray-level histogram of the infrared image is calculated, all possible thresholds are iterated, the image is divided into foreground and background classes, the inter-class variance is calculated, and the threshold that maximizes the inter-class variance is selected as the optimal threshold. The larger the inter-class variance, the more obvious the difference between the foreground and background, and the better the segmentation effect.

[0041] When extracting visible light water masks, for cameras with near-infrared capabilities, the calculation can be performed directly. index, The calculation formula is:

[0042] in, The normalized water index, For green light band reflectivity, This refers to the reflectivity in the near-infrared band. A higher value indicates that the pixel is more likely to belong to a water body area. For RGB cameras, which lack a near-infrared band, HSV color space analysis can be used to convert the RGB image to HSV space. Water bodies typically meet the following conditions: hue (H) is within the blue to green range, saturation (S) is below a preset threshold, and brightness (V) is above a preset threshold. Here, H is the hue in the HSV color space, S is the saturation in the HSV color space, and V is the brightness in the HSV color space. By setting a reasonable threshold range, a visible light water mask can be extracted.

[0043] When calculating the profile cross-correlation function, the distance function Either Euclidean distance or chamfered distance can be used. Chamfered distance has higher computational efficiency and is suitable for real-time processing scenarios. Distance threshold. The value can be determined based on the image resolution and the sharpness of the water body boundary, and is usually taken as 3 to 5 pixels. Let the contour point set of the infrared water body mask be... The outline point set of the visible light water mask is The optimal space transformation parameters are solved using the following formula:

[0044] in, For along Translation parameters of direction, For along Translation parameters of direction, For scaling parameters, To obtain the optimal solution Directional translation parameters, To obtain the optimal solution Directional translation parameters, To obtain the optimal scaling parameters, This indicates the value that makes the subsequent expression reach its maximum value. Values, For summation, For infrared water body mask contour point set One of the contour points, This is the set of points representing the outline of the infrared water thermal radiation region mask. For the outline point set of the visible light water body mask, This is an indicator function that takes the value 1 if the condition within the parentheses is true, and 0 otherwise. This is a distance function used to calculate the distance between two points or between a point and a set of points. Distance threshold Represents the contour points Scaling / Scaling Represents the contour points The scaling is followed by translation. This formula means that for each point on the infrared profile, after scaling and translation, if its distance to the nearest point on the visible light profile is less than a threshold... If the sum of the counts is 1, then the optimal solution is the parameter that maximizes the sum of the counts by iterating through all possible translation and scaling parameters.

[0045] When solving for translation and scaling parameters, exhaustive search or gradient descent optimization can be used. Exhaustive search is suitable for scenarios with a small search range, while gradient descent is suitable for scenarios with a large search range. To improve search efficiency, the image can be downsampled using pyramids first for a coarse search at low resolution, followed by a fine search at high resolution.

[0046] Unlike existing technologies that rely on matching ground feature points, this invention utilizes the boundary contours of the water body itself in both modalities as the registration benchmark, eliminating the need for additional calibration devices. Because water surface textures are sparse and change rapidly over time, traditional feature point matching methods are prone to failure in dynamic water surface scenes. In contrast, the water body boundary contours are less affected by water surface fluctuations and exhibit strong gradient features in both modalities. Therefore, the registration method of this invention is more robust, providing a reliable spatial benchmark for subsequent cross-modal feature alignment and unified image generation. Through Otsu adaptive thresholding and NDWI color space analysis, water body masks in both modalities can be robustly extracted. Spatial transformation parameters are solved using the contour cross-correlation function, eliminating the need for external calibration objects and making it suitable for rapid deployment in any water area.

[0047] Step S3: Dual-modal image enhancement and feature extraction.

[0048] In step S3, the extraction of visible light water quality features includes: Detect and repair highly saturated pixels in visible light images to remove specular reflections from water surfaces; A multi-scale illumination normalization algorithm based on Retinex theory is used to decompose a visible light image into a reflection component and an illumination component, while retaining the reflection component. Color features, texture features, and edge features are extracted from the reflection component, wherein the color features include a hue distribution histogram and chromaticity difference factor in the HIS color space, the texture features include water surface texture energy distribution based on Gabor filtering, and the edge features include water surface ripple edge density and direction distribution.

[0049] Meanwhile, in step S3, the extraction of infrared water quality features includes: Nonlocal mean denoising is performed on the registered infrared image; Calculate the local gray-level gradient tensor of the denoised infrared image; Principal component analysis was performed on the local gray-level gradient tensor, and the first three principal components were extracted as infrared water quality feature vectors. The first three principal components correspond to the thermal plume intensity, thermal plume direction, and thermal plume spatial scale, respectively.

[0050] The purpose of this step is to provide the cross-modal alignment network with physically meaningful visible light and infrared features that are unaffected by lighting conditions.

[0051] For visible light images, specular reflection removal preprocessing is first performed. Water surfaces are prone to specular reflection under sunlight, forming bright flares that severely interfere with the extraction of water color features. This invention utilizes the polarization characteristics of the water surface or a color saturation threshold to detect specular reflection pixels. Specifically, the visible light image is converted to HSV space; if the saturation S of a pixel is below a threshold... And the brightness V is higher than the threshold. If the value is 0, it is determined to be a specular reflection pixel. Here, S represents the pixel's saturation in the HSV color space, and V represents the pixel's brightness in the HSV color space. The threshold for saturation determination. This is the threshold for determining brightness. Typically... Take a value between 0.1 and 0.3. The value is set between 0.8 and 0.95. Specular reflection pixels are then repaired and filled using either neighborhood mean filling or image inpainting algorithms. Neighborhood mean filling has low computational complexity and is suitable for real-time processing; image inpainting algorithms offer better results and are suitable for scenarios with high image quality requirements. Image inpainting algorithms can employ the Criminisi algorithm or deep learning-based image inpainting networks.

[0052] Then, a multi-scale illumination normalization algorithm based on Retinex theory is employed. Retinex theory posits that an image is composed of the product of the reflection component and the illumination component, i.e.:

[0053] in, Visible light image, The reflective component reflects the inherent optical properties of the water body. This refers to the illumination component, which reflects ambient lighting conditions. This invention addresses this by analyzing the illumination component... Multi-scale Gaussian filtering estimation is performed to remove the illumination component and retain only the reflection component. The scale parameters of multi-scale Gaussian filtering can be determined based on the image resolution, typically choosing three scales: small scale (σ=15), medium scale (σ=80), and large scale (σ=250). Here, σ is the scale parameter of the Gaussian filter. The small scale is used to capture local illumination changes, the medium scale to capture medium-range illumination changes, and the large scale to capture global illumination trends. The illumination estimation results from the three scales are weighted and fused to obtain the final illumination components. Reflection component It reflects the inherent optical properties of water bodies and is not affected by solar altitude angle or cloud cover.

[0054] In the reflection component Extract three types of water quality-related image features: Regarding color characteristics, Convert to the HIS color space, calculate the histogram of hue H, and the chromaticity difference factor:

[0055] in, Hue in the HIS color space For reference hues used in cleaning water, a value of 200 to 220 degrees is typically chosen, which corresponds to the blue zone. This refers to the color difference factor. Color features are used to characterize color anomalies in water bodies, such as green shifts caused by algal blooms or color changes caused by industrial wastewater.

[0056] Regarding texture features, Gabor filtering is performed to calculate the texture energy distribution at different frequencies and directions. The frequency parameters of the Gabor filter can be taken as 0.1, 0.2, 0.3, 0.4, and 0.5, and the direction parameters can be... , , , For each frequency and direction combination, the energy of the filtered response is calculated, resulting in a 20-dimensional texture feature vector. Changes in water turbidity and suspended solids concentration cause changes in water surface texture, and the texture energy distribution can reflect these changes.

[0057] Regarding edge features, Canny edge detection is performed to calculate the edge density and orientation distribution of water surface ripples. The dual thresholds for Canny edge detection can be set to 50 and 150. When calculating edge density, the image is divided into several grids, and the number of edge pixels within each grid is counted to obtain the edge density distribution. When calculating the edge orientation distribution, the number of edge pixels within each orientation interval is counted to obtain an edge orientation histogram. Edge density and orientation distribution can help determine the water surface state and flow velocity.

[0058] The above color, texture, and edge features are concatenated into a visible light feature vector. .in, This is the visible light water quality feature vector.

[0059] For infrared images, nonlocal means denoising is performed first. Infrared images are affected by sensor thermal noise and atmospheric water vapor absorption, resulting in a low signal-to-noise ratio. Nonlocal means denoising calculates a weighted average of similar blocks in the image, suppressing noise while preserving the edges of the thermal plume. The search window size can be 21×21, the similarity window size can be 7×7, and the smoothing parameter h can be 10 to 15. Here, h is the smoothing parameter of nonlocal means denoising, used to control the denoising intensity. Compared with local denoising methods such as Gaussian filtering, nonlocal means denoising can better preserve the spatial structure of the thermal plume.

[0060] Then, the local gray-level gradient tensor of the denoised infrared image is calculated. For infrared images Its local gray-level gradient tensor G is:

[0061] in, For local gray-level gradient tensors, Infrared image, For infrared images in First-order partial derivatives in the direction For infrared images in First-order partial derivative in the direction, for The square of, for The square of, for and The product of . For each pixel, calculate the gradient tensor in its 3×3 neighborhood to obtain a 2×2 symmetric matrix.

[0062] For gradient tensor Eigenvalue decomposition yields two eigenvalues. and and its corresponding eigenvectors. For gradient tensor The first eigenvalue, For gradient tensor The second eigenvalue of the gradient tensor. The magnitude of the eigenvalue reflects the spatial rate of change of thermal radiation, and the direction of the eigenvector reflects the direction of the thermal plume. The greater the intensity of the thermal plume, the larger the eigenvalue of the gradient tensor; the more pronounced the direction of the thermal plume, the stronger the directionality of the eigenvector. During eigenvalue decomposition, the gradient tensor... Two eigenvalues and satisfy . The rate of change in the direction of the fastest change in thermal radiation. It reflects the rate of change perpendicular to this direction. The larger the eigenvalue, the more drastic the change in thermal radiation in that direction.

[0063] In principal component analysis, the gradient tensor of each pixel is expanded into a 4-dimensional vector. Principal component analysis (PCA) was performed on the 4D vectors of all pixels to extract the first three principal components. The reason for choosing the first three principal components is that the first principal component typically corresponds to the main direction of change of the thermal plume, the second principal component corresponds to the secondary direction of change, and the third principal component corresponds to spatial scale information; the combination of these three can comprehensively describe the spatial distribution characteristics of the thermal plume. Specifically, the first principal component corresponds to the main change pattern of the thermal plume, the second principal component corresponds to the secondary change pattern, and the third principal component corresponds to spatial scale information. These first three principal components were then used as the infrared water quality feature vector. .in, This is the infrared water quality feature vector.

[0064] Finally, for and Z-score standardization is performed on each feature vector to achieve a mean of 0 and a variance of 1, resulting in the standardized feature vectors. This is the visible light water quality feature vector. This represents the infrared water quality feature vector. The purpose of standardization is to eliminate differences in the dimensions and numerical ranges of different features, enabling the subsequent cross-modal alignment network to fairly process features across all dimensions.

[0065] This invention eliminates the feature distribution differences between two modalities caused by different imaging mechanisms through physically guided enhancement and feature extraction. Visible light features reflect the optical properties of water, while infrared features reflect the thermal radiation properties of water; both have clear physical meanings, laying a solid foundation for subsequent cross-modal alignment. Destigma eliminates the interference of solar flares on water color features, and Retinex illumination normalization preserves the reflection components reflecting the inherent optical properties of water, ensuring that the extracted color, texture, and edge features are unaffected by solar altitude angle and cloud cover. The HIS color space hue distribution histogram and chromaticity difference factor effectively characterize the green shift caused by algal blooms and chromaticity changes caused by industrial wastewater; Gabor filter texture energy distribution reflects changes in water turbidity and suspended solids concentration; and Canny edge density and direction distribution aid in determining water surface condition and flow velocity. Nonlocal mean denoising was used to suppress interference from sensor thermal noise and atmospheric water vapor absorption. Principal component analysis of thermal gradient tensor was used to extract feature vectors reflecting the intensity, direction and spatial scale of thermal plumes, providing physically meaningful infrared features for cross-modal alignment.

[0066] Step S4: Cross-modal feature alignment.

[0067] In step S4, the cross-modal feature alignment network includes an input layer, a first fully connected hidden layer, a second fully connected hidden layer, and an output layer. The first fully connected hidden layer has a dimension of 64, the second fully connected hidden layer has a dimension of 32, and the output layer outputs a D-dimensional unified water quality embedding vector. The loss training includes classification loss and modality consistency loss.

[0068] The cross-modal feature alignment network includes a modality discriminator, which is used to determine whether the input uniform water quality embedding vector originates from the visible light modality or the infrared modality; through adversarial training, the modality discriminator is unable to distinguish the source modality of the uniform water quality embedding vector.

[0069] In addition, in step S4, the unified water quality embedding vector is calculated using the following formula:

[0070] In the formula, for The unified water quality embedding vector after time-lapse fusion For visible light embedding vectors, For infrared embedding vectors, for The calculation of the fusion weight for the solar altitude angle at any given time; in,

[0071] In the formula, For the slope parameter, for The solar altitude angle at that moment, For reference solar altitude angle.

[0072] The purpose of this step is to achieve a unified semantic mapping of visible light features and infrared features, and to solve the modality switching problem during the day-night transition period.

[0073] The first fully connected hidden layer of the cross-modal feature alignment network has a dimension of 64, and the second fully connected hidden layer has a dimension of 32. The output layer outputs a D-dimensional unified water quality embedding vector. Here, D is the dimension of the unified water quality embedding vector, which can be determined based on the number of categories of water quality parameters. For example, when water quality is classified into four categories—clean, slightly eutrophic, moderately polluted, and heavily polluted—D can be between 8 and 16. The reason for choosing two hidden layers is that the first layer maps the input features to a higher-dimensional intermediate representation, and the second layer compresses them to a lower-dimensional semantic representation. This two-layer structure ensures expressive power while avoiding overfitting.

[0074] During training, daytime and nighttime samples are alternately input into the network. For daytime samples, the input... Using water quality labels (such as turbidity level and color level) as monitoring signals, the classification loss is calculated. .in, This is the visible light water quality feature vector. For classification loss.

[0075]

[0076] in, For classifying losses, This represents the number of samples taken during the day. For the daytime The first sample Each water quality label outputs a true probability. The cross-modal feature alignment network outputs a unified water quality embedding vector for the current daytime sample, which is then input into the classification layer to obtain the sample's output score for each water quality label. Subsequently, the Softmax function is used to normalize the output scores for each category, converting the scores into predicted probabilities for each category. Among these, the true water quality label... The corresponding prediction probability is .

[0077] Water quality labels can be obtained through concurrent in-situ sensor measurements or laboratory sampling analysis. For nighttime samples, input... Using visible light water quality labels obtained through time interpolation at the same time as the monitoring signal, the classification loss is calculated. .in, This is an infrared water quality feature vector. Since visible light images cannot be obtained at night, this invention uses a time interpolation method, interpolating the water quality labels of adjacent daytime periods before and after nighttime to obtain approximate water quality labels for nighttime.

[0078] Modal consistency loss Calculated during day-night transition, constraints and The cosine distance is the smallest:

[0079] in, For modal consistency loss, For visible light embedding vectors, For infrared embedding vectors, for and dot product, for norm, for The norm of the equation. The cosine distance ranges from 0 to 2, with a value of 0 when the two vectors are in the same direction and 2 when they are in opposite directions. By minimizing this loss, the visible light embedding vector and the infrared embedding vector of the same water body at the time of day-night transition are made as close as possible.

[0080] The total loss is:

[0081] in, For the total loss, For classifying losses, For modal consistency loss, These are weighting coefficients, which can be adjusted based on training results. The larger the value, the stronger the modal consistency constraint. The smaller the value, the higher the priority is given to classification accuracy.

[0082] Furthermore, to more thoroughly eliminate modal differences, this invention introduces a modality discriminator. The modality discriminator is a binary classification network, similar in structure to the cross-modal feature alignment network, including two fully connected hidden layers, and its output is a binary classification probability. Adversarial training is implemented using a gradient inversion layer, that is, a gradient inversion layer is inserted between the cross-modal feature alignment network and the modality discriminator, inverting the gradient during backpropagation, thereby achieving adversarial training. The adversarial loss is:

[0083] in, To combat the losses, For the expected operation, For logarithmic operations, For the modality discriminator to embed the uniform water quality vector The output, To unify the water quality embedding vector, the total loss becomes:

[0084] in, For the total loss, For classifying losses, For modal consistency loss, To combat the losses, The modal consistency loss weighting coefficient is used. To counteract the loss weights, adjustments can be made based on training performance. The adversarial training process involves the modality discriminator trying to distinguish the source modality of the embedding vectors, while the cross-modality feature alignment network tries to generate embedding vectors that the discriminator cannot distinguish. The two work alternately to optimize until a Nash equilibrium is reached.

[0085] During the day-night transition period, i.e., around sunrise and sunset, both visible light and infrared images have a certain degree of effectiveness, but relying solely on either mode is not reliable enough. Therefore, this invention calculates the fusion weight based on the solar altitude angle. The visible light embedding vector and the infrared embedding vector are weighted and fused:

[0086] in, for The unified water quality embedding vector after time-lapse fusion For visible light embedding vectors, For infrared embedding vectors, for The calculation of the fusion weight for the solar altitude angle at any given time. For time variables, unless otherwise specified in this invention, it is assumed to be within the time frame. time. Calculated using the following formula:

[0087] in, for Moment-wise fusion weights It is a natural exponential function. The slope parameter controls the steepness of the fusion weights as the solar altitude angle changes. for The solar altitude angle at that moment, For reference to the solar altitude angle. The physical meaning of this formula is: when the solar altitude angle... Much greater than the reference solar altitude angle hour, A value approaching 1 indicates that the visible light mode is dominant; when much smaller hour, Approaching 0 indicates that the infrared mode is dominant; when near hour, A smooth transition occurs around 0.5. Slope parameter. Control the steepness of the transition. The larger the size, the steeper the transition; The smaller the value, the smoother the transition, usually Use a value between 0.1 and 0.5. Refer to the solar altitude angle. Usually taken to This refers to the time around sunrise and sunset when the sun is near the horizon. It should be noted that this weighted fusion step obtains the visible light embedding vector in step S4. and infrared embedding vector This is performed after step S5, before pseudo-color mapping, to create the fused unified water quality embedding vector. Then, input the pseudo-color mapping function C from step S5 to generate a water quality assessment image for the day-night transition period.

[0088] This invention utilizes a shared network and modality consistency loss to map visible light and infrared features from different physical sources to the same water quality embedding space, ensuring that the same water quality state generates identical or similar embedding vectors in both modalities. The two-layer structure, with a first fully connected hidden layer of 64 dimensions and a second fully connected hidden layer of 32 dimensions, maintains expressive power while avoiding overfitting. Modality consistency loss, by constraining cosine distance minimization, makes the visible light and infrared embedding vectors of the same water body as close as possible to each other during day-night transitions, thus truly unifying day-night monitoring results at the feature level and overcoming the shortcomings of existing technologies that simply splice fully connected layers without considering modal physical differences. By introducing a modality discriminator and adversarial training, the embedding vectors generated by the cross-modal feature alignment network cannot be distinguished by the modality discriminator, further eliminating the interference of modality differences on water quality assessment. Through a weighted fusion mechanism based on solar altitude angle, continuous and uninterrupted monitoring is achieved during the day-night transition periods before and after sunrise and sunset. The fusion weights adaptively adjust with the continuous change of the solar altitude angle; the higher the solar altitude angle, the greater the visible light weight, and the lower the solar altitude angle, the greater the infrared weight. Unlike the fixed or simple fusion weight settings in existing technologies, the fusion weights of this invention continuously change with the solar altitude angle, which can adaptively reflect the effectiveness of the two modes under different lighting conditions, eliminating the monitoring blind spots and fusion discontinuities caused by mode switching.

[0089] Step S5: Unify pseudo-color mapping and generate water quality assessment images.

[0090] In step S5, the day-night unified pseudo-color mapping rule is a learnable color lookup table. The learnable color lookup table is optimized by minimizing the color difference between the visible light evaluation image and the infrared evaluation image under the same water quality label. The learnable color lookup table maps the unified water quality embedding vector to RGB color values, where different color values ​​correspond to clean, slightly eutrophic, moderately polluted and heavily polluted water quality states, respectively.

[0091] This step aims to achieve visual semantic consistency between day and night images. It defines a pseudo-color mapping function for day-night uniformity. The uniform water quality embedding vector Z is mapped to the RGB color space. Here, C is the pseudo-color mapping function. Represents a D-dimensional real space. Let Z represent the three-dimensional RGB color space, and Z be the unified water quality embedding vector. The mapping rules are determined by the correspondence between water quality labels and color semantics on the training set: blue corresponds to clean water, green corresponds to slightly eutrophic, yellow corresponds to moderate pollution, and red corresponds to heavily polluted.

[0092] The pseudo-color mapping function C can employ a learnable color lookup table. The color lookup table contains K color entries, each corresponding to a water quality embedding vector cluster center and its RGB color value. Here, K is the number of color entries in the color lookup table. For the input uniform water quality embedding vector Z, its distance to each cluster center is calculated, and the color value is determined through soft assignment or nearest neighbor assignment. The parameters of the learnable color lookup table are optimized by minimizing the color difference between visible light evaluation images and infrared evaluation images under the same water quality label.

[0093] in, The value represents the color difference loss, and ∑ is the summation symbol. For indexing samples with the same water quality label, This indicates that the summation is performed on all samples with the same water quality label. It is a pseudo-color mapping function. For the first A unified water quality embedding vector for each visible light sample. For the first A unified water quality embedding vector for each infrared sample. For the first The RGB color values ​​of a visible light sample after pseudo-color mapping For the first The RGB color values ​​of an infrared sample after pseudo-color mapping This is the square of the L2 norm. The meaning of this loss function is that for samples with the same water quality label, regardless of whether they originate from the visible light mode or the infrared mode, they should present the same or similar colors after pseudo-color mapping. By minimizing this loss, the parameters of the color lookup table are optimized, making the color semantics of day and night images tend to be consistent.

[0094] For each pixel or grid cell in the monitoring area, its unified water quality embedding vector is calculated, mapped by the pseudo-color mapping function C, and then filled with color to generate a unified water quality assessment image. In the final output water quality assessment image, the color of each pixel encodes the water quality status at that location, and the color semantics are completely consistent between daytime and nighttime.

[0095] This invention achieves color semantic consistency between daytime visible light assessment images and nighttime infrared assessment images through a unified pseudo-color mapping rule, ensuring that the same water quality state is represented by the same color code in day and night images. A learnable color lookup table optimizes the process by minimizing the color differences between visible light and infrared assessment images under the same water quality label, further guaranteeing the visual semantic consistency of day and night images. Maintenance personnel can interpret the water quality status throughout the day without switching analysis frameworks, allowing for continuous splicing and unified interpretation of day and night monitoring data within the same visual semantic framework.

[0096] Step S6: Time-series water quality analysis.

[0097] After generating the water quality assessment image in step S5, the following steps are also included: The water quality assessment images generated over multiple consecutive days are arranged in chronological order to form a time-series water quality assessment image sequence. The time-series water quality assessment image sequence is subjected to inter-frame difference or background modeling to identify the gradual trend and abrupt events of water quality status.

[0098] The purpose of this step is to perform long-term temporal change detection using the unified image sequence. After generating the water quality assessment image in step S5, the water quality assessment images generated over multiple consecutive days are arranged in chronological order to form a time-series water quality assessment image sequence. The time interval of the sequence can be determined according to monitoring requirements, such as one frame per hour, one frame per day, etc.

[0099] Inter-frame differencing or background modeling is performed on time-series water quality assessment image sequences. The specific process of inter-frame differencing is as follows: calculate the color difference at corresponding pixels in two adjacent water quality assessment images; if the difference exceeds a preset threshold, it is determined that a water quality change has occurred at that pixel location. The specific process of background modeling is as follows: establish a background model using a Gaussian mixture model or codebook model; compare the current frame with the background model; if the difference exceeds a preset threshold, it is determined to be foreground, i.e., a water quality change has occurred. Since the water quality assessment images have unified the day-night modality, the time-series analysis is no longer interfered with by spurious change signals caused by lighting changes. For example, in traditional schemes, a sudden brightening of the visible light image at sunrise may be misjudged as a sudden change in water quality by inter-frame differencing; however, in this invention, since the day-night images have unified to the same color semantics, the color will not change abruptly at sunrise, thus avoiding spurious change signals.

[0100] This invention achieves long-term water quality change detection unaffected by light switching by using inter-frame difference or background modeling of unified water quality assessment image sequences. It can accurately identify the gradual trend and abrupt events of water quality status, avoid the interference of spurious change signals caused by day-night switching on the time series analysis results, and improve the accuracy and reliability of long-term water quality change detection.

[0101] Based on the overall technical solution described above, the cross-modal feature alignment network outputs a multi-dimensional unified water quality embedding vector, corresponding to at least two water quality parameters, including turbidity, color, thermal pollution, and algal bloom. In step S5, a day-night unified pseudo-color mapping rule is used to map the multi-dimensional unified water quality embedding vector into a multi-channel water quality assessment image. For example, the red channel represents the degree of thermal pollution, the green channel represents the degree of algal bloom, and the blue channel represents turbidity. Maintenance personnel can simultaneously observe the spatial distribution of multiple water quality parameters through the multi-channel image. This extension method still belongs to the unified image semantic framework, and its role is to improve the invention's ability to simultaneously express multiple water quality parameters.

[0102] like Figure 2 As shown, based on the same inventive concept, this invention also proposes a multimodal fusion water conservancy monitoring system, comprising: The image acquisition module is used to acquire visible light images of the monitored water area during the day using a visible light camera, and to acquire infrared images of the monitored water area at night using an infrared thermal imager. The registration module is used to perform spatial registration of the visible light image and the infrared image based on the cross-correlation of water body contours, so that the two are aligned pixel by pixel. The feature extraction module is used to perform specular reflection removal and illumination normalization processing on the registered visible light image to extract visible light water quality features, and to perform denoising and thermal gradient tensor analysis on the registered infrared image to extract infrared water quality features. A cross-modal alignment module is used to input the visible light water quality features and the infrared water quality features into a shared cross-modal feature alignment network to obtain a unified water quality embedding vector; the cross-modal feature alignment network is trained with loss so that the same water quality state is mapped to the same embedding space in both the visible light mode and the infrared mode; The unified mapping module is used to map the unified water quality embedding vector into color values ​​using a day-night unified pseudo-color mapping rule, thereby generating a water quality assessment image, so that the water quality assessment images generated during the day and at night have the same color semantics.

[0103] The system provided by this invention transforms the method steps into functional modules for protection. In the image acquisition module, the visible light camera and infrared thermal imager are synchronously triggered via hardware to ensure consistent timestamps. The visible light camera can be an RGB camera or a hyperspectral camera, and the infrared thermal imager can be a long-wave infrared thermal imager, preferably operating in the 8 to 14 micrometer range. The registration module is used to perform spatial registration of the visible light image and the infrared image based on the cross-correlation of water body contours, aligning them pixel by pixel. The specific implementation of the registration module is the same as described in Embodiment 2, including sub-steps such as Otsu adaptive threshold segmentation, NDWI or color space analysis, contour cross-correlation function calculation, and spatial transformation. The feature extraction module is used to perform specular reflection removal and illumination normalization on the registered visible light image to extract visible light water quality features, and to perform denoising and thermal gradient tensor analysis on the registered infrared image to extract infrared water quality features. The specific implementation of the feature extraction module is the same as that described in Embodiments 3 and 4. The visible light features include the HIS color space hue distribution histogram, chromaticity difference factor, Gabor filter texture energy distribution, and Canny edge density and direction distribution. The infrared features include the first three principal components extracted by thermal gradient tensor principal component analysis, which correspond to thermal plume intensity, thermal plume direction, and thermal plume spatial scale, respectively. The cross-modal alignment module is used to input visible light water quality features and infrared water quality features into a shared cross-modal feature alignment network to obtain a unified water quality embedding vector. The cross-modal feature alignment network is trained using loss to map the same water quality state to the same embedding space under both visible light and infrared modes. The specific implementation of the cross-modal alignment module is the same as described in Examples 5 and 6, including a two-layer fully connected hidden layer structure, classification loss and modality consistency loss training, a modality discriminator, and adversarial training. The unified mapping module is used to map the unified water quality embedding vector to color values ​​using a day-night unified pseudo-color mapping rule to generate a water quality assessment image, ensuring that water quality assessment images generated during the day and night have the same color semantics. The specific implementation of the unified mapping module is the same as described in Example 7, including a learnable color lookup table and color difference minimization optimization. This system can be deployed on shore-based fixed stations, UAV airborne platforms, or unmanned surface vessel platforms. The registration module, feature extraction module, cross-modal alignment module, and unified mapping module can run on edge computing devices, such as the Jetson series platform, to achieve real-time image processing and water quality assessment image output.

[0104] This invention achieves all-weather, continuous, and unified water quality monitoring through the collaborative work of an image acquisition module, a registration module, a feature extraction module, a cross-modal alignment module, and a unified mapping module. The image acquisition module overcomes the inherent limitation of visible light images working effectively at night by employing a time-division acquisition strategy using a visible light camera during the day and an infrared thermal imager at night. The registration module provides a reliable spatial reference for subsequent unified processing of day and night images through spatial registration based on the cross-correlation of water body contours. The feature extraction module eliminates the feature distribution differences caused by the different imaging mechanisms of the two modalities through physically guided enhancement and feature extraction. The cross-modal alignment module, through a shared cross-modal feature alignment network and modal consistency loss training, maps visible light and infrared features from different physical sources to the same water quality embedding space, ensuring that the same water quality state produces the same or similar embedding vectors in both modalities. The unified mapping module, through a unified pseudo-color mapping rule for day and night, achieves color semantic consistency between daytime visible light assessment images and nighttime infrared assessment images, ensuring that the same water quality state presents the same color encoding in day and night images. This system provides an effective technical means for water environment protection and water resource management.

[0105] The above description is merely a preferred embodiment of the present invention and does not constitute any limitation on the present invention. Any simple modifications, alterations, or equivalent structural changes made to the above embodiments based on the technical essence of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A multimodal fusion method for water conservancy monitoring, characterized in that, Includes the following steps: S1: During the day, visible light images of the monitored water area are collected using a visible light camera, and at night, infrared images of the monitored water area are collected using an infrared thermal imager. S2: Perform spatial registration on the visible light image and the infrared image based on the cross-correlation of water body contours to align the two pixel by pixel; S3: Perform specular destigmatization and illumination normalization on the registered visible light image to extract visible light water quality features; perform denoising and thermal gradient tensor analysis on the registered infrared image to extract infrared water quality features. S4: Input the visible light water quality features and the infrared water quality features into a shared cross-modal feature alignment network to obtain a unified water quality embedding vector; the cross-modal feature alignment network is trained with loss to map the same water quality state to the same embedding space in both the visible light and infrared modes; S5: The unified water quality embedding vector is mapped to color values ​​using a day-night unified pseudo-color mapping rule to generate a water quality assessment image, so that the water quality assessment images generated during the day and at night have the same color semantics.

2. The multimodal fusion water conservancy monitoring method according to claim 1, characterized in that, In step S2, the spatial registration based on water body contour cross-correlation includes: Otsu adaptive thresholding is performed on the infrared image to extract the mask of the infrared water thermal radiation region; The visible light water mask is extracted by analyzing the normalized water index or color space of the visible light image. Calculate the cross-correlation function of the outlines of the infrared water thermal radiation region mask and the visible light water mask, and solve the translation and scaling parameters by maximizing the outline overlap. The infrared image is spatially transformed according to the translation parameters and the scaling parameters to align it pixel-by-pixel with the visible light image.

3. The multimodal fusion water conservancy monitoring method according to claim 1, characterized in that, In step S3, the extraction of visible light water quality features includes: Detect and repair highly saturated pixels in visible light images to remove specular reflections from water surfaces; A multi-scale illumination normalization algorithm based on Retinex theory is used to decompose a visible light image into a reflection component and an illumination component, while retaining the reflection component. Color features, texture features, and edge features are extracted from the reflection component, wherein the color features include a hue distribution histogram and chromaticity difference factor in the HIS color space, the texture features include water surface texture energy distribution based on Gabor filtering, and the edge features include water surface ripple edge density and direction distribution.

4. The multimodal fusion water conservancy monitoring method according to claim 1, characterized in that, In step S3, the extraction of infrared water quality features includes: Nonlocal mean denoising is performed on the registered infrared image; Calculate the local gray-level gradient tensor of the denoised infrared image; Principal component analysis was performed on the local gray-level gradient tensor, and the first three principal components were extracted as infrared water quality feature vectors. The first three principal components correspond to the thermal plume intensity, thermal plume direction, and thermal plume spatial scale, respectively.

5. The multimodal fusion water conservancy monitoring method according to claim 1, characterized in that, In step S4, the cross-modal feature alignment network includes an input layer, a first fully connected hidden layer, a second fully connected hidden layer, and an output layer. The first fully connected hidden layer has a dimension of 64, the second fully connected hidden layer has a dimension of 32, and the output layer outputs a D-dimensional unified water quality embedding vector. The loss training includes classification loss and modality consistency loss.

6. The multimodal fusion water conservancy monitoring method according to claim 5, characterized in that, The cross-modal feature alignment network includes a modality discriminator, which is used to determine whether the input uniform water quality embedding vector originates from the visible light mode or the infrared mode; through adversarial training, the modality discriminator is made unable to distinguish the source mode of the uniform water quality embedding vector.

7. The multimodal fusion water conservancy monitoring method according to claim 1, characterized in that, In step S5, the day-night uniform pseudo-color mapping rule is a learnable color lookup table, which is optimized by minimizing the color difference between the visible light evaluation image and the infrared evaluation image under the same water quality label. The learnable color lookup table maps a uniform water quality embedding vector to RGB color values, where different color values ​​correspond to clean, slightly eutrophic, moderately polluted, and heavily polluted water quality states, respectively.

8. The multimodal fusion water conservancy monitoring method according to claim 1, characterized in that, In step S4, the unified water quality embedding vector is calculated using the following formula: In the formula, for The unified water quality embedding vector after time-lapse fusion For visible light embedding vectors, For infrared embedding vectors, for The calculation of the fusion weight for the solar altitude angle at any given time; in, In the formula, For the slope parameter, for The solar altitude angle at that moment, For reference solar altitude angle.

9. The multimodal fusion water conservancy monitoring method according to claim 1, characterized in that, After generating the water quality assessment image in step S5, the following steps are also included: The water quality assessment images generated over multiple consecutive days are arranged in chronological order to form a time-series water quality assessment image sequence. The time-series water quality assessment image sequence is subjected to inter-frame difference or background modeling to identify the gradual trend and abrupt events of water quality status.

10. A multimodal fusion water conservancy monitoring system, characterized in that, include: The image acquisition module is used to acquire visible light images of the monitored water area during the day using a visible light camera, and to acquire infrared images of the monitored water area at night using an infrared thermal imager. The registration module is used to perform spatial registration of the visible light image and the infrared image based on the cross-correlation of water body contours, so that the two are aligned pixel by pixel. The feature extraction module is used to perform specular reflection removal and illumination normalization processing on the registered visible light image to extract visible light water quality features, and to perform denoising and thermal gradient tensor analysis on the registered infrared image to extract infrared water quality features. A cross-modal alignment module is used to input the visible light water quality features and the infrared water quality features into a shared cross-modal feature alignment network to obtain a unified water quality embedding vector; the cross-modal feature alignment network is trained with loss so that the same water quality state is mapped to the same embedding space in both the visible light mode and the infrared mode; The unified mapping module is used to map the unified water quality embedding vector into color values ​​using a day-night unified pseudo-color mapping rule, thereby generating a water quality assessment image, so that the water quality assessment images generated during the day and at night have the same color semantics.

Images