A low-altitude water monitoring system based on a learning model
By using an IMU and a multispectral camera to acquire attitude data in a low-altitude water monitoring system, performing affine and lighting corrections, and combining this with a learning model to recover missing pixels, the problems of image distortion and uneven illumination in low-altitude monitoring were solved, thus improving the accuracy and reliability of image monitoring.
Patent Information
- Application Number
- CN202511205622.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2025-08-27
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2045-08-27
AI Technical Summary
In low-altitude water monitoring systems, slight mechanical vibrations and displacements caused by atmospheric boundary layer turbulence during drone or light aircraft cruise can lead to affine distortion of images, disrupting spatial coordinate consistency and affecting the accuracy and reliability of image monitoring. At the same time, water surface reflections can cause uneven illumination, further reducing image quality.
Video streams and attitude data are acquired by drones equipped with IMU and multispectral cameras. Image correction is performed using affine correction units and shading correction units. Missing pixels are recovered by combining a learning model, including affine transformation matrix calculation, Fresnel reflection model improvement and adversarial generative model to generate high-quality images.
It effectively eliminates image distortion, improves the spatial coordinate consistency and illumination uniformity of images, enhances image quality, and ensures the accuracy and reliability of water area monitoring.
Smart Images

Figure CN121074722B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of unmanned aerial vehicle (UAV) water patrol technology, specifically relating to a low-altitude water monitoring system based on a learning model. Background Technology
[0002] Aquatic environment monitoring is a core task of ecological environmental protection, requiring real-time acquisition of high-precision water quality parameters (such as chlorophyll and suspended solids concentration). Currently, satellite and high-altitude remote sensing are insufficient in spatial resolution and are hampered by atmospheric interference (cloud scattering, aerosol attenuation), making it difficult to capture small-scale dynamic characteristics of aquatic pollution (such as algal blooms and oil spills). Ground-based monitoring methods (buoys, shipborne sensors) have limited coverage and slow response. Therefore, low-altitude cruise monitoring (10-500 meters altitude) is currently an irreplaceable solution. Using UAVs / light aircraft equipped with optical sensors, it can provide centimeter-resolution images, avoid atmospheric attenuation, and achieve large-scale, high-frequency observations.
[0003] However, low-altitude monitoring platforms inevitably encounter atmospheric boundary layer turbulence disturbances during cruise (induced by water surface thermal convection, wind shear, and platform motion), causing slight mechanical vibrations and displacements in UAVs / light aircraft. Airflow causes slight platform pitch or roll in cameras or other sensors, resulting in affine distortion of images, disrupting spatial coordinate consistency, and causing trajectory offsets that lead to failure in time-series image registration.
[0004] Furthermore, the reflection from the water surface of the monitored area can lead to uneven illumination. This uneven illumination distorts the color, texture, and other features of different areas of the image, further exacerbating the surface deformation caused by affine distortion. Subsequent processing of these defective images amplifies the detection error, leading to deviations in water surface identification based on image features. This makes it impossible to accurately determine the type of suspended matter on the water surface and water quality, ultimately reducing the reliability of image monitoring analysis. Summary of the Invention
[0005] To address the aforementioned problems in existing low-altitude water monitoring systems based on learning models, this invention provides a low-altitude water monitoring system based on a learning model. This system solves the problem that low-altitude currents cause slight platform pitch or roll of the camera, resulting in affine distortion of the image, which disrupts spatial coordinate consistency and affects the alignment of subsequent images. Furthermore, the unique surface reflection of the water in the water area further affects the accuracy of the acquired images.
[0006] The objective of this invention can be achieved through the following technical solution: a low-altitude water monitoring system based on a learning model, comprising: Data acquisition module: The UAV equipped with an IMU and a multispectral camera acquires video streams, ambient light intensity and incident angle direction of the monitored water area, as well as attitude data of the IMU. The video stream is then processed into multiple image blocks by frame segmentation. The attitude data includes pitch angle, roll angle and angular velocity. Image correction module: includes affine correction unit and luminance correction unit; The affine correction unit is used to calculate the affine transformation matrix using the OpenCV model based on the pose data, and to perform geometric correction on the image blocks by resampling based on the affine correction matrix. The light and shadow correction unit is used to improve the Fresnel reflection model according to the ambient light intensity and direction, calculate the reflection components of each image block after resampling, separate the reflection components from the image blocks, and output multiple corrected image blocks. Water area monitoring module: Performs gradient domain fusion and stitching on the optimized image blocks to generate a panoramic view of the water area, and uses the panoramic view of the water area to monitor the monitored water area.
[0007] Preferably, the correction module further includes an optimization unit, which is used to optimize the corrected image blocks. The optimization process of the optimization unit includes: S1: Define the pixels of multiple image blocks as nodes of a dynamic graph to obtain a graph structure, and determine the position of missing pixels based on the graph structure; S2: Encode the spatial and temporal information of the four adjacent nodes of the missing pixel to obtain the spatiotemporal coding matrix; S3: Input the spatiotemporal encoding into the first spatiotemporal attention model to perform feature encoding of pixels and obtain spatiotemporal encoded features; S4: Based on the spatiotemporal encoding, the spatiotemporal decoding features are obtained by decoding using the second spatiotemporal attention model; S5: Input the spatiotemporal decoded features into the generator of the adversarial generative model to generate missing pixels.
[0008] Preferably, the first spatiotemporal attention model is a two-branch structure combining spatial self-attention and cross-attention, specifically: Spatial attention weights between neighboring nodes are calculated using spatial self-attention. : Where Sim is the cosine similarity function. Let i and j be the spatial features of nodes i and j; Using the current frame as the query and the previous frame as the key, calculate the temporal cross-attention weights. : , where t is the current frame, k is the previous frame, and MLP is the similarity scoring function.
[0009] Preferably, the first spatiotemporal attention model further includes a gating mechanism, which takes spatial attention weights and temporal attention weights as inputs and calculates the fusion weights of spatial features and temporal features through a gating network. The gating network consists of two fully connected layers and a sigmoid activation function.
[0010] Preferably, the adversarial generative model uses U-Net as its network architecture, takes spatiotemporal decoding features as input, extracts global features through 4-layer downsampling, recovers details through 4-layer upsampling, and fuses spatial details in the encoding stage through skip connections; the output layer uses the Tanh activation function to generate pixel values.
[0011] Preferably, the loss function of the adversarial generative model is constructed using L1 loss and adversarial loss, wherein the calculation formula of the loss function is: ; in, Let L1 loss function be , where To counteract the generator's output in the generative model, i.e., the generated values of missing pixels, x represents the spatiotemporal decoding features output by the second spatiotemporal attention model, and y represents the "true pixel value that should exist" at the missing location. The L1 norm is the Manhattan distance. To provide an adversarial loss function, where, In an adversarial generative model, the output of the discriminator is D, where D is the feature dimension. The expected value is the average of the loss values for all samples input to the discriminator. It is the loss that minimizes the game objective between the generator and the discriminator.
[0012] Preferably, the affine correction unit is used to perform the following steps: Step 1: Obtain attitude data from the IMU during the current image frame exposure time. The attitude data includes pitch angle, roll angle, and angular velocity. Step 2: Calculate the static translation components using pitch and roll angles; Step 3: Calculate the pitch angle and yaw rate using angular velocity, and calculate the dynamic translation component based on the pitch angle and yaw rate; Step 4: Combine the static and dynamic translation components to construct the affine transformation matrix M using homogeneous coordinates; Step 5: Use the affine transformation matrix M to correct the coordinate transformation of each image block.
[0013] Preferably, the formula for calculating the affine matrix M is: ; in, and These are the angular velocity components about the Y-axis and Z-axis, respectively. The time interval between adjacent frames. and It is a proportionality constant, depending on the camera's focal length and altitude. It's the IMU roll angle. It's the IMU pitch angle.
[0014] Preferably, the construction of the improved Fresnel reflection formula includes: Step 1: Calculate the ambient incident vector: Extract the ambient light intensity and incident direction from the data acquisition module. The incident direction includes the horizontal angle. and vertical angle The ambient light incident vector is calculated based on the incident direction; Step 2: Construct an improved Fresnel reflection formula using ambient light intensity, ambient light incident vector, and Fresnel reflection coefficient. The calculation formula is as follows: ; in, For Fresnel reflection coefficient, Angle of incidence The angle of refraction is calculated using Snell's law. This represents the water surface reflectivity distribution.
[0015] Preferably, it also includes an early warning module, which compares the roll angle with a preset roll angle threshold and triggers a drone attitude alarm when the roll angle is not less than the preset roll angle threshold.
[0016] The beneficial effects of this invention are as follows: To address the issues of inconsistent spatial coordinates and failed temporal image registration caused by airflow disturbances, this solution uses IMU attitude data (pitch angle, roll angle, angular velocity). Based on this data, the affine correction unit uses an OpenCV model to calculate the affine transformation matrix of multiple image blocks, performing geometric correction on multiple image blocks. This effectively eliminates image distortion caused by slight changes in UAV attitude, ensuring alignment of subsequent temporal images. The lighting correction unit calculates an improved Fresnel reflection formula based on ambient light intensity and direction, calculates and separates the reflection components of image blocks, effectively solving the problem of uneven illumination and improving the color and texture feature reproduction of images in different regions. It locates missing pixels through dynamic graph structure, mines spatial and temporal context information using a spatiotemporal attention model, and combines an adversarial generative model to generate high-quality missing pixels, solving the problem of amplified subsequent processing errors caused by geometric and lighting correction in the previous steps, and further improving the image quality acquired by UAVs in the field of water monitoring. Attached Figure Description
[0017] To facilitate understanding by those skilled in the art, the present invention will be further described below with reference to the accompanying drawings.
[0018] Figure 1 The optimization steps of the optimization unit of the system of the present invention; Figure 2 This describes the correction steps for the affine correction unit of the system of the present invention. Detailed Implementation
[0019] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided.
[0020] Please see Figures 1-2 This embodiment provides a low-altitude water monitoring system based on a learning model, including: Data acquisition module: The UAV equipped with IMU acquires video streams, ambient light intensity and direction, and IMU attitude data of the monitored water area. The attitude data here includes pitch angle, roll angle and angular velocity. The data acquisition module performs frame-by-frame processing on the video stream to obtain multiple image blocks. Video streams of the monitored water surface are acquired using a drone equipped with an IMU and a camera, such as a spectral camera. The drone cruises at low altitudes over the monitored water surface, keeping its flight altitude below 100 meters to ensure video clarity and resolution. The drone is also equipped with a GPS positioning system to ensure the accuracy of the captured surface information, facilitating subsequent data processing and analysis.
[0021] The acquired video stream covers real-time visual information about the monitored water area, such as water surface ripples, the presence of floating objects, and changes in water color. Simultaneously, the IMU acquires the drone's own attitude data in real time. The pitch angle reflects the drone's tilt angle relative to the horizontal reference plane in the vertical direction, which is crucial for determining the vertical offset of the camera's shooting angle. The roll angle reflects the drone's rotation angle around its fuselage axis, helping to determine the camera's rotation state in the horizontal plane. Angular velocity represents the rate of change in the drone's attitude, i.e., the amount of change in pitch angle, roll angle, etc., per unit time. Precise time synchronization ensures a strict correspondence between the acquired video stream and the IMU attitude data.
[0022] Image correction module: includes affine correction unit and lighting correction unit. Affine correction unit is used to calculate affine transformation matrix based on pose data using OpenCV model, and resample image blocks based on affine correction matrix respectively. The process of performing affine correction on multiple image blocks after segmentation based on IMU pose data includes: using OpenCV's affine correction matrix function to perform affine correction transformation on the pixels of multiple images, then resampling the positions of the pixels in the image, and aligning the pixels using a pixel alignment method. This corrects the pixel offset in the acquired image caused by camera shake due to airflow during the acquisition process, and corrects the image based on the pixel alignment.
[0023] The Cv2.GetAffineTransform() function calculates the affine transformation matrix of the current frame to transform the current frame image and align it with the images of the preceding and following frames. Step 1: Obtain attitude data from the IMU during the current image frame exposure time. The attitude data includes pitch angle, roll angle, and angular velocity, where: Pitch angle (Radian measure): Rotation around the X-axis causes vertical distortion of the image; Roll angle (Radian measure): Rotation about the Y-axis causes horizontal distortion; angular velocity Used to compensate for motion blur, the correction steps include: Step 2: Calculate the static translation components using pitch and roll angles: The linear portion compensates for image distortion caused by roll and pitch angles, including cropping and scaling. Assume the camera coordinate system is: The x-axis represents the horizontal direction of the image, the y-axis represents the vertical direction of the image, and the z-axis is the optical axis. When the drone has a roll angle... and pitch angle At this time, the camera's rotation matrix is composed of basic rotation matrices. The standard rotation matrix is: Roll angle (rotation about the x-axis): ; Pitch angle (rotation about the y-axis): ; Since low-altitude water monitoring typically assumes an approximately flat water surface, and the UAV's altitude H is relatively large, an affine approximation (ignoring perspective effects) can be used. The combined rotation matrix R... ; Planar projection simplification: Projecting the 3D image onto a 2D image plane yields a simplified affine transformation matrix, suitable for OpenCV's affine model: ; First column This indicates scaling in the x-direction and shearing in the y-direction caused by roll. Second column This indicates the y-direction dependency and scaling caused by pitch. Step 3: Calculate the pitch angle and yaw rate using angular velocities, and calculate the dynamic translation component based on the pitch angle and yaw rate: Even if dynamic translation is used to partially compensate for angular velocity and Inter-frame time difference Image displacement caused by internal factors; In the camera model, changes in pitch angle primarily cause displacement along the x-axis, resulting in image point displacement. The formula for calculation is: , Changes in yaw angle primarily cause camera rotation, which is approximately equivalent to a translation in the y-direction (especially near the image center or at small angles). Displacement The calculation formula is: ; in, Specifically, this is obtained through camera calibration. ; s is the object distance, and H is the drone's cruising altitude, which is dynamically adjusted according to the specific drone's flight altitude. , .
[0024] Step 4: To facilitate the construction of the affine matrix using OpenCV's warpAffine function, the affine transformation matrix M is constructed using homogeneous coordinates, combining static attitude compensation and dynamic angular velocity compensation: ; Step 5: After the transformation is completed, the coordinate transformation of each image block is corrected using an affine transformation matrix. ; The affine correction unit uses an affine correction matrix to calculate the transformation of each image block, and OpenCV's resampling function is used to correct multiple image blocks, providing geometrically stable input for subsequent lighting correction and water monitoring.
[0025] The lighting correction unit is used to calculate the reflection components of each image block after resampling using an improved Fresnel reflection model. It separates and removes reflected light interference from the image blocks and outputs multiple corrected image blocks. The traditional Fresnel reflection model describes the reflection behavior of light at the interface between two different media. The calculation process ignores the directional differences of the actual ambient light, resulting in poor reflection calculation results when the actual light intensity is uneven. Therefore, ambient light intensity and direction compensation are introduced into the Fresnel model to improve the calculation effect of reflected light. The specific process is as follows: Step 1: Calculate the ambient incident vector: Extract the ambient light intensity (unit: lux) and incident direction (horizontal angle) from the data acquisition module. and vertical angle High-frequency fluctuations are smoothed through sliding window filtering, and the ambient light incident vector is calculated based on the horizontal and vertical angles. , The calculation formula is: Used to quantify the spatial distribution of light; Image block preprocessing: Gray-level normalization is performed on the affine-corrected image blocks (mapping pixel values to the 0-1 range), and the mean values of the RGB three channels are extracted as the basic brightness features; Canny edge detection is used to mark the boundary area between the water area and the shoreline, providing boundary basis for subsequent reflection model partitioning applications; Step 2: Construct the improved Fresnel reflection formula: The traditional Fresnel reflection model is improved by introducing ambient light intensity E and ambient incident light vector to obtain the reflection component. The calculation formula is: ; in, For Fresnel reflection coefficient, It is the angle of incidence (the angle between the ambient light and the normal to the water surface). The angle of refraction is calculated using Snell's law. The water surface reflectance distribution was determined by inverting the gray values of near-infrared images (near-infrared reflectance is positively correlated with water surface roughness), with values of 0.6-0.8 for rough regions and 0.3-0.5 for smooth regions. Step 3: Calculate the reflection components of each image block and subtract them from the original image: Apply the improved Fresnel reflection formula to each image block and calculate the reflection components pixel by pixel; Extracting intrinsic components: Subtracting the reflection component from the original image blocks yields the intrinsic components after removing reflections; Step 4: Output image blocks minus the reflection component.
[0026] Water area monitoring module: Performs gradient domain fusion and stitching on the optimized image patches using existing technology to generate a panoramic water area map, and uses the panoramic water area map to monitor the monitored water area.
[0027] The warning module compares the roll angle (pitch angle) with a preset roll angle threshold (pitch angle threshold). When the roll angle (pitch angle) is not less than the preset roll angle threshold (pitch angle threshold), a drone attitude alarm is triggered. The threshold setting is specifically determined based on whether the drone experiences slight shaking or relatively large shaking. This application's solution is used to address the problem of difficulty in stitching subsequent images caused by slight shaking of the drone due to low airflow.
[0028] This embodiment addresses the issues of inconsistent spatial coordinates and failed temporal image registration caused by airflow disturbances. The solution accurately acquires IMU attitude data (pitch angle, roll angle, angular velocity). Based on this data, the affine correction unit uses an OpenCV model to calculate the affine transformation matrix for multiple image blocks, performing geometric correction on these blocks to effectively eliminate image distortion caused by slight changes in UAV attitude and ensure alignment of subsequent temporal images. The lighting correction unit calculates an improved Fresnel reflection formula based on ambient light intensity and direction, calculating and separating the reflection components of image blocks, effectively solving the problem of uneven illumination. The affine correction unit uses OpenCV functions to perform linear transformations (rotation, scaling, or cropping) on the current frame to correct slight camera shake caused by airflow, resulting in geometric distortion in the captured image. During this process, pixels in edge regions may be cropped because their coordinate mapping exceeds the image boundary, or sub-pixel interpolation (such as bilinear or bicubic interpolation) may smooth high-frequency details (such as subtle textures and fine edges), leading to a decrease or even disappearance of the spatial position accuracy of the original features. The illustrative correction unit adjusts the brightness and contrast of the current frame image or eliminates uneven illumination (such as histogram equalization or gamma correction) through Fresnel reflection, which changes the original distribution of pixel grayscale values: brightening dark areas may amplify noise and smooth out subtle grayscale gradients, while darkening bright areas may compress feature differences in high-brightness areas. Especially for weak features that depend on the original illumination (such as shadow boundaries and textures formed by specular reflection), their grayscale relationships are reconstructed after correction, and the uniqueness of the features is weakened. Therefore, in one embodiment, to correct the feature loss problem in the image caused by affine correction and illustrative correction, the correction module further includes an optimization unit. The optimization unit is used to recover the missing feature values of multiple image blocks after affine correction and illustrative correction. The optimization process of the optimization unit on the corrected image includes: S1: Define the pixels of multiple image blocks as nodes of a dynamic graph to obtain a graph structure, and determine the position of missing pixels based on the graph structure; Press the corrected image as 16 The pixels are divided into 16 blocks with a 25% overlap rate (each block edge overlaps with adjacent blocks by 4 pixels) to ensure feature continuity between blocks; a binary mask is generated for each block to mark missing pixels (mask value 0 indicates missing, 1 indicates valid). Graph structure node definition: Each pixel (including valid and missing pixels) within a block is defined as a graph node. Node features include pixel coordinates. Grayscale values (or RGB values), timestamps (frame numbers for image sequences); edge pixels of adjacent blocks are used as cross-block nodes to establish adjacency relationships in the global graph structure; Adjacency matrix construction: The eight-neighbor rule is adopted, that is, adjacent pixels in the top, bottom, left, right and diagonal, to define node connections, calculate the spatial distance weight of adjacent nodes (the closer the distance, the higher the weight, weight = 1 / distance squared), and generate a sparse adjacency matrix to quantify the spatial dependencies between nodes. Missing pixel location: Based on the mask and adjacency matrix, nodes with a mask value of 0 are selected as missing pixels, and their coordinates are recorded. The valid nodes in its four-neighborhood and eight-neighborhood are marked as compensation reference nodes.
[0029] S2: Encode the spatial and temporal information of the four adjacent nodes of the missing pixel to obtain the spatiotemporal coding matrix: Spatial information encoding: for missing pixels Spatial features are extracted from the four effective neighboring nodes (up to four). These spatial features include relative coordinates, Euclidean distance, and orientation angle. The formula for calculating the relative coordinates is: The Euclidean distance is The formula for calculating the direction angle is: ; The coordinate distance is mapped to a 64-dimensional spatial feature vector through a spatial embedding layer, and the orientation angle is transformed into a periodic feature through sine and cosine encoding. Temporal information encoding: For consecutive frame images, extract the effective pixel values of the neighborhood of the frame containing the missing pixel (3 frames before and 1 frame after the frame), for a total of 4 frames, and calculate the inter-frame grayscale change rate. The formula for calculating the grayscale change rate is as follows: Furthermore, the frame number and rate of change are mapped into a 64-dimensional temporal feature vector through a temporal embedding layer to capture the evolution trend of pixels over time. The spatiotemporal coding matrix is concatenated by combining the spatial feature vector (64-dimensional) and temporal feature vector (64-dimensional) of each missing pixel in node order to form an N×128-dimensional matrix (N is the number of effective neighborhood nodes, N≤4); for missing pixels with fewer than 4 nodes, such as edge missing pixels, the matrix is filled with zero vectors to ensure that the input dimensions are consistent. S3: Input the spatiotemporal code into the first spatiotemporal attention model to perform feature encoding of the pixels and obtain the spatiotemporal code features. The first spatiotemporal attention model includes: The first spatiotemporal attention model here uses a two-branch structure that combines spatial self-attention with cross-attention, with the spatiotemporal encoding matrix as input; Specifically, spatial self-attention is used to calculate the spatial attention weights between neighboring nodes. The attention calculation formula for the weights is as follows: The spatial aggregation features of two nodes are obtained by weighted summation, where Sim is the cosine similarity. Let i be the spatial features of nodes i and j; where i is the spatial feature of nodes j. is used to represent the spatial information encoding vector of the i-th neighboring node, where For spatial feature dimensions, It includes the coordinate difference between the node and the missing pixel, as well as the spatial neighborhood texture gradient (such as horizontal / vertical gradient values) and spatial location confidence (reflecting whether the pixel is in a stable region such as the image edge). Temporal cross-attention: Using the current frame as the query and the previous frame as the key, calculate the temporal attention weights. The time aggregation features of the two nodes are obtained by weighting, where t is the current frame and k is the previous frame. The MLP outputs the similarity score. ; represents the time information encoding vector of the i-th neighboring node, where As a time feature dimension, Includes the time difference between the node and the pixel. The pixel value change rate in the time series (such as the pixel difference between the previous frame and the current frame) and the temporal stability weight (reflecting the degree of fluctuation of the node in the time series). The encoded output concatenates spatial and temporal aggregated features and outputs 128-dimensional spatiotemporal encoded features through a 2-layer fully connected network (256 hidden layer dimensions), while preserving the spatiotemporal dependencies of neighboring nodes.
[0030] S4: Based on the spatiotemporal encoding, the spatiotemporal decoding features are obtained by decoding through the second spatiotemporal attention model: The second spatiotemporal attention model adopts a decoding structure of "cross attention + positional encoding", the input is the spatiotemporal encoding features, and the goal is to restore the feature distribution of the missing pixels; In this method, cross-attention uses encoded features as keys and the location encoding of missing pixels as queries to calculate attention weights. The formula for calculating the attention weights is as follows: ; Where q is the missing location code, k is the coding feature, and m is the spatiotemporal coding matrix; the decoding features of the fusion context are obtained by weighting, and the absolute coordinate codes of the missing pixels (generated by a sine function) are injected into the decoding features to enhance the spatial positioning accuracy, Q is the temporal query matrix, and K is the key matrix; Decoding output: The 128-dimensional features are mapped to 256-dimensional spatiotemporal decoding features through a deconvolution layer, with the dimensions corresponding to the feature space of the missing pixels; S5: Input the spatiotemporal decoded features into the generator of the adversarial generative model to generate missing pixels. The Generative Adversarial Network (GAN) model here uses an improved U-Net architecture. It takes spatiotemporal decoded features as input, extracts global features through 4 layers of downsampling, recovers details through 4 layers of upsampling, and fuses spatial details in the encoding stage through skip connections. The output layer uses the Tanh activation function to generate pixel values (grayscale values 0-255 or RGB three-channel values). Discriminator and Adversarial Training: The discriminator uses a PatchGAN structure, taking a local block (32×32) of "real pixels + generated pixels" as input and outputting the realism probability of each patch. The loss function combines L1 loss (distance between generated and real pixels) with adversarial loss (minimizing the game loss between the generator and the discriminator). The expression for the loss function is as follows: ; in, Let L1 loss function be , where To counteract the generator's output in the generative model, i.e., the generated values (such as grayscale values or RGB three-channel values) of missing pixels, x represents the spatiotemporal decoded features of the input (from the output of the second spatiotemporal attention model), and y represents the "true pixel value that should exist" at the missing location. The L1 norm, also known as the Manhattan distance, is used to calculate the sum of the absolute differences between generated pixels and real pixels. , where is the weighting coefficient for the L1 loss (usually set to 100), used to balance the contributions of the two losses and ensure the numerical accuracy of the generated pixels. The L1 loss function forces the pixel values output by the generator to be as close as possible to the actual pixel values, thus constraining the accuracy of the generated pixels. For example, for gray pixels on a guardrail surface, L1 loss will suppress excessive brightness deviations in the generated pixels (such as generating pure white or pure black pixels), ensuring the grayscale continuity between the missing area and the surrounding valid pixels; To provide an adversarial loss function, where, In an adversarial generative model, the output of the discriminator represents the probability that an input pixel (or pixel block) is a "real pixel," and its value ranges from [value range missing]. (The closer to 1, the more likely it is a real pixel; the closer to 0, the more likely it is a generated pixel), D is the feature dimension; To achieve the mathematical expectation, the loss value is averaged over all samples (real pixel y and generated pixel G(x)) of the discriminator input, ensuring the stability of the loss across all samples. The goal of the game between the generator and the discriminator is to minimize the loss of the generator G (trying to make the discriminator unable to distinguish between generated pixels and real pixels), and maximize the loss of the discriminator D (trying to accurately distinguish between real pixels and generated pixels). Through the adversarial game between the generator and the discriminator, the generated pixels are forced to closely approximate the statistical characteristics of real pixels under the set standards, thereby improving the realism of the generated pixels. The optimization process of the loss function will affect the preceding spatiotemporal attention model through backpropagation. For example, if the generated pixels are distorted due to insufficient spatial features, the L1 loss will be fed back to the spatial attention mechanism to enhance the capture of the neighborhood spatial topology; if the lack of temporal features leads to dynamic discontinuity, the adversarial loss will drive the temporal attention mechanism to optimize the temporal weight allocation, and finally form a closed-loop optimization.
[0031] Post-processing optimization: Bilateral filtering (spatial sigma=1.0, grayscale sigma=20) is applied to the generated pixels to eliminate generation noise; cross-block fusion (averaging of overlapping regions) ensures the continuity of generated pixels between blocks, and finally outputs a complete compensated image; The optimization unit combines neighborhood spatial dependence and sequential temporal correlation to focus on key pixel neighborhood nodes through a multi-layer attention model, reducing interference from irrelevant information and achieving better compensation for missing pixels in complex areas such as edges and textures. First, by combining spatial topology and temporal dynamics, the compensated pixels are more in line with the physical laws of the scene. Second, the attention mechanism focuses on effective neighborhoods, reducing interference from irrelevant information. Third, adversarial generation improves the realism of texture details, ensuring that the compensated area is consistent with the original image at both the visual and feature levels, significantly improving the integrity and usability of the corrected image.
[0032] In one embodiment, the first spatiotemporal attention model uses a gating mechanism to adaptively adjust the fusion ratio of spatial and temporal features, dynamically allocating weights according to scene characteristics to improve the targeting of feature fusion. The specific execution process is as follows: Step A: Gating parameter calculation. Taking spatial and temporal features as input, the fusion weights are calculated through a gating network. The gating network consists of two fully connected layers (the hidden layer dimension is 1 / 2 of the spatial / temporal feature dimension) and a Sigmoid activation function. Step B: Perform dynamic feature fusion by weighting and fusing spatial and temporal features based on gating coefficients; Step C: Output the features. Use the fused features as the final output of the first spatiotemporal attention model and pass them to the second spatiotemporal attention model for decoding, providing accurate spatiotemporal fusion feature support for the generation of missing pixels.
[0033] Using gating mechanisms to further capture the commonalities of spatial and temporal features, the accuracy of the spatiotemporal coding features output by the first spatiotemporal attention model is further improved.
[0034] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.
Claims
1. A low-altitude water monitoring system based on a learning model, characterized in that: include: Data acquisition module: The UAV equipped with an IMU and a multispectral camera acquires video streams, ambient light intensity and incident angle direction of the monitored water area, as well as attitude data of the IMU. The video stream is then processed into multiple image blocks by frame segmentation. The attitude data includes pitch angle, roll angle and angular velocity. Image correction module: includes affine correction unit and luminance correction unit; The affine correction unit is used to calculate the affine transformation matrix using the OpenCV model based on the pose data, and to perform geometric correction on the image blocks by resampling based on the affine correction matrix. The light and shadow correction unit is used to improve the Fresnel reflection model according to the ambient light intensity and direction, calculate the reflection components of each image block after resampling, separate the reflection components from the image blocks, and output multiple corrected image blocks. Water area monitoring module: performs gradient domain fusion and stitching on the optimized image blocks to generate a panoramic view of the water area, and uses the panoramic view of the water area to monitor the monitored water area; The correction module further includes an optimization unit, which optimizes multiple image blocks after correction. The optimization process of the optimization unit includes: S1: Define the pixels of multiple image blocks as nodes of a dynamic graph to obtain a graph structure, and determine the position of missing pixels based on the graph structure; S2: Encode the spatial and temporal information of the four adjacent nodes of the missing pixel to obtain the spatiotemporal coding matrix; S3: Input the spatiotemporal encoding into the first spatiotemporal attention model to perform feature encoding of pixels and obtain spatiotemporal encoded features; S4: Based on the spatiotemporal encoding, the spatiotemporal decoding features are obtained by decoding using the second spatiotemporal attention model; S5: Input the spatiotemporal decoded features into the generator of the adversarial generative model to generate missing pixels.
2. The low-altitude water monitoring system based on a learning model according to claim 1, characterized in that: The first spatiotemporal attention model is a two-branch structure combining spatial self-attention and cross-attention, specifically: Spatial attention weights between neighboring nodes are calculated using spatial self-attention. : Where Sim is the cosine similarity function. Let i and j be the spatial features of nodes i and j; Using the current frame as the query and the previous frame as the key, calculate the temporal cross-attention weights. : , where t is the current frame, k is the previous frame, and MLP is the similarity scoring function.
3. The low-altitude water monitoring system based on a learning model according to claim 2, characterized in that: The first spatiotemporal attention model also includes a gating mechanism. The gating mechanism takes spatial attention weights and temporal attention weights as inputs and calculates the fusion weights of spatial features and temporal features through a gating network. The gating network consists of two fully connected layers and a sigmoid activation function.
4. A low-altitude water monitoring system based on a learning model according to claim 3, characterized in that: The adversarial generative model uses U-Net as its network architecture. It takes spatiotemporal decoding features as input, extracts global features through 4 layers of downsampling, recovers details through 4 layers of upsampling, and fuses spatial details in the encoding stage through skip connections. The output layer uses the Tanh activation function to generate pixel values.
5. A low-altitude water monitoring system based on a learning model according to claim 4, characterized in that: The loss function of the adversarial generative model is constructed using L1 loss and adversarial loss, where the formula for calculating the loss function is: ; in, Let L1 be the loss function, where, To counteract the generator's output in the generative model, i.e. the generated value of the missing pixel, x represents the spatiotemporal decoding feature output by the second spatiotemporal attention model, and y represents the "true pixel value that should exist" at the missing location. The L1 norm is the Manhattan distance. To provide an adversarial loss function, where, In an adversarial generative model, the output of the discriminator is D, where D is the feature dimension. The expected value is the average of the loss values for all samples input to the discriminator. It is the loss that minimizes the game objective between the generator and the discriminator.
6. The low-altitude water monitoring system based on a learning model according to claim 1, characterized in that: The affine correction unit is used to perform the following steps: Step 1: Obtain attitude data from the IMU during the current image frame exposure time. The attitude data includes pitch angle, roll angle, and angular velocity. Step 2: Calculate the static translation components using pitch and roll angles; Step 3: Calculate the pitch angle and yaw rate using angular velocity, and calculate the dynamic translation component based on the pitch angle and yaw rate; Step 4: Combine the static and dynamic translation components to construct the affine transformation matrix M using homogeneous coordinates; Step 5: Use the affine transformation matrix M to correct the coordinate transformation of each image block.
7. A low-altitude water monitoring system based on a learning model according to claim 6, characterized in that: The formula for calculating the affine transformation matrix M is as follows: ; in, and These are the angular velocity components about the Y-axis and Z-axis, respectively. The time interval between adjacent frames. and It is a proportionality constant, depending on the camera's focal length and altitude. It's the IMU roll angle. It's the IMU pitch angle.
8. A low-altitude water monitoring system based on a learning model according to claim 7, characterized in that: The construction of the improved Fresnel reflection formula includes: Step 1: Calculate the ambient incident vector: Extract the ambient light intensity and incident direction from the data acquisition module. The incident direction includes the horizontal angle. and vertical angle The ambient light incident vector is calculated based on the incident direction; Step 2: Construct an improved Fresnel reflection formula using ambient light intensity, ambient light incident vector, and Fresnel reflection coefficient. The calculation formula is as follows: ; in, For Fresnel reflection coefficient, Angle of incidence The angle of refraction is calculated using Snell's law. This represents the water surface reflectivity distribution.
9. A low-altitude water monitoring system based on a learning model according to claim 1, characterized in that: It also includes an early warning module, which compares the roll angle with a preset roll angle threshold and triggers a drone attitude alarm when the roll angle is not less than the preset roll angle threshold.
Citation Information
Patent Citations
Underwater image enhancement method for fishery monitoring
CN118674645A
Unmanned aerial vehicle image distortion correction and coordinate positioning compensation method
CN120495424A
