River inspection method, device and system based on multi-modal large model

Abstract

The invention discloses a river inspection method, device and system based on a multi-modal large model, and relates to the technical field of environment monitoring. The river inspection method comprises the following steps: acquiring a river aerial image, and dividing the river aerial image into a plurality of sub-blocks adaptive to the input size of a downstream model; and then inputting a semantic segmentation model to carry out pixel-level processing, and generating a water body mask enhancement graph which only retains water area content. Target detection, water quality turbidity evaluation and algae coverage evaluation are executed in parallel, and the category and position of an abnormal target, the turbidity grade and quantitative value and the algae coverage rate are obtained. And carrying out multi-modal data fusion on the river aerial image, the category and position of the abnormal target, the turbid grade and quantitative value, and the algae coverage rate. And inputting the fused data into a visual language large model, and performing reasoning by the visual language large model based on a preset cue word template to generate a natural language monitoring report containing an environment evaluation conclusion and a governance suggestion.

Description

Technical Field

[0001] This invention relates to the field of environmental monitoring technology, and more specifically, to a method, apparatus, and system for river inspection based on a multimodal large model. Background Technology

[0002] Traditional manual river patrols suffer from low efficiency, high cost, limited coverage, and strong subjectivity. Currently, some units have begun to use drones for aerial photography to assist in patrols, but most are still at the stage of manually reviewing videos or images, failing to achieve truly automated intelligent identification and analysis, and thus failing to meet the requirements of modern water environment supervision for real-time performance, accuracy, and comprehensiveness.

[0003] To address these challenges, several AI-based intelligent river inspection solutions have emerged in the current technology landscape. For example, some solutions fuse improved target detection models with large-scale visual-language models to detect targets and generate reports including risk levels. Other solutions attempt to directly utilize the zero-shot inference capabilities of large-scale visual-language models to analyze water area images, simplifying the system architecture. Still other solutions focus on improving the accuracy and real-time performance of floating object recognition by segmenting water areas before detection. Furthermore, there are also technologies that improve traditional target detection models to enhance the detection capability for small-scale floating objects.

[0004] However, these existing solutions still have significant limitations. First, most solutions are relatively simple in function, focusing only on target detection or report generation, lacking the comprehensive ability to quantitatively assess water quality parameters. Second, some solutions fail to accurately separate water and non-water areas before processing images, resulting in shoreline interference severely impacting identification and inversion accuracy. Furthermore, the systems lack sufficient automation, often failing to output professional and structured monitoring reports, requiring significant manual intervention for summarization and analysis. In addition, the functional modules of existing technologies are typically loosely coupled, failing to form a complete closed-loop system from data acquisition and intelligent analysis to decision support. Their visualization platform functionality is also relatively weak, making it difficult to support long-term trend analysis and scientific management decision-making. Summary of the Invention

[0005] This invention provides a method, apparatus, and system for river inspection based on a multimodal large model, in order to improve at least one of the aforementioned technical problems.

[0006] Firstly, the present invention provides a river inspection method based on a multimodal large model, which includes S1 to S7.

[0007] S1. Acquire aerial images of the river collected by the drone flying along the river route according to preset flight parameters; S2. Divide the aerial image of the river into multiple sub-blocks that are adapted to the input size of the downstream model; S3. Input the sub-map into the semantic segmentation model for pixel-level processing, extract the pure water area, and generate a water mask enhancement map that retains only the water content. S4. Based on the water mask enhancement map, perform target detection to obtain the category and location of abnormal targets; S5. Based on the water body mask enhancement map, perform water turbidity assessment to obtain turbidity level and quantitative value; S6. Based on the water body mask enhancement map, perform an algae coverage assessment to obtain the algae coverage rate; S7. Multimodal data fusion is performed on the river aerial images, the categories and locations of abnormal targets, the turbidity level and quantitative values, and the algae coverage rate; the fused data is then input into the visual language big model, which infers based on the preset prompt word templates to generate a natural language monitoring report containing environmental assessment conclusions and governance recommendations.

[0008] Secondly, the present invention provides a river inspection device based on a multimodal large model, which includes a data acquisition module, an image segmentation and preprocessing module, a water body region segmentation module, a target detection module, a turbidity assessment submodule, an algae coverage assessment submodule, and a report generation module.

[0009] The data acquisition module is used to acquire aerial images of the river collected by the UAV flying along the river route according to preset flight parameters; The image segmentation preprocessing module is used to divide the aerial image of the river into multiple sub-blocks that are adapted to the input size of the downstream model; The water body region segmentation module is used to input the sub-map tiles into the semantic segmentation model for pixel-level processing, extract pure water body regions, and generate a water body mask enhancement map that retains only the water body content. The target detection module is used to perform target detection based on the water mask enhancement map to obtain the category and location of abnormal targets; The turbidity assessment submodule is used to perform water turbidity assessment based on the water body mask enhancement map and obtain the turbidity level and quantitative value. The algae cover assessment submodule is used to perform algae cover assessment based on the water body mask enhancement map and obtain the algae cover rate; The report generation module is used to perform multimodal data fusion of aerial images of rivers, categories and locations of abnormal targets, turbidity levels and quantitative values, and algal coverage. The fused data is then input into a visual language big data model, which infers based on preset prompt word templates to generate a natural language monitoring report containing environmental assessment conclusions and governance recommendations.

[0010] Thirdly, the present invention provides a river inspection system device based on a multimodal large model, which includes a drone and a server connected by communication, for executing a river inspection method based on a multimodal large model as described in any paragraph of the first aspect.

[0011] By adopting the above technical solution, the present invention can achieve the following technical effects: The river inspection method based on a multimodal large model establishes a complete chain from data collection and intelligent analysis to decision support, avoiding information loss caused by fragmented modules. Through prior water area segmentation and dedicated model design, it significantly improves the accuracy of target identification and water quality assessment, demonstrating robust performance, especially in complex river scenarios. It supports multi-task joint perception of floating objects, water quality parameters, and human activities, providing more comprehensive scenario coverage. It innovatively applies a large visual language model to the water environment field, enabling professional-grade automated report generation and reducing manual intervention. Attached Figure Description

[0012] Figure 1 This is the overall architecture diagram of the river inspection method based on a multimodal large model.

[0013] Figure 2 This is an architecture diagram of the improved DeepLabV3+_ResNeSt-50 architecture.

[0014] Figure 3 This is a diagram of the improved EfficientNet-B7 architecture.

[0015] Figure 4 This is a flowchart of the river inspection method based on a multimodal large model. Figure 5 This is a flowchart of the river inspection method based on a multimodal large model. Detailed Implementation Example 1, please refer to Figures 1 to 2 The first embodiment of this invention provides a river inspection method based on a multimodal large model. This method uses drone aerial images to automatically identify, analyze, and generate reports on floating objects, water quality, and violations in rivers, and combines this with a Web platform for visualization.

[0016] S1. Acquire aerial images of the river collected by the drone flying along the river route according to preset flight parameters.

[0017] Preferably, the preset flight parameters are calculated based on a quantitative model established by the target recognition requirements and ground sampling distance constraints. The quantitative model includes a flight altitude formula, a maximum allowable sampling interval formula, and a maximum flight speed formula.

[0018] The data acquisition module is used to acquire high-resolution aerial images of the river area. All flight parameters are quantitatively modeled and optimized based on target recognition requirements, imaging geometry, and system dynamic response characteristics. This module includes a DJI Motrice 300 RTK UAV platform, an H20T multi-sensor gimbal camera, and a 5G transmission module. The H20T camera has 4K ultra-high-definition imaging capabilities and can capture visible light images at a resolution of 3840×2160 in wide-angle mode. It also records the GPS coordinates, timestamps, and attitude information of each frame of the image simultaneously, ensuring accurate spatial and temporal references for subsequent analysis.

[0019] To effectively identify various aquatic environmental anomalies, including domestic waste, construction waste, dead fish floating debris, algal patches, sewage outlets, and illegal activities such as angling / fishing / mining, this embodiment first defines the minimum identifiable physical size of each type of target. Then, combining the input feature extraction mechanism of a deep learning model, it is set that each target should occupy at least a 10×10 pixel spatial area in the image to ensure sufficient texture and contour information for classification and localization. Based on this, the required upper limit of the Ground Sample Distance (GSD) is derived as follows.

[0020] Floating objects (such as plastic bags and beverage bottles): minimum size approximately 30cm, GSD ≤ 1.5cm / px required.

[0021] Dead fish or small construction waste: minimum size approximately 50cm, GSD ≤ 5cm / px required.

[0022] Anglers or fishing gear: Human body dimensions ≥ 60cm, GSD ≤ 6cm / px.

[0023] Drainage outlet positioning: minimum opening size is approximately 80cm, and GSD is required to be ≤8cm / px.

[0024] Algal aggregation area identification: patch diameter ≥ 1m, GSD ≤ 10cm / px.

[0025] Based on the above requirements, the maximum allowable GSD value required by the system is 1.5cm / px to ensure that even the smallest target can be fully parsed.

[0026] It refers to the actual length of the ground corresponding to one pixel in the image, and its calculation formula is as follows.

[0027]

[0028] In the formula The altitude of the drone relative to the ground is expressed in meters (m). This refers to the physical width of the camera's imaging sensor, measured in millimeters (mm). This refers to the lens focal length, expressed in millimeters (mm). This represents the number of pixels in the horizontal direction of the image. The unit is centimeters per pixel (cm / px).

[0029] according to The expression for the flight altitude is obtained by solving the equation as follows.

[0030] .

[0031] In the formula This refers to the drone's flight altitude relative to the ground. This represents the actual length of the ground corresponding to one pixel in the image. This refers to the lens focal length. This represents the number of pixels in the horizontal direction of the image. This represents the physical width of the camera's imaging sensor.

[0032] Technical parameters based on the DJI H20T camera: Sensor width Equivalent focal length in wide-angle mode Image horizontal pixel count Substituting into the above formula and setting GSD≤1.5cm / px, we can solve for the flight altitude H, which should satisfy the condition. .

[0033] To further improve imaging quality and target recognition accuracy, especially in suppressing water surface reflection and shadow interference under complex lighting conditions, this embodiment preferably controls the flight altitude within the range of 20m to 22m. Within this range, GSD can be stably maintained between 1.4 and 1.5 cm / px, ensuring that a 30cm-sized floating object occupies approximately 20 to 21 pixels in the image, significantly better than the minimum recognition threshold of the AI ​​model. Preferably, the flight altitude is set to 21m in general inspection tasks.

[0034] To further prevent target misses due to image sparsity, this embodiment sets the image acquisition interval to one image automatically taken every 5 meters. This interval is determined based on constraints of the image ground coverage width and the minimum forward overlap rate. When the flight altitude is 21m, the calculated GSD ≈ 1.47cm / px, and the ground coverage width of a single image is:

[0035] If the minimum heading overlap rate is set to 40%, the maximum allowable data collection interval is as follows.

[0036]

[0037] Right now: . This represents the maximum allowed data collection interval. This represents the minimum heading overlap.

[0038] This embodiment uses an interval of 5 meters, which is much smaller than the limit value, to achieve highly redundant spatiotemporal sampling, effectively improving the probability of capturing sudden pollution events and supporting subsequent inter-frame correlation analysis and trajectory tracking.

[0039] Finally, flight speed Requires an interval with image acquisition Matching the system response cycle. Let the time interval between two photos be... It should be greater than or equal to the total delay of camera exposure, writing to the cache, and GPS refresh.

[0040] .

[0041] In the formula This is the maximum flight speed. This provides a safety margin for the system response cycle.

[0042] Actual measurements show that the H20T camera's average response time in automatic mode is 1.0–1.5 seconds. To ensure stable system operation and reduce the impact of flight jitter on image quality, a more conservative safety margin was adopted. =3.0s. Substitute =5m yields: .

[0043] This speed ensures clear, motion-free images while significantly reducing mechanical vibrations and attitude fluctuations during flight. It helps improve the stability of low-altitude aerial images and the reliability of AI recognition, representing the best balance between data quality and operational safety.

[0044] In summary, this embodiment scientifically determined the core flight parameters for UAV inspection by establishing a full-link mathematical model encompassing "target scale → GSD constraints → flight altitude → image coverage → acquisition density → flight speed": a flight altitude of 20–22 meters (preferably 21 meters), an image acquisition interval of 5 meters, and a flight speed of 6 km / h. All acquired images are transmitted back to the cloud server in real time via a 5G network, entering the next stage of the AI ​​intelligent analysis process. This achieves fully quantitative, automated, and highly reliable operation from data perception to decision output.

[0045] Specifically, by establishing a mathematical model of "target size - image resolution - flight parameters," the optimal flight parameters are precisely calculated. First, the minimum identifiable size of various detection targets is defined (e.g., a plastic bag of 30cm). Based on the requirements of the deep learning model, it is determined that the target must occupy at least 10×10 pixels in the image. Using the ground sampling distance formula GSD = (H × sensor width) / (f × image width), the upper limit of the flight altitude is calculated. Considering the GSD requirements of all targets, the flight altitude is optimized to 20-22 meters (preferably 21 meters), at which point the GSD stabilizes at 1.4-1.5 cm / pixel. Simultaneously, the acquisition interval is calculated based on the image coverage width and forward overlap rate. A 5-meter interval is used to achieve redundant sampling, and a flight speed of 6 km / h is matched to ensure image stability.

[0046] S2. Divide the aerial image of the river into multiple sub-maps adapted to the input size of the downstream model. Preferably, step S2 includes steps S21 to S23.

[0047] S21. Set the base size of the sliding window as follows: Set the sliding step size to ,in and This ensures that there are overlapping areas between adjacent sub-plots, thus preventing cross-plot targets from being missed. The height of the sliding window. The width of the sliding window. This is the sliding step size.

[0048] S22. When the sliding window exceeds the boundary of the original image, use reflection filling or zero filling to fill in the missing pixels.

[0049] S23. Record the global coordinate offset of the top left corner of each sub-plot in the original image, which is used to map the local detection results output by subsequent models back to the original image coordinate system.

[0050] The image segmentation preprocessing module divides the 4K resolution image (3840×2160) captured by the UAV into multiple sub-tiles adapted to the input size of the downstream AI model. Considering the different input size requirements of various analysis models, the target detection module YOLOv8-Worldv2 has the best detection performance at 640×640 resolution, which is particularly beneficial for the identification of small-scale targets. The water body segmentation module (DeepLabV3-ResNet50), the algae cover assessment sub-module (U-Net_ResNet34), and the turbidity assessment sub-module (EfficientNet-B7) support similar input sizes such as 512×512 or 600×600, respectively. To achieve a unified and efficient processing flow and avoid system complexity and coordinate mapping inconsistencies caused by multi-path segmentation, this embodiment adopts an image segmentation strategy based on a sliding window, specifically including the following steps.

[0051] Based on the above embodiments, in an optional embodiment of this embodiment, step S2 is as follows.

[0052] Determine the basic block size: Select 640×640 pixels as the basic size of the sliding window. This size can fully utilize YOLOv8-Worldv2's ability to detect small targets, and can also flexibly adapt to the input requirements of other models through bilinear interpolation scaling or center clipping.

[0053] Setting the sliding step size and overlap mechanism: The sliding step size is set to 512 pixels, so that there is an overlap area of ​​128 pixels in both the horizontal and vertical directions of adjacent sub-blocks. This ensures that target objects distributed across blocks (such as fishing nets, ribbon-like floating garbage, and other slender targets) are completely contained in at least one sub-block, effectively preventing missed detections caused by cutting and breaking, and improving the detection stability of boundary areas.

[0054] Edge filling and full coverage processing: When the sliding window exceeds the boundary of the original image, missing pixels are filled using reflection filling or zero filling to ensure that all generated sub-patterns are of standard 640×640 size. By traversing the entire 3840×2160 image, a total of [(3840-640) / 512]+1=8 columns × [(2160-640) / 512]+1=4 rows, totaling 32 sub-patterns, are generated, achieving seamless full coverage of the original image.

[0055] Record coordinate offsets: Record the global coordinate offset of the top-left corner of each sub-tile in the original image. , This offset information is used to accurately map the local detection results (such as bounding boxes and segmentation masks) output by subsequent AI modules back to the original image coordinate system, ensuring the spatial consistency and traceability of the detection location.

[0056] Organize batches for parallel inference: Organize all sub-plots into batches and send them to the subsequent multimodal AI analysis module for parallel inference, thereby improving overall processing efficiency.

[0057] Through the above-mentioned block-based method, this embodiment achieves comprehensive and complete processing of high-resolution aerial images without reducing the original image resolution. It not only meets the compatibility requirements of multiple models for input formats, but also ensures the effective capture of small and complex-shaped targets. This provides reliable data support for the subsequent accurate identification of environmental anomalies such as domestic waste, construction waste, dead fish floating debris, algae accumulation areas, sewage outlets, and illegal activities.

[0058] Specifically, an improved DeepLabV3 model (backbone network ResNeSt-50) is used to segment aerial images into pixel-level water regions, generating a "water mask enhancement map" that retains only pure water areas. Subsequent target detection (YOLOv8-Worldv2) and water quality assessment (turbidity, algae cover) are based on this mask map, effectively shielding non-water body interference such as shoreline buildings and vegetation.

[0059] S3. Input the sub-map into the semantic segmentation model for pixel-level processing, extract the pure water area, and generate a water mask enhancement map that retains only the water content.

[0060] The water body region segmentation module, acting as a "spatial attention guide" for the intelligent river inspection system, focuses on pixel-level semantic segmentation of high-resolution aerial images captured by drones. This precise extraction of pure water body regions (ROIs) provides clean spatial prior information for subsequent tasks such as floating object identification, water turbidity assessment, algae cover analysis, and violation detection. The module employs a deep learning-based semantic segmentation model and has undergone systematic improvements for the specific low-altitude aerial water monitoring scenario described in this embodiment, significantly enhancing segmentation accuracy and robustness under complex lighting conditions, shadow interference, and blurred boundaries.

[0061] Preferably, the semantic segmentation model adopts an improved DeepLabV3+_ResNeSt-50 architecture, with its backbone network using ResNeSt-50 and its basic architecture being a DeepLabV3 network. A CBAM attention mechanism is embedded at the output of the DeepLabV3 ASPP module. The ResNeSt-50 is located at the front end of the model for feature extraction. The ASPP module is located after the backbone network to receive the features extracted by the backbone network.

[0062] The core algorithm model used in this module is based on the DeepLabV3 network, with a ResNet-50 backbone. The standard DeepLabV3 model, through an encoder-decoder structure combining atrous convolution and spatial pyramid pooling (ASPP) modules, can expand the receptive field without reducing feature map resolution, effectively capturing multi-scale contextual information. Simultaneously, skip connections introduce shallow detail features into the decoder, achieving high-precision edge recovery and upsampling output, suitable for water extraction tasks in remote sensing imagery. The ASPP (Atrous Spatial Pyramid Pooling) module is a core component of the DeepLabV3 part.

[0063] To meet the requirements of this embodiment for high-precision and strong generalization capabilities in water segmentation, the following key improvements have been made to the standard DeepLabV3_ResNet-50 model.

[0064] First, at the data level, a dedicated training dataset for real-world inspection tasks was constructed. This dataset consists of self-collected drone aerial imagery, covering various typical landforms such as plains rivers, mountain streams, urban polluted water bodies, reservoirs, and tidal wetlands, totaling no fewer than 2,000 labeled images. All images were meticulously manually labeled, clearly distinguishing between "water bodies" and "non-water bodies," and adhering to the "water surface inclusiveness principle." This means that any targets located above or within the water surface (such as floating garbage, dead fish, and discolored water areas near sewage outlets) were classified as water bodies, ensuring that no key pollution information was missed in subsequent analysis. Based on this, data augmentation techniques such as illumination simulation and ripple perturbation were implemented to improve the model's adaptability to complex imaging conditions.

[0065] Secondly, in terms of network structure, two key improvements were made to the original model.

[0066] First, the backbone network was upgraded from the original ResNet-50 to ResNeSt-50. ResNeSt introduces the Split-Attention mechanism to achieve finer-grained feature selection in the channel dimension, which enhances the model's ability to represent water bodies with different turbidity and color variations, and improves classification consistency under uneven lighting conditions.

[0067] Secondly, the CBAM (Convolutional Block Attention Module) attention mechanism is embedded at the output of the ASPP module. CBAM consists of two sub-modules: channel attention and spatial attention. Channel attention generates channel weights by performing global average pooling along the spatial dimension of the feature map and then passing it through a fully connected layer, highlighting the feature channels that contribute more to water body identification. Spatial attention takes the maximum and mean values ​​along the channel dimension, concatenates them, and then generates a spatial weight matrix through a 7x7 convolution, allowing the model to focus on regions in the image that are more likely to belong to water bodies.

[0068] Let the ASPP module output feature map be , For channel dimensions. Represents a real number.

[0069] The output after CBAM processing is: .

[0070] In the formula Here is the channel attention weight vector. Here is the spatial attention weight matrix. This indicates element-wise multiplication. This mechanism significantly improves the model's ability to locate water bodies under interference such as shadow occlusion and cluttered backgrounds.

[0071] The input to this module comes from the output of the previous "Image Tile Preprocessing Module," specifically a sequence of 32 georeferenced tile sequences generated from the segmentation of the original aerial image. Each tile is 640×640 pixels in size. This sequence serves as the input batch for this module, and is sequentially fed into the improved DeepLabV3 model for independent segmentation inference.

[0072] The model outputs a water-mask enhanced image with the same resolution as the input sub-plots. It is generated as follows: for each predicted water segmentation mask, it is used as a binary template and fused bitwise with the original input sub-plots. Only the pixel values ​​classified as "water" are retained, while all pixels in non-water areas are forcibly set to zero (i.e., filled with a black background). The resulting output image visually appears as "only water content is retained, with the rest of the area completely black."

[0073] This design offers dual technical advantages: Firstly, by removing redundant information from land areas, it significantly reduces the computational load on subsequent water quality analysis models (such as floating object identification and algae coverage calculation). Secondly, it effectively avoids misidentification caused by non-water body elements such as shoreline reflected light, building outlines, and vegetation textures, thereby improving the overall system's accuracy and reliability.

[0074] Furthermore, to optimize the convergence stability and boundary segmentation quality during model training, this embodiment designs a joint loss function. Because the proportion of water pixels in aerial images is usually much smaller than that of non-water pixels, there is a serious class imbalance problem, and the correct classification of boundary pixels directly affects the accuracy of subsequent analysis. Therefore, using cross-entropy loss alone is insufficient to meet the requirements.

[0075] Therefore, the total loss function is defined as the weighted sum of the weighted cross-entropy loss and the Dice Loss: .

[0076]

[0077] In the formula The weighted cross-entropy loss is used to assign higher weights to water body categories in order to alleviate sample imbalance. for The weight. (i.e., Dice Loss) is a loss function based on the Dice coefficient. for The weight. This is a mask for the real label. This is the predicted result. This is a smoothing term used to prevent the denominator from being zero. Preferably, .

[0078] Directly measuring the degree of overlap between the predicted and actual regions is particularly beneficial for improving boundary fit. Experiments have verified that when... , At that time, the model achieved optimal segmentation performance on the test set.

[0079] In summary, this embodiment proposes a high-precision water body segmentation method for UAV inspection scenarios through end-to-end innovation in dataset construction, network structure design, and loss function optimization. This module takes the geographically aligned sub-map sequence generated by the previous stage as input, processes it through the improved DeepLabV3+ model, and outputs a water body mask enhancement map. This achieves accurate identification of land-water boundaries in complex environments, exhibits good real-time performance and stability, and provides an efficient and reliable data foundation for subsequent multi-task intelligent water quality analysis.

[0080] S4. Based on the water mask enhancement map, perform target detection to obtain the category and location of abnormal targets.

[0081] The object detection adopts an open vocabulary object detection model based on the YOLOv8-Worldv2 architecture, and employs a context-enhanced text prompting mechanism and a lightweight end-to-end fine-tuning strategy.

[0082] The target detection module is used to accurately identify and locate abnormal targets in the aquatic environment in UAV aerial images. The water mask enhancement map output from the preceding "water region segmentation module" is used as input and fed into an open-vocabulary target detection model based on an improved YOLOv8-Worldv2 architecture for inference. This model has zero-shot recognition capability and supports dynamically defining detection categories through natural language text prompts. New categories can be added without retraining (such as adding "oil slick" or "illegal sand dredger"), significantly improving the system's flexibility and maintainability in complex inspection scenarios.

[0083] In this embodiment, the preset detection keywords include, but are not limited to: "household waste" (such as plastic bags, foam boards), "construction waste" (such as plastic pipes, wood), "dead fish", "sewage outlet", "fisherman", "fishing net", "fishing boat", "sand mining machinery", etc.

[0084] Users can customize or add / delete the list of prompt words through the web platform, and the system will take effect immediately. The model outputs the bounding box coordinates [x, y, w, h], semantic category label, and confidence score for each detected target. For targets identified as suspected illegal activities such as "fishing," "fishing," or "mining," the system automatically associates the timestamp of the frame image with GPS coordinates to generate an event record with time and spatial markers.

[0085] To further improve the model's detection accuracy and robustness for small-scale, long-distance, and partially occluded targets in low-altitude aerial river scenes, this embodiment performs domain adaptation fine-tuning on the original YOLOv8-Worldv2 model, specifically including the following aspects.

[0086] In terms of dataset construction, this embodiment constructs a dedicated labeled dataset for smart river and lake inspection tasks to support few-shot learning of the model. This dataset covers various typical landforms, including plain rivers, urban polluted water bodies, mountain streams, and reservoirs, and contains no fewer than 1500 high-resolution images. All images have been finely annotated by professional annotators using Roboflow, covering various floating objects, areas with abnormal water quality, and human violations, ensuring that the bounding boxes accurately fit the target contours. Specifically, the annotation process follows the "water surface presence priority" principle, meaning that all objects located above or suspended in the water are considered valid detection targets to avoid missed detections due to shoreline interference. Furthermore, to enhance the model's generalization ability, the dataset includes at least 20% negative samples—clean water surface images without any pollution or violations—to suppress the false alarm rate. The dataset also employs data augmentation techniques such as Mosaic, random lighting perturbation, and ripple simulation to improve the model's adaptability to complex lighting, reflections, shadows, and cluttered background conditions.

[0087] Text prompt design and integration. This embodiment innovatively adopts a context-enhanced text prompt mechanism, making the text description closer to the actual inspection context and narrowing the semantic gap between the visual and linguistic modalities. Unlike the traditional approach that only uses isolated category names (such as "plastic bag"), the prompt template constructed in this embodiment explicitly introduces spatial location relationships and existence status information, as shown below.

[0088] A photo of a plastic bag floating on the water surface.

[0089] "A person engaged in illegal fishing activity by the riverside with a rod."

[0090] Construction debris drifting in slow-moving river water.

[0091] "Sewage outfall discharging colored wastewater into the stream."

[0092] These context-rich cue words, after being encoded by the CLIP text encoder, generate more discriminative text embedding vectors, significantly improving the model's semantic understanding of "water-related targets".

[0093] During the fine-tuning phase, this set of enhanced prompts serves as part of the supervision signal, guiding the model to learn the text-image alignment relationship in specific contexts, thereby improving the accuracy of understanding key actions and states such as "floating," "discharge," and "shoreline operations."

[0094] Fine-tuning methods and model structure optimization. This embodiment adopts a lightweight end-to-end fine-tuning strategy, focusing on optimizing multimodal alignment capabilities while maintaining the efficiency of the YOLOv8-Worldv2 backbone network. The specific fine-tuning scheme is as follows.

[0095] First, most parameters of the image backbone network (CSPDarknet53) are fixed (i.e., parameters other than the neck network and the detection head), and only the parameters of the neck network and the detection head are updated to reduce computational overhead, prevent overfitting, and accelerate convergence.

[0096] Secondly, the CLIP text encoder is unfrozen, allowing it to participate in training with a smaller learning rate (e.g., 1e-6). This enables the text representations of technical terms such as "sewage outlet," "algal bloom," and "illegal fishing" to be adjusted based on local visual data, achieving precise alignment between the text feature space and the visual feature space in the water monitoring scenario.

[0097] Furthermore, a multi-granularity semantic mapping layer is introduced into the classification part of the YOLOv8-Worldv2 model's detection head. This layer consists of a learnable fully connected network used to map general text embeddings to a semantic subspace suitable for water quality monitoring. This mapping layer is jointly optimized during fine-tuning, further improving the model's ability to recognize ambiguous semantic targets (such as "turbid area" and "garbage accumulation zone").

[0098] Finally, a water spatial attention mechanism is integrated into the classification part of the YOLOv8-Worldv2 model's detection head: the high-resolution water mask enhancement map from the "water region segmentation module" is downsampled to the current feature map size (e.g., 80×80), and a spatial attention weight matrix is ​​generated by Sigmoid activation. The algorithm then multiplies element-wise (⊙) the visual feature map of the corresponding level to suppress responses in non-aquatic areas and enhance attention to targets within water bodies. This prior guidance mechanism effectively reduces false detections in land areas and improves overall detection purity.

[0099] In the post-processing stage, the NMS (Non-Maximum Suppression) threshold is set to 0.45 to avoid redundant selection. Results are only reported when the target confidence level is ≥0.5 to ensure output quality.

[0100] In summary, this embodiment achieves high-precision and high-recall detection of multi-category, small, and dynamic targets in river environments through structural optimizations of the YOLOv8-Worldv2 model, including the construction of domain-specific datasets, the design of context-enhanced text prompts, and fine-tuning of the text encoder, multi-granularity semantic mapping, and spatial attention fusion. This module not only inherits the flexibility advantages of open-vocabulary detection but also significantly improves its practicality and reliability in real-world UAV inspection tasks through targeted improvements, providing an accurate data foundation for subsequent water quality assessment and report generation.

[0101] The water quality assessment module is used to quantitatively analyze and determine the turbidity and algae cover in UAV aerial images. This module consists of two sub-modules: a turbidity assessment sub-module and an algae cover assessment sub-module. Both modules use the water mask enhancement map output from the preceding "water body region segmentation module" as input, meaning they retain and enhance only the actual water areas, removing non-water interference such as land, buildings, and vegetation. Based on this, deep learning models are used to intelligently identify and quantify water quality parameters, providing crucial data support for the subsequent comprehensive report generation.

[0102] S5. Based on the water body mask enhancement map, perform water turbidity assessment to obtain turbidity level and quantitative value.

[0103] The turbidity assessment submodule described in this embodiment is used to classify the visual turbidity level of the pure water area image after pre-processing, and supplement it with continuous NTU value regression estimation to output four levels of qualitative results (clear, slightly turbid, moderately turbid, and heavily turbid) and corresponding confidence levels, so as to realize non-contact rapid assessment of water transparency.

[0104] This module uses EfficientNet-B7 as the backbone network architecture. Based on the original ImageNet pre-trained weights, the structure was optimized and the task was reconstructed to meet the actual needs of turbidity discrimination in aerial water images. The improved model no longer performs general object classification tasks, but is specifically designed to extract color, texture, and optical scattering features related to suspended particulate matter concentration from high-resolution water images.

[0105] Compared to the original EfficientNet-B7 model, its typical structure includes a Stem convolutional layer, an encoder consisting of seven MBConv modules, a global average pooling layer, and a final fully connected classification head. It is suitable for RGB image inputs with resolutions from 224×224 to 600×600, and outputs 1000 ImageNet class scores. However, this structure does not consider the spatial consistency and illumination sensitivity of water images in remote sensing scenarios, resulting in insufficient generalization ability when directly applied to water quality assessment.

[0106] This embodiment makes targeted adjustments to the original model, resulting in an improved EfficientNet-B7 model, the structure of which is shown in the figure below. Figure 3 As shown. First, the input size is uniformly set to 600×600 pixels to optimize inference performance for B7 scale. Second, a Convolutional Block Attention (CBAM) module is introduced at the end of the backbone network, integrated after the last MBConv block and before global pooling. This module enhances the response to attenuation features in the blue-green bands through channel attention mechanism, while using spatial attention mechanism to focus on the central region of the water body, suppressing misjudgments caused by ripples or local reflections, and significantly improving the model's stability under complex lighting conditions.

[0107] Furthermore, the original 1000-dimensional output layer is removed and replaced with a dual-task prediction head structure: on one hand, a 4-class classification head is set as a classification branch, which outputs the probability distribution of the four turbidity levels through the Softmax activation function. On the other hand, a regression head is added as a regression branch, which outputs the result of a single neuron activated by the Sigmoid function and linearly maps it to the [0,200] NTU range to provide a continuous turbidity index reference. Here, NTU is an abbreviation for "Nephelometric Turbidity Unit".

[0108] The two tasks share core features, and a weighted loss function is used for optimization during joint training.

[0109]

[0110] In the formula The training loss function for the turbidity assessment submodule, This represents the mean squared error loss, used in regression tasks. This represents the cross-entropy loss, used in classification tasks. for Hyperparameters. for Hyperparameters.

[0111] The hyperparameters were determined experimentally to be: , In order to balance the influence of the two types of monitoring signals.

[0112] During training, a transfer learning strategy was employed. ImageNet pre-trained weights were used for initialization, and approximately 70% of the shallow network parameters (including the MBConv modules in the first five stages) were frozen. Only the deep feature extraction part, attention module, and prediction head were fine-tuned to avoid overfitting under small sample conditions. The optimizer used was AdamW, with an initial learning rate set to [value missing]. The batch size is 16, the training cycle is about 100 rounds, and an early stopping mechanism is used to monitor the loss changes on the validation set to ensure that the model converges to the optimal state.

[0113] Ultimately, the system determines the turbidity level based on the highest probability output by the classification head. The specific classification criteria are as follows: when the predicted NTU < 5, it is judged as "clear"; 5 ≤ NTU < 20, it is "slightly turbid"; 20 ≤ NTU < 50, it is "moderately turbid"; and NTU ≥ 50, it is "severely turbid". This classification result, combined with the regression value, forms a highly interpretable and accurate turbidity assessment conclusion.

[0114] In summary, this embodiment introduces an attention mechanism and multi-task output design into the high-level structure of EfficientNet-B7, and combines it with a hierarchical fine-tuning strategy to construct a specialized deep learning model for water turbidity identification, which has good environmental adaptability while ensuring high accuracy.

[0115] S6. Based on the water body mask enhancement map, perform algae coverage assessment and obtain algae coverage rate.

[0116] The algae coverage assessment submodule in this embodiment is used to perform pixel-level semantic segmentation on the water mask enhancement map output by the previous stage, accurately identify the algae aggregation areas, generate an algae distribution heat map, and further calculate the "algae coverage rate" to provide a quantitative basis for judging the degree of eutrophication of the water body.

[0117] This module is built on the U-Net_ResNet34 architecture, fully leveraging its encoder-decoder structure's advantages in dense prediction tasks. The original U-Net uses a hand-designed stack of convolutions as the encoder, lacking the ability to represent prior information about natural images and making it difficult to effectively capture the faint and scattered algal patch features in aerial images.

[0118] This embodiment uses ResNet34 as the encoder backbone, leveraging its rich hierarchical feature representation capabilities obtained from pre-training on ImageNet to significantly improve the model's ability to perceive typical patterns such as abnormal algal colors and blurred edges.

[0119] The overall network structure is as follows Figure 4As shown. The input is a water mask enhancement map provided by the previous stage, scaled down to a size of 512×512 to eliminate interference from all non-water regions, ensuring the model focuses on processing targets within the water. The encoder consists of five stages of ResNet34, progressively downsampling to extract multi-scale semantic features. The bottleneck layer receives features from the deepest layer and then enters a symmetrical decoding path. During each upsampling stage, the encoder features from the corresponding level are fused through skip connections to recover spatial detail information.

[0120] To improve the segmentation accuracy of small targets, this embodiment embeds an attention gate module in the skip connection path to dynamically generate a spatial weight map. This suppresses irrelevant responses from waterside shadows, reflections, or residual noise, retaining only feature channels highly correlated with algae regions to be passed to the decoder, thereby enhancing boundary localization accuracy. Furthermore, to reduce computational overhead and improve the feasibility of edge deployment, this embodiment uses depthwise separable convolutions instead of standard 3×3 convolution operations in each convolutional layer of the network. This significantly reduces the number of parameters and FLOPs while maintaining the receptive field, enabling efficient inference on devices with limited computing power.

[0121] The decoder uses a 1×1 convolutional layer to compress the number of channels to 1, followed by a sigmoid activation function to output the probability value of each pixel belonging to the algae region, forming a continuous algae probability map. Finally, thresholding (default threshold 0.5) is applied to obtain the binary segmentation result, and connected component analysis is used to remove isolated noise points. After segmentation, the system automatically calculates the algae coverage rate. .

[0122]

[0123] In the formula This represents the total number of pixels identified as algae in the segmentation results. R is the total number of valid water pixels in the current image (determined by the input mask). For example, if the valid water area contains 10,000 pixels, of which 2,300 are identified as algae, then R = 23%.

[0124] During training, the optimizer uses the Adam algorithm with an initial learning rate of The batch size is set to 8. The loss function is a weighted combination of Dice Loss and Binary Cross-Entropy Loss: .

[0125] In the formula This is the Binary Cross-Entropy Loss. (i.e., DiceLoss) is a loss function based on the Dice coefficient.

[0126] This composite loss function balances class imbalance and segmentation boundary overlap, maintaining stable gradient updates even with small algal patch areas. Training continues until the validation set metrics converge, during which online data augmentation techniques such as random rotation, affine transformation, and HSV perturbation are applied to improve model robustness.

[0127] In summary, the algae coverage assessment submodule constructed in this embodiment relies on the U-Net_ResNet34 architecture and combines attention gating and depth-separable convolution technology to achieve high-precision, low-latency segmentation of tiny algal patches within a limited water area, meeting the real-time and accuracy requirements of the UAV inspection system.

[0128] The water quality assessment module described in this embodiment achieves dual intelligent perception of water transparency and eutrophication status by collaboratively running a turbidity assessment model based on an improved EfficientNet-B7 and an algae cover assessment model based on U-Net_ResNet34. Both models take a "water mask enhancement map" as input, and the effective region has already been extracted in the previous stage. Under this premise, this embodiment improves the model's recognition accuracy and practicality in real aerial photography scenarios by introducing feasible techniques such as attention mechanisms, multi-task learning, lightweight convolution, and gated skip connections.

[0129] Specifically, multiple dedicated models are used in parallel to process different tasks: YOLOv8-Worldv2 is responsible for target detection, EfficientNet-B7 is used to assess turbidity, and U-Net_ResNet34 is used to analyze algal cover. The outputs of each model are fused and input into the Qwen3-VL visual language model, which has been fine-tuned by QLoRA. Combined with a standardized Prompt template, a structured professional report containing a problem overview, causal analysis, risk level, and governance recommendations is automatically generated.

[0130] S7. Multimodal data fusion is performed on the river aerial images, the categories and locations of abnormal targets, turbidity levels and quantitative values, and algae coverage. The fused data is then input into a visual language model, which uses preset prompt word templates to infer and generate a natural language monitoring report containing environmental assessment conclusions and governance recommendations.

[0131] The report generation module integrates drone aerial images, target detection results, water quality assessment data, and metadata to automatically generate professional, structured, and actionable river ecological environment assessment reports. Built upon the Qwen3-VL visual language model, this module serves as the system's "intelligent decision-making hub," possessing multimodal information fusion, causal reasoning, and natural language generation capabilities, achieving full automation from raw sensory data to governance recommendations.

[0132] At the input level, this module receives structured outputs from the preceding AI sub-modules and performs unified organization and preprocessing. Specific inputs include: (1) 4K resolution aerial images of the river collected by UAVs. (2) Target detection results, including the category labels of abnormal targets, global bounding box coordinates relative to the original image, and confidence scores. (3) Water quality assessment data, including water turbidity levels (divided into four levels: clear, slightly turbid, moderately turbid, and heavily turbid), corresponding NTU estimates, and algae coverage (expressed as a percentage). (4) Inspection metadata, such as collection time, GPS location, and weather conditions. The above multi-source heterogeneous data are uniformly encoded into a standardized JSON format and used together with the original images as inputs to the Qwen3-VL model.

[0133] To further enhance the model's focus on key regions, a visual annotation layer can be optionally overlaid on the original image during the inference phase. This layer includes target detection boxes, algae segmentation mask heatmaps, and turbidity distribution pseudo-color maps. If necessary, automatically generated image descriptions (such as "a large blue-green patch exists in the upper left corner of the water surface") can be added to narrow the semantic gap between vision and language.

[0134] To enable the general visual language model to possess professional knowledge in the water environment field and the ability to generate accurate reports, this embodiment employs the QLoRA (Quantized Low-Rank Adaptation) efficient fine-tuning strategy to adapt Qwen3-VL to the relevant domain. The fine-tuning process first constructs a professional training dataset containing no fewer than 200 real-world inspection cases. Each sample consists of drone aerial images, structured detection and water quality data, and a corresponding assessment report written by a senior water environment engineer or environmental expert. The data covers different seasons, landform types (such as plain rivers, urban black and odorous water bodies, mountain streams, etc.), and typical pollution scenarios (such as floating debris, illegal sewage discharge, algal blooms, and illegal shoreline operations, etc.) to ensure the model has good generalization ability. The report content strictly covers problem description, cause inference, risk level determination (low / medium / high), and no fewer than three specific remediation recommendations, using standardized language and accurate terminology.

[0135] For fine-tuning, the Qwen3-VL pedestal model was loaded using 4-bit quantization to reduce memory usage, the original weights were frozen, and low-rank adapters (LoRAs) were inserted into each Transformer layer, training only newly added parameters. The rank (r) was set to 64 and the scaling parameter α to 16 to minimize the number of trainable parameters while ensuring performance. The optimizer was AdamW, and the learning rate was set to... The batch size is 4, the training cycle is 5 rounds, and an early stopping mechanism is used to prevent overfitting.

[0136] The core objectives of fine-tuning are to enable the model to: 1. Identify key visual cues from images (such as green flocculent algal patches, oil film reflections on the water surface, and fishing gear on the shore). 2. Correlate NTU estimates with the "Surface Water Environmental Quality Standard" (GB 3838-2002) to accurately determine water quality categories. 3. Assess ecological risks based on algal coverage thresholds (e.g., ≥15% is considered a eutrophication warning line). 4. Perform cross-objective causal reasoning by combining spatial location and time series (e.g., identify potential associations between upstream sewage outlets and downstream high turbidity).

[0137] To guide the model in generating reports according to professional logic, this embodiment designs a standardized system prompt template, explicitly assigns the model the role of "senior water environment engineer", and forces it to make rigorous reasoning by combining image evidence and quantitative data.

[0138] The preset prompt template includes role settings, observation data fields, image fields, and output constraint fields. The observation data fields include time, location, weather, list of detected targets, turbidity level, NTU estimate, and algal coverage. The output constraint fields require the model to perform causal inference based on image evidence and quantitative data, and output a report in JSON or Markdown format that includes a problem overview, causal analysis, risk level, and remediation recommendations.

[0139] Example prompts are shown below.

[0140] You are a senior water environment engineer. Based on the following drone inspection information and image content, please conduct a comprehensive assessment of the current ecological environment of the river section.

[0141] [Observational Data]: Time: ... Location: ... Weather: ... Detected: ... Turbidity: ... (NTU estimate: ...). Algal Coverage: ...%.

[0142] [Image]: Image.

[0143] Please generate a report containing the following: Problem Overview: ... Cause Analysis: ... Risk Level (Low / Medium / High). Governance Recommendations (at least 3). Requirements: Formal tone, standardized terminology, and description incorporating visual evidence.

[0144] During the inference phase, the fine-tuned Qwen3-VL model performs three core tasks: First, it achieves visual-linguistic alignment, accurately locating key regions in the image (such as sewage outlets, algae accumulation areas, and floating garbage belts) and establishing spatial semantic associations with structured detection data. Second, it conducts causal inference, analyzing the potential logical relationships between different anomalies (e.g., determining whether a sewage discharge leads to a significant increase in turbidity in downstream water). Finally, it generates a structurally complete and professionally contented natural language report based on a preset template. This ensures the professionalism and consistency of the output.

[0145] This embodiment introduces a constraint mechanism in the decoding stage: it prohibits the use of vague terms such as "possibly" and "probably," and mandates the embedding of industry-specific terms such as "eutrophication," "point source pollution," "suspended solids concentration," and "NTU." It also supports dual-format output—generating readable Markdown reports for manual review, or outputting structured JSON data (including fields such as risk level and recommended items) for subsequent automatic system parsing, task assignment, or integration and display on a web platform. Through this design, this module effectively connects the final link from AI perception to business decision-making, significantly improving the closed-loop governance capabilities of the intelligent river inspection system.

[0146] S8. The natural language monitoring report, target recognition results and water quality parameters are synchronously updated to the Web visualization platform for dynamic display.

[0147] The web visualization platform adopts a left-right split-screen layout, with the river management area on the left and the main display area on the right. The left side features a list of river names, containing all river names entered into the system. This list supports scrolling and search filtering. Users can click on any river name to trigger the dynamic loading and updating of the content on the right.

[0148] The right-hand area is divided into several functional sections, from top to bottom: a satellite map display area, a timeline control area, an image annotation and data visualization area, and a comprehensive assessment report area. The satellite map display area, located at the top right, integrates a high-resolution satellite base map. Based on real-time latitude and longitude coordinate data obtained from drone inspections, the system automatically fits and generates the river's center trajectory line, accurately marking the river's direction and geographical shape on the map with green lines. This trajectory line is divided into 5-meter analysis unit segments for refined assessment of water quality in different river sections. When the user selects a specific time point, the system colors each analysis unit segment according to the detection results at that moment: if the AI ​​model determines that the river section requires immediate treatment, the corresponding trajectory segment is marked in red. If there is potential risk or the indicator is close to the threshold requiring attention, it is marked in yellow. Other river sections with normal conditions remain in their original green color, achieving a spatial visualization of treatment needs.

[0149] A timeline control area is set below the satellite map, providing an interactive time selector that supports switching inspection time points by day, week, and month. Users can dynamically replay historical inspection results at different times by dragging the slider or clicking on a specific date, enabling temporal evolution analysis of the river's health status.

[0150] The satellite map displays image annotations and multi-dimensional data charts simultaneously on the right. The image annotation section presents original image screenshots of key anomalies identified by the system, including algae aggregation, floating garbage, dead fish, illegal sewage outlets, and violations. Bounding boxes and category labels are overlaid on the images, allowing users to intuitively verify the AI ​​recognition results. The data visualization section includes multiple chart components: a pie chart displays the maximum algae coverage rate; a bar chart lists the total inspection length, total number of river sections, number and percentage of red and yellow warning river sections, average turbidity value, and number of anomalies detected. A line chart is also included for historical trend analysis, dynamically plotting the change curve with the test time on the x-axis and the number of each type of identified target on the y-axis, reflecting the pollution development trend.

[0151] Below the images and charts is a comprehensive assessment report area. Based on a comparison of current and historical data, the system automatically generates an overall health assessment of the entire river, including a summary of water quality changes, a pollution risk level assessment, and targeted remediation recommendations, forming a complete closed loop from data perception to decision support. All displayed content supports a responsive layout, adapting to both desktop and mobile devices, ensuring efficient and user-friendly information delivery.

[0152] Specifically, a complete closed-loop system is constructed, encompassing drone data acquisition, image segmentation and preprocessing, multi-model intelligent analysis, automatic report generation, and visualization on a web platform. The platform integrates satellite maps, timeline controls, and multi-dimensional statistical charts, supporting multi-river management, historical data backtracking, trend analysis, and PDF / JSON report export.

[0153] This embodiment, through innovation in four key technologies, significantly outperforms existing technologies in terms of data acquisition quality, analysis accuracy, report professionalism, and system integrity, forming a highly practical intelligent river inspection solution.

[0154] The workflow of this embodiment of a river inspection method based on a multimodal large model is an automated closed-loop process from data perception to decision support. After the system starts, it first controls a drone to conduct inspection flights over the target river along a preset route, using its onboard high-definition camera equipment to collect aerial image data, and then transmitting the images and corresponding metadata such as geographic location and timestamps to the cloud server in real time via a high-speed wireless network.

[0155] After receiving the data, the cloud server first preprocesses the raw imagery, dividing it into standardized tiles suitable for computational analysis. Then, the process moves to the core analysis stage: the system first calls the water body segmentation module to accurately identify and extract pure water bodies from the preprocessed tiles, generating a water body mask-enhanced map that shields against non-target interference such as land. This map lays a precise spatial foundation for subsequent specialized analyses.

[0156] Following this, the system uses the enhanced water mask image as a unified input to drive two parallel analysis engines. The first is a target detection module, responsible for identifying various abnormal targets within the water body, such as floating debris, sewage outlets, and illegal human activities, and outputting their category, location, and confidence level. The second is a water quality assessment module, which further executes two sub-tasks in parallel: quantitative analysis and grade assessment of the water's transparency and turbidity, and algae coverage, outputting key water quality parameters.

[0157] Subsequently, the system integrates and aligns all the above analysis results—including target detection information, water quality parameters, and raw metadata—and inputs them together into the report generation module based on a large-scale visual language model. This module, acting as the system's intelligent hub, performs deep understanding and causal reasoning on multimodal information, ultimately automatically generating a well-structured and professionally worded comprehensive assessment report of the river's ecological environment.

[0158] The final step in the process is to synchronize the generated report, along with all intermediate results and raw data, to a web-based visualization platform. This platform integrates and displays satellite maps of the river, historical data trends, snapshots of identified targets, and a complete assessment report through an interactive interface. This provides users with a comprehensive understanding from macroscopic situations to microscopic details, enabling intelligent, automated, and visualized river inspection work.

[0159] This embodiment aims to overcome the shortcomings of existing intelligent river inspection technologies, such as limited functionality, insufficient accuracy, isolated analysis, reliance on manual reporting, and weak visualization capabilities.

[0160] This embodiment presents a river inspection method based on a multimodal large model, which can automatically acquire aerial images of rivers, accurately identify floating garbage, sewage outlets, and various illegal human activities, and conduct qualitative and quantitative assessments of water turbidity and algae coverage. Based on a visual language large model, it automatically generates a comprehensive monitoring report in natural language with a clear structure and professional content. Finally, it realizes the visualization of multi-dimensional data and historical trend analysis through a web platform, thereby constructing an end-to-end, full-process, and highly integrated intelligent river ecological environment monitoring solution.

[0161] The overall architecture of the river inspection method comprises seven functional modules. First, a multi-rotor drone equipped with a 4K high-definition camera and a 5G communication module periodically flies along a preset route at a fixed altitude and speed, automatically capturing images and uploading the raw images to a cloud server in real time via the 5G network. Next, an image segmentation preprocessing module intelligently crops and normalizes the raw images according to the input size requirements of different downstream AI models (e.g., 640×640, 512×512, or 600×600), ensuring each model receives suitable high-quality input. Then, a water body region segmentation module, based on DeepLabV3+ and a ResNet-50 backbone network, performs pixel-level semantic segmentation of the images, accurately extracting pure water areas and effectively shielding non-water surface interference such as riverside buildings and vegetation. Finally, a target detection module uses the YOLOv8-World v2 model to identify various targets and violations at a 640×640 resolution, including household waste, dead fish, sewage outlets, and activities such as fishing, illegal sand mining, etc. The water quality assessment module is divided into two sub-modules: the turbidity assessment sub-module uses the EfficientNet-B7 model to perform NTU value regression or four-level turbidity classification on the segmented water body image patches; the algae cover assessment sub-module uses U-Net combined with a ResNet34 encoder to perform pixel-level algae segmentation and calculate the coverage ratio of the water body. The report generation module introduces the Qwen3-VL visual language model, combined with a standardized Prompt template, to fuse the original images, detection results, and water quality data, automatically generating a structured Chinese report containing a problem overview, causal analysis, risk level, and remediation recommendations. Finally, the web visualization module, based on a B / S architecture, integrates satellite maps, target annotations, statistical charts (pie charts, bar charts, line charts), timeline controls, and a report display area, supporting multi-river management, historical data review, and PDF export.

[0162] The river inspection method in this embodiment relies on the powerful computing power of 5G high-speed networks and cloud servers to achieve a closed-loop cloud-based processing of the entire chain, from image uploading and intelligent analysis to report generation and visualization.

[0163] This embodiment presents a river inspection method based on a multimodal large model, which establishes a complete chain from data collection and intelligent analysis to decision support, avoiding information loss caused by module fragmentation. Through prior water area segmentation and dedicated model design, the accuracy of target identification and water quality assessment is significantly improved, especially demonstrating robust performance in complex river scenarios. It supports multi-task joint perception of floating objects, water quality parameters, and human activities, providing more comprehensive scenario coverage. It innovatively applies a large visual language model to the water environment field, achieving professional-grade automatic report generation and reducing manual intervention. The web platform possesses powerful spatiotemporal data analysis and visualization capabilities, facilitating trend analysis and emergency response by regulatory authorities.

[0164] In addition, the system adopts a modular architecture, with replaceable models and expandable categories (such as adding detection of oil pollution and red tide in the future), which has good adaptability and upgrade potential, providing reliable technical support for smart water conservancy and ecological protection.

[0165] Example 2: This invention provides a river inspection device based on a multimodal large model, which includes a data acquisition module, an image segmentation and preprocessing module, a water body region segmentation module, a target detection module, a turbidity assessment submodule, an algae cover assessment submodule, and a report generation module.

[0166] The data acquisition module is used to acquire aerial images of the river collected by the UAV flying along the river route according to preset flight parameters; The image segmentation preprocessing module is used to divide the aerial image of the river into multiple sub-blocks that are adapted to the input size of the downstream model; The water body region segmentation module is used to input the sub-map tiles into the semantic segmentation model for pixel-level processing, extract pure water body regions, and generate a water body mask enhancement map that retains only the water body content. The target detection module is used to perform target detection based on the water mask enhancement map to obtain the category and location of abnormal targets; The turbidity assessment submodule is used to perform water turbidity assessment based on the water body mask enhancement map and obtain the turbidity level and quantitative value. The algae cover assessment submodule is used to perform algae cover assessment based on the water body mask enhancement map and obtain the algae cover rate; The report generation module is used to perform multimodal data fusion of river aerial images, the categories and locations of abnormal targets, turbidity levels and quantitative values, and algae coverage. The fused data is then input into a visual language big data model, which infers based on preset prompt word templates to generate a natural language monitoring report containing environmental assessment conclusions and governance recommendations.

[0167] Example 3: The present invention provides a river inspection system device based on a multimodal large model, which includes a drone and a server connected by communication, for executing a river inspection method based on a multimodal large model as described in any paragraph of Example 1.

Claims

1. A river inspection method based on a multimodal large model, characterized in that, Include: S1. Acquire aerial images of the river collected by the drone flying along the river route according to preset flight parameters; S2. Divide the aerial image of the river into multiple sub-blocks that are adapted to the input size of the downstream model; S3. Input the sub-map into the semantic segmentation model for pixel-level processing, extract the pure water area, and generate a water mask enhancement map that retains only the water content. S4. Based on the water mask enhancement map, perform target detection to obtain the category and location of abnormal targets; S5. Based on the water body mask enhancement map, perform water turbidity assessment to obtain turbidity level and quantitative value; S6. Based on the water body mask enhancement map, perform an algae coverage assessment to obtain the algae coverage rate; S7. Multimodal data fusion is performed on the river aerial images, the categories and locations of abnormal targets, the turbidity level and quantitative values, and the algae coverage rate; the fused data is then input into the visual language big model, which infers based on the preset prompt word templates to generate a natural language monitoring report containing environmental assessment conclusions and governance recommendations.

2. The river inspection method based on a multimodal large model according to claim 1, characterized in that, The preset flight parameters are calculated based on a quantitative model established by the target recognition requirements and ground sampling distance constraints; the quantitative model includes the flight altitude formula, the maximum allowable sampling interval formula, and the maximum flight speed formula. The formula for flight altitude is: In the formula This refers to the drone's flight altitude relative to the ground. This represents the actual length of the ground corresponding to one pixel in the image. The focal length of the lens; This represents the number of pixels in the horizontal direction of the image. The physical width of the camera's imaging sensor; The formula for the maximum allowable acquisition interval is: ; This is the maximum allowed data collection interval; Minimum heading overlap; The formula for the maximum flight speed is: In the formula Maximum flight speed; Image acquisition interval; This provides a safety margin for the system response cycle.

3. The river inspection method based on a multimodal large model according to claim 1, characterized in that, The semantic segmentation model adopts an improved DeepLabV3+_ResNeSt-50 architecture. Its backbone network uses ResNeSt-50, and the basic architecture is a DeepLabV3 network. The CBAM attention mechanism is embedded at the output of the ASPP module of DeepLabV3. The ResNeSt-50 is located at the front end of the model for feature extraction. The ASPP module is located after the backbone network to receive the features extracted by the backbone network. Let the ASPP module output feature map be The output after processing by the CBAM attention mechanism for: In the formula This is the channel attention weight vector; This is the spatial attention weight matrix; This is element-wise multiplication; The method for generating a water body mask enhancement map that retains only water body content is as follows: the predicted water body segmentation mask is used as a binary template and fused bitwise with the input sub-map, retaining only the pixel values ​​that are determined to be water bodies, and setting the pixels in non-water body areas to zero. The semantic segmentation model is trained using a joint loss function. Optimize; ； ； In the formula The weighted cross-entropy loss; for The weights; The loss function is based on the Dice coefficient; for The weights; A mask for the real label; For the predicted results; This is a smoothing term.

4. The river inspection method based on a multimodal large model according to claim 1, characterized in that, In S4, object detection adopts an open vocabulary object detection model based on the YOLOv8-Worldv2 architecture, and uses a context-enhanced text prompting mechanism and a lightweight end-to-end fine-tuning strategy. The context-enhanced text prompt mechanism explicitly introduces spatial location and existence status information into the prompt template; The fine-tuning strategy includes the following steps; By fixing the parameters of the neck network and the detection head part of the image backbone network, and updating the parameters only for the neck network and the detection head part, the computational cost is reduced, overfitting is prevented, and convergence is accelerated. Unfreeze the CLIP text encoder to allow it to participate in training to align the text feature space with the water visual feature space; In the classification part of the YOLOv8-Worldv2 model detection head, a learnable multi-granularity semantic mapping layer is introduced. This layer consists of a learnable fully connected network used to map general text embeddings to a semantic subspace applicable to the field of water quality monitoring. The classification part of the YOLOv8-Worldv2 model's detection head is constructed as follows: after downsampling the water mask enhancement map, it is activated by Sigmoid to generate a spatial attention weight matrix, which is then multiplied element-wise with the visual feature map to suppress the response of non-water regions.

5. A river inspection method based on a multimodal large model according to claim 1, characterized in that, The improved EfficientNet-B7 model was used to assess water turbidity in S5. The improvements were: introducing a CBAM attention module at the end of the backbone network and removing the original output layer, replacing it with a dual-task prediction head. The CBAM attention module is integrated after the last MBConv block and before global pooling. The dual-task prediction head includes: a classification branch, which outputs the probability distribution of turbidity levels through the Softmax function; and a regression branch, which outputs continuous NTU estimates through the Sigmoid function and linear mapping. The training loss function for the turbidity assessment submodule is defined as follows: In the formula The training loss function for the turbidity assessment submodule, This represents the mean squared error loss, used in regression tasks; This represents the cross-entropy loss, used in classification tasks. for hyperparameters; for Hyperparameters.

6. The river inspection method based on a multimodal large model according to claim 1, characterized in that, The algae cover evaluation in S6 adopts an improved U-Net architecture; the improvements are: the encoder uses a pre-trained ResNet34; the decoder is constructed through depthwise separable convolutions; and attention gating modules are embedded in the skip connection paths. At the end of the decoder, the number of channels is compressed to 1 through a 1×1 convolutional layer, followed by a sigmoid activation function, which outputs the probability value of each pixel belonging to the algae region. The probability values ​​of algal regions are binarized and segmented according to the threshold, and then the algal coverage rate is calculated. In the formula Algae coverage; This represents the total number of pixels identified as algae in the segmentation results; The total number of valid water pixels in the current image The training loss function of the algal cover assessment model is a weighted combination of Dice loss and binary cross-entropy loss.

7. A river inspection method based on a multimodal large model according to claim 1, characterized in that, In S7, the visual language large model uses the QLoRA strategy for domain adaptation fine-tuning. During fine-tuning, a low-rank adapter is inserted into the Transformer layer and the original weights are frozen. The preset prompt word template includes role settings, observation data fields, image fields, and output constraint fields; The observation data fields include time, location, weather, list of detected targets, turbidity level, estimated NTU value, and algal coverage. The output constraint field requires the model to perform causal reasoning based on image evidence and quantitative data, and to output a report in JSON or Markdown format that includes an overview of the problem, causal analysis, risk level, and governance recommendations.

8. A river inspection method based on a multimodal large model according to claim 1, characterized in that, S2 employs a sliding window-based block strategy; S2 specifically includes the following steps; Set the base size of the sliding window to Set the sliding step size to ,in and This ensures that there are overlapping areas between adjacent sub-plots, preventing cross-plot targets from being missed. The height of the sliding window; The width of the sliding window; When the sliding window extends beyond the boundary of the original image, missing pixels are filled using either reflection fill or zero fill. Record the global coordinate offset of the top left corner of each sub-patch in the original image, which is used to map the local detection results output by subsequent models back to the original image coordinate system; The river inspection method also includes S8; S8, synchronously updating the natural language monitoring report, target identification results and water quality parameters to the Web visualization platform for dynamic display.

9. A river inspection device based on a multimodal large model, characterized in that, Include: The data acquisition module is used to acquire aerial images of the river collected by the UAV flying along the river route according to preset flight parameters; The image segmentation preprocessing module is used to divide the aerial image of the river into multiple sub-blocks that are adapted to the input size of the downstream model; The water body region segmentation module is used to input the sub-map tiles into the semantic segmentation model for pixel-level processing, extract pure water body regions, and generate a water body mask enhancement map that retains only the water body content. The target detection module is used to perform target detection based on the water mask enhancement map to obtain the category and location of abnormal targets; The turbidity assessment submodule is used to perform water turbidity assessment based on the water body mask enhancement map and obtain the turbidity level and quantitative value. The algae cover assessment submodule is used to perform algae cover assessment based on the water body mask enhancement map and obtain the algae cover rate; The report generation module is used to perform multimodal data fusion of aerial images of rivers, categories and locations of abnormal targets, turbidity levels and quantitative values, and algal coverage. The fused data is then input into a visual language big data model, which infers based on preset prompt word templates to generate a natural language monitoring report containing environmental assessment conclusions and governance recommendations.

10. A river inspection system based on a multimodal large model, characterized in that, The system includes a drone and a server with communication connectivity, used to execute a river inspection method based on a multimodal large model as described in any one of claims 1 to 8.

Images