A reservoir safety intelligent inspection method and system based on fusion of YOLO and VLM

Abstract

The application discloses a kind of reservoir safety intelligent inspection method and system based on YOLO and VLM fusion, comprising the following steps: S1, multi-source data acquisition and pretreatment: unmanned aerial vehicle and ground equipment are used to collect image / video data, and the data is processed by noise reduction, enhancement and space-time alignment;S2, improved YOLO target detection: optimization network structure and training strategy;S3, VLM post-link analysis: VLM is used to realize target / scene correlation judgment;S4, report generation.The application sets up the reservoir safety intelligent inspection method and system based on YOLO and VLM fusion;Using the cross-modal semantic understanding, zero-sample inference and video global analysis capability of visual language large model, it is used as the post-processing tool of YOLO, and it is mutually fused and works in parallel, can realize the whole process intelligentization of "target detection-semantic analysis-report generation", effectively solve the prior art problem, enhance the comprehensiveness of water conservancy project safety monitoring system.

Description

Technical Field

[0001] This invention relates to the field of water conservancy project safety monitoring technology, and more specifically, to a reservoir safety intelligent inspection method and system based on the integration of YOLO and VLM. Background Technology

[0002] As a core hub for water resource regulation, the safe operation of reservoirs directly affects the flood control safety, water supply, and ecological environment stability of the basin. Traditional reservoir safety inspection methods rely on manual foot patrols, fixed-point observations, and periodic aerial photography, which have inherent drawbacks such as insufficient spatial and temporal coverage, delayed hazard identification, and high labor costs. With the deep integration of drone technology and computer vision, drone inspection, with its advantages of wide coverage and flexibility, has become an important means of reservoir safety monitoring.

[0003] In existing technologies, the processing of drone inspection data still has significant limitations: First, while relying solely on the YOLO algorithm can achieve rapid target detection, it cannot perform scenario-based judgments such as "whether floating objects are close to the spillway". Second, reservoir scenarios have diverse types of hazards, and some new targets (such as special aquaculture equipment and precursors to new geological disasters) are difficult to collect complete datasets due to the scarcity of samples, resulting in insufficient generalization ability of traditional detection models. Third, drone inspection data and ground monitoring data lack semantic-level correlation, making it difficult to generate structured reports containing in-depth analysis. Summary of the Invention

[0004] To overcome the aforementioned deficiencies of the prior art, embodiments of the present invention provide a reservoir safety intelligent inspection method and system based on the fusion of YOLO and VLM. By setting up a reservoir safety intelligent inspection method and system that integrates YOLO and VLM, and utilizing the cross-modal semantic understanding, zero-shot reasoning, and video global analysis capabilities of the Visual Language Model (VLM), it is used as a post-processing tool for YOLO and integrated with it to work in parallel. This enables intelligent processing of the entire process from "target detection to semantic analysis to report generation," effectively solving the problems of the prior art, enhancing the comprehensiveness of the water conservancy project safety monitoring system, and thus addressing the issues raised in the background art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a reservoir safety intelligent inspection method and system based on the integration of YOLO and VLM, comprising the following steps:

[0006] S1. Multi-source data acquisition and preprocessing:

[0007] Image / video data is collected using drones and ground equipment, and the data is then processed for noise reduction, enhancement, and spatiotemporal alignment.

[0008] S2. Improved YOLO object detection:

[0009] Optimize network structure and training strategy to achieve high-precision detection of dam cracks and floating objects in UAV scenarios;

[0010] S3 and VLM back-link analysis:

[0011] Extract the visual feature vector of the target in step S2, and at the same time extract the global visual feature vector of the whole image. Construct a text dictionary containing the core semantic concepts of reservoir inspection, and use VLM to realize target / scene association judgment, zero-shot target recognition and video temporal understanding.

[0012] S4. Report Generation:

[0013] Based on the detection results and semantic analysis, a structured report containing hazard information, risk assessment, and disposal suggestions is generated.

[0014] In a preferred embodiment, step S1 specifically includes:

[0015] (1) Data acquisition operation: A multi-rotor drone equipped with an industrial-grade camera was used to collect images / videos of the reservoir dam, water surface, spillway and surrounding dangerous areas. Data on the bottom of the dam and gate area was supplemented by a ground-based high-definition fixed camera. The timestamp and GPS coordinates of each frame of data were recorded during the acquisition.

[0016] (2) Noise reduction: Gaussian filtering algorithm is used to remove high-frequency noise from the original image / video, and dynamic noise in the video data is further eliminated by inter-frame difference method;

[0017] (3) Enhancement processing: For complex lighting scenes, the Retinex algorithm is used to adjust the image brightness and contrast to highlight the key target features of dam cracks and floating objects;

[0018] (4) Spatiotemporal alignment processing: Based on timestamps and GPS coordinates, the RANSAC algorithm is used to realize the spatial registration of UAV and ground equipment data. The video frame time synchronization is completed by linear interpolation to form a standardized multi-source fusion dataset.

[0019] In a preferred embodiment, step S2 specifically includes:

[0020] (1) Network structure optimization: Based on the YOLOv8 architecture, the convolutions in the backbone part are replaced with depthwise separable convolutions to reduce the number of parameters. An attention module is added to the neck part to enhance the feature extraction capability for small-sized dam crack targets.

[0021] (2) Training strategy optimization: Construct a special dataset for reservoir inspection containing multiple targets such as dam cracks and floating objects, and use a cosine annealing learning rate strategy combined with Focal Loss loss function to train the model until the accuracy of the validation set is stable.

[0022] (3) Target detection execution: Input the preprocessed dataset into the improved YOLO model, set a reasonable confidence level and IOU threshold, remove duplicate detection boxes through non-maximum suppression, realize the target detection of dam cracks, floating objects and illegal personnel, and output the target category, bounding box coordinates and confidence level.

[0023] In a preferred embodiment, in step S3, the ROI region corresponding to the target detection bounding box is input into the pre-trained ResNet model, and then the visual feature vector is extracted to extract the global visual feature vector of the entire image.

[0024] In step S3, a text dictionary containing the core semantic concepts of reservoir inspection is constructed. The text concepts are input into the pre-trained BERT model to generate text embedding vectors, which are then linearly transformed to unify the dimensions and matched with the visual feature vectors.

[0025] In a preferred embodiment, step S3 specifically includes:

[0026] (1) Target / scene association judgment: Calculate the cosine similarity between the visual feature vector of the target ROI and the text embedding vector of the key area of ​​the scene. When the similarity reaches the set threshold, the target and the scene are determined to be associated.

[0027] (2) Zero-shot target recognition: For unknown targets that are not detected, extract their ROI visual feature vectors, calculate the distance between them and the text embedding vectors of candidate categories in the text dictionary, and select the category with the smallest distance as the recognition result, without the need for additional training;

[0028] (3) Video temporal understanding: For the detection results of multiple consecutive video frames, analyze the target motion trajectory and state changes, and output the target temporal correlation information.

[0029] In a preferred embodiment, step S4 specifically includes:

[0030] (1) Data integration: The target detection results, VLM analysis results and original spatiotemporal data are linked and integrated to form structured data containing target information, scene association information and spatiotemporal information;

[0031] (2) Risk assessment: The fuzzy comprehensive evaluation method is used to calculate the risk level of the hidden danger from three dimensions: the severity of the hidden danger, the scope of its impact, and the development trend;

[0032] (3) Generation of disposal suggestions: Based on the risk level, the preset disposal suggestion knowledge base is called to generate targeted disposal suggestions in combination with the spatiotemporal information of the hidden danger;

[0033] (4) Structured report output: Generate a standardized report containing basic inspection information, a list of hidden dangers, risk assessment and disposal suggestions, and store it in the database to support subsequent queries.

[0034] In a preferred embodiment, the cosine similarity calculation between visual features and text embeddings in step S3 after VLM link analysis specifically includes:

[0035] Extract visual feature vectors from the target ROI region and generate text embedding vectors for text concepts in key areas of the scene.

[0036] Unify the dimensions of text embedding vectors and visual feature vectors through a linear transformation matrix;

[0037] The system calculates the cosine similarity of vectors. When the similarity reaches a set threshold, it determines that the target is associated with a key area of ​​the scene, thus enabling scene association judgment between floating objects / outlets and personnel / dangerous areas, meeting the accuracy requirements of actual inspections.

[0038] In a preferred embodiment, the cosine similarity calculation between visual features and text embeddings in step S3 after VLM link analysis specifically includes:

[0039] Extract visual feature vectors from the target ROI region and generate text embedding vectors for text concepts in key areas of the scene.

[0040] Unify the dimensions of text embedding vectors and visual feature vectors through a linear transformation matrix;

[0041] The system calculates the cosine similarity of vectors. When the similarity reaches a set threshold, it determines that the target is associated with a key area of ​​the scene, thus enabling scene association judgment between floating objects / outlets and personnel / dangerous areas, meeting the accuracy requirements of actual inspections.

[0042] In a preferred embodiment, the reservoir safety intelligent inspection system includes the following functional layers, each working in sequence and collaboratively:

[0043] Data acquisition layer: Composed of a hexacopter drone with camera, GPS, and timestamp recording functions and a ground-based high-definition fixed camera, it collects images / videos of key areas of the reservoir and transmits them to the preprocessing module;

[0044] Preprocessing sublayer: Integrates Gaussian filtering, Retinex enhancement, and spatiotemporal alignment modules, performs the preprocessing operation in step S1, and outputs a standardized multi-source fusion dataset;

[0045] YOLO detection layer: includes a model training module and a real-time detection module, receives preprocessed data and performs the detection operation in step S2, and outputs the target detection results;

[0046] VLM Analysis Layer: Includes modules for visual feature extraction, text embedding generation, association judgment, zero-shot recognition, and time series understanding. It receives detection results, performs step S3 analysis, and outputs association, recognition, and time series information.

[0047] Report generation layer: Composed of data integration, risk assessment, handling recommendations and report output modules, it receives analysis results and raw data, executes step S4, and outputs a standardized report;

[0048] Storage layer: It adopts an architecture that combines relational databases and object storage to store structured data and raw data, model files and reports respectively, and supports data storage and expansion;

[0049] Collaborative logic across layers: Data acquisition layer → Preprocessing sub-layer → YOLO detection layer → VLM analysis layer → Report generation layer, with all data ultimately stored in the storage layer, forming a closed loop of data flow.

[0050] The technical effects and advantages of this invention are as follows:

[0051] This invention establishes a reservoir safety intelligent inspection method and system that integrates YOLO and VLM. Utilizing the cross-modal semantic understanding, zero-shot reasoning, and global video analysis capabilities of the Visual Language Model (VLM), it is used as a post-processing tool for YOLO and integrated with it in parallel. This enables intelligent operation of the entire process from "target detection to semantic analysis to report generation," effectively addressing several issues in existing technologies, such as the inability to perform scenario-based judgments, insufficient generalization ability of traditional detection models, and difficulty in generating structured reports with in-depth analysis. This further enhances the comprehensiveness of the water conservancy project safety monitoring system. Attached Figure Description

[0052] Figure 1 This is a flowchart of the image data preprocessing in step S1 of the present invention;

[0053] Figure 2 This is a flowchart of the post-link analysis and video understanding of the visual language big model of the present invention. Detailed Implementation

[0054] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0055] As attached Figure 1 To be continued Figure 2 The method and system for intelligent reservoir safety inspection based on the integration of YOLO and VLM, as shown, include the following steps:

[0056] S1: Multi-source data acquisition and preprocessing

[0057] Data acquisition operation: Multi-rotor drones equipped with industrial-grade cameras are used to collect images / videos of the reservoir dam, water surface, spillway and surrounding dangerous areas; at the same time, high-definition fixed ground cameras are used to supplement the data collection of the bottom of the dam and the gate area. The timestamp and GPS coordinates of each frame of data are recorded during the acquisition.

[0058] Noise reduction: Gaussian filtering algorithm is used to remove high-frequency noise from the original image / video. Video data is further processed by inter-frame difference method to eliminate dynamic noise, ensuring that the image clarity meets the requirements of subsequent detection.

[0059] Enhancement processing: For complex lighting scenes, the Retinex algorithm is used to adjust the image brightness and contrast, highlighting key target features such as dam cracks and floating objects.

[0060] Spatiotemporal alignment processing: Based on timestamps and GPS coordinates, the RANSAC algorithm is used to achieve spatial registration of data between UAVs and ground equipment, and the video frame time synchronization is completed by linear interpolation to form a standardized multi-source fusion dataset.

[0061] S2: Improved YOLO object detection

[0062] Network structure optimization: Based on the YOLOv8 architecture, the convolutions in the backbone part are replaced with depthwise separable convolutions to reduce the number of parameters, and an attention module is added to the neck part to enhance the feature extraction capability for targets such as small-sized dam cracks.

[0063] Training strategy optimization: Construct a dedicated dataset for reservoir inspection that includes multiple targets such as dam cracks and floating objects. Use a cosine annealing learning rate strategy combined with the Focal Loss loss function to train the model until the accuracy on the validation set is stable.

[0064] Target detection execution: Input the preprocessed dataset into the improved YOLO model, set reasonable confidence and IOU thresholds, remove duplicate detection boxes through non-maximum suppression, realize the detection of targets such as dam cracks, floating objects, and illegal personnel, and output the target category, bounding box coordinates and confidence.

[0065] S3: VLM Post-Link Analysis

[0066] Visual feature extraction: Input the ROI region corresponding to the object detection bounding box into the pre-trained ResNet model to extract visual feature vectors, and at the same time extract the global visual feature vector of the entire image.

[0067] Text embedding generation: Construct a text dictionary containing core semantic concepts of reservoir inspection (such as dam cracks and spillway), input the text concepts into the pre-trained BERT model to generate text embedding vectors, unify the dimensions through linear transformation, and match them with visual feature vectors.

[0068] Target-scene association judgment: Calculate the cosine similarity between the visual feature vector of the target ROI and the text embedding vector of the key area of ​​the scene. When the similarity reaches the set threshold, the target is determined to be associated with the scene (such as a floating object approaching the outlet). The accuracy of association judgment meets the actual inspection needs.

[0069] Zero-shot target recognition: For undetected unknown targets, extract their ROI visual feature vectors, calculate the distance between them and the text embedding vectors of candidate categories in the text dictionary, and select the category with the smallest distance as the recognition result. No additional training is required, making it suitable for new types of hidden danger scenarios.

[0070] Video temporal understanding: For the detection results of multiple consecutive video frames, analyze the target motion trajectory (such as the direction and speed of floating objects) and state changes (such as whether the crack is expanding), and output the target temporal correlation information.

[0071] S4: Report Generation

[0072] Data integration: The target detection results, VLM analysis results and raw spatiotemporal data are linked and integrated to form structured data containing target information, scene association information and spatiotemporal information.

[0073] Risk assessment: The fuzzy comprehensive evaluation method is used to calculate the risk level (low, medium, high, and extremely high) of the hidden danger from three dimensions: the severity of the hidden danger, the scope of its impact, and its development trend.

[0074] Disposal suggestion generation: Based on the risk level, the system calls up the preset disposal suggestion knowledge base and generates targeted disposal suggestions by combining the spatiotemporal information of the potential hazards.

[0075] Structured report output: Generates a standardized report containing basic inspection information, a list of potential hazards, risk assessment, and handling recommendations, and stores it in the database for subsequent retrieval.

[0076] In step S3, the VLM post-link analysis specifically includes the calculation of cosine similarity between visual features and text embeddings:

[0077] Visual feature vectors are extracted from the target ROI region, and text embedding vectors are generated for text concepts in key areas of the scene.

[0078] The dimensions of text embedding vectors and visual feature vectors are unified by a linear transformation matrix (the parameters of which are obtained by transfer learning from a pre-trained CLIP model).

[0079] The system calculates the cosine similarity of vectors. When the similarity reaches a set threshold, it determines that the target is associated with a key area of ​​the scene, enabling scene association judgments such as floating objects-outlets and personnel-dangerous areas, thus meeting the accuracy requirements of actual inspections.

[0080] The zero-shot target recognition in step S3 specifically includes: for undetected unknown targets, extracting the complete ROI region by improving the YOLO candidate box generation module, extracting the visual feature vector of the unknown target ROI region, selecting candidate categories from the text dictionary, generating text embedding vectors for each category and unifying the dimensions, calculating the distance between the unknown target feature vector and the candidate category vector, and selecting the category with the smallest distance as the recognition result. This process does not require additional training, is suitable for new hidden danger scenarios where datasets are difficult to construct, meets actual recognition needs, and significantly increases its practicality.

[0081] A reservoir safety intelligent inspection system based on the fusion of YOLO and a large visual language model includes the following functional layers, each working in sequence and collaboratively:

[0082] Data acquisition layer: Composed of a hexacopter drone with camera, GPS, and timestamp recording functions and a ground-based high-definition fixed camera, it collects images / videos of key areas of the reservoir and transmits them to the preprocessing module.

[0083] Preprocessing sublayer: Integrates Gaussian filtering, Retinex enhancement, and spatiotemporal alignment modules to perform the preprocessing operations in step one and output a standardized multi-source fusion dataset.

[0084] YOLO detection layer: Includes a model training module (supporting dataset training and network optimization) and a real-time detection module. It receives preprocessed data, performs the detection operation in step two, and outputs the target detection results.

[0085] VLM Analysis Layer: Includes modules for visual feature extraction, text embedding generation, association judgment, zero-shot recognition, and time series understanding. It receives detection results, performs the analysis operation in step three, and outputs association, recognition, and time series information.

[0086] Report generation layer: It consists of data integration, risk assessment, disposal recommendations and report output modules. It receives analysis results and raw data, performs step four operations, and outputs a standardized report.

[0087] Storage layer: It adopts an architecture that combines relational databases and object storage to store structured data and raw data, model files and reports respectively, and supports data storage and expansion.

[0088] Collaborative logic across layers: Data acquisition layer → Preprocessing sub-layer → YOLO detection layer → VLM analysis layer → Report generation layer, with all data ultimately stored in the storage layer, forming a closed loop of data flow.

[0089] System architecture workflow

[0090] 1. Data Acquisition Layer: The UAV inspection unit (5K camera + LiDAR) and ground monitoring equipment form a three-dimensional network, collecting ≥8 hours of high-definition video daily;

[0091] 2. Processing layer: Includes the YOLO object detection module and the VLM analysis module. The former outputs the object coordinates and category, while the latter realizes association judgment, zero-shot recognition, and video understanding.

[0092] 3. Application Layer: Generate special reports on drone inspections, including time-series comparisons of potential hazards, slope stability assessments (based on VLM continuous three-week deformation analysis), and air-to-ground data correlation cases;

[0093] Storage layer: NAS array is used to store raw videos, PostgreSQL is used to store detection results and reports, and HDFS is used to store model parameters, supporting retrieval by time, region, and target type.

[0094] Multi-source heterogeneous data acquisition and preprocessing

[0095] Constructing a three-dimensional data collection network combining "drones + ground monitoring":

[0096] Unmanned aerial vehicle (UAV) inspection unit: Equipped with multi-rotor UAVs (endurance ≥ 50 minutes) carrying 5K high-definition optical cameras, thermal imagers and lidar, it conducts two daily patrols of areas such as the top of the reservoir dam, slopes and center of the reservoir area, collecting oblique photography data (heading overlap rate ≥ 85%, side overlap rate ≥ 75%), thermal infrared images and 3D point cloud data.

[0097] Ground monitoring equipment: High-definition bullet cameras (one every 50 meters, 4K resolution) are deployed along the dam body, and 360° spherical cameras are installed around the reservoir area to achieve all-weather monitoring of the near-shore area;

[0098] Data preprocessing: UAV data undergoes distortion correction, image stitching (SIFT feature matching, error ≤2 pixels) and point cloud noise reduction (statistical filtering); ground data is denoised using BM3D and enhanced with Retinex; air-to-ground data is unified to the UTM coordinate system through GPS / IMU fusion positioning, with a spatiotemporal alignment error ≤0.5 meters.

[0099] Optimization of UAV target detection based on improved YOLO

[0100] YOLO, a classic algorithm in real-time object detection, innovatively transforms the object detection task into a single regression problem. Its core lies in dividing the input image into an S×S grid, with each grid cell responsible for predicting multiple bounding boxes and their corresponding class confidence scores. Taking the common YOLOv8 model as an example, after the input image (e.g., 640×640 pixels) enters the network, it first passes through a backbone network composed of a series of convolutional and pooling layers to extract features. The convolutional layers in the backbone network capture rich texture, shape, and other feature information in the image through convolutional kernels of different sizes, while the pooling layers reduce the resolution of the feature maps, decreasing computation while preserving key features.

[0101] The YOLO algorithm is used to achieve efficient detection of potential hazards in reservoirs, and optimizations are made for drone aerial photography scenarios as follows:

[0102] 1. Network adaptation and optimization: The backbone network adopts a hybrid architecture of CSPDarknet and MobileNetV3, supporting dynamic resolution input from 2K to 8K; an improved BiFPN structure is introduced in the neck section, which enhances the detection capability of small targets at high altitudes (such as floating objects ≤30 pixels and fine cracks) by fusing multi-scale features through attention weighting.

[0103] 2. Dataset Construction: 150,000 drone inspection images were collected, and 12 typical targets such as dam cracks, slope hazards, floating objects, and illegal personnel were labeled. The oblique view sample accounted for 60%, which solved the problem of scarce target labeling under the drone view and increased the coverage of the target collection area.

[0104] 3. Training Strategy: Mosaic+CutMix data augmentation is used to simulate aerial occlusion and scale change scenarios; the loss function integrates CIoU (localization accuracy) and Focal Loss (class balance), with small targets given a 2.5x weight; transfer learning based on the COCO pre-trained model improves convergence speed by 40%.

[0105] Real-time inference: Deploy the quantization model on the UAV's onboard edge unit (computing power ≥ 20 TOPS). After TensorRT optimization, the detection time for a single frame of 5K image is ≤ 30ms, and the output is the target category, bounding box and coordinates (confidence ≥ 0.7).

[0106] Post-link analysis and video understanding of large visual language models

[0107] Visual Language Models (VLMs) integrate techniques from computer vision and natural language processing to establish semantic relationships between visual information (such as images and videos) and natural language. Their architecture typically comprises two main parts: a visual encoder and a language encoder. The visual encoder is responsible for extracting important visual attributes such as color, shape, and texture from image or video input and converting them into vector embeddings that can be processed by machine learning models. Early VLMs often used convolutional neural networks (CNNs) for feature extraction, while more advanced models today utilize visual transducers (ViTs). ViTs segment images into patches and treat these patches as sequences, similar to words in a language transducer. They then perform self-attention mechanisms on the patches to create transducer-based representations of the input image. The language encoder is responsible for capturing the semantic and contextual relationships between words and phrases and converting them into text embeddings. Most VLMs use transducer models (such as Google's BERT and OpenAI's GPT series) as their language encoders. During training, VLM uses strategies such as contrastive learning, occlusion, and generative model training to align and fuse information from the visual and language encoders, learn how to associate images with text, and thus acquire the ability to complete various visual language tasks, thereby improving its performance.

[0108] Leveraging the cross-modal understanding capabilities of VLM, deep analysis of YOLO detection results is performed to achieve global video understanding and semantic reasoning:

[0109] 1. Target-Scene Association Judgment: The VLM receives the target coordinates and category information output by YOLO, and combines them with the text description of key areas of the reservoir (such as "the coordinate range of the spillway is (X0,Y0)-(X1,Y1)" and "the restricted area of ​​the dam is within 5 meters of the edge"). Through the calculation of cosine similarity between visual features and text embedding, it determines whether floating objects are close to the spillway, whether abnormal personnel are in dangerous areas, and whether illegal vessels have entered the restricted navigation area, etc., and the accuracy of association judgment is ≥85%.

[0110] 2. Zero-shot target recognition: For novel unlabeled targets (such as new surveying equipment and illegal breeding facilities), VLM does not require additional training. It achieves zero-shot classification (confidence ≥ 0.7) by comparing the distance between the visual features of the unknown target and the text embedding of the candidate category, thus solving the pain point of the difficulty in constructing special hidden danger datasets.

[0111] 3. Video temporal understanding: VLM performs inter-frame correlation analysis on UAV cruise videos and captures the dynamic trend of targets through temporal attention: such as tracking the drift trajectory of floating objects over 2 hours to predict whether they will enter the spillway area; analyzing the morphological changes of slope cracks in consecutive frames to determine whether there is a risk of expansion.

[0112] 4. Global semantic integration: The system integrates multi-frame detection results to construct a dynamic scene map, integrates the frequency of target occurrence, location changes and interaction relationships, and forms a global understanding of the inspection area, such as identifying potential patterns such as "the correlation between floating object accumulation areas and wind direction" and "the correlation between the activities of illegal personnel and holidays".

[0113] Multimodal report generation and air-to-ground data fusion

[0114] Based on YOLO detection results and VLM semantic analysis, a structured inspection report is generated:

[0115] 1. Report generation logic: VLM converts YOLO's target data into natural language descriptions (such as "a 1.2-meter diameter plastic floating object was found 30 meters from the spillway"), and combines the time series analysis results to generate a report that includes a list of hazards, risk levels (high / medium / low), and disposal recommendations;

[0116] 2. Air-to-Ground Data Association: VLM uses feature matching to associate distant drone targets with close-up ground details. For example, it can merge the drone's "suspicious floating object" with the ground camera's "2-meter diameter plastic bucket" attribute to supplement key information such as target size and material.

[0117] Visualization Enhancement: Automatically generates 3D hazard distribution maps (based on point clouds, with an accuracy of ≤0.1 meters), target motion trajectory time series maps, and air-ground image linkage comparison maps, and supports clicking on the reported target to jump to the corresponding video segment.

[0118] Finally, the following points should be noted: First, in the description of this application, it should be noted that, unless otherwise specified and limited, the terms "installation", "connection", and "linkage" should be interpreted broadly, and can be mechanical or electrical connections, or internal connections between two components, or direct connections. "Up", "down", "left", "right", etc. are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may change.

[0119] Secondly: The accompanying drawings of the embodiments disclosed in this invention only involve the structures involved in the embodiments disclosed in this invention. Other structures can refer to the general design. In the absence of conflict, the same embodiment and different embodiments of this invention can be combined with each other.

[0120] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A reservoir safety intelligent inspection method based on the integration of YOLO and VLM, characterized in that: Includes the following steps, S1. Multi-source data acquisition and preprocessing: Image / video data is collected using drones and ground equipment, and the data is then processed for noise reduction, enhancement, and spatiotemporal alignment. S2. Improved YOLO object detection: Optimize network structure and training strategy to achieve high-precision detection of dam cracks and floating objects in UAV scenarios; S3 and VLM back-link analysis: Extract the visual feature vector of the target in step S2, and at the same time extract the global visual feature vector of the whole image. Construct a text dictionary containing the core semantic concepts of reservoir inspection, and use VLM to realize target / scene association judgment, zero-shot target recognition and video temporal understanding. S4. Report Generation: Based on the detection results and semantic analysis, a structured report containing hazard information, risk assessment, and disposal suggestions is generated. Step S2 specifically includes: Network structure optimization: Based on the YOLOv8 architecture, the convolutions in the backbone part are replaced with depthwise separable convolutions to reduce the number of parameters, and an attention module is added to the neck part to enhance the feature extraction capability for small-sized dam crack targets; Training strategy optimization: Construct a special dataset for reservoir inspection containing multiple targets such as dam cracks and floating objects, and use a cosine annealing learning rate strategy combined with Focal Loss loss function to train the model until the accuracy of the validation set is stable; Target detection execution: Input the preprocessed dataset into the improved YOLO model, set reasonable confidence and IOU thresholds, remove duplicate detection boxes through non-maximum suppression, realize the target detection of dam cracks, floating objects, and illegal personnel, and output the target category, bounding box coordinates and confidence. Step S3 specifically includes: Target / scene association determination: Calculate the cosine similarity between the visual feature vector of the target ROI and the text embedding vector of the key area of ​​the scene. When the similarity reaches a set threshold, the target and the scene are determined to be associated. Zero-shot target recognition: For undetected unknown targets, extract their ROI visual feature vectors, calculate the distance between them and the text embedding vectors of candidate categories in the text dictionary, and select the category with the smallest distance as the recognition result, without the need for additional training; Video temporal understanding: Analyze the target motion trajectory and state changes from the detection results of multiple consecutive video frames, and output the target temporal correlation information.

2. The intelligent reservoir safety inspection method based on the fusion of YOLO and VLM as described in claim 1, characterized in that: Step S1 specifically includes: Data acquisition operation: A multi-rotor drone equipped with an industrial-grade camera is used to collect images / videos of the reservoir dam, water surface, spillway and surrounding dangerous areas. Ground-based high-definition fixed cameras are used to supplement the data collection of the bottom of the dam and the gate area. The timestamp and GPS coordinates of each frame of data are recorded during the acquisition. Noise reduction: Gaussian filtering algorithm is used to remove high-frequency noise from the original image / video, and dynamic noise in the video data is further eliminated by inter-frame differencing. Enhancement processing: For complex lighting scenes, the Retinex algorithm is used to adjust the image brightness and contrast, highlighting key target features such as dam cracks and floating objects; Spatiotemporal alignment processing: Based on timestamps and GPS coordinates, the RANSAC algorithm is used to achieve spatial registration of data between UAVs and ground equipment, and the video frame time synchronization is completed by linear interpolation to form a standardized multi-source fusion dataset.

3. The intelligent reservoir safety inspection method based on the fusion of YOLO and VLM as described in claim 1, characterized in that: In step S3, the ROI region corresponding to the target detection bounding box is input into the pre-trained ResNet model, and then the visual feature vector is extracted to extract the global visual feature vector of the entire image. In step S3, a text dictionary containing the core semantic concepts of reservoir inspection is constructed. The text concepts are input into the pre-trained BERT model to generate text embedding vectors, which are then linearly transformed to unify the dimensions and matched with the visual feature vectors.

4. The intelligent reservoir safety inspection method based on the fusion of YOLO and VLM as described in claim 1, characterized in that: Step S4 specifically includes: Data integration: The target detection results, VLM analysis results and raw spatiotemporal data are linked and integrated to form structured data containing target information, scene association information and spatiotemporal information; Risk assessment: The fuzzy comprehensive evaluation method is used to calculate the risk level of the hidden danger from three dimensions: severity of the hidden danger, scope of impact, and development trend. Disposal suggestion generation: Based on the risk level, a preset disposal suggestion knowledge base is invoked, and targeted disposal suggestions are generated by combining the spatiotemporal information of the potential hazards; Structured report output: Generates a standardized report containing basic inspection information, a list of potential hazards, risk assessment, and handling recommendations, and stores it in the database for subsequent retrieval.

5. The intelligent reservoir safety inspection method based on the fusion of YOLO and VLM as described in claim 1, characterized in that: In step S3, the VLM post-link analysis specifically includes the calculation of cosine similarity between visual features and text embeddings: Extract visual feature vectors from the target ROI region and generate text embedding vectors for text concepts in key areas of the scene. Unify the dimensions of text embedding vectors and visual feature vectors through a linear transformation matrix; The system calculates the cosine similarity of vectors. When the similarity reaches a set threshold, it determines that the target is associated with a key area of ​​the scene, thus enabling scene association judgment between floating objects / outlets and personnel / dangerous areas, meeting the accuracy requirements of actual inspections.

6. The intelligent reservoir safety inspection method based on the fusion of YOLO and VLM as described in claim 1, characterized in that: The zero-shot target identification in step S3 specifically includes: For undetected unknown targets, the complete ROI region is extracted by improving the YOLO candidate box generation module; Extract visual feature vectors of unknown target ROI regions; Candidate categories are selected from the text dictionary, text embedding vectors for each category are generated, and the dimensions are unified. Calculate the distance between the feature vector of the unknown target and the candidate category vector, and select the category with the smallest distance as the recognition result; This process requires no additional training, is suitable for novel hazard scenarios where datasets are difficult to construct, and meets practical identification needs.

7. A reservoir safety intelligent inspection method based on the fusion of YOLO and VLM according to any one of claims 1-6, wherein the reservoir safety intelligent inspection system is as follows, characterized in that: It includes the following functional layers, and each layer works in sequence and in coordination: Data acquisition layer: Composed of a hexacopter drone with camera, GPS, and timestamp recording functions and a ground-based high-definition fixed camera, it collects images / videos of key areas of the reservoir and transmits them to the preprocessing module; Preprocessing sublayer: Integrates Gaussian filtering, Retinex enhancement, and spatiotemporal alignment modules, performs the preprocessing operation in step S1, and outputs a standardized multi-source fusion dataset; YOLO detection layer: includes a model training module and a real-time detection module, receives preprocessed data and performs the detection operation in step S2, and outputs the target detection results; VLM Analysis Layer: Includes modules for visual feature extraction, text embedding generation, association judgment, zero-shot recognition, and time series understanding. It receives detection results, performs step S3 analysis, and outputs association, recognition, and time series information. Report generation layer: Composed of data integration, risk assessment, handling recommendations and report output modules, it receives analysis results and raw data, executes step S4, and outputs a standardized report; Storage layer: It adopts an architecture that combines relational databases and object storage to store structured data and raw data, model files and reports respectively, and supports data storage and expansion; Collaborative logic across layers: Data acquisition layer → Preprocessing sub-layer → YOLO detection layer → VLM analysis layer → Report generation layer, with all data ultimately stored in the storage layer, forming a closed loop of data flow.

Images