Dam termite image recognition method based on multi-modal fusion and dynamic feature enhancement

Abstract

The invention discloses a dam termite image recognition method based on multi-modal fusion and dynamic feature enhancement. The method comprises the steps that image data, environmental parameters and vibration signals of a dam monitoring area are collected in real time through a multi-modal sensor network; carrying out the preprocessing of the image data, and carrying out the target detection through an improved ACP-YOLOv5s model; processing the continuous video frames by adopting Gaussian background modeling and a difference algorithm to generate a binary mask graph representing dynamic activities of termites, and separating a moving target area from a static background; and the target detection result and the dynamic mask graph are fused, the collected environment parameters and vibration signals are combined, the termite damage level is calculated through an AI engine, and graded early warning or automatic killing response is triggered. The method breaks through the limitation of traditional single-mode detection. And a multi-dimensional hazard grade evaluation model is established, so that the automation level of dam termite prevention and control is greatly improved.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and Internet of Things (IoT) technology, specifically to a method for recognizing termite images in dams based on multimodal data fusion and dynamic feature enhancement, applicable to intelligent monitoring and early warning of termite activity in water conservancy projects. Background Technology

[0002] Termites are a major biological threat to water conservancy projects. Their long-term, covert activity can easily damage the dam structure and even lead to dam failure. Traditional monitoring methods mainly rely on manual inspections, using visual observation and sound analysis to determine signs of termite activity. However, this method has significant limitations: 1. Inefficient and lacking in timeliness: Manual inspection has a limited coverage area and relies on the experience of inspectors, making it difficult to achieve real-time monitoring of the entire dam area, especially making it difficult to detect hidden features such as early tiny ant trails and shallow nests. 2. Poor environmental adaptability: Single visual recognition models (such as traditional YOLO, Faster R-CNN, etc.) are easily affected by changes in lighting, vegetation occlusion, and soil color interference in complex outdoor environments, leading to termites being confused with background colors (such as light-colored termites against a sandy background, and reflective noise in humid environments), with a false detection rate as high as 18%-25%; 3. Lack of multi-dimensional data fusion: Existing technologies are mostly based on independent analysis of single-modal data (such as images or temperature and humidity sensors), lacking the ability to jointly model "image visual features + environmental parameters + termite dynamic behavior", and cannot comprehensively assess the correlation between termite activity intensity (such as nest construction trajectory and colony movement pattern) and environmental factors (such as abnormal temperature and humidity, soil vibration signals), resulting in an early warning accuracy rate of less than 75%. Summary of the Invention

[0003] The technical problem to be solved by the present invention is that the existing monitoring methods are inefficient, lack timeliness, have poor environmental adaptability, and lack multi-dimensional data fusion. In order to solve the above problems, an image recognition method for dam termites based on multimodal fusion and dynamic feature enhancement is provided.

[0004] The object of this invention is achieved in the following manner: A method for termite image recognition on dams based on multimodal fusion and dynamic feature enhancement, the method comprising the following steps: Step 1: Real-time acquisition of image data, environmental parameters, and vibration signals of the dam monitoring area using a multimodal sensor network; Step 2: Preprocess the image data and perform target detection using the improved ACP-YOLOv5s model; Step 3: Use Gaussian background modeling and difference algorithm to process continuous video frames to generate a binary mask image representing the dynamic activity of termites, and separate the moving target area from the static background; Step 4: Fuse the target detection results from Step 2 with the dynamic mask image from Step 3, and combine them with the environmental parameters and vibration signals collected in Step 1. Calculate the termite damage level using the AI ​​engine, triggering a graded early warning or automatic pest control response.

[0005] The multimodal sensor network includes: High-resolution cameras are used to capture visible light and infrared images; Temperature and humidity sensor, used to collect ambient temperature and humidity; Soil vibration sensors are used to collect vibration signals with a frequency of 5-15Hz, corresponding to the frequency band of termite activity. Edge computing nodes are used to perform spatiotemporal alignment and preprocessing of multi-source data and transmit it to the cloud via wireless communication networks.

[0006] The preprocessing in step 2 includes: Resize and normalize pixel values ​​of image data; The sensor data is Z-score normalized to form a uniform input vector.

[0007] The improved ACP-YOLOv5s model embeds an adaptive color perception module (ACP-Module) and a CARFE deformable upsampling module. The ACP-Module dynamically separates the similar color distributions of termites and the background through a color gamut clustering algorithm, enhancing the feature response of low-contrast targets. The CARFE module expands the receptive field through deformable convolution, strengthening the feature extraction of small targets and blurred edges.

[0008] The Gaussian background modeling and difference algorithm described in step 3 specifically includes: A Gaussian mixture model (GMM) is used to model the dynamic background of consecutive video frames, generating background images. ; The current frame is calculated using inter-frame difference. The difference between the image and the background image is segmented into a binary mask image M using Otsu thresholding. t Mark the target area for movement: .

[0009] The AI ​​engine fusion decision in step 4 specifically includes: Data-level fusion: The target detection results are concatenated with sensor data to form a multi-dimensional feature vector; the target detection results include the number of targets, confidence level, and area of ​​the active region; the sensor data includes temperature and humidity, vibration signal amplitude, and frequency data. Feature-level fusion: Learn the nonlinear correlation of multimodal features through a fully connected neural network to extract key features (such as the correlation between ant colony density and vibration frequency). Decision-level fusion: Based on a rule engine and a random forest classifier, it outputs the hazard level as low, medium, or high and triggers the corresponding response.

[0010] The rule engine includes: triggering a high-level alert when continuous activity lasts for more than 30 minutes and humidity is >90%; triggering a medium-level alert when ≥5 active targets are detected for 10 consecutive minutes and humidity is >85%; and triggering a basic alert when ≥3 active targets are detected for 5 consecutive minutes and humidity is ≥85%.

[0011] The corresponding responses are divided into primary warning, intermediate warning, and advanced warning. The primary warning pushes real-time monitoring data to management personnel via SMS and APP, along with the location coordinates of the activity area. The intermediate warning automatically retrieves historical 72-hour data for the area and generates trend analysis reports, such as temperature and humidity change curves and ant colony activity frequency, to assist manual judgment. The advanced warning triggers linkage equipment to execute disposal measures, such as remotely controlling pest control devices or atomizing sprayers to perform targeted spraying on the target area; it also activates an encrypted monitoring mode, increases the video acquisition frame rate to 30fps, and the sensor sampling frequency to 20Hz, to track the disposal effect in real time.

[0012] The beneficial effects of this invention are as follows: This invention constructs a multimodal data fusion architecture of "image vision + environmental parameters + vibration signal," breaking through the limitations of traditional single-modal detection. It captures visual features such as termite morphology and nest structure in real time using a high-resolution camera, and combines this with environmental factors collected by temperature and humidity sensors and soil vibration sensors to form a multi-dimensional data input system. By integrating dynamic feature extraction technology and adaptive color perception technology, it achieves accurate separation of dynamic targets from static backgrounds, addressing the challenges of lighting changes, color confusion, and static environmental noise in complex dam environments. This fundamentally solves the problem of high false negative rates in traditional methods for early, subtle signs of activity.

[0013] A target detection model adapted to dam environments, the ACP-YOLOv5s, is proposed, achieving performance breakthroughs through two core modules. The Adaptive Color Perception Module (ACP-Module) is embedded in the backbone network, dynamically separating the similar color distributions of termites from complex backgrounds based on a color gamut clustering algorithm, enhancing the robustness of target color under different lighting conditions. The Cascaded Deformable Feature Enhancement (CARFE) technique is integrated into the neck network, dynamically expanding the receptive field through deformable convolutions to strengthen the feature extraction capabilities for small targets (such as individual worker ants) and blurred edges (such as mud lines). In practical tests, the model achieves an average detection accuracy (mAP) of 92.3% in complex scenes, a 6.7% improvement over the traditional YOLOv5s model, with a reduction of over 40% in the false negative rate for low-contrast targets.

[0014] A closed-loop system of "data fusion - risk assessment - automatic response" is constructed. Through an AI engine, image recognition results (target location, density, activity trajectory) are deeply integrated with sensor data (temperature and humidity anomaly thresholds, vibration signal frequency characteristics) to establish a multi-dimensional hazard level assessment model. For different risk levels (e.g., triggering an advanced warning when continuous activity exceeds 30 minutes and humidity > 90%), the system can automatically link with pest control equipment (such as directional spraying devices), realizing a shift from "manual inspection - passive response" to "intelligent monitoring - proactive treatment." Compared to traditional solutions, this reduces manual intervention by 90%, significantly improving the automation level of termite control on dams. Attached Figure Description

[0015] Figure 1 This is the system architecture diagram of the present invention.

[0016] Figure 2 This is a structural diagram of the ACP-YOLOv5s model of the present invention.

[0017] Figure 3 This is a flowchart of the dynamic feature extraction process of the present invention. Detailed Implementation

[0018] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0019] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used herein have the same technical meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0020] The present invention provides a method for termite image recognition of dams based on multimodal fusion and dynamic feature enhancement, the method comprising the following steps: Step 1: Real-time acquisition of image data, environmental parameters, and vibration signals of the dam monitoring area using a multimodal sensor network; the image data includes visible light images and infrared images, the environmental parameters include temperature, humidity, and soil moisture content, and the vibration signals are used to characterize the weak vibrations generated by ant colony activity; Step 2: Preprocess the image data and perform target detection using the improved ACP-YOLOv5s model; Step 3: Use Gaussian background modeling and difference algorithm to process continuous video frames to generate a binary mask image representing the dynamic activity of termites, and separate the moving target area from the static background; Step 4: Fuse the target detection results from Step 2 with the dynamic mask image from Step 3, and combine them with the environmental parameters and vibration signals collected in Step 1. Calculate the termite damage level using the AI ​​engine, triggering a graded early warning or automatic pest control response.

[0021] The multimodal sensor network includes: constructing a multimodal sensor array in the dam monitoring area to achieve real-time acquisition of multi-dimensional data. Visual data acquisition: Deploy visible light cameras (resolution ≥1920×1080, frame rate 15-30fps) and infrared thermal imaging cameras to cover the surface of the dam and areas prone to ant trails (such as back slopes and cracks), and support simultaneous acquisition of RGB and infrared dual-spectrum images. Environmental parameter sensing: It integrates temperature and humidity sensors (accuracy ±2% RH, ±0.5℃), soil vibration sensors (frequency response 5-15Hz, used to capture weak vibrations generated by ant colony activities) and soil moisture sensors to collect environmental data in real time at a frequency of 10Hz. Edge computing nodes: Low-power embedded devices (such as NVIDIA Jetson AGX Orin) are deployed at the monitoring site to perform spatiotemporal alignment (based on millisecond-level timestamp synchronization) and noise reduction processing (image denoising and sensor outlier filtering) on ​​the raw data, and transmit it to the cloud server via 4G / 5G network.

[0022] The preprocessing in step 2 includes: Image data is normalized (pixel values ​​are normalized to [0,1]) and resized (input size 640×640). Sensor data is Z-score normalized to form a uniform input vector. ,in The images are visible light and infrared, with T, H, S, and V representing temperature, humidity, soil moisture content, and vibration signal characteristic values, respectively.

[0023] The improved ACP-YOLOv5s model incorporates two core modules to enhance features. The Adaptive Color Perception Module (ACP-Module) introduces a color gamut separation branch into the backbone network. It learns the color distribution of termites and the background using K-means clustering (RGB color gamut cluster centers ≤ 3 classes), generating a pixel-wise color weight matrix to enhance the feature response of low-contrast targets (color difference ≤ 15 gray levels). The formula is as follows: Where F is the original feature map, W color The color weight matrix is ​​used to suppress interference from similar color gamuts in the background through an attention mechanism. The CARFE deformable upsampling module replaces the traditional upsampling layer with cascaded deformable convolutions in the neck network, dynamically expanding the receptive field to the target region, enhancing feature extraction for small targets (worker ant individual size ≤ 8×8 pixels) and blurred edges, and improving multi-scale target detection capabilities.

[0024] The Gaussian background modeling and difference algorithm described in step 3 specifically includes: A Gaussian mixture model (GMM) is used to model the background of consecutive video frames, generating dynamic background images. The current frame is calculated using an inter-frame difference algorithm. The difference between the background model and the scene threshold segmentation (Otsu algorithm) generates a binary mask image. Mark the target area of ​​movement (such as ant colony movement trajectory, mud line construction behavior): This step effectively filters out static interference (such as fixed stones and vegetation) while preserving the spatiotemporal characteristics of dynamic targets.

[0025] Target detection process: After preprocessing (normalization and data augmentation) of the input image, the target bounding box (including category, confidence, and location coordinates) is output by the ACP-YOLOv5s model. Combined with the dynamic mask map Mt, the real termite activity area is selected, and static false detection targets (such as stones and dead leaves) are suppressed. The processing speed of a single frame is controlled within ≤30ms to meet the real-time monitoring requirements.

[0026] The AI ​​engine fusion decision in step 4 specifically includes: Data-level fusion: Integrating target detection results (number of targets N, average confidence level C, activity area A) with sensor data (real-time temperature and humidity T, vibration signal amplitude V) amp Frequency V freq ) concatenate to form feature vector F fusion =[N,C,A,T,H,V amp、 V freq ].

[0027] Feature-level fusion: Learn the non-linear relationship between features through a fully connected neural network (FCN, 128 hidden layer nodes) to extract key related features (such as the positive correlation between ant colony density and vibration frequency, and the probability of nest construction in a high humidity environment). Decision-level fusion: A decision-making model combining a rule engine and machine learning is adopted. Basic rules such as "detecting N≥3 active targets for 5 consecutive minutes and H≥85%" trigger a primary warning. The machine learning model is trained on a hazard level classifier based on the random forest algorithm. The input is the fused feature vector, and the output is the risk level (low, medium, high). The criteria for determining the high risk level include "the target trajectory shows clustering (trajectory overlap ≥60%) and the vibration signal frequency is concentrated in 5-15Hz (the frequency band characteristic of termite activity)".

[0028] The rule engine includes: triggering a high-level alert when continuous activity lasts for more than 30 minutes and humidity is >90%; triggering a medium-level alert when ≥5 active targets are detected for 10 consecutive minutes and humidity is >85%; and triggering a basic alert when ≥3 active targets are detected for 5 consecutive minutes and humidity is ≥85%.

[0029] Based on the decision-making results, the response is tiered into primary, intermediate, and advanced warnings. Primary warnings push real-time monitoring data to management personnel via SMS and an app, along with the location coordinates of the activity area. Intermediate warnings automatically retrieve 72 hours of historical data for the area, generating trend analysis reports (such as temperature and humidity change curves and ant colony activity frequency) to assist manual judgment. Advanced warnings trigger coordinated equipment to implement control measures, such as remotely controlling pest control devices (e.g., atomized spraying equipment) to perform targeted spraying of the target area; and activating an encrypted monitoring mode (increasing the video capture frame rate to 30fps and the sensor sampling frequency to 20Hz) to track the effectiveness of the control measures in real time.

[0030] The system implementation architecture of this invention: The hardware layer consists of front-end sensor nodes (cameras, sensors), edge computing devices (data preprocessing), and cloud servers (model training and decision-making), supporting solar power and wireless transmission (LoRa / NB-IoT). The software layer includes a data acquisition module (Python multi-threaded real-time data acquisition), a feature processing module (ACP-YOLOv5s model inference implemented based on PyTorch), and a fusion decision-making module (Java-developed rule engine and API interface), supporting cross-platform deployment and remote operation and maintenance.

[0031] This invention pioneers a multimodal fusion dynamic perception architecture: A multimodal data fusion architecture integrating "image vision + environmental parameters + vibration signals" is constructed to overcome the limitations of traditional single-modal detection. High-resolution cameras capture real-time visual features such as termite morphology and nest structure, while temperature and humidity sensors and soil vibration sensors collect environmental triggers (such as sudden humidity changes and regular vibration signals), forming a multi-dimensional data input system. Dynamic feature extraction technology (Gaussian background modeling and difference algorithm) and adaptive color perception technology (ACP-Module) are integrated to address complex dam scenarios involving lighting changes, color confusion (such as low-contrast interference between light-colored termites and sandy soil backgrounds), and static environmental noise. This achieves accurate separation of dynamic targets (termite colony activity trajectories and nest construction behavior) from the static background, fundamentally solving the problem of high false negative rates in traditional methods for early, subtle activity signs.

[0032] This invention employs an improved ACP-YOLOv5s detection model: A target detection model adapted to dam environments, the ACP-YOLOv5s, is proposed, achieving performance breakthroughs through two core modules. The Adaptive Color Perception Module (ACP-Module) is embedded in the backbone network, dynamically separating similar color distributions between termites and complex backgrounds based on a color gamut clustering algorithm (e.g., solving the challenge of identifying low-contrast targets with color differences ≤15 gray levels), enhancing the color robustness of targets under different lighting conditions (strong light reflection, weak light shadow). The Cascaded Deformable Feature Enhancement (CARFE) technique is integrated into the neck network, dynamically expanding the receptive field through deformable convolutions to strengthen the feature extraction capabilities for small targets (such as individual worker ants) and blurred edges (such as mud lines). In practical tests, the model achieves an average detection accuracy (mAP) of 92.3% in complex scenes, a 6.7% improvement over the traditional YOLOv5s model, with a reduction of over 40% in the false negative rate for low-contrast targets.

[0033] This invention relates to a multimodal collaborative decision-making and intelligent response mechanism: A closed-loop system of "data fusion - risk assessment - automatic response" is constructed. Through an AI engine, image recognition results (target location, density, activity trajectory) are deeply integrated with sensor data (temperature and humidity anomaly thresholds, vibration signal frequency characteristics) to establish a multi-dimensional hazard level assessment model. For different risk levels (e.g., triggering an advanced warning when continuous activity exceeds 30 minutes and humidity > 90%), the system can automatically link with pest control equipment (such as directional spraying devices), realizing a shift from "manual inspection - passive response" to "intelligent monitoring - proactive treatment." Compared to traditional solutions, this reduces manual intervention by 90%, significantly improving the automation level of termite control on dams.

[0034] The technical solution of this invention has been verified for its practical application value through engineering practice: 1. Reliable performance in complex environments In a pilot water conservancy project in XX province, 137 termite nests were successfully identified, of which 42% were early-stage shallow nests (depth < 30cm). Traditional manual inspections had a miss rate of up to 60% for these nests. However, this invention effectively captured subtle signs of activity through dynamic masking technology (Gaussian modeling + differential algorithm), improving the early detection rate of potential hazards by more than 40% and achieving an early warning accuracy rate of 91.2%, demonstrating its engineering applicability in complex outdoor scenarios.

[0035] 2. Edge computing and distributed deployment capabilities The processing time for a single frame image is ≤30ms. It supports real-time operation on low-power edge devices (such as the NVIDIA Jetson series). With the solar power supply and wireless transmission solution (battery life ≥72 hours), it can work stably for a long time in remote dams without human intervention. It solves the problems of traditional monitoring equipment relying on mains power and high deployment costs, and meets the monitoring needs of water conservancy projects with wide distribution and harsh environments.

[0036] 3. Potential for technology transfer and industrial application The core technologies (multimodal fusion framework and dynamic feature enhancement algorithm) have good scalability. By adjusting the sensor type and model parameters, they can be quickly adapted to fields such as termite monitoring of ancient buildings, identification of agricultural and forestry pests, and detection of pests in storage facilities, forming a cross-industry intelligent monitoring solution with significant industrial promotion value and economic benefits.

[0037] The above description is only a preferred embodiment of the present invention. It should be noted that those skilled in the art can make several changes and improvements without departing from the overall concept of the present invention, and these should also be considered within the scope of protection of the present invention.

Claims

1. A dam termite image recognition method based on multi-modal fusion and dynamic feature enhancement, characterized in that: The method comprises the following steps: Step 1: Real-time acquisition of image data, environmental parameters and vibration signals of the dam monitoring area by a multi-modal sensor network; Step 2: Preprocessing of the image data, target detection by an improved ACP-YOLOv5s model; Step 3: Processing of consecutive video frames by Gaussian background modeling and difference algorithm to generate a binary mask graph representing termite dynamic activity, separating the moving target area from the static background; Step 4: Fusion of the target detection results of Step 2 and the dynamic mask graph of Step 3, combined with the environmental parameters and vibration signals collected in Step 1, calculation of the termite damage level by an AI engine, triggering a graded warning or automatic killing response.

2. The dam termite image recognition method based on multi-modal fusion and dynamic feature enhancement according to claim 1, characterized in that: The multi-modal sensor network comprises: High-resolution cameras for acquiring visible light images and infrared images; Temperature and humidity sensors for acquiring environmental temperature and humidity; Soil vibration sensors for acquiring vibration signals with a frequency of 5-15 Hz, corresponding to the characteristic frequency band of termite activity; Edge computing nodes for spatio-temporal alignment and preprocessing of multi-source data, and transmission to the cloud through a wireless communication network.

3. The dam termite image recognition method based on multi-modal fusion and dynamic feature enhancement according to claim 1, characterized in that: The preprocessing in Step 2 includes: Size adjustment and pixel value normalization of the image data; Z-score standardization of the sensor data to form a uniform format input vector.

4. The dam termite image recognition method based on multi-modal fusion and dynamic feature enhancement according to claim 1, characterized in that: The improved ACP-YOLOv5s model embeds an adaptive color perception module ACP-Module and a CARFE deformable upsampling module; ACP-Module dynamically separates the similar color distribution of termites and background through color cluster algorithm, enhancing the feature response of low-contrast targets; CARFE module expands the receptive field through deformable convolution, strengthening the feature extraction of small targets and fuzzy edges.

5. The dam termite image recognition method based on multi-modal fusion and dynamic feature enhancement according to claim 1, characterized in that: The Gaussian background modeling and difference algorithm in Step 3 specifically includes: A Gaussian Mixture Model (GMM) is used to model the dynamic background of consecutive video frames to generate a background image ; The current frame is calculated by inter-frame difference The difference from the background image is binarized into a mask map M by Otsu thresholding t Marking the moving object region: .

6. The dam termite image recognition method based on multi-modal fusion and dynamic feature enhancement according to claim 1, characterized in that: The AI engine fusion decision in Step 4 specifically includes: Data-level fusion: concatenating the target detection results and sensor data into a multi-dimensional feature vector; the target detection results include target number, confidence, and active area; the sensor data includes temperature and humidity, vibration signal amplitude and frequency data; Feature-level fusion: learning the non-linear correlation of multi-modal features through a fully connected neural network, extracting key features (such as the correlation between termite density and vibration frequency); Decision-level fusion: based on rule engine and random forest classifier, outputting damage level: low, medium, high, and triggering corresponding response.

7. The dam termite image recognition method based on multi-modal fusion and dynamic feature enhancement according to claim 6, characterized in that: The rule engine includes: continuous activity for more than 30 minutes and humidity > 90% triggers high-level warning, detection of ≥5 active targets for 10 consecutive minutes and humidity > 85% triggers medium-level warning, detection of ≥3 active targets for 5 consecutive minutes and humidity ≥85% triggers primary warning.

8. The dam termite image recognition method based on multi-modal fusion and dynamic feature enhancement according to claim 1, characterized in that: The corresponding response is divided into primary early warning, medium early warning and high early warning; the primary early warning monitors data to the management personnel in real time through short message and APP push, with the activity area positioning coordinates; the medium early warning automatically calls the historical 72-hour data of the area, generates a trend analysis report, such as a temperature and humidity change curve and an ant colony activity frequency, and assists manual research and judgment; the high early warning triggers a linkage device to execute a disposal measure, such as remotely controlling a killing device, such as a atomizing pesticide spraying device, to perform directional pesticide spraying on a target area; an encryption monitoring mode is started, the video acquisition frame rate is increased to 30fps, the sensor sampling frequency is increased to 20Hz, and the disposal effect is tracked in real time.

Images