A multi-modal large model-based granary pest intelligent detection method and system

Abstract

This invention discloses an intelligent detection method and system for grain storage pests based on a multimodal large-scale model. The method employs a hierarchical cascade architecture of "lightweight model for coarse detection + large-scale model for fine identification." First, a lightweight model such as YOLO is used for rapid initial detection. When the confidence level is low, a multimodal large-scale model is triggered for precise identification. The large-scale model receives images and structured text prompts, outputs pest species, morphological descriptions, and confidence level assessments, and possesses self-reflection capabilities, enabling it to identify uncertain results and trigger manual verification. The system continuously optimizes through an active learning mechanism, supporting rapid adaptation to new pests with limited samples. Compared to traditional methods, this invention improves accuracy by 7.5%, reduces false detection rate by 60%, and maintains controllable computational costs, demonstrating significant practical value.

Description

Technical Field

[0001] This invention relates to the fields of intelligent agriculture and computer vision technology, and in particular to a hierarchical cascade detection method and system for grain storage pests that combines a lightweight target detection model with a multimodal large language model. Background Technology

[0002] During grain storage, pest infestation can lead to significant grain losses. Traditional pest detection relies on regular manual inspections, with workers examining samples visually or with magnifying glasses. This method is inefficient; manual inspections cannot cover the entire grain silo and are susceptible to fatigue and experience differences, leading to the omission of small or deep-seated pests. Furthermore, frequent entry into the grain silo can affect the stability of the storage environment. In recent years, with the development of deep learning technology, deep learning-based target detection techniques (such as the YOLO series models) have been applied to pest identification. These methods, through training deep neural networks, can automatically detect pest targets in images. However, in practical applications, these methods still have the following problems:

[0003] 1. Insufficient small target detection capability: Grain warehouse pests are small in size, usually between 2-8mm, and are easily masked by noise in complex backgrounds. In particular, when pests are obscured by grain particles, traditional models have difficulty in accurately identifying them, resulting in a high false negative rate.

[0004] 2. High false detection rate: There are many interfering objects in the grain warehouse environment that are similar to pests, such as grain debris and impurities. These are similar to pests in terms of color, size, and shape, and traditional models are prone to misidentification.

[0005] 3. Difficulty in identifying new pests: Traditional deep learning models require a large amount of labeled data for training. For rare or new pests, there is a lack of sufficient samples, which makes it impossible for the model to identify them accurately. In addition, a lot of time and manpower are needed to collect data and retrain the model.

[0006] 4. Lack of semantic understanding: Traditional models can only output bounding boxes and category labels, and cannot make comprehensive judgments by combining environmental information (such as food type, temperature and humidity, light, etc.).

[0007] 5. Weak uncertainty handling capability: Traditional models lack a self-evaluation mechanism, cannot determine whether results with medium confidence levels (such as 0.5-0.7) are reliable, and cannot trigger manual verification, which may lead to false alarms or missed alarms.

[0008] In recent years, multimodal large language models have provided new solutions. These models can process images and text simultaneously and possess powerful visual understanding, semantic reasoning, and few-shot learning capabilities. However, these models suffer from slow inference speed and high computational cost in real-time pest detection, necessitating improved solutions to balance speed, accuracy, and cost.

[0009] Therefore, a novel method is needed that combines rapid detection with lightweight models with accurate identification with large models, which can improve identification accuracy, reduce false positives and false negatives, and quickly adapt to new pests while ensuring speed and controlling computational costs. Summary of the Invention

[0010] The purpose of this invention is to provide a method and system for intelligent detection of grain pests based on a multimodal large model, which can significantly improve the accuracy of pest identification, reduce false detection and false negative rates, and has the ability to quickly adapt to new pests while ensuring detection speed.

[0011] Technical solution:

[0012] To achieve the above-mentioned objectives, this invention first provides an intelligent detection method for grain storage pests based on a multimodal large model, comprising the following steps:

[0013] (1) A lightweight target detection model is used to quickly perform preliminary detection on the grain warehouse image, and the candidate target bounding box, first-stage category and first-stage confidence are output;

[0014] (2) When the confidence level in the first stage is lower than the first threshold, the region of interest (ROI) of the corresponding candidate target is extracted, and the multimodal large model is triggered; otherwise, the preliminary detection result is output as the final detection result.

[0015] (3) The multimodal large model receives visual input ROI and structured text prompts; the structured text includes at least task instructions and output format constraints, and includes key knowledge points about pests;

[0016] (4) Trigger the local deployment of a multimodal large model to reason about the input and output structured results, including at least the pest species, morphological description, second-stage confidence and uncertainty assessment;

[0017] (5) When the confidence level of the second stage is higher than the second threshold, the structured result is used as the final detection result; when the confidence level of the second stage is lower than the second threshold, a self-reflection mechanism is triggered, and the corresponding labeled sample is marked as pending manual confirmation and pushed to the user interaction terminal.

[0018] (6) Based on human feedback and historical data, dynamically adjust the detection threshold and write manually confirmed samples and labeling information into the pest knowledge base.

[0019] Preferably, the lightweight object detection model is a YOLO series model; the model training includes the following key improvements:

[0020] (1) Network framework adjustment: P2 detection head is used for small-scale target detection. The P2 detection head corresponds to a feature map with 4 times downsampling, which has higher spatial resolution than the traditional detection head;

[0021] (2) Feature selection optimization: By using the shallow feature map of the P2 layer to retain more spatial details and texture information, it is possible to identify subtle differences such as the beak features of the rice weevil and the antennae morphology of the grain beetle;

[0022] (3) Key parameter settings: Data augmentation includes operations such as random flipping and random scaling; Test-time augmentation (TTA) technology is used, including horizontal flipping, vertical flipping, and multi-scale transformation (0.8×, 1.0×, 1.2×), and the weighted nonmaximum suppression algorithm is used to fuse multiple prediction results;

[0023] (4) Slice-assisted reasoning: The high-resolution image is sliced ​​into overlapping slices of 640×640 pixels using SAHI technology.

[0024] Preferably, the multimodal large model is a vision-language pre-trained model, capable of simultaneously processing image and text inputs and outputting structured JSON format results; the system deploys a localized inference framework on the grain warehouse field server to run the open-source multimodal large model, thereby achieving offline inference; the structured JSON format results include at least one of the following: pest detection markers, pest species, life stage, quantity, morphological description, confidence level, key feature objects, uncertainty markers, uncertainty causes, and suggested measures fields, for subsequent decision-making and manual confirmation.

[0025] Preferably, the structured text prompts include at least one of the following: environmental description, task instructions, few-sample examples, and domain knowledge prompts; the environmental description information includes at least one of the following: grain type, sampling time, ambient temperature, ambient humidity, and grain pile surface condition; the task instructions explicitly specify the recognition task and output format requirements of the multimodal large model; the few-sample examples include at least one reference image and its corresponding text annotation; the domain knowledge prompts include distinguishing features between pests and interfering objects and key points for morphological recognition.

[0026] Preferably, the self-reflection mechanism includes: generating a confidence report from a large model; the confidence report includes at least one of the following: a location description of the identified ambiguous areas, the results of feature conflict analysis, an assessment of the impact of environmental factors, and improvement suggestions.

[0027] Preferably, the intelligent detection method further includes active learning, wherein the active learning triggering conditions include: the confidence level of the large model is lower than the second threshold, the first stage category is inconsistent with the second stage category, and an unknown pest type is detected; the active learning involves manually confirming or correcting samples and annotation information, and storing the information in a pest knowledge base for learning with few samples; after a preset number of new samples are stored in the pest knowledge base, the lightweight target detection model is fine-tuned.

[0028] Preferably, the method further includes an environmental context awareness step: using a large model to identify grain type, lighting conditions, and grain pile surface condition, and combining historical detection data for correlation reasoning, and dynamically adjusting the detection strategy according to environmental characteristics; the dynamically adjusted detection strategy includes at least one of the following: adjusting the classification confidence threshold for different pest species according to grain type, adjusting the false detection filtering threshold according to lighting conditions, and adding targeted domain knowledge to the structured text prompts according to the grain pile surface condition.

[0029] Preferably, the dynamic threshold adjustment strategy is as follows: while ensuring that the detection accuracy is not lower than the preset value, the large model call rate is minimized to reduce the computational cost; the dynamic adjustment is based on the statistical analysis of historical detection data, including the accuracy statistics of samples in different confidence intervals in the first stage, the large model call rate statistics in the second stage, and the accuracy statistics of manually confirmed samples.

[0030] To achieve the above detection method, the present invention also provides an intelligent detection system for grain warehouse pests. The system adopts an edge-cloud collaborative architecture and includes five main components: a data acquisition module, an edge computing module, a local intelligent analysis module, a user interaction module, and an adaptive optimization module.

[0031] 1. Data Acquisition Module

[0032] The data acquisition module is responsible for acquiring images of the grain silo and uploading them to the backend system. This module consists of three parts: the detection rod hardware, the timed sampling mechanism, and the data upload interface.

[0033] The detection rod hardware integrates a UV insect-attracting lamp, a supplementary light, and a high-definition camera. The UV lamp (365nm, 5W) attracts phototactic pests, the supplementary light (6000K, 3W) provides uniform illumination, and the high-definition camera (12MP) supports macro photography, capable of capturing details of pests longer than 2mm. The detection rod features a waterproof and dustproof design, suitable for the high humidity and dusty environments of grain warehouses. Timed sampling mechanism: Allows setting sampling intervals (e.g., every 2 hours, daily), automatically triggering image acquisition as needed. Each sampling can capture multiple photos from different angles (e.g., 5 photos), and the sampling frequency can be adjusted according to the season. Data upload interface: Supports multiple communication methods such as 4G, 5G, and WiFi, adapting to different network environments. Images are uploaded to edge computing devices via encrypted transmission, ensuring data security.

[0034] 2. Edge computing module

[0035] Edge computing modules are deployed on or near edge devices at the grain depot site to perform the first phase of rapid initial inspection. Commonly used edge devices include NVIDIA Jetson Nano (4GB RAM) and Raspberry Pi 4B, which have GPU acceleration capabilities and can run lightweight YOLO models in real time.

[0036] This module receives images uploaded by the data acquisition module, performs preprocessing (denoising, enhancement, ROI extraction), and then calls a locally deployed lightweight YOLO model for rapid detection. For detection results with a confidence level higher than the threshold θ1, the edge computing module directly outputs and pushes them to the user interaction module; for detection results with a confidence level lower than θ1, the module will crop the ROI and perform accurate identification through the localized large model intelligent analysis module.

[0037] 3. Local Intelligent Analysis Module

[0038] The local intelligent analysis module is deployed on the grain depot's on-site server, responsible for performing the second stage of accurate identification and the fourth stage of environmental context enhancement. This module adopts an edge-local collaborative architecture, bringing AI inference capabilities down to the grain depot site for fully offline operation. The module comprises three components: a multimodal large-model inference engine, a pest knowledge base, and historical data management.

[0039] Localized deployment solution based on the Ollam framework

[0040] To address the specific needs of real-world grain depot applications, the system adopts a localized large-scale model deployment solution based on the Ollam framework. Ollam is a lightweight, open-source large-scale model runtime framework specifically optimized for local deployment, enabling efficient execution of large-scale visual and language models on ordinary servers. Its core advantages are:

[0041] (1) Simplified deployment process: Ollam provides a unified command-line interface and RESTful API, supporting one-click download, loading and running of various open-source visual models without the need for complex Python environment configuration, dependency management or model format conversion. System administrators can complete model deployment simply by executing simple commands.

[0042] (2) Intelligent Resource Management: Ollam has a built-in model quantization engine that supports INT8 and INT4 quantization technologies. Through quantization, large models with 70B parameters can be compressed to 25-50% of their original size, significantly reducing memory usage. For example, the original Qwen2-VL-72B model requires about 144GB of video memory, but after INT4 quantization, it only requires 48GB, which can run smoothly on a single A100 80GB GPU.

[0043] (3) Completely offline operation: All model files, inference engine and dependent components are stored and run entirely locally without any internet connection. This is crucial for grain depot scenarios, as many grain depots are located in remote areas with unstable network conditions or are prohibited from connecting to the external network for security reasons.

[0044] (4) Low latency response: Local inference completely eliminates network transmission latency, with typical inference latency between 1 and 3 seconds. Compared to cloud solutions that require uploading images to the cloud, waiting for cloud processing, and downloading results, the local solution offers a 3-5 times faster response time, making it more suitable for pest monitoring scenarios that require real-time feedback.

[0045] (5) Data security guarantee: All grain depot images and inspection data are stored entirely on local servers and will not be uploaded to any third-party servers, fundamentally eliminating the risk of data leakage. This is especially important for grain reserves that are related to national food security.

[0046] Supported open-source large vision models:

[0047] The system supports a variety of mainstream open-source vision models, which can be flexibly selected according to actual hardware conditions, performance requirements, and language preferences.

[0048] Qwen2-VL series (Ali Tongyi Qianwen): With parameter scales of 2B, 7B and 72B, it performs excellently in Chinese understanding and fine-grained visual recognition, and is particularly suitable for processing Chinese pest names, morphological descriptions and professional terms.

[0049] Llama 3.2-Vision series (Meta): With parameter scales of 11B and 90B, it has powerful multimodal reasoning capabilities, performs well in few-shot learning scenarios, and is particularly suitable for rapid adaptation to new pests.

[0050] The Mistral-Small series features approximately 22 bytes of parameters, providing excellent visual understanding capabilities while maintaining a small model size. It also boasts fast inference speed (approximately 1.5 seconds per inference) and low power consumption.

[0051] Gemma2 series (Google): It has two parameter sizes: 9B and 27B. It adopts an efficient model architecture design and its inference speed is 20-30% faster than models with the same parameter size under the same hardware conditions.

[0052] Multimodal large model inference engine:

[0053] The inference engine is the core component of the local intelligent analysis module, responsible for inference using large models running within the Ollam framework. The engine's main functions include:

[0054] (1) Model loading and management: When the system starts, it automatically loads the pre-configured large visual model (such as Qwen2-VL-72B) into the GPU memory. It supports hot switching of models, switching to different backup models without stopping the system.

[0055] (2) Request queue management: Receives ROI images and environmental information uploaded by the edge computing module and maintains a first-in-first-out request queue. Supports a priority mechanism, with urgent samples (such as suspected samples of new pests) being processed first.

[0056] (3) Prompt word construction: Structured prompt words are dynamically generated based on the detection scenario, environmental characteristics, and historical data. Prompt word templates are stored in a local database and support online editing and version management.

[0057] (4) Inference call: Send an inference request to the local large model through Ollama's RESTful API (http: / / localhost:11434 / api / generate), pass Base64 encoded images and text prompts, and obtain recognition results in JSON format.

[0058] (5) Result parsing and verification: Parse the JSON results returned by the large model and verify the integrity of the fields and the validity of the data. Perform legality checks on key fields such as confidence level and pest species to ensure that the results are usable.

[0059] (6) Batch processing optimization: Supports batch inference, packaging multiple ROIs to be detected into batches and using the parallel computing power of GPUs to process them simultaneously, thereby improving throughput. In a dual-GPU configuration, pipeline parallelism can be achieved, with one GPU processing the current batch while the other GPU processes the next batch.

[0060] (7) Performance monitoring: Real-time recording of performance indicators such as inference latency, GPU utilization, and video memory usage. When a performance bottleneck is detected, an alarm is issued to assist in system optimization.

[0061] Pest Knowledge Base:

[0062] The pest knowledge base is a structured local database (using SQLite or PostgreSQL) that stores multi-dimensional information about pests. The data structure of the knowledge base includes:

[0063] (1) Basic information table: Stores the Chinese name, Latin name, other names, and classification status (order, family, genus) of pests.

[0064] (2) Morphological characteristics table: Record in detail the body length range, body color, body shape, special structure (such as the beak of the rice weevil and the serrated teeth of the sawtooth beetle) and other visual characteristics of the pests. This information is used to construct domain knowledge prompts.

[0065] (3) Life cycle table: Records the developmental stages of pests (egg, larva, pupa, adult), duration of each stage, reproductive cycle, etc., for time series data correlation analysis.

[0066] (4) Harm characteristics table: Record the harm level of the pest, host preference (e.g., rice weevil prefers rice and wheat, corn weevil prefers corn), harm symptoms, etc.

[0067] (5) Reference Image Library: Stores standard reference images (multiple angles, multiple life stages) for each pest, used for few-sample learning scenarios. Each image is accompanied by detailed text annotations.

[0068] (6) List of Easily Confused Species: Record other pests or disturbances that are easily confused with the target pest (such as grain beetles and grain debris), as well as the key points of differentiation, to build targeted domain knowledge tips.

[0069] The knowledge base supports CRUD operations via a management interface. When the active learning mechanism discovers new pests and completes manual annotation, the relevant information is automatically added to the knowledge base. During inference, the large model queries the knowledge base to obtain relevant domain knowledge and embeds it into prompts to assist in recognition and decision-making.

[0070] Historical data management:

[0071] The historical data management module stores and analyzes all detection records, and supports the following functions:

[0072] (1) Data storage: The complete information of each detection (sampling time, location coordinates, original image, ROI image, first stage result, second stage result, environmental conditions, processing measures, etc.) will be persistently stored in the local database and file system.

[0073] (2) Time series analysis: Provides time series analysis function to statistically analyze the changing trend of pest numbers and the evolution of pest species distribution in a specific location or region. Supports the generation of trend curves and forecast reports.

[0074] (3) Related queries: Supports complex multi-dimensional related queries, such as "the detection rate of rice weevils when the temperature is higher than 28℃ and the humidity is higher than 70% in the past 30 days", providing data support for environmental condition optimization.

[0075] (4) Data export: Supports exporting historical data to CSV, Excel or PDF formats, which is convenient for administrators to perform offline analysis or report to superiors.

[0076] (5) Data backup: Regularly and automatically back up the database and image files to backup storage devices or remote servers to prevent data loss.

[0077] 4. User Interaction Module

[0078] The user interaction module provides a management platform (mini-program or web application) for grain depot administrators to view test results, perform manual labeling, receive early warning notifications, and view analysis reports. The management platform includes the following functions:

[0079] Detection results are displayed in the form of image annotations, including the original image, detection box, pest species, quantity, confidence level, morphological description and other information. It supports filtering and querying by time, location and pest species.

[0080] Pest density assessment: Calculate the pest density at each sampling point (e.g., the number of pests per kilogram of grain) and display the distribution of pest density in different areas of the grain warehouse using heat maps or bar charts to help identify key areas for prevention and control.

[0081] Trend prediction: Based on historical data and pest reproduction patterns, predict future trends in pest numbers, provide early warnings of potential pest outbreaks, and offer a basis for prevention and control decisions.

[0082] Manual annotation interface: For samples that trigger manual confirmation, administrators can view detailed information (original image, ROI image, detection results, confidence report, etc.) and confirm or correct the annotations. After annotation is completed, the samples are automatically...

[0083] Warning notification: When the pest density exceeds the threshold, new pests are detected, or the results are abnormal, the system will automatically send a warning notification, which can be sent via SMS, email, mini-program push, and other methods.

[0084] Analysis Report: The system regularly generates pest detection and analysis reports, which include statistics on the number of pests detected, distribution of pest species, changes in pest density, evaluation of control effectiveness, system performance indicators, etc., providing data support.

[0085] 5. Adaptive Optimization Module

[0086] The adaptive optimization module is responsible for monitoring the detection performance at each stage of the system, automatically adjusting threshold parameters, and generating model optimization suggestion reports. This module includes three functions: performance monitoring, adaptive threshold adjustment, and optimization suggestion generation.

[0087] Performance monitoring: Real-time collection of system operation data, such as detection count, accuracy, recall, F1 score, inference time, large model call rate, and manual intervention rate, is displayed through a visual dashboard. An anomaly detection mechanism will issue alerts when metrics are abnormal.

[0088] Threshold adaptive adjustment: The confidence threshold for the first stage is dynamically adjusted based on historical detection data and human feedback. Second-stage confidence threshold The goal is to minimize the call rate of large models and reduce computational costs while ensuring that the accuracy is not lower than a preset value (e.g., 95%). Threshold adjustments are performed automatically every week and recorded in the system log.

[0089] Optimization suggestion generation: Based on performance monitoring data and threshold adjustment results, a model optimization suggestion report is generated. The report includes: a current system performance assessment, identifying bottlenecks and areas for improvement; and threshold adjustment suggestions, such as optimization... , The report includes model fine-tuning suggestions, such as recommending fine-tuning the YOLO model when new samples are added to the knowledge base; and data collection suggestions, such as adjusting the sampling frequency or lighting conditions in certain areas. These optimization suggestions help administrators and technicians continuously improve system performance.

[0090] Key technological innovations

[0091] 1. Hierarchical Cascaded Detection Architecture

[0092] This paper proposes a novel two-stage cascaded detection scheme: a lightweight model for coarse detection and a large model for fine recognition. The first stage uses a lightweight YOLO model for rapid initial detection, capable of real-time operation on edge devices with an inference time of less than 100ms per image. The second stage employs a locally deployed multimodal visual language model based on the Ollam framework for accurate recognition, capable of handling complex scenes and challenging samples. Through a confidence-driven dynamic triggering mechanism, the system determines whether to directly output the result or call the large model for secondary verification based on the detection confidence of the first stage, achieving a balance between speed and accuracy.

[0093] 2. Multimodal input design

[0094] In addition to input images, the system incorporates environmental descriptions, task instructions, few-shot examples, and domain knowledge hints to construct structured prompt word templates for grain warehouse pest detection. Environmental descriptions include information such as grain type, temperature and humidity, light conditions, and the surface condition of the grain pile, helping the large-scale model understand the detection scenario. Task instructions clearly inform the model of the tasks to be completed and the output format. Domain knowledge hints provide key points for pest identification, such as distinguishing features between different pests and interfering objects, and key points for identifying dead pests. Few-shot examples provide reference images when detecting new pests, enabling the large-scale model to quickly learn the features of new pests. This multimodal input design fully leverages the large-scale model's visual understanding, semantic reasoning, and few-shot learning capabilities, significantly improving recognition accuracy.

[0095] 3. Self-reflection and proactive learning mechanism

[0096] The large model output includes not only detection results (pest species, quantity, and confidence level), but also confidence level assessment and uncertainty analysis. Confidence level assessment identifies ambiguous areas, analyzes feature conflicts, evaluates the impact of environmental factors, and provides suggestions. If the detection result is uncertain, the system automatically triggers an active learning mechanism, marking the sample as "awaiting manual confirmation" and pushing it to the administrator. After marking, the sample enters the knowledge base for few-sample learning and model fine-tuning. Through human-machine collaboration, the system continuously optimizes without retraining the model. In a real-world deployment at a grain depot, the system identified the new pest "millet beetle" within one hour using the active learning mechanism, while traditional methods required 2-4 weeks.

[0097] 4. Contextual Awareness

[0098] Leveraging the scene understanding capabilities of large-scale models, the system identifies environmental features in images, such as grain type, grain pile surface condition, and lighting quality, and performs correlation reasoning by combining historical detection data. The system dynamically adjusts its detection strategy based on environmental characteristics; for example, it increases sensitivity to corn weevils in corn silos or adds domain knowledge hints when damaged grains are present on the grain pile surface. The time-series data correlation function analyzes changes in pest numbers, species consistency, and environmental conditions to determine the reasonableness of the current results. If the current results significantly deviate from the expected trend, the system marks them as "abnormal and requires manual verification." Through environmental context awareness, the system adapts to different grain silo environments, improving detection accuracy and robustness. After enabling environmental context enhancement, the false detection rate decreased from 5.2% to 1.8%.

[0099] 5. Knowledge-enhanced reasoning

[0100] Knowledge in the field of entomology is encoded into a structured knowledge base, storing information such as the morphological characteristics, life cycle, damage level, host preferences, and easily confused species of common pests. The large-scale model retrieves relevant domain knowledge from the knowledge base to assist in identification and decision-making. The knowledge base supports dynamic updates, and an active learning mechanism can add manually labeled new pest samples to the knowledge base, enriching its content. Knowledge-enhanced reasoning enables the large-scale model to utilize domain-specific knowledge, improving identification accuracy, particularly in distinguishing similar pests from interfering species.

[0101] Compared with existing technologies, the present invention has the following significant advantages:

[0102] 1. High detection accuracy: The two-stage architecture combines the speed of a lightweight model with the accuracy of a large model, achieving an overall accuracy of over 95%.

[0103] 2. Low false positive rate: The semantic understanding ability of the large model can effectively distinguish between pests and similar interference objects (grain debris).

[0104] 3. Rapid adaptation with few samples: Through a few-sample learning mechanism, new pests can be identified with only 2-5 samples, without the need for retraining;

[0105] 4. Controllable computational cost: Lightweight models handle most simple scenarios, while large models are only called when necessary. For example, in Example 1, only 103 large model inferences are required per 100 images on average.

[0106] 5. High interpretability: The output of the large model includes morphological description, confidence assessment, and uncertainty analysis, which facilitates administrator understanding and decision-making;

[0107] 6. Continuous optimization capability: Through an active learning mechanism, the system can continuously optimize based on feedback from actual use, without the need for manual retraining;

[0108] 7. Good environmental adaptability: The context-aware mechanism enables the system to adapt to different grain types, light conditions, seasonal changes, etc. Attached Figure Description

[0109] Figure 1: Schematic diagram of the overall system architecture.

[0110] Figure 2: Flowchart of hierarchical cascaded detection.

[0111] Figure 3: Input-output structure diagram of multimodal large model.

[0112] Figure 4: Flowchart of self-reflection and proactive learning mechanism.

[0113] Figure 5: Schematic diagram of environmental context awareness.

[0114] Figure 6: An example of entering the second stage of inputting into the large model, including the cut roi and prompt, in an embodiment of localized pest detection based on the Ollam framework and Qwen2-VL.

[0115] Figure 7: Output of the large model in an example of localized pest detection based on the Ollam framework and Qwen2-VL.

[0116] Figure 8: In an example of few-shot learning to deal with new pests, relevant information about the new pests is added to the input of the large model when faced with new pests.

[0117] Figure 9: In an example of learning from few samples to deal with new pests, the large model successfully identified and output the results after learning the cue words.

[0118] Figure 10: In an embodiment of the environmental context-aware optimization, an enhanced multimodal input is constructed to address the confusion between grain beetles and grain debris.

[0119] Figure 11: In the embodiment of the environmental context-aware optimization, the large model successfully identified the output as grain debris to address the confusion between grain beetles and grain debris.

[0120] Figure 12: In the implementation of the environmental context-aware optimization, the large model successfully identified the grain beetle as the grain beetle in the output, addressing the confusion between grain beetle and grain debris.

[0121] Figure 13: In an embodiment of the environmental context-aware optimization, an enhanced multimodal input is constructed to address the confusion between round dark grains and bean weevils / grain thieves.

[0122] Figure 14: In the embodiment of the environmental context-aware optimization, the large model successfully identified the output as black beans to address the confusion between round dark grains and bean weevils / grain thieves. Detailed Implementation

[0123] This invention provides an intelligent detection method for grain storage pests based on a multimodal large model, combining... Figure 1 and Figure 2 This method employs a hierarchical cascaded detection architecture, comprising four stages:

[0124] Phase 1: Rapid Initial Detection Based on a Lightweight Model

[0125] The purpose of the first stage is to perform a rapid preliminary detection on the grain warehouse images, screen out potential pest targets, and decide whether to output the results directly or proceed to the second stage for precise identification based on the detection confidence level.

[0126] 1. Image Acquisition and Preprocessing

[0127] This method uses a specially designed detection device to acquire images of the grain silo's interior. The device is equipped with an ultraviolet (UV) insect-attracting lamp and a supplemental light, enabling it to acquire clear images under varying lighting conditions. The UV lamp attracts phototactic pests, increasing their probability of appearance; the supplemental light provides uniform illumination in low-light conditions, avoiding the influence of shadows and highlights on detection. The device is also equipped with a high-resolution camera that supports macro photography, capable of capturing details of pests larger than 2mm.

[0128] The acquired images undergo a series of preprocessing steps, including:

[0129] Denoising: Use Gaussian filtering or bilateral filtering to remove image noise;

[0130] Enhancement processing: Improve image sharpness through histogram equalization or contrast enhancement;

[0131] Region extraction: Use edge detection or saliency detection algorithms to extract regions of interest, remove irrelevant background, and focus on areas where pests may appear.

[0132] These preprocessing steps can effectively improve the accuracy and efficiency of subsequent detection.

[0133] 2. Lightweight target detection

[0134] In the first stage, a lightweight object detection model is used to quickly detect objects in the preprocessed images. Lightweight versions of the YOLO series, such as YOLOv8-nano or YOLOv11-nano, are preferred. These models have fewer than 5 million parameters, can run in real-time on edge computing devices, and have an inference time of less than 100 milliseconds per image. The model is fine-tuned on a self-built grain storage pest dataset, which contains images of common pests and distractions. Through fine-tuning, the model learns specific features of grain storage pests, improving detection accuracy.

[0135] The model outputs detection results for each image, including:

[0136] (1) Bounding box coordinates This indicates the location of the detected target in the image;

[0137] (2) Preliminary category label, indicating the pest species to which the target is most likely to belong;

[0138] (3) Confidence score , which represents the model's confidence level in the detection result, and its value ranges from 0 to 1.

[0139] 3. Initial screening

[0140] Based on the set first-stage confidence threshold If the confidence level in the first stage exceeds the threshold, it indicates that the lightweight model has high confidence in the results, and the system directly outputs the detection results. If the confidence level is below the threshold, it indicates that the model has uncertainty about the results (such as small targets, occlusions, or similar targets with interference), and the system then enters the second stage for precise identification. Through this confidence-based dynamic triggering mechanism, the system can achieve a balance between speed and accuracy by calling a large model for precise identification of difficult samples while maintaining speed.

[0141] Phase Two: Accurate Identification Based on Multimodal Large Models

[0142] The purpose of the second stage is to accurately identify the difficult samples with low confidence in the first stage, and to improve the detection accuracy and reduce the false positive and false negative rates by leveraging the powerful visual understanding and semantic reasoning capabilities of the multimodal large model.

[0143] 4. Region of Interest Enhancement Processing

[0144] For detection boxes with a confidence level below the threshold in the first stage, the system first performs region of interest (ROI) enhancement processing to improve image quality and provide clearer input for larger models. The processing steps are as follows:

[0145] Cropping ROI: Based on the bounding box coordinates output from the first stage, the target region (ROI) is cropped from the original image. To preserve contextual information, the margin is extended outward by 10-30% from the bounding box during cropping. For example, when the detection box width is 100 pixels, the cropping region width is extended to 120-160 pixels to preserve contextual information such as grain morphology and lighting conditions.

[0146] Super-resolution algorithm enhancement: Deep learning-based super-resolution algorithms (such as ESRGAN, Real-ESRGAN, etc.) are used to enlarge ROI images, increasing the image resolution to 512×512 pixels or higher. This algorithm can restore image details, making features such as the legs, antennae, and body texture of pests clearer.

[0147] Extracting local contextual information: The system analyzes environmental features such as grain morphology and lighting conditions around the ROI image and encodes this information into structured text, which serves as one of the inputs to the large model to help improve the accuracy of precise recognition.

[0148] 5. Multimodal large-scale model inference

[0149] Figure 3This section demonstrates the input structure of the multimodal large model (detailed structures of visual and text inputs), the model processing (visual encoding, text encoding, cross-modal fusion), and the output structure (the various fields in JSON format and their meanings). This step is the core of the second stage, constructing multimodal inputs and calling the large model for inference, outputting the pest's species, morphological description, confidence level, and uncertainty assessment. The input of the multimodal large model includes two parts: visual input and text input. The visual input is the enhanced ROI image from step 4, transmitted in Base64 encoded format. The text input is structured prompts, including the following components:

[0150] (1) Role definition: Inform the big model of its role, such as "You are a grain warehouse pest identification expert with knowledge of pest morphology and ecology", to help activate the big model's relevant domain knowledge.

[0151] (2) Environment description: Provide background information, such as grain type, sampling time, ambient temperature and humidity and grain pile surface condition, to help large models combine environmental context for reasoning.

[0152] (3) Task instructions: Clearly tell the large model the task to be completed, such as "identify the types of pests in the image, output the pest name, life stage (adult / larva), quantity and confidence level", and require the output format (such as JSON format).

[0153] (4) Domain knowledge prompts: Provide key points and domain knowledge for pest identification to help large models focus on key features for judgment. For the identification of dead pests, the prompts can emphasize that "the legs and antennae of dead pests may be detached, but body segmentation structure, appendage connection traces or special structures can still be observed".

[0154] (5) Few-sample examples (optional): When a new or rare pest is detected, the system can extract 2-3 reference images from the pest knowledge base and input them as few-sample examples into the large model. Each reference image is accompanied by a brief text description. By providing few-sample examples, the large model can quickly learn the characteristics of the new pest and achieve the recognition of the new pest without retraining the model.

[0155] Large model outputs structured results:

[0156] After performing inference on the multimodal input, the large model outputs a structured JSON format result, including the following fields:

[0157] Fields Data types must illustrate For example pest_detected Boolean value yes Indicates whether pests have been detected. true / false pest_type String yes Pest types Rice weevil / Grain beetle life_stage String no Pest life stages Adult / larvae count Integer no Number of pests 2 morphology String no Morphological characteristics of pests Body length approximately 3mm, dark brown description String no Describe in detail the features observed by the large model The target is cylindrical and approximately 2.5 mm long. confidence floating-point numbers yes Confidence of the large model on the recognition results 0.92 key_features object no Key characteristics of pests, such as whether they have legs or antennae. {has_legs: true, has_antennae: true} uncertainty_flag Boolean value no Is there uncertainty? true / false uncertainty_reason String no Reasons for uncertainty The target is obscured, making it impossible to observe the appendages. suggestion String no Model Recommendations Multi-angle sampling is recommended.

[0158] 6. Outcome Integration and Decision Making

[0159] The system outputs confidence scores based on the large model. Decisions are made based on the uncertainty flag (uncertainty_flag). A second-stage confidence threshold is set. (Typical value is 0.85), if Furthermore, since uncertainty_flag is false, it indicates that the large model has high confidence in the recognition results, and the system uses the output of the large model as the final detection result.

[0160] if If uncertainty_flag is true, it indicates that the large model has uncertainty about the recognition results. The system then triggers the third-stage self-reflection mechanism to further evaluate the results and decide whether manual intervention is needed.

[0161] Phase Three: Self-Reflection and Active Learning

[0162] The purpose of the third stage is to conduct self-evaluation of uncertain detection results, trigger manual intervention for annotation, and continuously optimize system performance through an active learning mechanism. Figure 4 This demonstrates the complete closed-loop process of uncertainty assessment, active learning triggering, manual annotation, knowledge base updating, and model optimization, as well as the triggering conditions and processing logic for each stage. Specifically, it includes:

[0163] 7. Uncertainty Assessment

[0164] When the confidence level of the second-stage test results is low or uncertainty markers are present, the large model generates a confidence report to analyze the reliability of the test results. The confidence report includes the following:

[0165] (1) Identify blurry areas: The large model indicates which areas in the image have unclear features, making it impossible to make an accurate judgment. For example, "the target's legs are completely covered by grains, and the appendage features cannot be observed" and "the image is unevenly lit, the target is in a shadow area, and the color and texture cannot be accurately judged".

[0166] (2) Feature conflict analysis: The large model analysis is used to determine whether there are contradictions in the observed features. For example, "The target's body size is consistent with that of the maize weevil, but its color is lighter and no typical beak-like protrusions are observed" and "The target partially matches the features of both the maize weevil and the corn weevil, and cannot be clearly distinguished".

[0167] (3) Environmental factors: The large model analyzes the impact of environmental factors on the identification results. For example, "There are a lot of broken corn kernels around, the target may be grain debris" and "Bean weevils have never been detected in this area before, but a lot of suspected bean weevil targets have suddenly appeared, which may be black beans".

[0168] (4) Recommended measures: The large model provides recommended measures based on the causes of uncertainty. For example, "It is recommended to sample from multiple angles and observe the side and ventral features of the target", "It is recommended to increase the light intensity and reacquire the image", and "It is recommended to examine it manually with a microscope".

[0169] The confidence report is generated by prompting the large model to perform uncertainty analysis. For example, the task instruction might include, "If uncertain, please explain why, including which features are unclear, which features are contradictory, the influence of environmental factors, and recommended measures." The large model then uses its language generation capabilities to output detailed uncertainty analysis results.

[0170] 8. Active learning trigger

[0171] The system automatically marks a sample as "awaiting manual confirmation" and triggers the active learning mechanism when one of the following conditions is met:

[0172] (1) Confidence of large models 2: This indicates that the large model lacks confidence in the recognition results.

[0173] (2) There is a contradiction between the results of the first and second stages: for example, the first stage classifies the target as rice elephant, but the second stage judges it as grain debris, which is obviously contradictory.

[0174] (3) Unknown pest type detected: The large model judges it as a pest, but cannot classify it into a known pest type.

[0175] (4) uncertainty_flag is true: The large model explicitly marks the existence of uncertainty.

[0176] Samples marked as "awaiting manual confirmation" are pushed to the administrator via the management terminal (mini-program or web platform). The administrator can view the images, detection results, and the large model confidence report, and annotate them based on professional knowledge, confirming or correcting information such as pest species, life stage, and quantity. After annotation, the samples and annotation information are added to the pest knowledge base for subsequent few-sample learning to improve recognition accuracy. Once enough new pest samples have accumulated in the knowledge base, the system can fine-tune the lightweight YOLO model. Through an active learning mechanism, the system achieves human-machine collaboration, combining automatic recognition with manual confirmation, significantly reducing system maintenance workload and enabling continuous optimization.

[0177] 9. Adaptive threshold adjustment

[0178] The system dynamically adjusts the confidence threshold for the first stage based on historical testing data and human feedback. 1. Second-stage confidence threshold The goal of adaptive threshold adjustment is to minimize the large model call rate and reduce computational costs while ensuring that the detection accuracy is not lower than a preset value (such as 95%).

[0179] The system automatically analyzes the detection logs weekly and compiles the following statistics:

[0180] (1) The proportion that was finally confirmed to be correct among the samples that passed directly in the first stage.

[0181] (2) The accuracy of samples with different confidence intervals (such as 0.6-0.7, 0.7-0.8) in the first stage after confirmation in the second stage.

[0182] (3) The call rate and accuracy of the large model in the second stage.

[0183] (4) The number of samples that trigger manual verification and the accuracy of manual annotation.

[0184] (5) Key performance indicators such as overall system accuracy, false detection rate, and false negative rate.

[0185] Based on these statistics, the system uses the following strategy to adjust the threshold:

[0186] If the accuracy of samples within a certain confidence interval (e.g., 0.7-0.8) in the first stage reaches over 95% after confirmation in the second stage, it indicates that the sample quality in that interval is high, and the accuracy can be appropriately increased. This allows these samples to pass directly through the first stage, reducing the need for large model calls. For example, the accuracy rate for samples in the 0.7-0.8 range is 95%, while the accuracy rate for the 0.8-0.85 range is 99%. The system can then... The value increased from 0.7 to 0.75, with more samples passing the first phase.

[0187] If the overall accuracy of the system is lower than the preset value (e.g., 95%), it indicates that the threshold setting is too lenient and needs to be lowered. or improve This increases the proportion of large models being called, thereby improving detection accuracy.

[0188] If the large model's call rate is too high (e.g., exceeding 30%), resulting in excessively high computational costs or slow detection speeds, the speed can be appropriately increased. Reduce the number of large model calls, but monitor the accuracy to ensure it is not lower than the preset value.

[0189] Phase 4: Enhanced Contextual Environment

[0190] The fourth stage is an optional enhancement that leverages the scene understanding capabilities of large models to identify environmental features and combine them with historical data for correlation reasoning, thereby further improving the accuracy and robustness of detection.

[0191] 10. Environmental Feature Extraction

[0192] The large model can not only identify pests but also environmental features in images. When constructing multimodal inputs, the system can request the large model to output environmental feature information, including:

[0193] (1) Grain type: Identify the types of grains in the image (such as wheat, corn, etc.). Different grain types are inhabited by different pests. Identifying grain types helps to narrow down the range of pest species and improve the accuracy of identification.

[0194] (2) Illumination quality: Evaluate the image illumination conditions, such as shadows, highlights, or uneven illumination. Illumination quality directly affects the detection results; insufficient or uneven illumination may lead to unclear features.

[0195] The method for extracting environmental features is to explicitly request the large model to output environmental information in the prompt words.

[0196] 11. Contextual Reasoning

[0197] The system dynamically adjusts its detection strategy based on extracted environmental features, achieving self-adaptation to different grain storage environments. Specific strategies include:

[0198] (1) Grain type association: In the corn granary, the system increases the sensitivity to corn weevil and decreases the sensitivity to rice weevil; in the wheat granary, the system increases the sensitivity to rice weevil and grain borer. This strategy is based on the host preference of pests and reduces false detections.

[0199] (2) Lighting condition adaptation: Under low light conditions, the system increases the false detection filtering threshold and strictly screens to avoid false identification of shadows; under high light conditions, the threshold is reduced to enhance detection sensitivity and avoid missed detection.

[0200] (3) Adaptation to grain pile condition: If there are a large number of damaged grains on the surface of the grain pile, the system adds targeted domain knowledge prompts to the prompt words, emphasizing the difference between damaged grains and pests, and helping the large model avoid false detection.

[0201] By dynamically incorporating environmental features and knowledge cues into the prompts, the system enables context-based reasoning. For example, when an image comes from a corn silo and has damaged corn kernels on the surface, the prompt could include: "Background information: The grain type is corn, and there are damaged corn kernels on the surface of the grain pile. Damaged corn kernels are easily confused with corn weevils, but corn weevils have obvious legs and antennae, while damaged grain kernels have no leg structure."

[0202] 12. Time-series data correlation

[0203] The system retrieves historical detection records, analyzes the results of the previous N samplings (N is typically 5-10), and performs time-series correlation analysis in conjunction with the current detection results. The large model judges the reasonableness of the current results, mainly focusing on the following aspects:

[0204] (1) Pest population trend: Analyze whether the pest population is in line with the expected growth or decline trend. For example, if historical data shows that the pest population is gradually increasing, it is reasonable to detect more pests at present; if historical data shows that the pest population is decreasing (possibly due to control measures), it is also reasonable to detect fewer pests at present. If the current result is seriously inconsistent with the expected trend (such as a sudden detection of a large number of new pests), it is marked as "abnormal and requires manual verification".

[0205] (2) Consistency of pest species: Analyze whether the currently detected pest species are consistent with historical records. If rice weevils were mainly detected in a certain area in the past, and a large number of maize weevils are suddenly detected now, there may be false detections or a new outbreak of pests, which need to be further confirmed.

[0206] (3) Environmental condition correlation: Analyze whether changes in environmental conditions (such as temperature, humidity, and season) reasonably explain changes in the number or types of pests. For example, it is reasonable to detect more pests when the temperature rises in summer, as pests reproduce faster; it is reasonable to detect fewer pests when the temperature drops in winter, as pest activity weakens.

[0207] Time-series data correlation is achieved by incorporating historical detection records into the prompts. For example, the background information might include: "Historical detection records: In the past 7 days, the number of rice weevils detected at this location each day was 5, 8, 12, 15, 20, 28, and 35, showing an exponential growth trend. Currently, 45 rice weevils have been detected, consistent with the expected growth trend." The large model analyzes the time-series information to determine the reasonableness of the current result and adjusts the confidence level accordingly. If the current result matches the expected trend, the large model increases the confidence level; if it is abnormal, it decreases the confidence level and marks it as requiring manual verification.

[0208] By enhancing the environmental context, the system can gain a more comprehensive understanding of the detection scenario and make a comprehensive judgment by combining environmental features and historical data, which significantly improves the accuracy and robustness of the detection.

[0209] Example 1: Localized pest detection based on the Ollama framework and Qwen2-VL

[0210] As shown in Figure 1, this task is mainly divided into three stages: rapid coarse detection using a lightweight model, accurate identification of large multimodal models, and an adaptive optimization module. The detailed configuration is as follows:

[0211] Hardware configuration:

[0212] The detection rod features an integrated design, equipped with a 365nm ultraviolet LED (5W, wavelength 365±5nm) to attract phototactic pests, and a 6000K white light supplemental lamp (3W) to provide uniform illumination. The camera uses a Sony IMX477 sensor (12MP, 1 / 2.3-inch, supports autofocus), supporting macro shooting (minimum focal length 3cm), and can clearly capture details of pests larger than 2mm. The detection rod's casing has an IP65 protection rating, making it suitable for the high humidity (up to 80%) and high dust environments of grain silos.

[0213] The edge computing device uses the NVIDIA Jetson Nano Developer Kit (4GB RAM, 128-core Maxwell GPU), is small in size (69.6mm × 45mm), and has low power consumption (5W-10W), making it suitable for long-term stable operation in grain silos. The device connects to a camera via a USB 3.0 interface and to a local server via Gigabit Ethernet.

[0214] Software configuration:

[0215] The first-stage model uses YOLOv8-nano with approximately 3.2 million parameters and a model file size of 6MB, capable of real-time execution on Jetson Nano. The model was fine-tuned on a self-built grain storage pest dataset containing eight common pests (such as rice weevils, maize weevils, and grain borers) and 5000 negative sample images. The model was trained for 100 epochs using the AdamW optimizer with a learning rate of 0.001 and a batch size of 16. Data augmentations included random flipping and random scaling. After training, the model achieved an mAP@0.5 of 89.3% on the test set, with a single-image inference time of approximately 80ms.

[0216] Confidence threshold in the first stage The confidence level was set to 0.7. Based on the analysis of the validation set data, the detection accuracy was 89% for confidence levels ≥ 0.7 and 65% for confidence levels below 0.7. Therefore, setting 0.7 as the cutoff point effectively filters out difficult samples that require the intervention of a large model.

[0217] The second stage uses the locally deployed Qwen2-VL-72B-Instruct model (INT4 quantization version). After loading, the model resides in GPU memory, occupying approximately 48GB. Inference is performed via Ollama's RESTful API (http: / / localhost:11434 / api / generate), with the parameter temperature set to 0.2 to ensure output stability. The confidence threshold for the second stage is... The value was set to 0.85. Based on the output characteristics of Qwen2-VL-72B-Instruct, the accuracy is about 97% when the confidence level is ≥0.85 and about 82% when it is below 0.85. Therefore, 0.85 was chosen as the threshold for manual verification to ensure that the overall accuracy reaches more than 95%.

[0218] The testing process is shown in Figure 2, and is as follows:

[0219] 1. The probe automatically samples every 2 hours, taking 5 photos from different angles each time. During the shooting process, the ultraviolet insect-attracting lamp and the supplementary light are turned on simultaneously, and the exposure time is automatically adjusted to obtain a clear image.

[0220] 2. After shooting, the image is transferred to Jetson Nano via USB 3.0. Jetson Nano preprocesses the image: first, it uses Gaussian filtering (kernel size 5×5, standard deviation 1.0) to remove noise; then, it uses the CLAHE algorithm to enhance contrast (clipLimit=2.0, tileGridSize=8×8); finally, it uses edge detection to extract the region of interest and remove the background and probe parts.

[0221] 3. YOLOv8-nano was used to detect preprocessed images on a Jetson Nano. The model outputs bounding boxes, categories, and confidence scores. The average processing time per image is 80ms, and it can detect 0-20 potential pest targets (depending on pest density). The coordinate format of the bounding boxes is as follows: The category label is the type of pest (such as "rice weevil", "grain beetle", etc.), and the confidence level is a floating-point number between 0 and 1.

[0222] 4. Regarding confidence level The system directly marks the test result of 0.7 (accounting for approximately 80%) as "Phase 1 Passed" and uploads it to the local server via TCP / IP protocol. The server then pushes it to the user interaction module for display.

[0223] 5. Regarding confidence level For the detection results of 0.7 (approximately 20% of the total detections), Jetson Nano performs the following processing:

[0224] (1) Clip the ROI according to the coordinates of the detection box and extend the margin outside the box by 20% to preserve the context information;

[0225] (2) Use the Real-ESRGAN model to perform super-resolution processing on the ROI, and increase the image resolution to 512×512 pixels;

[0226] (3) Analyze the environmental characteristics around the ROI (such as food type and light conditions) and encode them into text descriptions;

[0227] (4) Package the enhanced image and environment description into JSON format and upload it to the local server via HTTPS protocol.

[0228] 6. The local server calls the Qwen2-VL model running on the Ollam framework. The server constructs a multimodal input, as shown in Figure 3, including:

[0229] (1) Visual input: Base64 encoded ROI image;

[0230] (2) Text input: Structured prompts, including role definitions, environment descriptions, task instructions, domain knowledge prompts, etc. See Figure 6 for specific prompt examples.

[0231] 7. The Qwen2-VL model processes the request and returns the result in JSON format, as shown in the right half of Figure 3. A typical example of the returned result is shown in Figure 7.

[0232] 8. The local server parses and verifies the return results of Qwen2-VL, proceeding to the third stage. The detailed process is shown in Figure 4. First, it checks if the JSON format is correct, ensuring that all required fields are complete. Then, it makes a decision based on the confidence level and the uncertainty flag.

[0233] (1) If Furthermore, since uncertainty_flag is false, the system marks it as "Phase Two Passed" and pushes the result to the user interaction module for display.

[0234] (2) If If uncertainty_flag is true, the system marks it as "awaiting manual confirmation" and pushes information such as the original image of the sample, ROI image, first-stage results, and second-stage recognition results to the administrator via a mini-program.

[0235] 9. After receiving the notification in the mini-program, the administrator can view and annotate the sample details, and select the following actions:

[0236] (1) Confirm that the AI ​​recognition result is correct;

[0237] (2) Correct the AI ​​recognition results and select the correct pest species;

[0238] (3) Marked as grain debris or other interfering materials;

[0239] (4) Supplement morphological feature descriptions and suggested measures. After annotation, the samples and annotation information are entered into the pest knowledge base for subsequent few-sample learning and model fine-tuning.

[0240] Test results:

[0241] In a real-world deployment at a national grain reserve, the system ran continuously for 30 days, and the test results are as follows:

[0242] Total number of images detected: 3600 (an average of 120 images per day)

[0243] Number of potential pest targets detected: 18,500

[0244] Phase 1 passed directly: 14,800 (80%), average confidence level 0.85, accuracy 89.3%.

[0245] Second-stage large-scale model intervention: 3700 cases (20%), average confidence level 0.91, accuracy 97.2%.

[0246] Human verification was triggered in 185 cases (1%). After manual annotation, 18 AI errors were found, resulting in a human correction accuracy rate of 90.3%.

[0247] Overall accuracy: 96.8% (17,880 correct / 18,500 total tests)

[0248] False detection rate: 1.5% (the proportion of grain debris and other interfering objects that are mistaken for pests)

[0249] False negative rate: 1.7% (the proportion of pests that were not actually detected)

[0250] Average detection time: 80ms / image in the first stage, 2.3 seconds / target in the second stage, and an overall average of 250ms / image.

[0251] System availability: 99.9% (unaffected by network fluctuations, running completely offline)

[0252] Data security: All images and monitoring data are completely localized, with no risk of leakage.

[0253] Example 2: Learning from Few Samples to Cope with New Pests

[0254] A provincial grain reserve located in southern China primarily stores rice and wheat. In December 2025, while using the system of this invention for pest detection, the reserve discovered the locally rare "sawtooth grain beetle." The sawtooth grain beetle is a flat beetle, 2-3 mm in length, with hammer-shaped antennae, primarily damaging stored grains and flour. Because this pest is extremely rare in local grain reserves, neither the first-stage YOLO model nor the second-stage Qwen2-VL model of the system could accurately identify it. The YOLO model misclassified it as a grain borer (confidence 0.45), and the Qwen2-VL model classified it as an "unknown pest type" (confidence 0.52). In traditional methods, researchers would need to collect hundreds of sawtooth grain beetle images and retrain the YOLO model, taking several weeks. In this solution, the administrator uploads three standard images of the sawtooth grain beetle (from a pest atlas) to the mini-program; the system adds these three images to the "few sample examples" section of the prompt, as shown in the format of the left half of Figure 3, with the specific prompt in Figure 8.

[0255] When the probe captured images of sawgrass thieves again, the large model successfully identified them through few-sample learning, and the output of the large model is shown in Figure 9. After 10 successful identifications, the system accumulated enough sawgrass thief samples, allowing for selective fine-tuning of the YOLO model.

[0256] Effect:

[0257] From the discovery of a new pest to the system's ability to identify it, it takes only about 1 hour (compared to 2-4 weeks for traditional methods), with a few-shot recognition accuracy of 87% (compared to only 60% in the cold start phase of traditional methods). The key to few-shot learning lies in the design of the prompts. By explicitly stating in the prompts "Please refer to the following example image to identify the current image," and providing detailed morphological feature descriptions and key recognition points, Qwen2-VL can be guided to focus on key features (such as the serrated protrusions and hammer-shaped antennae of the sawtooth miller), improving recognition accuracy. In addition, providing example images from multiple angles (top view, side view, close-up) allows Qwen2-VL to understand target features from different perspectives, enhancing recognition robustness.

[0258] Example 3: Context-Aware Optimization

[0259] This embodiment demonstrates the environmental context awareness capability of the present invention. By combining environmental features and domain knowledge, the system can effectively distinguish pests from similar interfering objects, significantly reducing the false detection rate. The embodiment includes three typical scenarios: confusion between grain weevils and grain debris, confusion between bean weevils and black beans, and identification of dead pests.

[0260] Scenario A: The problem of confusing grain weevils with grain debris

[0261] A grain depot was storing wheat, and a large amount of broken wheat grains and bran were generated during mechanical handling. During the detection process, the system easily misidentified these grain fragments as grain beetles. Grain beetles are cylindrical beetles, 2-3 mm in length, and brownish in color, which are very similar in color and size to wheat fragments, making them difficult to distinguish against a complex background.

[0262] In the YOLO model detection results, 12 suspected cereal beetle targets were detected. After manual verification, 7 of them were indeed cereal beetles, and 5 were grain debris (false detection rate of 41.7%). The confidence level was... Below the threshold This triggers the second phase.

[0263] Phase Two: Multimodal Large Model Inference

[0264] If standard cue words are used for reasoning, Qwen2-VL still produces false positives, resulting in unsatisfactory performance. Therefore, we will construct an enhanced multimodal input (Figure 10).

[0265] Example of large model inference results:

[0266] Case 1: As shown in Figure 5, the large model successfully identified the negative sample that was originally identified as a grain beetle as grain debris. The identification result is shown in Figure 11.

[0267] Case 2: As shown in Figure 12, the large model was successfully identified as a grain beetle.

[0268] Scene B: Confusing round, dark grains with bean weevils / grain thieves

[0269] A grain depot stores mixed grains (wheat + black beans). The round, dark-colored black beans are easily mistaken for bean weevils or grain beetles. Black beans are approximately 5-8 mm in diameter, with a smooth surface, and are black or dark brown; bean weevils are 3-5 mm long, oval-shaped, and dark brown to black; grain beetles are 3-4 mm long, flat, and reddish-brown to dark brown. From a top-down view, they exhibit some similarities in appearance.

[0270] The YOLO model detected 18 suspected large grain beetle / bean weevil targets. After verification, 7 of them were found to be black beans (false detection rate of 38.9%).

[0271] Phase Two: Multimodal Large Model Inference

[0272] As shown in Figure 13, the system constructs targeted prompts emphasizing the distinguishing features between the bean weevil / grain thief and black beans. The large-scale model inference results are shown in Figure 14, successfully identifying it as a black bean.

Claims

1. A method for intelligent detection of grain storage pests based on a multimodal large model, characterized in that, Includes the following steps: (1) A lightweight target detection model is used to quickly perform preliminary detection on the grain warehouse image, and the candidate target bounding box, first-stage category and first-stage confidence are output; (2) When the confidence level in the first stage is lower than the first threshold, the region of interest (ROI) of the corresponding candidate target is extracted, and the multimodal large model is triggered; Conversely, the preliminary detection result is output as the final detection result; (3) The multimodal large model receives visual input ROI and structured text prompts; The structured text includes at least task instructions and output format constraints, and includes key knowledge points about pests; (4) Trigger the local deployment of a multimodal large model to reason about the input and output structured results, including at least the pest species, morphological description, second-stage confidence and uncertainty assessment; (5) When the confidence level of the second stage is higher than the second threshold, the structured result is used as the final detection result; When the confidence level in the second stage is lower than the second threshold, a self-reflection mechanism is triggered, and the corresponding labeled sample is marked as awaiting manual confirmation and pushed to the user interaction terminal. (6) Based on human feedback and historical data, dynamically adjust the detection threshold and write manually confirmed samples and labeling information into the pest knowledge base.

2. The method according to claim 1, characterized in that, The lightweight object detection model is a YOLO series model; the model training includes the following key improvements: (1) Network framework adjustment: P2 detector head is used for small-scale target detection, and the P2 detector head corresponds to a feature map downsampled by 4 times; (2) Feature selection optimization: By utilizing the shallow feature map of the P2 layer, more spatial details and texture information are preserved, which can identify subtle differences; (3) Key parameter settings: Data augmentation includes operations such as random flipping and random scaling; Test-time augmentation (TTA) technology is adopted, including horizontal flipping, vertical flipping, and multi-scale transformation, and the weighted nonmaximum suppression algorithm is used to fuse multiple prediction results; (4) Slice-assisted reasoning: The high-resolution image is sliced ​​into overlapping slices of 640×640 pixels using SAHI technology.

3. The method according to claim 1, characterized in that, The multimodal large model is a vision-language pre-trained model that can process image and text inputs simultaneously and output structured JSON format results. The system deploys a local inference framework on the grain warehouse field server to run an open-source multimodal large model, thereby achieving offline inference; the structured JSON format results include at least one of the following: pest detection markers, pest species, life stage, quantity, morphological description, confidence level, key feature objects, uncertainty markers, uncertainty causes, and suggested measures fields, for subsequent decision-making and manual confirmation.

4. The method according to claim 1, characterized in that, The structured text prompts include at least one of the following: environmental description, task instructions, few-sample examples, and domain knowledge prompts; the environmental description information includes at least one of the following: grain type, sampling time, ambient temperature, ambient humidity, and grain pile surface condition; the task instructions explicitly specify the recognition task and output format requirements of the multimodal large model; the few-sample examples include at least one reference image and its corresponding text annotation; the domain knowledge prompts include distinguishing features between pests and interfering objects and key points for morphological recognition.

5. The method according to claim 1, characterized in that, The self-reflection mechanism includes: a large model generating a confidence report; the confidence report includes at least one of the following: a location description of the identified ambiguous areas, the results of feature conflict analysis, an assessment of the impact of environmental factors, and improvement suggestions.

6. The method according to claim 1, characterized in that, The intelligent detection method also includes active learning. The active learning triggering conditions include: the confidence of the large model is lower than the second threshold, the first stage category is inconsistent with the second stage category, and an unknown pest type is detected. The active learning involves manually confirming or correcting samples and annotation information, and storing the information in a pest knowledge base for learning with a small number of samples. After storing a preset number of new samples in the pest knowledge base, the lightweight target detection model is fine-tuned.

7. The method according to claim 1, characterized in that, It also includes an environmental context awareness step: using a large model to identify grain type and lighting conditions, and combining historical detection data for correlation reasoning, and dynamically adjusting the detection strategy according to environmental characteristics; the dynamically adjusted detection strategy includes at least one of the following: adjusting the classification confidence threshold of different pest species according to grain type, adjusting the false detection filtering threshold according to lighting conditions, and adding targeted domain knowledge to the structured text prompts according to the surface state of the grain pile.

8. The method according to claim 1, characterized in that, The dynamic threshold adjustment strategy is as follows: while ensuring that the detection accuracy is not lower than the preset value, minimize the large model call rate and reduce the computational cost; the dynamic adjustment is based on the statistical analysis of historical detection data, including the accuracy statistics of samples in different confidence intervals in the first stage, the large model call rate statistics in the second stage, and the accuracy statistics of manually confirmed samples.

9. A smart detection system for grain warehouse pests based on a multimodal large model, characterized in that, include: (1) Data acquisition module: Equipped with a detection device with ultraviolet insect-attracting lamp and supplementary light lamp, used to acquire images of grain warehouses; (2) Edge computing module: Deploys a lightweight target detection model for rapid local detection; (3) Local intelligent analysis module: Locally deployed multimodal large models for accurate recognition and inference; the local intelligent analysis module is deployed on the grain warehouse field server, equipped with GPU computing resources, and supports model management, quantization inference, offline operation and concurrent processing functions; unified management of multiple large visual models, supporting online download and update of models, offline loading, switching and running; adopting INT8 or INT4 quantization technology to reduce memory usage and inference latency while maintaining recognition accuracy; supporting multi-GPU parallel inference, and can process multiple detection requests at the same time; (4) User interaction module: Provides interface for displaying test results and manual annotation; (5) Adaptive optimization module: monitors detection performance and dynamically adjusts threshold parameters.

10. The system according to claim 9, characterized in that, The local intelligent analysis module also includes a pest knowledge base, which stores domain knowledge such as the morphological characteristics, life cycle, and hazard level of common pests for large model retrieval and reasoning. The pest knowledge base is stored in a structured database and supports dynamic updates. An active learning mechanism can add manually labeled new pest sample information to the pest knowledge base.

Images