Greenhouse pest early warning and targeted spraying system based on multi-modal perception

Abstract

The application discloses a greenhouse disease and pest early warning and targeted pesticide spraying system based on multi-modal perception and belongs to the technical field of agricultural pesticide application. The system comprises a multi-modal perception module, a data preprocessing and registration module, a disease and pest identification and early warning module, a target area positioning and path planning module, a pesticide proportioning and dosage regulation module, a pesticide spraying execution and feedback module and a system general control module. The multi-modal perception module fuses visual, environmental and spectral perception data, and constructs a disease and pest characteristic tensor after data preprocessing and registration. The disease and pest identification and early warning module adopts a multi-branch fusion network to complete disease and pest type, grade and position identification, and generates early warning information in combination with growth cycle data. The application solves the problems of traditional greenhouse disease and pest monitoring lag, blind pesticide application, pesticide waste and environmental pollution and the like, improves disease and pest control efficiency and precision, and reduces agricultural production cost.

Description

Technical Field

[0001] This invention relates to the field of agricultural pesticide application technology, specifically to a greenhouse pest and disease early warning and targeted spraying system based on multimodal perception. Background Technology

[0002] Greenhouse cultivation, which allows for the artificial control of the growing environment and enables off-season planting with high and stable yields, has become an important planting model in modern agriculture. However, the enclosed, high-temperature, and high-humidity environment inside greenhouses makes them highly susceptible to pests and diseases, which spread rapidly. If these pests and diseases are not detected and controlled in time, they can cause severe crop yield reduction or even total crop failure.

[0003] Traditional greenhouse pest and disease monitoring mainly relies on manual inspections, judging the occurrence of pests and diseases by visually observing crop morphological characteristics. This has problems such as low monitoring efficiency, strong subjectivity, and difficulty in detecting early infection, often leading to delayed detection of pests and diseases and missing the best control opportunity. In the pesticide application process, manual backpack sprayers are mostly used for general spraying of the entire garden. This is not only labor-intensive, but also has problems such as blind application, serious waste of pesticides, and inaccurate dosage. Excessive application of pesticides not only increases agricultural production costs, but also causes soil and water pollution, damages the greenhouse ecological environment, and easily leads to pesticide resistance in pests and diseases, affecting the subsequent control effect.

[0004] Existing intelligent pest and disease control technologies mostly use a single visual perception method for pest and disease identification, which is greatly affected by lighting and shooting angle, resulting in low identification accuracy and difficulty in achieving early and accurate identification. Some precision spraying systems can only achieve simple area application, lacking pixel-level positioning and precise path planning of pest and disease target areas, resulting in insufficient spraying accuracy.

[0005] To address the aforementioned problems, we propose an early warning and targeted spraying system for greenhouse pests and diseases based on multimodal perception. Summary of the Invention

[0006] The purpose of this invention is to provide an early warning and targeted spraying system for greenhouse pests and diseases based on multimodal perception, so as to solve the problems mentioned in the background art.

[0007] To achieve the above objectives, the present invention provides the following technical solution: A greenhouse pest and disease early warning and targeted spraying system based on multimodal perception includes: The multimodal sensing module is used to deploy visual, environmental, and spectral sensing units to collect image information, environmental parameters, and spectral feature data of greenhouse crops as the basic sensing input of the system. The data preprocessing and registration module is used to perform noise reduction, enhancement, and normalization on multimodal sensing data, complete the spatial and temporal registration of multi-source data, and construct the pest and disease feature tensor. The pest and disease identification and early warning module is used to construct a multi-branch fusion network, receive pest and disease feature tensors, combine them with crop growth cycle data, complete the identification of pest and disease types, occurrence levels and spatial locations, generate early warning information of pests and diseases and push it out. The target area localization and path planning module is used to perform pixel-level segmentation of the recognition results, determine the precise geographical coordinates of the target area of ​​pests and diseases, and plan the optimal operation path of the unmanned spraying equipment by combining the layout of greenhouse crops and equipment movement constraints. The pesticide formulation and dosage control module is used to establish a pesticide formulation matching library based on pest and disease type, occurrence level and target area, to match the optimal pesticide formulation and calculate the precise application dosage. The spraying execution and feedback module is used to control the unmanned spraying equipment to complete the targeted spraying operation according to the operation path, pesticide formula and dosage, and at the same time collect multimodal perception data after spraying to form feedback on the spraying effect.

[0008] Preferably, the multimodal sensing module includes a visual sensing unit, an environmental sensing unit, and a spectral sensing unit. The visual sensing unit consists of high-definition cameras and infrared cameras deployed in different areas of the greenhouse to collect visible light and infrared images of crop leaves and stems. The environmental sensing unit consists of a temperature and humidity sensor, a light sensor, a CO2 sensor, and a soil moisture sensor, which collect data on air temperature and humidity, light intensity, CO2 concentration, and soil temperature, humidity, and moisture content inside the greenhouse. The spectral sensing unit consists of a portable hyperspectral instrument and a spectral sensor, which collects the reflectance spectral characteristics data of parts of crops infected by pests and diseases.

[0009] Preferably, in the data preprocessing and registration module, the data preprocessing process is as follows: Perform dehazing, noise reduction, scale normalization, and feature point extraction on visual images; Environmental parameters are processed by outlier removal, smoothing filtering, and normalization; spectral data are processed by baseline correction, spectral normalization, and feature band extraction. The data registration process is as follows: Using the greenhouse geographic coordinate system as a reference, spatial registration is completed by matching feature points of visual images. Using the system's unified timestamp as a reference, time registration of multi-source data is completed. The registered visual features, environmental features, and spectral features are stacked by channel to construct a three-dimensional pest and disease feature tensor.

[0010] Preferably, in the pest and disease identification and early warning module, the multi-branch fusion network includes a visual feature extraction branch, an environmental feature fusion branch, a spectral feature recognition branch, and a fusion decision layer. The visual feature extraction branch uses an encoder-multi-scale decoder structure to extract the morphological and texture features of crop pests and diseases. The environmental feature fusion branch fuses environmental parameters with visual features through a fully connected layer to uncover environmental correlation features of pest and disease occurrence; The spectral feature recognition branch extracts the spectral features of pests and diseases through convolutional neural networks, enabling accurate identification of early infection sites. The fusion decision layer uses an attention mechanism to weightedly fuse the features of the three branches. Combined with crop growth cycle data, it outputs the types and occurrence levels of pests and diseases through a classifier and the spatial coordinates of pests and diseases through a regressor. Based on the occurrence level and crop growth stage, it generates red, yellow and blue three-level early warning information and pushes it to the terminal.

[0011] Preferably, in the target area localization and path planning module, the target area localization process is as follows: The results of pest and disease identification are subjected to pixel-level semantic segmentation to generate a target area mask map of pests and diseases. The coordinate mapping of the mask map is completed by combining the greenhouse geographical coordinate system to determine the precise geographical coordinates and area of ​​the target area. The path planning process is as follows: With the goals of full target coverage, shortest operating path, and collision-free equipment movement, an improved A* algorithm is used, combined with constraints such as greenhouse crop row spacing, equipment turning radius, and spray width, to plan the starting point, waypoints, end point, and optimal operating path of the unmanned spraying equipment, and generate a path coordinate sequence.

[0012] Preferably, the pesticide formulation and dosage control module establishes a pesticide type-pesticide formula matching library containing pest and disease types, pesticide types, ratios, application dosages, and application methods. Based on the pest and disease type and occurrence level, the optimal pesticide formula is retrieved from the matching library. According to the target area, crop height, and pest and disease infection degree, a precise application dosage is calculated using a dosage calculation model. Manual adjustment of the formula and dosage is also supported. The dosage calculation formula is as follows: in, This is the dosage for drug administration. The target area, This represents the pest and disease severity coefficient. This is the crop plant height coefficient. This represents the coefficient of the active ingredient in the drug.

[0013] Preferably, the spraying execution and feedback module includes unmanned spraying equipment and a spraying effect sensing unit. The unmanned spraying equipment is a wheeled or tracked mobile spraying robot equipped with a multi-channel agent mixing device, a precision metering pump, an adjustable atomizing nozzle, and a GPS positioning module. It completes autonomous navigation according to the work path coordinate sequence, completes multi-channel precise agent mixing according to the agent formula, and controls the metering pump and nozzle flow rate according to the application dosage to achieve targeted and precise spraying of the target area. Preferably, the pesticide application effect sensing unit collects crop images, environmental parameters, and spectral data at 12h, 24h, and 48h after pesticide application through a multimodal sensing module, transmits them to the pest and disease identification and early warning module for secondary identification, compares the pest and disease status before and after pesticide application, and forms quantitative feedback data on the pesticide application effect.

[0014] Preferably, it also includes a system master control module, which is used to coordinate the data interaction and collaborative work of various modules, realize the full-process automated control of perception, identification, early warning, positioning, planning, application and feedback, and store all system operation data and historical records.

[0015] Preferably, the system's central control module adopts an industrial-grade controller, equipped with an embedded operating system, and establishes data connections with each sensing unit and unmanned spraying equipment through a wireless communication module. It supports edge computing and cloud data interaction, and has functions such as data storage, status monitoring, parameter setting, and manual / automatic mode switching. It can display greenhouse environmental data, pest and disease identification results, early warning information, spraying equipment location and operation status in real time, store all operating data and historical records, and support data query, export and analysis.

[0016] Compared with the prior art, the beneficial effects of the present invention are: This invention overcomes the limitations of single-sensor methods by fusing visual, environmental, and spectral data through a multimodal perception module. It utilizes spectral features to achieve accurate identification of early-stage pest and disease infection, solving the problem of difficulty in early detection in traditional monitoring and improving the accuracy and timeliness of pest and disease identification. Through data preprocessing and registration modules, it completes the spatial and temporal registration of multi-source data and constructs pest and disease feature tensors, providing high-quality data support for subsequent accurate identification. The multi-branch fusion network, combined with an attention mechanism, achieves weighted fusion of multi-source features. By combining crop growth cycle data, it accurately identifies the types, levels, and locations of pests and diseases, generates three levels of early warning information, and pushes it out in a timely manner, enabling early and accurate warnings of pests and diseases and securing the best opportunity for pest and disease control. The target area positioning and path planning module achieves pixel-level positioning of pest and disease target areas and optimal operation path planning, ensuring that unmanned spraying equipment can perform full-coverage, non-repetitive, and collision-free operations, thereby improving spraying efficiency. The pesticide formulation and dosage control module establishes a matching library and uses a dosage calculation model to achieve precise matching of pesticide formulations and precise calculation of application dosage, avoiding blind application, reducing pesticide waste, and lowering agricultural production costs; the unmanned spraying equipment is equipped with a multi-channel pesticide mixing device, a precision metering pump, and an adjustable atomizing nozzle to achieve precise mixing of pesticides and precise targeted spraying of the target area, improving pesticide utilization and reducing environmental pollution; The application execution and feedback module collects multi-source data after application to generate quantitative feedback on the application effect, realizing closed-loop control of pest and disease control and providing data support for subsequent control and pesticide formulation optimization. The system's overall control module coordinates the collaborative work of all modules to achieve fully automated control of the entire process of perception, identification, early warning, positioning, planning, application, and feedback, reducing manual labor intensity and improving the intelligence and precision of pest and disease control in greenhouses. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the structure of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Example 1:

[0019] like Figure 1 As shown, the greenhouse pest and disease early warning and targeted spraying system based on multimodal perception includes a multimodal perception module, a data preprocessing and registration module, a pest and disease identification and early warning module, a target area positioning and path planning module, a pesticide ratio and dosage control module, a spraying execution and feedback module, and a system control module. The multimodal perception module collects multi-source sensing data from the greenhouse, and after processing by the data preprocessing and registration module, it constructs a pest and disease feature tensor. The pest and disease identification and early warning module completes pest and disease identification and generates early warning information. The target area positioning and path planning module determines the target area coordinates and plans the optimal operation path. The pesticide ratio and dosage control module matches the pesticide formula and calculates the precise application dosage. The spraying execution and feedback module controls the equipment to complete targeted spraying and collects feedback data after application. The system control module coordinates the collaborative work of all modules to achieve fully automated control of the entire process.

[0020] The multimodal sensing module includes a visual sensing unit, an environmental sensing unit, and a spectral sensing unit. The visual sensing unit consists of a high-definition camera and an infrared camera, deployed on the greenhouse roof and pillars to achieve full coverage imaging of greenhouse crops, collecting visible light and infrared images, and capturing the morphology, texture, and temperature anomalies of crop diseases and pests. The environmental sensing unit consists of temperature and humidity sensors, light sensors, CO2 sensors, and soil moisture sensors, evenly distributed throughout the greenhouse area, collecting environmental parameters such as air temperature and humidity, light intensity, CO2 concentration, and soil temperature, humidity, and moisture content in real time to identify environmental factors that induce disease and pest occurrence. The spectral sensing unit consists of a portable hyperspectral analyzer and spectral sensors, which can collect reflectance spectral characteristic data of crop leaves and stems, and use the spectral differences between infected and healthy parts of the plant to achieve accurate early identification of disease and pest infection.

[0021] In the data preprocessing and registration module, the data preprocessing process is as follows: Gaussian filtering and bilateral filtering are used to denoise the visual image, and the dark channel prior method is used to remove fog, completing scale normalization and SIFT feature point extraction; outliers are removed from environmental parameters using the 3σ criterion, and smoothing filtering is performed using the moving average method, completing 0-1 normalization processing; baseline correction is performed on the spectral data using polynomial fitting, spectral normalization is performed using standard normal variable transformation, and feature bands are extracted through the continuous projection algorithm to reduce data redundancy.

[0022] The data registration process is as follows: using the greenhouse geographic coordinate system as a reference, spatial registration of multi-source images is completed through SIFT feature point matching of visual images. Environmental perception data, spectral data and feature points of visual images are associated to achieve spatial registration of multi-source data. Using the unified timestamp of the system as a reference, interpolation and resampling are performed on multi-source data with different acquisition frequencies to complete time registration. The registered visual features, environmental features and spectral features are stacked by channel to construct a three-dimensional pest and disease feature tensor with dimensions [H,W,C], where H and W are the height and width of the feature map, and C is the number of feature channels.

[0023] In the pest and disease identification and early warning module, the visual feature extraction branch of the multi-branch fusion network uses ResNet50 (Residual Network 50) as the encoder to extract deep morphological and texture features of crop images. It then uses a multi-scale decoder for upsampling to restore the spatial resolution of the feature map. The environmental feature fusion branch maps the normalized environmental parameters into feature vectors through a fully connected layer and fuses them with the mid-level features of the visual feature extraction branch to explore the correlation between environmental parameters and the occurrence of pests and diseases. The spectral feature recognition branch is composed of a one-dimensional convolutional neural network to extract pest-specific spectral features from the spectral feature data, enabling accurate identification of early infection sites.

[0024] The fusion decision layer employs a channel attention mechanism to weightedly fuse the output features of the three branches, highlighting key pest and disease characteristics. Combined with crop growth cycle data (such as sowing time, growth stage, and growth period), it outputs pest and disease types (such as aphids, powdery mildew, and downy mildew) through a Softmax classifier, pest and disease occurrence levels (mild, moderate, and severe) through a multi-classifier, and spatial coordinates of pests and diseases through a coordinate regressor. Based on the pest and disease occurrence level and crop growth stage, it generates three levels of early warning information: red, yellow, and blue. A red warning indicates severe pest and disease occurrence, requiring immediate pesticide application; a yellow warning indicates moderate pest and disease occurrence, requiring timely pesticide application; and a blue warning indicates mild pest and disease occurrence or a risk of occurrence, requiring enhanced monitoring. Warning information is pushed to agricultural production terminals via SMS and an app.

[0025] In the target area localization and path planning module, the target area localization process is as follows: the U-Net++ network (U-shaped network++) is used to perform pixel-level semantic segmentation on the pest and disease identification results, generate a pest and disease target area mask map, map the pixels of the pest and disease area in the mask map to the greenhouse geographic coordinate system, complete the conversion from pixel coordinates to geographic coordinates, determine the precise geographic coordinates, outline and area of ​​the target area, and realize the fine positioning of the target area.

[0026] The path planning process is as follows: taking the starting point of the unmanned spraying equipment as the initial point, all pest and disease target areas as the target points, and the spacing between greenhouse crop rows, equipment turning radius, spraying width, and the location of greenhouse obstacles as constraints, the heuristic function of the traditional A algorithm is improved by introducing a target area coverage coefficient and a path smoothing coefficient to make the heuristic function more suitable for the greenhouse spraying operation scenario; the optimal path from the starting point to each target area is searched by the improved A algorithm, and a path coordinate sequence is generated to ensure that the unmanned spraying equipment can fully cover all target areas with the shortest path, without collisions or repeated operations.

[0027] In the pesticide formulation and dosage control module, a pesticide formula matching library is established for pest and disease types. The library stores information such as the type of pesticide, the optimal ratio, the recommended dosage, the application method, and the application interval for common greenhouse pests and diseases (such as aphids, spider mites, powdery mildew, downy mildew, etc.). The matching library supports manual addition, modification, and deletion, and can be updated according to the actual crops and pest and disease types.

[0028] Based on the pest and disease identification results, the optimal pesticide formulation is automatically retrieved from the matching database. If it is a compound pest or disease, a compound pesticide formulation is retrieved. The precise application dosage is calculated using a dosage calculation model based on the target area, crop height, and pest and disease infestation level. The dosage calculation model is as follows: in This refers to the dosage (unit: L). Target area (unit: m) 2 ), The severity coefficients are for pest and disease levels (mild k=0.8, moderate k=1.0, severe k=1.2). The crop plant height coefficient is calculated as follows: (plant height < 50cm, ρ = 0.9; 50cm ≤ plant height ≤ 150cm, ρ = 1.0; plant height > 150cm, ρ = 1.1). The effective ingredient coefficient of the pesticide (determined based on the content of the effective ingredient, with a value ranging from 0.9 to 1.1); the system supports manual adjustment of the pesticide formulation and dosage to meet different operational needs.

[0029] The unmanned spraying equipment in the spraying execution and feedback module is a wheeled or tracked mobile spraying robot, adapted to the complex ground environment inside greenhouses. The equipment is equipped with a multi-channel pesticide mixing device, a precision metering pump, an adjustable atomizing nozzle, a GPS positioning module, a wireless communication module, and obstacle avoidance sensors. The GPS positioning module collects the equipment's position coordinates in real time and matches them with the path coordinate sequence generated by the path planning module to achieve autonomous navigation. The multi-channel pesticide mixing device accurately mixes 2-4 pesticides according to the pesticide formula, with a mixing ratio error ≤ ±2%. The precision metering pump accurately controls the pesticide output flow rate according to the dosage, with a flow rate adjustment range of 0-5L / min and an error ≤ ±1%. The adjustable atomizing nozzle can adjust the droplet size (50-300μm) and spray width (1-5m) to achieve uniform atomization and precise spraying of droplets, improving pesticide utilization.

[0030] The pesticide application effect sensing unit collects visible light images, infrared images, spectral data, and environmental parameters of the crop at 12h, 24h, and 48h after pesticide application through a multimodal sensing module. The collected data is then transmitted to the pest and disease identification and early warning module for secondary identification and analysis. By comparing the types, levels, infected areas, and crop growth status of pests and diseases before and after pesticide application, quantitative feedback data on the pesticide application effect (such as pest and disease reduction rate and crop recovery rate) is generated. This feedback data is transmitted to the system's central control module, providing data support for subsequent pest and disease control and pesticide formulation optimization.

[0031] The system's main control module adopts an industrial-grade controller and is equipped with a Linux embedded operating system, possessing powerful computing capabilities and stable operating performance. It establishes bidirectional data connections with the various sensing units of the multimodal sensing module and the unmanned spraying equipment through WiFi and 4G / 5G wireless communication modules, enabling real-time data transmission and remote command issuance. The system supports edge computing and cloud data interaction, allowing locally collected data to be uploaded to a cloud server for remote storage and analysis.

[0032] The system's central control module is equipped with a human-machine interface, featuring data storage, status monitoring, parameter setting, and manual / automatic mode switching. It can display real-time environmental parameters within the greenhouse, pest and disease identification results, early warning information, the real-time location, operational status, and application progress of the unmanned spraying equipment. It can store all system operational data and historical records, including sensing data, identification results, early warning information, application records, and feedback data, for a storage time of ≥3 years. It supports data querying, exporting, and analysis based on time, pest and disease type, target area location, and other criteria, providing data support for agricultural production decisions. The system supports both manual and automatic operation modes. In automatic mode, all modules work collaboratively to achieve fully automated operation. In manual mode, operators can manually control the movement of the unmanned spraying equipment, pesticide mixing ratio, and application dosage through the human-machine interface to meet special operational needs. Example 2:

[0033] Please see Figure 1 As shown, the present invention is a greenhouse pest and disease early warning and targeted spraying system based on multimodal perception, including a multimodal perception module, a data preprocessing and registration module, a pest and disease identification and early warning module, a target area positioning and path planning module, a pesticide ratio and dosage control module, a spraying execution and feedback module, and a system control module.

[0034] Multimodal sensing module: In a tomato greenhouse at a vegetable greenhouse planting base, 10 high-definition cameras and 5 infrared cameras are deployed on the greenhouse roof at a spacing of 3m×4m to achieve full-coverage visual perception of the greenhouse tomato crops, collecting visible light and infrared images of tomato leaves and stems in real time. Eight environmental sensing units are evenly distributed throughout the greenhouse, each including a temperature and humidity sensor, a light sensor, a CO2 sensor, and a soil moisture sensor, collecting real-time data on air temperature and humidity, light intensity, CO2 concentration, and soil temperature, humidity, and moisture content. Two portable hyperspectral analyzers are equipped to periodically collect the reflectance spectral characteristics of tomato leaves, focusing on areas showing slight yellowing or spots, enabling spectral perception of early pest and disease infection. All sensing units establish a wireless data connection with the system's central control module, transmitting sensing data in real time.

[0035] Data preprocessing and registration module: Gaussian filtering is used to denoise the visible light images of tomatoes captured by the high-definition camera, and dark channel prior method is used to remove fog. The image scale is normalized to 640×480 pixels, and SIFT (Scale Invariant Feature Transform) feature points are extracted. Infrared images are denoised and temperature calibrated to complete registration with visible light images. Outliers are removed from environmental perception data using the 3σ criterion, and smoothing filtering is performed using the moving average method to normalize all environmental parameters to the 0-1 range. Spectral data collected by the hyperspectral analyzer are normalized by polynomial fitting baseline correction and standard normal variable transformation. Feature bands such as 450nm, 550nm, 680nm, and 760nm are extracted using the continuous projection algorithm to reduce data redundancy.

[0036] Based on the greenhouse geographic coordinate system, spatial registration of visual images, environmental perception data, and spectral data is completed through SIFT feature point matching. Time registration of multi-source data is completed with the system 1s as the timestamp. The registered visual features, environmental features, and spectral features are stacked by channel to construct a three-dimensional pest and disease feature tensor with dimensions [640, 480, 12], which is then transmitted to the pest and disease identification and early warning module.

[0037] The pest and disease identification and early warning module: The visual feature extraction branch of the multi-branch fusion network uses ResNet50 as the encoder to extract morphological and textural features such as leaf spots, curling, and wilting in tomato images, and restores the spatial resolution of the feature map through a multi-scale decoder; the environmental feature fusion branch maps normalized environmental parameters such as temperature, humidity, and light into feature vectors, and fuses them with the mid-layer features of the visual features to explore the correlation between high temperature and high humidity environment and tomato downy mildew; the spectral feature recognition branch extracts the spectral features of early infected parts of tomato leaves through a one-dimensional convolutional neural network to achieve early identification of downy mildew.

[0038] The fusion decision layer uses a channel attention mechanism to weightedly fuse the features of the three branches. Combined with the tomato growth cycle data (budding stage), the Softmax classifier identifies the pest as tomato downy mildew, the multi-classifier determines the occurrence level as moderate, and the coordinate regressor outputs the spatial coordinates of the downy mildew occurrence. Based on the moderate occurrence level and the tomato budding stage, the system generates a yellow early warning message and pushes it to the grower's terminal via the APP to remind them to apply pesticides in time.

[0039] Target area localization and path planning module: The U-Net++ network is used to perform pixel-level semantic segmentation on the identification results of tomato downy mildew, generating a downy mildew target area mask map. The pixel coordinates in the mask map are mapped to the greenhouse geographic coordinate system to determine the precise geographic coordinates, outline, and area of ​​three downy mildew target areas, namely S1=25m. 2 S2=32m 2 S3=18m 2.

[0040] Starting from the unmanned spraying equipment parking point at the greenhouse entrance, and with three target areas as objectives, and constrained by a tomato row spacing of 0.8m, a spraying robot turning radius of 1.2m, a spraying width of 2m, and the location of obstacles such as pillars inside the greenhouse, the heuristic function of the A algorithm is improved by introducing a target area coverage coefficient and a path smoothness coefficient. The optimal operation path is searched through the improved A algorithm, generating a path coordinate sequence: starting point → target area S1 → target area S3 → target area S2 → starting point, ensuring that the spraying robot can fully cover all target areas with the shortest path, without collisions or repeated operations.

[0041] The pesticide formulation and dosage control module: The system automatically retrieves the optimal pesticide formulation from the pest / disease type-pesticide formulation matching library, based on tomato downy mildew and its moderate occurrence level: 800x dilution of 72% cymoxanil / mancozeb wettable powder + 1500x dilution of 25% azoxystrobin suspension; based on the total area of ​​the three target zones S = 25 + 32 + 18 = 75m². 2 The tomato plant height is 80cm (ρ=1.0), the moderate occurrence level is k=1.0, the effective ingredient coefficient of the pesticide is α=1.0, and the precise application dose is calculated to be 75L using the dosage calculation model Q=75×1.0×1.0×1.0=75L. The grower can confirm the formula and dosage through the system's human-computer interaction interface without manual adjustment.

[0042] The spraying execution and feedback module: The unmanned spraying robot receives the operation path, pesticide formula, and dosage instructions issued by the system's central control module. The GPS positioning module collects the robot's position coordinates in real time and matches them with the path coordinate sequence to achieve autonomous navigation. The multi-channel pesticide mixing device accurately mixes 72% cymoxanil-mancozeb wettable powder and 25% azoxystrobin suspension according to the formula, with a mixing ratio error of ±1.5%. The precision metering pump controls the pesticide output flow rate according to a dosage of 75L, with the flow rate adjusted to 3L / min. The adjustable atomizing nozzle adjusts the droplet size to 100μm and the spraying width to 2m to achieve targeted and precise spraying of the tomato downy mildew target area.

[0043] After the pesticide application is completed, the pesticide application effect sensing unit collects tomato leaf images, spectral data, and greenhouse environmental parameters at 12h, 24h, and 48h after the application through the multimodal sensing module. These data are then transmitted to the pest and disease identification and early warning module for secondary identification. By comparing the infection area of ​​downy mildew before and after the application, the reduction rate of tomato downy mildew at 48h after the application is calculated to be 85%, and the crop recovery rate is 80%. This generates quantitative feedback data on the pesticide application effect, which is then transmitted to the system's central control module for storage.

[0044] The system's central control module, an industrial-grade controller, coordinates data interaction and collaborative work among various modules via a 4G wireless communication module, achieving fully automated control from sensing, identification, early warning, positioning, and planning to application and feedback. The human-machine interface displays real-time environmental parameters such as temperature, humidity, and light within the greenhouse, the identification results of tomato downy mildew, yellow warning information, and the real-time location, operating status, and application progress of the spraying robot. The system stores all data from this operation, including sensing data, identification results, early warning information, target area parameters, pesticide formulation, application dosage, and application effect feedback data. Growers can query detailed records of this operation by time through the interface, providing data support for subsequent tomato pest and disease control.

[0045] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A greenhouse pest and disease early warning and targeted spraying system based on multimodal perception, characterized in that, include: The multimodal sensing module is used to deploy visual, environmental, and spectral sensing units to collect image information, environmental parameters, and spectral feature data of greenhouse crops as the basic sensing input of the system. The data preprocessing and registration module is used to perform noise reduction, enhancement, and normalization on multimodal sensing data, complete the spatial and temporal registration of multi-source data, and construct the pest and disease feature tensor. The pest and disease identification and early warning module is used to construct a multi-branch fusion network, receive pest and disease feature tensors, combine them with crop growth cycle data, complete the identification of pest and disease types, occurrence levels and spatial locations, generate early warning information of pests and diseases and push it out. The target area localization and path planning module is used to perform pixel-level segmentation of the recognition results, determine the precise geographical coordinates of the target area of ​​pests and diseases, and plan the optimal operation path of the unmanned spraying equipment by combining the layout of greenhouse crops and equipment movement constraints. The pesticide formulation and dosage control module is used to establish a pesticide formulation matching library based on pest and disease type, occurrence level and target area, to match the optimal pesticide formulation and calculate the precise application dosage. The spraying execution and feedback module is used to control the unmanned spraying equipment to complete the targeted spraying operation according to the operation path, pesticide formula and dosage, and at the same time collect multimodal perception data after spraying to form feedback on the spraying effect.

2. The greenhouse pest and disease early warning and targeted spraying system based on multimodal perception according to claim 1, characterized in that, The multimodal perception module includes a visual perception unit, an environmental perception unit, and a spectral perception unit. The visual perception unit consists of high-definition cameras and infrared cameras deployed in different areas of the greenhouse to collect visible light and infrared images of crop leaves and stems. The environmental sensing unit consists of a temperature and humidity sensor, a light sensor, a CO2 sensor, and a soil moisture sensor, which collect data on air temperature and humidity, light intensity, CO2 concentration, and soil temperature, humidity, and moisture content inside the greenhouse. The spectral sensing unit consists of a portable hyperspectral instrument and a spectral sensor, which collects the reflectance spectral characteristics data of parts of crops infected by pests and diseases.

3. The greenhouse pest and disease early warning and targeted spraying system based on multimodal perception according to claim 1, characterized in that, In the data preprocessing and registration module, the data preprocessing process is as follows: Perform dehazing, noise reduction, scale normalization, and feature point extraction on visual images; Environmental parameters are processed by outlier removal, smoothing filtering, and normalization; spectral data are processed by baseline correction, spectral normalization, and feature band extraction. The data registration process is as follows: Using the greenhouse geographic coordinate system as a reference, spatial registration is completed by matching feature points of visual images. Using the system's unified timestamp as a reference, time registration of multi-source data is completed. The registered visual features, environmental features, and spectral features are stacked by channel to construct a three-dimensional pest and disease feature tensor.

4. The greenhouse pest and disease early warning and targeted spraying system based on multimodal perception according to claim 1, characterized in that, In the pest and disease identification and early warning module, the multi-branch fusion network includes a visual feature extraction branch, an environmental feature fusion branch, a spectral feature recognition branch, and a fusion decision layer. The visual feature extraction branch uses an encoder-multi-scale decoder structure to extract the morphological and texture features of crop pests and diseases. The environmental feature fusion branch fuses environmental parameters with visual features through a fully connected layer to uncover environmental correlation features of pest and disease occurrence; The spectral feature recognition branch extracts the spectral features of pests and diseases through convolutional neural networks, enabling accurate identification of early infection sites. The fusion decision layer uses an attention mechanism to weightedly fuse the features of the three branches. Combined with crop growth cycle data, it outputs the types and occurrence levels of pests and diseases through a classifier and the spatial coordinates of pests and diseases through a regressor. Based on the occurrence level and crop growth stage, it generates red, yellow and blue three-level early warning information and pushes it to the terminal.

5. The greenhouse pest and disease early warning and targeted spraying system based on multimodal perception according to claim 1, characterized in that, In the target area localization and path planning module, the target area localization process is as follows: The results of pest and disease identification are subjected to pixel-level semantic segmentation to generate a target area mask map of pests and diseases. The coordinate mapping of the mask map is completed by combining the greenhouse geographical coordinate system to determine the precise geographical coordinates and area of ​​the target area. The path planning process is as follows: With the goals of full target coverage, shortest operating path, and collision-free equipment movement, an improved A* algorithm is used, combined with constraints such as greenhouse crop row spacing, equipment turning radius, and spray width, to plan the starting point, waypoints, end point, and optimal operating path of the unmanned spraying equipment, and generate a path coordinate sequence.

6. The greenhouse pest and disease early warning and targeted spraying system based on multimodal perception according to claim 1, characterized in that, The pesticide formulation and dosage control module establishes a pesticide type-pesticide formula matching library containing pest and disease types, pesticide types, ratios, application dosages, and application methods. Based on the pest and disease type and occurrence level, the optimal pesticide formula is retrieved from the matching library. According to the target area, crop height, and pest and disease infection severity, a precise application dosage is calculated using a dosage calculation model. Manual adjustment of the formula and dosage is also supported. The dosage calculation formula is as follows: in, This is the dosage for drug administration. The target area, This represents the pest and disease severity coefficient. This is the crop plant height coefficient. This represents the coefficient of the active ingredient in the drug.

7. The greenhouse pest and disease early warning and targeted spraying system based on multimodal perception according to claim 1, characterized in that, The spraying execution and feedback module includes unmanned spraying equipment and a spraying effect sensing unit. The unmanned spraying equipment is a wheeled or tracked mobile spraying robot equipped with a multi-channel agent mixing device, a precision metering pump, an adjustable atomizing nozzle, and a GPS positioning module. It completes autonomous navigation according to the work path coordinate sequence, completes multi-channel precise agent mixing according to the agent formula, and controls the metering pump and nozzle flow rate according to the application dosage to achieve targeted and precise spraying of the target area.

8. The greenhouse pest and disease early warning and targeted spraying system based on multimodal perception according to claim 7, characterized in that, The pesticide application effect sensing unit collects crop images, environmental parameters, and spectral data at 12h, 24h, and 48h after pesticide application through a multimodal sensing module. These data are then transmitted to the pest and disease identification and early warning module for secondary identification. The module compares the status of pests and diseases before and after pesticide application to generate quantitative feedback data on the pesticide application effect.

9. The greenhouse pest and disease early warning and targeted spraying system based on multimodal perception according to claim 1, characterized in that, It also includes a system master control module, which coordinates the data interaction and collaborative work of various modules to achieve fully automated control of the entire process of perception, identification, early warning, positioning, planning, application and feedback, and stores all system operation data and historical records.

10. The greenhouse pest and disease early warning and targeted spraying system based on multimodal perception according to claim 9, wherein the system's main control module adopts an industrial-grade controller, is equipped with an embedded operating system, establishes data connections with each sensing unit and unmanned spraying equipment through a wireless communication module, supports edge computing and cloud data interaction, and has data storage, status monitoring, parameter setting, and manual / automatic mode switching functions. It can display greenhouse environmental data, pest and disease identification results, early warning information, spraying equipment location and operation status in real time, store all operating data and historical records, and support data query, export and analysis.

Images