Multi-mode fusion orchard intelligent pest monitoring, early warning and repelling system

Abstract

The invention relates to the technical field of agricultural monitoring and plant protection, and discloses a multi-modal fusion orchard intelligent pest monitoring, early warning and repelling system, which comprises a data acquisition module, an integrated optical imaging and millimeter wave radar unit, and an image and a radar point cloud of a monitoring area are synchronously acquired; the data processing module executes heterogeneous data coordinate registration, maps radar three-dimensional space data to a two-dimensional image plane, and dynamically generates a region of interest by using depth information; the identification decision module is used for carrying out double check classification on the target by fusing the visual features extracted by the convolutional neural network and the radar micro-Doppler motion features; and the repelling module is used for controlling an acoustic insect driver to emit a specific signal according to a classification result, and dynamically adjusting an output strategy based on the pests fed back by the radar. According to the method, the radar is used for guiding vision to focus an effective target, false alarms generated by ambient light and shadows and wind blowing branches and leaves are effectively eliminated, the algorithm operand is reduced, and accurate and anti-adaptive closed-loop repelling is realized.

Description

Technical Field

[0001] This invention relates to the field of agricultural monitoring and plant protection technology, specifically to a multimodal fusion intelligent pest monitoring, early warning, and repellency system for orchards. Background Technology

[0002] Orchard pests are one of the main factors affecting fruit yield and quality, and timely and accurate monitoring and effective repellency interventions are key to achieving refined management. With the development of machine vision technology, image recognition-based monitoring equipment has gradually replaced traditional manual inspections and physical trapping and counting, becoming the mainstream solution for smart orchard construction. This type of technology typically uses cameras deployed in the orchard to collect images and uses deep learning algorithms to identify and count the insects in the images.

[0003] Existing single-optical-vision monitoring technologies face severe challenges in complex real-world field environments. Orchard environments are unstructured, with drastic changes in lighting conditions over time, and significant foliage obstruction and background clutter. Optical sensors, being passive imaging devices, are highly susceptible to insufficient light or shadow interference, leading to degraded image quality and difficulty in capturing the characteristics of minute pests. More critically, wind-driven swaying of branches and leaves is common in orchards, and purely vision-based motion detection algorithms often struggle to distinguish between wind-driven vegetation movement and the active flight of pests. This results in numerous false alarms in windy weather, increasing the data processing burden and reducing the reliability of early warnings.

[0004] Monocular vision solutions lack depth information and are affected by perspective imaging principles, making it difficult to accurately infer the actual physical size of a target from image pixel dimensions. This can easily lead to confusion between nearby tiny flying insects and other distant disturbances. In the repellency execution phase, existing physical insect repellent devices mostly employ open-loop control modes with timed activation or fixed frequency operation, failing to detect whether pests have truly left. Long-term use of a single mode of sound and light stimulation can easily lead to pest adaptation, causing the repellency effect to significantly diminish over time. Therefore, there is an urgent need for an intelligent monitoring and repellency system that can integrate multi-dimensional sensory information, possess high anti-interference capabilities, and achieve closed-loop feedback control. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a multimodal fusion intelligent pest monitoring, early warning, and repellency system for orchards. This system solves the problems of high false alarm rates in existing orchard pest monitoring technologies due to the susceptibility of single-visual perception to interference from lighting and background movement, and poor repellency effects caused by open-loop repellency methods that allow pests to adapt.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a multimodal fusion intelligent pest monitoring, early warning, and repellency system for orchards, comprising; The data acquisition module is configured to acquire multi-dimensional physical information of the orchard environment. The data acquisition module includes an optical imaging unit and a radar detection unit. The optical imaging unit is used to acquire visible light images, and the radar detection unit is used to transmit and receive radar wave signals to analyze the spatial distance, radial velocity and physical size of targets within the monitoring area. The data processing module, connected to the data acquisition module, is configured to receive the visible light image and the radar wave signal, perform data cleaning and coordinate system registration, map the three-dimensional point cloud data in the radar detection unit coordinate system to the camera coordinate system of the optical imaging unit and project it onto the two-dimensional image plane, and generate a region of interest on the visible light image. The identification and decision-making module is connected to the data processing module and is configured to use a convolutional neural network model to extract features from the region of interest, classify the target by combining radar micro-Doppler features, and generate control commands containing pest categories and avoidance parameters based on the classification results and the distance information measured by radar. The avoidance module is connected to the identification and decision module and is configured to receive the control command and drive the physical device to selectively emit a beam of light of a specific wavelength or a sound wave of a specific frequency according to the pest category.

[0007] Preferably, the data acquisition module further includes a synchronization trigger circuit; The synchronization trigger circuit is connected to the optical imaging unit and the radar detection unit respectively, and is used to generate a unified timestamp signal to control the exposure time of the optical imaging unit to keep synchronized with the signal transmission time of the radar detection unit. The field of view of the radar detection unit is configured to cover the field of view of the optical imaging unit, and the main beam axis of the radar detection unit and the optical axis of the optical imaging unit are physically parallel or intersect at a preset angle.

[0008] Preferably, the data processing module includes a coordinate transformation unit, which stores pre-calibrated rigid body transformation parameters, including rotation matrices and translation vectors. The coordinate transformation unit is configured to perform a linear transformation operation on the target three-dimensional spatial coordinates output by the radar detection unit, and to rotate the target coordinate vector using the rotation matrix and superimpose the translation vector, thereby converting the target coordinates in the radar coordinate system into the target coordinates in the camera coordinate system.

[0009] Preferably, the data processing module further includes an image projection mapping unit and a region of interest generation unit; The image projection mapping unit is configured to project the target coordinates in the camera coordinate system onto the two-dimensional image pixel plane using a pre-stored camera intrinsic parameter matrix, thereby obtaining the pixel center coordinates of the target. The region of interest generation unit is configured to calculate the size of the region of interest based on optical perspective laws, and its calculation logic is as follows: The pixel width and pixel height of the region of interest are directly proportional to the product of the target reference physical size and the camera focal length, and inversely proportional to the depth value of the target in the camera coordinate system. The calculation results are expanded according to a preset safety redundancy coefficient.

[0010] Preferably, the recognition decision module includes a visual feature extraction unit, a motion feature analysis unit, and a fusion classification unit; The visual feature extraction unit is used to perform convolution operations on the region of interest to extract the texture and edge features of the target and generate a visual feature vector. The motion feature analysis unit is used to perform feature encoding on the raw data output by the radar detection unit, extract velocity amplitude and frequency features to generate motion feature vectors; The fusion classification unit is used to concatenate the visual feature vector and the motion feature vector to generate a joint feature vector.

[0011] Preferably, the fusion classification unit is configured to perform a weighted mapping on the joint feature vector using a fully connected layer, and to calculate the probability of the mapping result using a normalized exponential function; the probability calculation process includes: Calculate the index value of the output node of the fully connected layer, divide the index value by the sum of the index values ​​of all category nodes, and obtain the probability value of the target belonging to the preset pest category.

[0012] Preferably, the identification decision module is configured with anti-false alarm determination logic, which specifically includes: When the confidence level of the classification result exceeds a preset threshold, the radar radial velocity corresponding to the target is further detected to see if it is within a preset biologically effective velocity range. If the radar radial velocity indicates that the target is stationary or oscillating, it is determined to be environmental interference and the generation of the control command is prohibited. The control command is generated only when the classification result points to a pest category and the radar radial velocity matches the flight characteristics.

[0013] Preferably, the repelling module includes an acoustic and light insect repellent device, which integrates a multispectral light-emitting diode array board and a dual-mode sound-emitting component. The multispectral light-emitting diode array board includes ultraviolet-inducing LEDs with wavelengths from 365nm to 395nm, yellow light-repelling LEDs with wavelengths of 585nm, and full-spectrum white light-strobe LEDs. The dual-modal sound generation component includes a piezoelectric ceramic ultrasonic transducer with a frequency response range of 20kHz to 60kHz and a full-range loudspeaker with a frequency response range of 500Hz to 10kHz.

[0014] Preferably, the avoidance module has a built-in closed-loop feedback strategy: During the operation of the sound and light insect repellent, the radar detection unit continuously monitors the radial velocity and distance of the target; If the monitoring data shows that the target's radial velocity is pointing in a direction away and the distance is increasing, the avoidance module stops outputting; If the monitoring data shows that the target is stuck, the avoidance module automatically increases the light intensity of the multispectral light-emitting diode array board or switches the sound wave frequency of the dual-mode sound-emitting component.

[0015] A multimodal fusion intelligent pest monitoring, early warning, and repellency device for orchards, comprising: The control box has a rotating shaft structure fixedly connected to its lower surface, a camera fixedly connected inside the control box, an adjustment knob fixedly connected to its upper surface, a sound and light insect repellent fixedly connected inside the control box, and a bottom heat sink fixedly connected to its lower surface.

[0016] This invention provides a multimodal fusion intelligent pest monitoring, early warning, and repellency system for orchards. It has the following beneficial effects: 1. This invention utilizes heterogeneous data fusion between radar detection and optical imaging units, leveraging depth and position information measured by radar to guide the generation of visual regions of interest (ROIs). By establishing a rigid transformation mapping from the radar coordinate system to the camera coordinate system, the system can directly locate physical targets on the image, allowing the convolutional neural network to extract features from specific areas instead of scanning the entire image. This approach significantly reduces the computational demands of image processing and physically eliminates visual background interference caused by wind-blown leaves or changes in light and shadow in the orchard environment, improving the system's response speed and detection efficiency.

[0017] 2. This invention constructs a dual logic verification mechanism based on visual texture and motion features, effectively reducing the false alarm rate. While using visual algorithms to identify the appearance of pests, it combines radial velocity and micro-Doppler frequency shift features collected by radar to determine the target's motion state. An alarm is triggered only when the visual classification result points to a pest and the radar detects a velocity signal that conforms to biological flight characteristics. This logic can accurately distinguish between real pests with directional flight capabilities and background interference objects that are static or making reciprocating swinging movements, ensuring high accuracy of monitoring and early warning in complex field environments.

[0018] 3. This invention constructs a closed-loop repellency strategy encompassing perception, decision-making, execution, and feedback. It not only precisely intervenes by calling upon targeted sensitive wavelength beams or sound frequencies based on the identified pest type, but also utilizes radar to continuously monitor the radial velocity changes of the target during the repellency process. When radar feedback detects that pests have developed tolerance or lingering towards the current signal, it automatically adjusts the intensity of the sound and light or switches the signal frequency. This dynamic adjustment mechanism overcomes the problem of pests adapting due to the single signal mode in traditional pest control devices, ensuring a long-term and stable repellency effect. Attached Figure Description

[0019] Figure 1 This is a perspective view of the present invention; Figure 2 This is a schematic diagram of the present invention; Figure 3 This is a diagram of the apparatus of the present invention.

[0020] The components include: 1. Control box; 2. Camera; 3. Adjustment knob; 4. Bottom heat sink; 5. Rotary shaft structure; and 6. Sound and light insect repellent. Detailed Implementation

[0021] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] Please see the appendix Figure 1 This invention provides a multimodal fusion intelligent pest monitoring, early warning, and repellency system for orchards, comprising: The data acquisition module is configured to acquire multi-dimensional physical information of the orchard environment. This module includes an optical imaging unit and a radar detection unit. The optical imaging unit acquires visible light images, and the radar detection unit transmits and receives radar wave signals to analyze the target's spatial distance, radial velocity, and physical size. The data processing module, connected to the data acquisition module, is configured to receive visible light images and radar wave signals, perform data cleaning, signal enhancement and coordinate system registration, map the three-dimensional point cloud data in the radar coordinate system to the camera coordinate system and project it onto the two-dimensional image plane, and generate the region of interest on the visible light image. The identification and decision-making module is connected to the data processing module and is configured to use a convolutional neural network model to extract features from the region of interest, combine radar micro-Doppler features to classify targets, and generate control commands containing pest categories and avoidance parameters based on the classification results and radar distance information. The repelling module, connected to the identification and decision module, is configured to receive control commands and drive the physical device to selectively emit beams of specific wavelengths or sound waves of specific frequencies according to the type of pest.

[0023] The following section will provide a detailed explanation of each of the above modules and the core technical principles involved, taking into account specific implementation details.

[0024] The multimodal data acquisition module, serving as the physical front end for acquiring environmental information, is configured to acquire optical image information and radar point cloud information within the monitoring area under a unified time reference. This module specifically includes an optical imaging unit and a radar detection unit.

[0025] The optical imaging unit is installed inside the housing of camera 2, with its optical axis facing the orchard area to be monitored. The optical imaging unit includes an optical lens group and an image sensor. The optical lens group consists of multiple corrected optical glass lenses used to collect light reflected from the external environment and converge it onto the photosensitive surface of the image sensor. The image sensor can employ a complementary metal-oxide-semiconductor (CMOS) or charge-coupled device (CCD) photosensitive element to convert the received light signal into a digital electrical signal. To adapt to the complex lighting environment of the orchard, the optical imaging unit can be configured to support a wide dynamic range (WDR) mode to ensure clear texture features are acquired even under backlight or shade conditions.

[0026] The radar detection unit is housed inside the casing of camera 2, adjacent to the optical imaging unit. Specifically, the radar detection unit employs a millimeter-wave radar sensor, such as a frequency-modulated continuous-wave radar operating in the 24GHz or 77GHz band. The radar detection unit includes a transmitting antenna array, a receiving antenna array, and an RF front-end circuit. The transmitting antenna array is configured to transmit electromagnetic wave signals into the monitoring area. These signals are capable of penetrating smoke, dust, and fine raindrops, overcoming the perception limitations of the optical imaging unit in low-visibility environments. The receiving antenna array is configured to receive echo signals reflected back from the target object. The RF front-end circuit performs mixing, filtering, and analog-to-digital conversion on the echo signals to obtain the radial distance, radial velocity, and azimuth angle information of the target object relative to the radar detection unit.

[0027] The optical imaging unit and the radar detection unit follow the principle of field-of-view coverage in their spatial geometric relationship. The optical imaging unit has a first field of view, and the radar detection unit has a second field of view. In hardware selection and layout, the second field of view of the radar detection unit is configured to cover or exceed the first field of view of the optical imaging unit, or at least ensure that their central detection areas coincide. Specifically, the optical axis of the optical imaging unit and the main beam axis of the radar detection unit are physically parallel, or designed to intersect at a preset monitoring distance. This spatial geometric constraint ensures that for any target seen by the optical imaging unit, the radar detection unit can simultaneously detect its depth and velocity information, thus providing a physical basis for subsequent data fusion processing that maps three-dimensional radar data to a two-dimensional image plane.

[0028] The multimodal data acquisition module also integrates a synchronization trigger circuit. This circuit connects the optical imaging unit and the radar detection unit to generate a unified timestamp signal. It controls the exposure time of the optical imaging unit to be synchronized with the signal transmission time of the radar detection unit, or records a fixed time difference between the two acquisition times. This ensures that each frame of optical image data corresponds strictly with a frame of radar point cloud data in the time dimension, eliminating moving target matching errors caused by asynchronous acquisition.

[0029] The data fusion and processing module is primarily responsible for the spatiotemporal registration of heterogeneous data and the generation of regions of interest (ROIs). It establishes a mapping relationship between the radar detection space and the visual imaging plane, utilizing the radar's active detection capabilities to guide visual focusing on the target. Logically, the data processing module is further divided into a coordinate transformation unit, an image projection mapping unit, and a ROI generation unit. The coordinate transformation unit is responsible for converting the target space coordinates output by the radar detection unit into the camera coordinates of the optical imaging unit. Since the radar detection unit and the optical imaging unit are rigidly connected in their physical structure, their relative positions and attitudes remain constant. Before leaving the factory, joint calibration is performed to determine the rigid body transformation parameters from the radar coordinate system to the camera coordinate system, including the rotation matrix. Translation vector For any target point detected by radar Its coordinates in the radar coordinate system are represented as The coordinate transformation unit calculates the target's coordinates in the camera coordinate system based on the following linear transformation model. ; ; In the formula, This represents the three-dimensional coordinate vector of the target point in the camera coordinate system. This represents the three-dimensional coordinate vector of the target point in the radar coordinate system. This indicates a rotation from the radar coordinate system to the camera coordinate system. Rotation matrix, This represents the three-dimensional translation vector of the radar coordinate system origin in the camera coordinate system.

[0030] The image projection mapping unit 202, based on the pinhole imaging model, projects the three-dimensional target points in the camera coordinate system. The image is projected onto the two-dimensional image pixel plane. The internal parameters of the optical imaging unit 101 are pre-acquired through calibration, forming the camera intrinsic parameter matrix. The image projection mapping unit 202 calculates the pixel center coordinates of the target in the image according to the following projection relationship. ; ; In the formula, This represents the horizontal (x-axis) coordinate of a 3D point in the camera coordinate system. This represents the vertical (y-axis) coordinate of a 3D point in the camera coordinate system. This represents the depth (z-axis) coordinate of a 3D point in the camera coordinate system. The pixel x-coordinate of the projection point. Represents the pixel ordinate of the projection point. Indicates the scale factor. Represents the camera intrinsic parameter matrix. This indicates the camera's effective focal length in the horizontal (x-direction). This indicates the camera's effective focal length in the longitudinal (y-direction). Indicates the skewness factor of the coordinate axes. This represents the horizontal coordinates of the principal point within a pixel. This represents the vertical coordinate of the principal point within the pixel.

[0031] The Region of Interest (RO) generation unit uses the target depth information and physical size estimation obtained from radar to delineate a rectangular detection box containing the target on the visible light image (RO). Since optical imaging follows the perspective law of near objects appearing larger and farther objects smaller, the image size of a target is inversely proportional to its distance. The RO generation unit uses projected pixel coordinates... Centered on the target depth Calculate the width of the region of interest and height ; ; In the formula, Indicates the pixel width of the region of interest. Indicates the pixel height of the region of interest. Indicates the safety redundancy coefficient. Indicates the target reference physical width. Indicates the target's reference physical altitude. This indicates the camera's lateral equivalent focal length / internal parameter. Indicates the camera's longitudinal equivalent focal length / inner diameter. This indicates the depth of the target in the camera coordinate system.

[0032] The data processing module outputs a series of region-of-interest (ROI) image patches with depth information through the aforementioned processing flow. This processing mechanism leverages the radar's high sensitivity to moving and solid targets, eliminating visual interference from the orchard background caused by swaying leaves and changes in light and shadow. Subsequent recognition algorithms only need to perform convolution operations on these ROIs, rather than scanning the entire image, thus reducing computational requirements and eliminating the risk of false detections in non-target areas. For the temporal registration of radar and visual data, based on the synchronization timestamp provided by the multimodal data acquisition module, the radar point cloud frame with the smallest time difference is selected and associated with an image frame, ensuring the spatiotemporal consistency of high-speed moving pests.

[0033] See attached document Figure 2 , Figure 2 A schematic diagram of the internal logic structure of the recognition decision module is shown. The recognition decision module 300 operates within the computing unit of control box 1, and is used to perform deep feature analysis and multimodal logic determination on the region of interest output by the data processing module. This module specifically includes a visual feature extraction unit, a motion feature analysis unit, and a fusion classification unit.

[0034] The visual feature extraction unit receives normalized region of interest (ROI) image patches output by the data processing module. Since these image patches have undergone spatial guidance from radar data, their background noise is significantly reduced. A lightweight convolutional neural network model is deployed within the visual feature extraction unit. This model contains several cascaded convolutional layers, pooling layers, and activation layers. The convolutional layers use convolutional kernels of specific sizes to slide across the image patches, extracting high-dimensional visual features such as insect texture, wing veins, and body edges; the pooling layers reduce data dimensionality while maintaining the translation invariance of features. After multi-layer network operations, the visual feature extraction unit transforms the two-dimensional image into a one-dimensional visual feature vector. .

[0035] The motion feature analysis unit receives the target radial velocity synchronously output by the radar detection unit. And micro-Doppler frequency shift characteristics. In actual orchard environments, wind-blown leaves and flying insects are easily confused visually, but they are fundamentally different in kinematic characteristics. Leaves under wind force typically exhibit in-situ reciprocating swaying, and their radial velocity... It exhibits low-frequency sinusoidal oscillation characteristics and an average velocity close to zero; while flying pests typically possess continuous directional movement speed or generate specific micro-Doppler spectra due to high-frequency wing vibrations. The motion feature analysis unit performs feature encoding on the raw radar data, extracts velocity amplitude and frequency features, and constructs a motion feature vector. The motion feature analysis unit provides a second dimension of discrimination, distinct from optical texture, by quantifying the target's motion state.

[0036] The fusion classification unit connects the visual feature extraction unit and the motion feature analysis unit, performing multimodal fusion at the feature layer. The fusion classification unit uses a concatenation strategy to combine visual feature vectors... With motion feature vector Merge to generate a joint feature vector This joint feature vector contains both the target's appearance and motion behavior information. Subsequently, the joint feature vector is input into a fully connected layer for weighted mapping, and the Softmax classification function is used to calculate the probability value of the target belonging to each preset category. Assume there are a total of... The target belongs to the first preset category. class probability The calculation is as follows: ; In the formula, Indicates that the target belongs to the first The probability value of the class. The output of the fully connected layer represents the first... The logistic regression value of each node, Indicates the total number of preset categories. This represents the index variable in the summation symbol. Represents the natural constant. This indicates that during the summation process, the output of the fully connected layer corresponds to the first... The original logistic regression values ​​for each category.

[0037] The identification decision module executes anti-false alarm judgment logic based on the above probability calculation results. Preset classification confidence thresholds and motion validity thresholds are provided. When the classification result points to a certain pest category, and the corresponding probability... If the confidence threshold is exceeded, a secondary verification is initiated: the radar radial velocity corresponding to the target is checked. Does it conform to the biological flight characteristics of this type of pest?

[0038] If visual classification identifies it as a pest, but radar detects its radial velocity... If the target is stationary or oscillating, the identification decision module will classify the identification as a false positive or environmental interference and will not trigger an alarm. Conversely, if both visual and motion features point to a pest category, or if visual features are blurred but the radar captures typical insect flight trajectories and micro-Doppler wingbeat signals, the identification decision module will generate a definite pest category label and location information, and send this information as a control command to subsequent modules.

[0039] The avoidance module, acting as the end effector, constructs an automated closed loop from environmental perception to physical intervention. This module mainly consists of a strategy mapping unit, a drive circuit unit, and an acoustic and visual insect repellent device 6, which serves as the physical execution end.

[0040] The acoustic and optical insect repellent 6 is installed inside the control box 1 to ensure that its acoustic and optical coverage range is consistent with the monitoring field of view of the multimodal data acquisition module. The acoustic and optical insect repellent 6 includes a waterproof shell made of high-strength engineering plastic, a high-transmittance optical mask, and a dustproof and waterproof acoustic grille.

[0041] In terms of optical emission structure, the sound and light insect repellent device 6 integrates a multispectral LED array board. This array board uses a high thermal conductivity aluminum substrate as a carrier, on which various specific wavelength light-emitting diode (LED) wafers are soldered using surface mount technology (SMT). Specific light source combinations include: Ultraviolet-induced band: Ultraviolet LED beads with a wavelength range of 365nm to 395nm are arranged in a ring on the outer edge of the array plate to induce or interfere with the visual navigation of insects by taking advantage of their phototaxis. Specific repelling wavelengths: Yellow LED beads with a center wavelength of 585nm and green LED beads with a center wavelength of 520nm are distributed in a matrix in the central area of ​​the array plate to create a light barrier against light-avoiding pests such as noctuid moths. High-brightness strobe band: Full-spectrum white LEDs are used to generate high-intensity instantaneous flashes. The optical mask is injection molded from polycarbonate (PC) material with an anti-UV coating. Its inner side is designed with Fresnel lens texture to reshape the point light source emitted by the LED into a scattered beam that covers the monitoring area, thereby expanding the effective avoidance area.

[0042] In terms of sound wave emission structure, the sound and light insect repellent 6 has a built-in dual-mode sound emission component.

[0043] The first component is a piezoelectric ceramic ultrasonic transducer, fixed in the resonant cavity inside the housing, with an operating frequency response range covering 20kHz to 60kHz. This transducer utilizes the piezoelectric effect to convert high-frequency electrical signals into mechanical vibrations, generating high-sound-pressure level ultrasonic waves that are inaudible to the human ear but can cause neurological disorders in pests.

[0044] The second component is a high-magnetic full-range loudspeaker, mounted facing the acoustic grille on the surface of the housing, used to play audible sound waves in the frequency range of 500Hz to 10kHz, simulating the calls of birds hunting.

[0045] The strategy mapping unit receives the pest category labels output by the identification decision module. Based on location information, a pre-set biological characteristic database is queried to calculate the corresponding light and sound wave control parameters.

[0046] The light wave driving circuit is connected to the multispectral LED array board and uses multiple independent constant current driving sources. When the strategy mapping unit determines that the target is a phototactic pest, the driving circuit selects the ultraviolet LED channel and loads a low-frequency pulse signal. When the pest is identified as a light-avoiding pest, the 585nm yellow LED channel is selected and a constant high brightness output is maintained; For situations requiring strong repulsion, control the white LED to flash at a frequency of 10Hz to 50Hz with a high duty cycle, using the drastic change in luminous flux to force the pests to fly away.

[0047] The acoustic drive circuit connects to a dual-mode sound-generating component and uses direct digital frequency synthesis (DDS) technology to generate the drive waveform. For ultrasonically sensitive pests, the circuit drives the piezoelectric ceramic transducer to output frequency-converted ultrasonic waves and executes a random frequency sweep strategy to prevent pests from developing adaptations. For birds or small rodents, the circuit drives a full-range speaker to play stored acoustic samples of natural enemies.

[0048] The entire avoidance process is a dynamic closed loop. While the acoustic and visual insect repellent device 6 is operating, the radar detection unit continuously monitors the target's radial velocity. and distance ; If the feedback data shows that the pests are moving away or have disappeared, the output will automatically stop to reduce power consumption. If pests remain, the strategy mapping unit will automatically upgrade the control strategy, such as increasing light intensity. Or switch the sound wave frequency Until the repelling effect is achieved.

[0049] See attached document Figure 2 A multimodal fusion intelligent pest monitoring, early warning, and repellency device for orchards, comprising: Control box 1, a rotating shaft structure 5 is fixedly connected to the lower surface of control box 1, a camera 2 is fixedly connected inside control box 1, an adjustment knob 3 is fixedly connected to the upper surface of control box 1, a sound and light insect repellent 6 is fixedly connected inside control box 1, and a bottom heat sink 4 is fixedly connected to the lower surface of control box 1.

[0050] Specifically, control box 1 has a rotating shaft structure 5 fixedly connected to its lower surface, a camera 2 fixedly connected inside control box 1, an adjustment knob 3 fixedly connected to its upper surface, a sound and light insect repellent 6 fixedly connected inside control box 1, and a bottom heat sink 4 fixedly connected to its lower surface.

Claims

1. A multimodal fusion intelligent pest monitoring, early warning, and repellency system for orchards, characterized in that, include: The data acquisition module is configured to acquire multi-dimensional physical information of the orchard environment. The data acquisition module includes an optical imaging unit and a radar detection unit. The optical imaging unit is used to acquire visible light images, and the radar detection unit is used to transmit and receive radar wave signals to analyze the spatial distance, radial velocity and physical size of targets within the monitoring area. The data processing module, connected to the data acquisition module, is configured to receive the visible light image and the radar wave signal, perform data cleaning and coordinate system registration, map the three-dimensional point cloud data in the radar detection unit coordinate system to the camera coordinate system of the optical imaging unit and project it onto the two-dimensional image plane, and generate a region of interest on the visible light image. The identification and decision-making module is connected to the data processing module and is configured to use a convolutional neural network model to extract features from the region of interest, classify the target by combining radar micro-Doppler features, and generate control commands containing pest categories and avoidance parameters based on the classification results and the distance information measured by radar. The avoidance module is connected to the identification and decision module and is configured to receive the control command and drive the physical device to selectively emit a beam of light of a specific wavelength or a sound wave of a specific frequency according to the pest category.

2. The multimodal fusion intelligent pest monitoring, early warning, and repellency system for orchards according to claim 1, characterized in that, The data acquisition module also includes a synchronization triggering circuit; The synchronization trigger circuit is connected to the optical imaging unit and the radar detection unit respectively, and is used to generate a unified timestamp signal to control the exposure time of the optical imaging unit to keep synchronized with the signal transmission time of the radar detection unit. The field of view of the radar detection unit is configured to cover the field of view of the optical imaging unit, and the main beam axis of the radar detection unit and the optical axis of the optical imaging unit are physically parallel or intersect at a preset angle.

3. The multimodal fusion intelligent pest monitoring, early warning, and repellency system for orchards according to claim 1, characterized in that, The data processing module includes a coordinate transformation unit, which stores pre-calibrated rigid body transformation parameters, including rotation matrices and translation vectors. The coordinate transformation unit is configured to perform a linear transformation operation on the target three-dimensional spatial coordinates output by the radar detection unit, and to rotate the target coordinate vector using the rotation matrix and superimpose the translation vector, thereby converting the target coordinates in the radar coordinate system into the target coordinates in the camera coordinate system.

4. The multimodal fusion intelligent pest monitoring, early warning, and repellency system for orchards according to claim 1, characterized in that, The data processing module further includes an image projection mapping unit and a region of interest generation unit; The image projection mapping unit is configured to project the target coordinates in the camera coordinate system onto the two-dimensional image pixel plane using a pre-stored camera intrinsic parameter matrix, thereby obtaining the pixel center coordinates of the target. The region of interest generation unit is configured to calculate the size of the region of interest based on optical perspective laws, and its calculation logic is as follows: The pixel width and pixel height of the region of interest are directly proportional to the product of the target reference physical size and the camera focal length, and inversely proportional to the depth value of the target in the camera coordinate system. The calculation results are expanded according to a preset safety redundancy coefficient.

5. The multimodal fusion intelligent pest monitoring, early warning, and repellency system for orchards according to claim 1, characterized in that, The recognition decision module includes a visual feature extraction unit, a motion feature analysis unit, and a fusion classification unit; The visual feature extraction unit is used to perform convolution operations on the region of interest to extract the texture and edge features of the target and generate a visual feature vector. The motion feature analysis unit is used to perform feature encoding on the raw data output by the radar detection unit, extract velocity amplitude and frequency features to generate motion feature vectors; The fusion classification unit is used to concatenate the visual feature vector and the motion feature vector to generate a joint feature vector.

6. The multimodal fusion intelligent pest monitoring, early warning, and repellency system for orchards according to claim 1, characterized in that, The fusion classification unit is configured to use a fully connected layer to perform a weighted mapping on the joint feature vector and to use a normalized exponential function to calculate the probability of the mapping result; The probability calculation process includes: Calculate the index value of the output node of the fully connected layer, divide the index value by the sum of the index values ​​of all category nodes, and obtain the probability value of the target belonging to the preset pest category.

7. The multimodal fusion intelligent pest monitoring, early warning, and repellency system for orchards according to claim 1, characterized in that, The identification decision module is configured with anti-false alarm determination logic, which specifically includes: When the confidence level of the classification result exceeds a preset threshold, the radar radial velocity corresponding to the target is further detected to see if it is within a preset biologically effective velocity range. If the radar radial velocity indicates that the target is stationary or oscillating, it is determined to be environmental interference and the generation of the control command is prohibited. The control command is generated only when the classification result points to a pest category and the radar radial velocity matches the flight characteristics.

8. The multimodal fusion intelligent pest monitoring, early warning, and repellency system for orchards according to claim 1, characterized in that, The repelling module includes an acoustic and light insect repellent (6), which integrates a multi-spectral light-emitting diode array board and a dual-mode sound-emitting component. The multispectral light-emitting diode array board includes ultraviolet-inducing LEDs with wavelengths from 365nm to 395nm, yellow light-repelling LEDs with wavelengths of 585nm, and full-spectrum white light-strobe LEDs. The dual-modal sound generation component includes a piezoelectric ceramic ultrasonic transducer with a frequency response range of 20kHz to 60kHz and a full-range loudspeaker with a frequency response range of 500Hz to 10kHz.

9. The multimodal fusion intelligent pest monitoring, early warning, and repellency system for orchards according to claim 1, characterized in that, The avoidance module has a built-in closed-loop feedback strategy: During the operation of the sound and light insect repellent (6), the radar detection unit continuously monitors the radial velocity and distance of the target; If the monitoring data shows that the target's radial velocity is pointing in a direction away and the distance is increasing, the avoidance module stops outputting; If the monitoring data shows that the target is stuck, the avoidance module automatically increases the light intensity of the multispectral light-emitting diode array board or switches the sound wave frequency of the dual-mode sound-emitting component.

10. A multimodal fusion intelligent orchard pest monitoring, early warning, and repellency device, applied to the multimodal fusion intelligent orchard pest monitoring, early warning, and repellency system as described in any one of claims 1-9, characterized in that, include; A control box (1) is fixedly connected to a rotating shaft structure (5) on its lower surface. A camera (2) is fixedly connected inside the control box (1). An adjustment knob (3) is fixedly connected to the upper surface of the control box (1). A sound and light insect repellent (6) is fixedly connected inside the control box (1). A bottom heat sink (4) is fixedly connected to the lower surface of the control box (1).

Images