Agricultural pest forecasting method and device

Abstract

According to the agricultural pest forecasting method and device provided by the embodiment of the invention, a fine linkage analysis mechanism between the RGB image sensor and the near infrared spectrum imaging sensor is constructed, the high-resolution RGB image is firstly utilized to carry out rapid coarse screening, and then only when a specific difficult target is identified, the high-resolution RGB image is utilized to carry out rapid coarse screening; and the near infrared spectrum image analysis unit is automatically triggered to carry out deep analysis. According to the target processing mechanism of coarse first and fine second, the spatial resolution of the RGB image and the spectral information dimension advantage of the near-infrared spectral image are fully utilized, the recognition precision of the agricultural pests of the similar species is remarkably improved, the calculation load and the system delay are greatly reduced, and a solution is provided for developing efficient and accurate agricultural pest forecasting equipment.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence, and in particular to a method and apparatus for agricultural pest monitoring and forecasting based on RGB images and near-infrared spectral images. Background Technology

[0002] Agricultural pests are a major biological threat to food security. According to statistics from the Ministry of Agriculture and Rural Affairs, crop yield losses due to pests and their secondary disasters reach as high as 15% annually. Achieving precise pest control hinges on obtaining accurate and timely pest monitoring information. Agricultural pest monitoring devices based on the principle of insect phototaxis have long been the most important tool for targeted monitoring of agricultural pests.

[0003] Agricultural pest monitoring technology has evolved through three generations: The first generation primarily relied on basic light trapping, supplemented by chemical fumigation, with subsequent manual identification and counting of insects by professionals. The second generation introduced modern photoelectric and CNC technologies, enabling functions such as timed automatic light switching, far-infrared insect inactivation treatment, and automatic switching of insect-collecting devices, significantly improving the automation level of the equipment. However, the identification and counting of pest species still required manual work. The third generation achieved a major technological breakthrough by integrating RGB image recognition technology and equipping it with a killing and drying device, an automatic insect separation and dispersing device, and an image acquisition device, enabling the entire process from trapping to identification to be automated. Currently, this type of equipment can accurately identify more than 100 common agricultural insects, covering all major agricultural pests listed as Class I and Class II in China.

[0004] Despite significant advancements in third-generation agricultural pest monitoring devices, their core technology—RGB image-based recognition methods—has fundamental limitations. While RGB imaging offers high spatial resolution and effectively captures the external morphological features of insects, its accuracy drops significantly when distinguishing between closely related species within the same genus or even family. For example, the brown planthopper (a closely related species within the same genus)... Nilaparvata lugens ), false brown planthopper ( Nilaparvata muiri ) and brown planthopper ( Nilaparvata bakeri ), a closely related species in the same family, the beet armyworm ( Spodoptera litura ) and the fall armyworm ( Spodoptera frugiperda These subtle morphological differences between similar species are difficult to reliably capture using RGB images, severely impacting the accuracy of monitoring data and the scientific basis of subsequent prevention and control decisions.

[0005] Spectral imaging recognition systems, capable of capturing the chemical composition information of insects, can theoretically achieve high-precision insect classification, including approximate species. However, these systems suffer from drawbacks such as high data processing computation, high computational resource requirements, system complexity, and high cost, limiting their potential for widespread application. Furthermore, pest monitoring devices attract large numbers of insects of diverse species and sizes, and contain various impurities. Relying solely on spectral image analysis technology makes it difficult to construct a universal multi-target recognition model in complex backgrounds. Even with a full-band spectral imager including RGB channels, its spatial resolution is still far lower than that of a dedicated RGB image sensor, preventing the full extraction of crucial subtle morphological features for identification. Therefore, it is necessary to integrate high-resolution RGB image sensors and high-information-dimensional spectral imaging sensors to simultaneously acquire both external morphological features and internal chemical composition characteristics, constructing an agricultural pest monitoring method and device based on multi-source image fusion analysis technology. Summary of the Invention

[0006] This disclosure addresses the aforementioned technical bottlenecks faced by traditional RGB image analysis technology in identifying pests with extremely similar morphologies, and the challenges of spectral imaging recognition systems, such as high computational resource requirements, large data processing volumes, system complexity, and high costs, making it difficult to develop a universal model for multi-target identification. It proposes an innovative method and device for agricultural pest monitoring and forecasting. This method effectively integrates the high spatial resolution of RGB images with the high information dimensionality of near-infrared spectral images. By constructing a refined collaborative working mechanism between the two imaging sensors, it significantly improves identification accuracy, especially for similar agricultural pest species. It overcomes the shortcomings of existing spectral systems, such as excessive computational load and difficulty in field deployment, thus providing key core technical support for building a new generation of efficient and accurate digital monitoring and early warning systems for crop diseases and pests.

[0007] In a first aspect, embodiments of this disclosure provide a method for monitoring and forecasting agricultural pests, including: The process involves acquiring images of agricultural pests from an image acquisition device; identifying the types of agricultural pests to obtain a first identification result; performing secondary identification on the acquired first type of agricultural pest to obtain a second identification result; and sending the first identification result and / or the second identification result to a client. This method achieves a refined collaborative working mechanism between two imaging sensors, significantly improving identification accuracy.

[0008] The image acquisition device in the above method includes an RGB image sensor and a near-infrared spectral imaging sensor; the first type of agricultural pest includes agricultural pests with similar appearances that are pre-defined by the first identification result.

[0009] The acquisition of agricultural pest images by the image acquisition device specifically involves: acquiring the external morphological features of the agricultural pests acquired by the RGB image sensor; and acquiring the chemical composition features of the agricultural pests acquired by the near-infrared spectral imaging sensor. The RGB image sensor and the near-infrared spectral imaging sensor are connected to an FPGA chip that integrates an RGB image recognition model and a near-infrared spectral image classification model. This solution overcomes the shortcomings of existing spectral systems, such as excessive computational load and difficulty in field deployment, through a refined collaborative working mechanism of the two image sensors. Simultaneously, the FPGA chip integrating the RGB image recognition model and the near-infrared spectral image classification model is connected to the RGB image sensor and the near-infrared spectral imaging sensor. This method reduces the computational load of existing spectral systems, decreases the computational power requirements, and improves the efficiency of accurate digital identification of crop diseases and pests.

[0010] In one alternative embodiment of the first aspect, identifying agricultural pest species to obtain a first identification result includes extracting the external morphological features of the pest acquired by the RGB image sensor to determine high spatial resolution features of the pest's external morphological features; and determining the species of the pest based on the high spatial resolution features of the pest's external morphological features. The agricultural pest images acquired by the RGB image sensor have high spatial resolution, and the RGB image sensor can effectively capture the external morphological features of insects for coarse classification of agricultural pests. Most pest species do not require secondary identification from images acquired by the near-infrared spectral imaging sensor.

[0011] In another alternative embodiment of the first aspect, the step of performing secondary identification on the acquired first type of pest to obtain a second identification result further includes: determining the first position of the first type of agricultural pest based on the agricultural pest image acquired by the RGB image sensor; matching the coordinates of the first image of the first type of agricultural pest acquired by the RGB image sensor to the coordinates of the second image of the first type of agricultural pest acquired by the near-infrared spectral imaging sensor using affine transformation spatial calibration technology, ensuring that the insect target is positioned consistently in both image coordinate systems, and outputting the registered image pair to provide accurate spatial information for subsequent target detection; detecting the registered first type of agricultural pest target acquired by the RGB image sensor, obtaining the first position coordinates, mapping the first position coordinates to the coordinates of the second image of the first type of agricultural pest acquired by the near-infrared spectral image sensor, extracting the region of interest (ROI) of the first type of agricultural pest image acquired by the RGB image sensor, and determining the recognition region of the second image of the first type of agricultural pest acquired by the near-infrared spectral imaging sensor. The near-infrared image processing system analyzes the recognition region data to achieve fine classification of difficult species.

[0012] In another possible implementation, after extracting the region of interest (ROI) from the first type of agricultural pest image acquired by the RGB image sensor and determining the recognition region from the image acquired by the near-infrared spectral imaging sensor, the method further includes preprocessing the recognition region of the first type of agricultural pest image by using Savitzky-Gore filtering to remove noise and applying first-order derivative processing to enhance spectral feature contrast; identifying the optimized feature set of the first type of agricultural pest using a pre-set agricultural pest species classification model; and determining the second recognition result for the first type of agricultural pest species. By constructing a refined collaborative working mechanism between the two image processing systems using the above method, the recognition accuracy and efficiency are significantly improved.

[0013] In another alternative embodiment of the first aspect, before acquiring the image of the agricultural pest from the image acquisition device, the method further includes: attracting the agricultural pest with light; collecting the agricultural pest; and transferring the collected agricultural pest to the acquisition area of ​​the image acquisition device. The above method, by integrating image recognition technology and equipping it with devices such as a killing and drying device and an automatic insect separation and dispersing device, can automatically complete the entire process from trapping to identification.

[0014] Secondly, embodiments of this disclosure provide an agricultural pest monitoring and forecasting system, including an insect-attracting device for attracting agricultural pests using light; a collection device for collecting the agricultural pests; a transmission device for transmitting the collected agricultural pests to the collection area of ​​the image acquisition device; an image acquisition device for acquiring images of the agricultural pests; an image analysis device for identifying the type and quantity of the agricultural pests to obtain a first identification result, and performing secondary identification on the acquired first type of agricultural pests to obtain a second identification result; and a data transmission device for sending the first identification result and / or the second identification result to a client.

[0015] In the aforementioned agricultural pest monitoring system, the image acquisition device includes an RGB image sensor and a near-infrared spectral imaging sensor. The first type of agricultural pest includes: agricultural pests or suspected agricultural pests whose first identification result is a pre-defined category with similar appearance.

[0016] Thirdly, this disclosure also provides an agricultural pest monitoring and forecasting device, including a processor and a memory; the processor is connected to the memory; the memory is used to store executable program code; the processor reads the executable program code stored in the memory to run a program corresponding to the executable program code, so as to implement the agricultural pest monitoring and forecasting method provided by the first aspect or any implementation of the first aspect of this disclosure.

[0017] Fourthly, this disclosure provides a computer-readable storage medium storing computer program code that, when run on a computer, causes the computer to perform any of the methods described in the first aspect and its possible implementations. Such computer-readable storage includes, but is not limited to, one or more of the following: read-only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), flash memory, electrically EPROM (EEPROM), and hard drive.

[0018] Fifthly, this disclosure provides a computer program product containing instructions that, when run on a computer, cause the computer to perform any of the methods described in the first aspect and possible implementations above.

[0019] The beneficial effects of the embodiments disclosed herein include: This disclosure provides an agricultural pest monitoring method, apparatus, and system that establishes a refined linkage analysis mechanism between an RGB image sensor and a near-infrared spectral imaging sensor, effectively solving the technical bottleneck of identifying similar species of agricultural pests. The system employs a hierarchical triggering and collaborative optimization strategy. First, high-resolution RGB images are used for rapid coarse screening, such as distinguishing between white-backed planthoppers, gray planthoppers, or classifying them into the genus *Brassica oleracea*. Then, only when specific difficult targets, such as those in the genus *Brassica oleracea*, are identified, the near-infrared spectral image analysis unit is automatically triggered for in-depth analysis to accurately distinguish between *Brassica oleracea*, *Brassica pseudo-Brassica*, or *Brassica pseudo-Brassica*. This targeted processing mechanism, which combines coarse-to-fine analysis, fully utilizes the spatial resolution of RGB images and the spectral information dimensionality of near-infrared spectral images, significantly improving the accuracy of identifying similar species of agricultural pests while greatly reducing computational load and system latency. This provides a solution for developing efficient and accurate agricultural pest monitoring equipment. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this disclosure, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is an overall flowchart of an agricultural pest monitoring and forecasting method provided in an embodiment of the present disclosure; Figure 2 A flowchart for obtaining a first identification result by identifying agricultural pests provided in an embodiment of this disclosure; Figure 3 A flowchart for obtaining a second identification result by identifying agricultural pests provided in this embodiment of the disclosure; Figure 4 This is a schematic diagram of the architecture of an agricultural pest monitoring and forecasting system provided in an embodiment of the present disclosure; Figure 5 This is a schematic diagram of the structure of an agricultural pest monitoring device provided in an embodiment of this disclosure. Detailed Implementation

[0022] The technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0023] In the following description, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance. The following description provides multiple embodiments of this disclosure, which can be substituted or combined with each other. Therefore, this disclosure should also be considered to include all possible combinations of the same and / or different embodiments described. Thus, if one embodiment includes features A, B, and C, and another embodiment includes features B and D, then this disclosure should also be considered to include embodiments containing one or more other possible combinations of A, B, C, and D, even if such embodiments are not explicitly described in the following text.

[0024] The terminology used in the following embodiments is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to also include expressions such as “one or more,” unless the context clearly indicates otherwise. It should also be understood that in the following embodiments of this application, “at least one” and “one or more” refer to one or more (including two). The term “and / or” is used to describe the relationship between related objects, indicating that three relationships can exist; for example, A and / or B can indicate: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character “ / ” generally indicates that the preceding and following related objects are in an “or” relationship.

[0025] References to "one embodiment" or "some embodiments" as used in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized. The term "connection" includes both direct and indirect connections, unless otherwise stated.

[0026] In the embodiments of this application, the words "exemplarily" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplarily" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of the words "exemplarily" or "for example" is intended to present the relevant concepts in a specific manner.

[0027] The following description provides examples and does not limit the scope, applicability, or examples set forth in the claims. Changes may be made to the function and arrangement of the described elements without departing from the scope of this disclosure. Various processes or components may be appropriately omitted, substituted, or added to the examples. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Furthermore, features described with respect to some examples may be combined into other examples.

[0028] Overall Plan Please see Figure 1 , Figure 1 A flowchart illustrating the overall process of an agricultural pest monitoring and forecasting method provided in this embodiment is shown.

[0029] like Figure 1 As shown, the agricultural pest monitoring and forecasting method may include at least the following steps: Step S102: Acquire the image of the agricultural pest captured by the image acquisition device.

[0030] In this embodiment of the disclosure, the agricultural pest monitoring and forecasting method can be applied to, but is not limited to, edge computing devices, which can establish communication connections with both the image acquisition device and the mobile terminal corresponding to the client.

[0031] Agricultural pests are lured to the vicinity of the collection device using light, and the live insects are adsorbed onto the collection panel of the image acquisition device using a negative pressure device. Alternatively, a thermal infrared insecticidal drying device is used to kill the pests, and a conveyor belt is used to separate the insects individually and lay them flat on the collection panel. The conveyor platform is configured to evenly spread the agricultural pests by vibration, with an amplitude of 2mm and a frequency of 5Hz, so that the insects are laid in a single layer (spacing ≥5mm). The bottom of the conveyor belt integrates a calibration scale grid (1mm×1mm). When an insect passes through the collection area of ​​the RGB image sensor, it triggers position encoding, generating two-dimensional coordinates (X,Y). For example, after dispersing 12 white-backed planthoppers, gray planthoppers, and brown planthoppers in a uniform material layer, they are conveyed to the image acquisition area by a conveyor belt at a speed of 10mm / s. When the RGB image sensor detects an insect, it automatically records the position coordinates, such as: P0 (X0, Y0), P1 (X1, Y1)...P11 (X11, Y11).

[0032] An RGB image sensor is used to acquire the external morphological features of agricultural pests and determine the high spatial resolution features of these features. The RGB image sensor is used to acquire the external morphology of the pests and extract their morphological features, such as color, body length, and texture.

[0033] Near-infrared spectroscopy, based on the principle of molecular vibrational spectroscopy, non-destructively obtains vibrational energy level transition information of chemical bonds within insect samples by detecting the characteristic absorption and reflection responses of light in the 1000-2500 nm wavelength range. This technology can accurately analyze the hydrocarbon composition characteristics of insect epidermis. It offers advantages such as in-situ non-destructive processing, no sample pretreatment required, and rapid detection, providing a high-precision biochemical fingerprinting method for real-time monitoring of field pests.

[0034] This solution establishes a sophisticated linkage analysis mechanism between RGB image sensors and near-infrared spectral imaging sensors, effectively addressing the technical bottleneck in identifying similar species of agricultural pests. The system employs a hierarchical triggering and collaborative optimization strategy. This coarse-to-fine targeted processing mechanism fully leverages the spatial resolution of RGB and the dimensionality of near-infrared spectral chemical composition information, significantly improving the accuracy of similar species identification while drastically reducing computational load and system latency. This provides a solution for developing efficient and accurate agricultural pest monitoring equipment.

[0035] Solution Implementation Details Please see Figure 2 , Figure 2 A flowchart illustrating the process of obtaining a first identification result for identifying agricultural pests provided in an embodiment of this disclosure is shown.

[0036] Step S104: Identify the types of agricultural pests to obtain the first identification result.

[0037] Specifically, the process includes: S201 extracting the external morphological features of the pest collected by the RGB image sensor to determine the high spatial resolution features of the external morphological features of the pest; S202 determining the species of the pest based on the high spatial resolution features of the external morphological features of the pest to obtain a first identification result, wherein the first identification result includes common insect species and similar-looking difficult insect species.

[0038] In this embodiment, an agricultural pest species classification model pre-installed in an edge computing device, such as an FPGA chip integrating an RGB image recognition model and a near-infrared spectral image classification model, is used to match the high spatial resolution features of the external morphological characteristics of the agricultural pests to determine the pest species. The edge computing device can input each frame of agricultural pest image into a trained convolutional neural network model to label each frame with species of the genera *White-backed Planthopper*, *Grey Planthopper*, and *Brown Planthopper*. Then, the species is determined based on the high spatial resolution features of the identified agricultural pest external morphological characteristics, and the counting results can be sent to the corresponding mobile terminal of the client for quick viewing by the user. For example, for an insect at coordinate P0, if the measured morphological characteristics—body length, color, and texture—conform to the *White-backed Planthopper* genus, the system labels it as "White-backed Planthopper". The edge computing device can also send agricultural pest images to the corresponding mobile terminal of the client, ensuring not only a correspondence between agricultural pest species and images but also traceability of the results.

[0039] In this embodiment, the RGB image sensor can be composed of a high-precision imaging device, such as a single fixed-focus industrial camera, along with its auxiliary control hardware and data processing unit. The entire RGB image sensor and edge computing device execute a precise workflow for agricultural pest identification. This system is capable of extracting single insect features within 200ms in high-temperature and high-humidity farmland environments, ensuring a 98.7% identification accuracy rate for planthoppers.

[0040] In this embodiment, the Web service platform can also be used to receive agricultural pest images collected by an RGB image sensor, extract at least two frames from the agricultural pest images, and input each frame of the agricultural pest image into a trained convolutional neural network model to label the white-backed planthopper, gray planthopper, and brown planthopper genera in each frame. It can also count the identified agricultural pest species and send the counting results to the corresponding mobile terminal of the client, allowing the user to quickly view the data on the mobile terminal. Here, the Web service platform can also send the agricultural pest images along with the counting results to the corresponding mobile terminal of the client, ensuring that the counting results correspond to the agricultural pest images and guaranteeing the traceability of the counting results.

[0041] The mobile terminal corresponding to the client can, but is not limited to, having an agricultural pest monitoring and survey app installed. This app is used to receive agricultural pest identification and counting results, or identification results and corresponding agricultural pest images, sent from edge computing devices or web service platforms. The identification results and images can be displayed on the agricultural pest monitoring and survey app. Here, the mobile terminal can be accessed by the user who logs into their account on the agricultural pest monitoring and survey app and receives agricultural pest identification and counting results, or identification results and corresponding agricultural pest images, sent from edge computing devices or web service platforms, either manually or automatically. It is understood that the mobile terminal can also obtain real-time time, weather, and geographic information, and users can manually fill in information such as field location, rice variety, and survey personnel.

[0042] This embodiment utilizes the high spatial resolution of RGB imaging, which effectively captures the external morphological features of insects, to identify the species of most agricultural pests. For agricultural pests identified as having similar appearances to pre-defined species, images acquired using a near-infrared spectral imaging sensor need to be re-analyzed to further distinguish the similar species. For common insect species of agricultural pests, there is no need to re-analyze images acquired by the near-infrared spectral imaging sensor, thus balancing accuracy and cost-effectiveness in identification.

[0043] Please see Figure 3 , Figure 3 A flowchart illustrating the process of obtaining a second identification result by performing secondary identification on the first type of agricultural pests obtained in this embodiment of the present disclosure.

[0044] As an optional embodiment of this disclosure.

[0045] Step S106: Perform secondary identification on the obtained first type of pest to obtain a second identification result.

[0046] Specifically, this includes: matching the coordinates of the first image acquired by the S301 RGB image sensor to the coordinates of the second image acquired by the near-infrared spectral imaging sensor; The affine transformation spatial calibration technique is used to match the coordinates of the first image of the first type of agricultural pest acquired by the RGB image sensor to the coordinates of the second image of the first type of agricultural pest acquired by the near-infrared spectral imaging sensor, ensuring that the insect target is in the same position in the two image coordinate systems, and outputting the registered image pair to provide accurate spatial information for subsequent target detection. Traditional affine calibration algorithms assume that two images are in the same spectrum and can directly solve the global 6-DOF affine matrix using SIFT / SURF feature points. However, in visible-near-infrared spectral imaging registration scenarios, due to the large differences in texture and contrast between cross-spectral images, conventional descriptions struggle to find sufficient common points, leading to a significant drop in accuracy. To address this, we first perform PCA on the spectral cube, selecting the first principal component with the closest contrast to the agricultural pest image acquired by the RGB image sensor and compressing it into a single-band grayscale image. Then, we perform histogram matching or DoG normalization on this grayscale image and the RGB grayscale image to equalize brightness and contrast. Finally, we extract SURF keypoints from the two normalized grayscale images, use RANSAC to eliminate mismatches, and obtain a 3 × 3 homography matrix, thereby achieving pixel-level high-precision registration of cross-spectral images. A global 6-DOF affine matrix refers to a single set of 6 parameters used to describe the mapping relationship of all pixels from source coordinates (X1,Y1), (X2,Y2), (X3,Y3) to target coordinates (X1',Y1'), (X2',Y2'), (X3',Y3') across the entire 2D image.

[0047] S302. Extract the region of interest (ROI) from the first type of agricultural pest image acquired by the RGB image sensor, and determine the recognition region of the image acquired by the near-infrared spectral imaging sensor. Use the YOLO algorithm to detect the first type of agricultural pest target acquired by the registered RGB image sensor, obtain the first position coordinates, and map the first position coordinates to the second image coordinates of the first type of agricultural pest acquired by the near-infrared spectral imaging sensor. Extract the ROI from the first type of agricultural pest image acquired by the RGB image sensor, and determine the recognition region of the second image of the first type of agricultural pest acquired by the near-infrared spectral imaging sensor.

[0048] S303. Preprocess the recognition region of the first type of agricultural pest image to obtain the feature set of the first type of agricultural pest.

[0049] The YOLO algorithm is used to output a single six-tuple (xmin, ymin, w, h, conf, classid) for each insect, where (xmin, ymin) are the normalized coordinates of the top-left corner of the detection box in the original image, w and h are the width and height of the box, classid represents the insect category, and conf is the recognition confidence score. First, the near-infrared spectral data is compressed using PCA to a narrow band that best matches the contrast of RGB features, resulting in a grayscale image. Then, histogram matching or DoG normalization is performed on the two images to reduce differences in illumination and contrast. Next, SURF keypoints are extracted, and RANSAC is used to filter out incorrect matches, identifying inliers and fitting a 3 × 3 homography matrix H. Finally, the inliers are refined using the least squares method to reduce the mean square error of reprojection.

[0050] The regions of interest in the images of the first type of agricultural pests are preprocessed, and noise is removed by Savitsky-Gore filtering. First derivative processing is applied to enhance the contrast of spectral features. First derivative is used to emphasize spectral peaks and key features, thereby enhancing the interpretability of the data and achieving better classification performance.

[0051] In the quantitative analysis, we selected a wavelength range of 1000-2500nm. By effectively minimizing the influence of irrelevant information and covariates, and by identifying the spectral features with the most information content, we improved the accuracy and efficiency of the model, thereby promoting data dimensionality reduction.

[0052] S304. Identify the optimized first type of agricultural pest feature set using a pre-set agricultural pest species classification model.

[0053] Images of the first type of agricultural pests in the 1000-2500 nm band are selected, and the feature set of the first type of agricultural pests is further optimized; the optimized feature set of the first type of agricultural pests is identified using a pre-set agricultural pest species classification model; and the second identification result of the species of the first type of agricultural pests is determined.

[0054] By constructing a refined collaborative recognition mechanism for the two imaging sensors using the above method, the recognition accuracy and efficiency are significantly improved.

[0055] As an optional embodiment of this disclosure, step S108, sending the first identification result and / or the second identification result to the client, includes: Edge computing devices or web service platforms can send the identification and counting results to the corresponding mobile terminals of the clients, allowing users to quickly view them on their mobile devices. Edge computing devices or web service platforms can also send images of agricultural pests along with the counting results to the corresponding mobile terminals of the clients, ensuring a correlation between the counting results and the agricultural pest images, and guaranteeing the traceability of the counting results.

[0056] The mobile terminal corresponding to the client can, but is not limited to, having an agricultural pest monitoring and survey app installed. This app is used to receive agricultural pest identification and counting results, or identification results and corresponding agricultural pest images, sent from edge computing devices or web service platforms. The identification results and images can be displayed on the agricultural pest monitoring and survey app. Here, the mobile terminal can be accessed by the user who logs into their account on the agricultural pest monitoring and survey app and receives agricultural pest identification and counting results, or identification results and corresponding agricultural pest images, sent from edge computing devices or web service platforms, either manually or automatically. It is understood that the mobile terminal can also obtain real-time time, weather, and geographic information, and users can manually fill in information such as field location, rice variety, and survey personnel.

[0057] Please see Figure 4 , Figure 4 This is a schematic diagram of the architecture of an agricultural pest monitoring and forecasting system provided in an embodiment of this disclosure.

[0058] like Figure 4 As shown, the agricultural pest monitoring system includes at least an insect-attracting device 401, a collection device 402, a transmission device 403, an image acquisition device 404, an image analysis device 405, and a data transmission device 406. Specifically: the insect-attracting device 401 is used to attract agricultural pests using light; the collection device 402 is used to collect the agricultural pests; the transmission device 403 is used to transmit the collected agricultural pests to the acquisition area of ​​the image acquisition device; the image acquisition device 404 is used to acquire images of the agricultural pests; the image analysis device 405 is used to identify the type and quantity of the agricultural pests to obtain a first identification result, and to perform a second identification on the acquired first type of agricultural pest to obtain a second identification result; the data transmission device 406 is used to send the first identification result and / or the second identification result to a client.

[0059] The aforementioned agricultural pest monitoring and forecasting system integrates image recognition technology and is equipped with LED trapping light sources, automatic collection devices, and image acquisition devices. It can automatically complete the entire process from trapping to identification, and establishes a fine linkage analysis mechanism between RGB image sensors and near-infrared spectral imaging sensors, effectively solving the technical bottleneck of identifying closely related species of agricultural pests.

[0060] In some possible embodiments, the image acquisition device 404 is used for: The external morphological features of the agricultural pests collected by the RGB image sensor are acquired; the chemical composition features of the agricultural pests collected by the near-infrared spectral imaging sensor are acquired.

[0061] The external morphology of the pest is captured by an image acquisition device, and the morphological characteristics of the agricultural pest, such as body length, color, and texture, are extracted.

[0062] In some possible embodiments, the image analysis device 405 is also used for: The type of agricultural pest is determined by matching the high spatial resolution features of the external morphological characteristics of the pests using a pre-set agricultural pest species classification model in an edge computing device. The edge computing device can input each frame of agricultural pest image into a trained convolutional neural network model to label the white-backed planthopper, gray planthopper, and brown planthopper in each frame. Then, it counts the pests based on their identified status and sends the results to the corresponding mobile terminal for quick viewing by the user.

[0063] In some possible embodiments, the image analysis device 405 is also used for: The first type of agricultural pest is located at a first position on the agricultural pest transport platform based on the agricultural pest image acquired by the RGB image sensor. The coordinates of the first image of the first type of agricultural pest acquired by the RGB image sensor are matched to the coordinates of the second image of the first type of agricultural pest acquired by the near-infrared spectral imaging sensor using affine transformation spatial calibration technology. This ensures that the insect target is positioned consistently in both image coordinate systems and outputs the registered image pair, providing accurate spatial information for subsequent target detection. Traditional affine calibration algorithms assume that the two images are in the same spectrum and can directly solve the global 6-DOF affine matrix using SIFT / SURF feature points. However, in the visible-near-infrared spectral registration scenario, due to the large differences in texture and contrast between the cross-spectral images, conventional descriptors struggle to find sufficient common points, resulting in a significant decrease in accuracy. To this end, we first perform PCA on the spectral cube, select the first principal component with the closest contrast to RGB, and compress it into a single-band grayscale image. Then, we perform histogram matching or DoG normalization on this grayscale image and the RGB grayscale image to equalize brightness and contrast. Finally, we extract SURF keypoints from the two normalized grayscale images, use RANSAC to remove mismatches, and obtain a 3 × 3 homography matrix, thus achieving sub-pixel-level high-precision registration of cross-spectral images. A global 6-DOF affine matrix refers to a single set of 6-parameter affine transformations used to describe the mapping relationship of all pixels from source coordinates (X1,Y1), (X2,Y2), (X3,Y3) to target coordinates (X1',Y1'), (X2',Y2'), (X3',Y3') across the entire 2D image.

[0064] The YOLO algorithm is used to detect the first type of agricultural pest target acquired by the registered RGB image sensor, obtain the first position coordinates, and map the first position coordinates to the second image coordinates of the first type of agricultural pest acquired by the near-infrared spectral imaging sensor. The region of interest of the first type of agricultural pest image acquired by the RGB image sensor is extracted, and the recognition region of the second image of the first type of agricultural pest acquired by the near-infrared spectral imaging sensor is determined.

[0065] The YOLO algorithm is used to output a single six-tuple (xmin, ymin, w, h, conf, class_id) for each insect, where (xmin, ymin) are the normalized coordinates of the top-left corner of the detection box in the original image, w and h are the width and height of the box, class_id represents the insect category, and conf is the recognition confidence score. First, the multispectral data is compressed using PCA to a narrow band that best matches the RGB feature contrast, resulting in a grayscale image. Then, histogram matching or DoG normalization is performed on the two images to reduce differences in illumination and contrast. Next, SURF keypoints are extracted, and RANSAC is used to filter out incorrect matches, identifying inliers and fitting a 3 × 3 homography matrix H. Finally, the inliers are refined using least squares to reduce the mean square error of reprojection.

[0066] The identification regions of the first type of agricultural pest images are preprocessed, and noise is removed by Savitsky-Gore filtering. First derivative processing is applied to enhance the contrast of spectral features. First derivative is used to emphasize spectral peaks and key features, enhance the interpretability of data, and thus achieve better classification performance.

[0067] In the quantitative analysis, we selected images of the first type of agricultural pests with a wavelength range of 1000-2500 nm, further optimized the feature set of the first type of agricultural pests, and used a pre-set agricultural pest species classification model to identify the optimized feature set of the first type of agricultural pests; and determined the second identification result of the species and quantity of the first type of agricultural pests.

[0068] The system adopts a hierarchical triggering and collaborative optimization strategy. This coarse-to-fine targeted processing mechanism makes full use of the spatial resolution of RGB and the dimensionality of near-infrared spectral information. While significantly improving the accuracy of identifying closely related species, it greatly reduces the computational load and system latency, providing a solution for developing efficient and accurate agricultural pest monitoring equipment.

[0069] Those skilled in the art will clearly understand that the above embodiments can be implemented entirely or partially by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state disks, SSDs), etc.

[0070] Please see Figure 5 , Figure 5 A schematic diagram of the structure of an agricultural pest monitoring device provided in an embodiment of this disclosure is shown.

[0071] like Figure 5 As shown, the agricultural pest monitoring device 500 may include at least a processor 501, at least one network interface 504, a user interface 503, a memory 505, and at least one communication bus 502.

[0072] The communication bus 502 can be used to realize the connection and communication of the above components.

[0073] The user interface 503 may include buttons, and the optional user interface may also include a standard wired interface or a wireless interface.

[0074] The network interface 504 may include, but is not limited to, Bluetooth modules, NFC modules, Wi-Fi modules, etc.

[0075] The processor 501 may include one or more processing cores. The processor 501 connects to various components within the agricultural pest monitoring device 500 via various interfaces and lines. It executes instructions, programs, code sets, or instruction sets stored in the memory 505, and calls data stored in the memory 505 to perform various functions and process data within the agricultural pest monitoring device 500. Optionally, the processor 501 may be implemented using at least one hardware form of DSP, FPGA, or PLA. The processor 501 may integrate one or more of the following: CPU, GPU, and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content to be displayed on the screen; and the modem handles wireless communication. It is understood that the modem may also not be integrated into the processor 501 and may be implemented as a separate chip.

[0076] The memory 505 may include RAM or ROM. Optionally, the memory 505 may include a non-transitory computer-readable medium. The memory 505 may be used to store instructions, programs, code, code sets, or instruction sets. The memory 505 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above-described method embodiments, etc.; the data storage area may store data involved in the above-described method embodiments, etc. Optionally, the memory 505 may also be at least one storage device located remotely from the aforementioned processor 501. Figure 5 As shown, the memory 505, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and an agricultural pest monitoring application.

[0077] Specifically, the processor 501 can be used to call the agricultural pest monitoring application stored in the memory 505, and specifically perform the following operations: acquire images of the agricultural pests collected by the image acquisition device; identify the type and quantity of the agricultural pests to obtain a first identification result; perform secondary identification on the acquired first type of agricultural pests to obtain a second identification result; and send the first identification result and / or the second identification result to the client. The image acquisition device includes an RGB image sensor and a near-infrared spectral imaging sensor. The first type of agricultural pest includes: agricultural pests with similar appearances that are pre-defined in the first identification result.

[0078] In some possible embodiments, acquiring the agricultural pest image acquired by the image acquisition device includes: acquiring the external morphological features of the agricultural pest acquired by the RGB image sensor; and acquiring the chemical composition features of the agricultural pest acquired by the near-infrared spectral imaging sensor.

[0079] In some possible embodiments, identifying the species and quantity of the agricultural pest to obtain a first identification result further includes: extracting the external morphological features of the pest collected by the RGB image sensor to determine the high spatial resolution features of the external morphological features of the pest; and determining the species of the pest based on the high spatial resolution features of the external morphological features of the pest.

[0080] In some possible embodiments, a second identification result is obtained by performing secondary identification on the acquired first type of agricultural pest, including: determining the first position of the first type of agricultural pest on the agricultural pest conveying platform based on the agricultural pest image acquired by the RGB image sensor; matching the first image coordinates of the first type of agricultural pest acquired by the RGB image sensor to the second image coordinates of the first type of agricultural pest acquired by the near-infrared spectral imaging sensor using affine transformation spatial calibration technology, ensuring that the insect target is in the same position in the two image coordinate systems, and outputting the registered image pair to provide accurate spatial information for subsequent target detection; using the YOLO algorithm to detect the first type of agricultural pest target acquired by the registered RGB image sensor, obtaining the first position coordinates, mapping the first position coordinates to the second image coordinates of the first type of agricultural pest acquired by the near-infrared spectral imaging sensor, extracting the region of interest of the first type of agricultural pest image acquired by the RGB image sensor, and determining the recognition region of the second image of the first type of agricultural pest acquired by the near-infrared spectral imaging sensor.

[0081] In some possible embodiments, after extracting the region of interest from the first type of agricultural pest image acquired by the RGB image sensor and determining the recognition region of the image acquired by the near-infrared spectral imaging sensor, the method further includes: preprocessing the recognition region of the first type of agricultural pest image by using Savitsky-Gore filtering to remove noise and applying first derivative processing to enhance spectral feature contrast; selecting the first type of agricultural pest image with a wavelength range of 1000-2500 nm to further optimize the first type of agricultural pest feature set; identifying the optimized first type of agricultural pest feature set using a pre-set agricultural pest species classification model; and determining the second recognition result of the first type of agricultural pest species.

[0082] This disclosure also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method. The computer-readable storage medium may include, but is not limited to, any type of disk, including floppy disks, optical disks, DVDs, CD-ROMs, microdrives, as well as magneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs, DRAMs, VRAMs, flash memory devices, magnetic cards or optical cards, nanosystems (including molecular memory ICs), or any type of medium or device suitable for storing instructions and / or data.

[0083] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this disclosure is not limited to the described order of actions, because according to this disclosure, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this disclosure.

[0084] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0085] In the several embodiments provided in this disclosure, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the shown or discussed mutual couplings or direct couplings or communication connections may be through some service interfaces; indirect couplings or communication connections between apparatuses or units may be electrical or other forms.

[0086] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0087] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0088] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this disclosure. The aforementioned memory includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0089] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by instructing related hardware through a program. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

Claims

1. A method for monitoring and forecasting agricultural pests, characterized in that, include: Acquire images of the agricultural pests captured by the image acquisition device; The first identification result is obtained by identifying the species of agricultural pests; A second identification result is obtained by performing secondary identification on the first type of agricultural pests; Send the first identification result and / or the second identification result to the client; The image acquisition device includes: an RGB image sensor and a near-infrared spectral imaging sensor; The first type of agricultural pest includes: agricultural pests with similar appearances that are pre-defined by the first identification result.

2. The method according to claim 1, characterized in that, The acquisition of agricultural pest images by the image acquisition device specifically involves: acquiring the external morphological features of the agricultural pests acquired by the RGB image sensor; and acquiring the chemical composition features of the agricultural pests acquired by the near-infrared spectral imaging sensor.

3. The method according to any one of claims 2, characterized in that, The identification of agricultural pest species to obtain the first identification result includes: Extract the external morphological features of the pest collected by the RGB image sensor to determine the high spatial resolution features of the external morphological features of the pest; The species of the pest is determined based on the high spatial resolution characteristics of its external morphological features.

4. The method according to claim 2 or 3, characterized in that, The second identification result obtained by performing secondary identification on the first type of pest includes: The first location of the first type of agricultural pest is determined based on the agricultural pest image acquired by the RGB image sensor. The affine transformation spatial calibration technique is used to match the coordinates of the first image of the first type of agricultural pest acquired by the RGB image sensor to the coordinates of the second image of the first type of agricultural pest acquired by the near-infrared spectral imaging sensor, ensuring that the insect target is in the same position in the two image coordinate systems, and outputting the registered image pair to provide accurate spatial information for subsequent target detection. The first type of agricultural pest target acquired by the registered RGB image sensor is detected, the first position coordinates are obtained, and the first position coordinates are mapped to the second image coordinates of the first type of agricultural pest acquired by the near-infrared spectral imaging sensor. The region of interest of the first type of agricultural pest image acquired by the RGB image sensor is extracted, and the recognition region of the second image of the first type of agricultural pest acquired by the near-infrared spectral imaging sensor is determined.

5. The method according to claim 4, characterized in that, After extracting the region of interest from the first type of agricultural pest image acquired by the RGB image sensor and determining the recognition region of the second image of the first type of agricultural pest acquired by the near-infrared spectral imaging sensor, the method further includes: The identification region of the first type of agricultural pest image is preprocessed, noise is removed by Savitsky-Gore filter, and the contrast of spectral features is enhanced by first derivative processing. The first type of agricultural pest images in key spectral bands are screened using a competitive adaptive reweighted sampling algorithm. Feature importance analysis is performed by combining adaptive gradient boosting decision tree and adaptive gradient boosting classification algorithm. The first type of agricultural pest feature set is obtained by retaining the top N most distinguishable band features. The optimized feature set of the first type of agricultural pests is identified using a pre-set agricultural pest species classification model. The second identification result determines the species of the first type of agricultural pest.

6. The method according to claim 1 or 5, characterized in that, Before the image acquisition device acquires the image of the agricultural pest, the following is also included: The agricultural pests were attracted using light. Collect the aforementioned agricultural pests; The collected agricultural pests are transmitted to the acquisition area of ​​the image acquisition device.

7. The method according to any one of claims 1-6, characterized in that, The RGB image sensor and the near-infrared spectral imaging sensor are connected to an FPGA chip that integrates an RGB image recognition model and a near-infrared spectral image classification model.

8. An agricultural pest monitoring and forecasting system, characterized in that, include: Insect-attracting devices are used to lure the agricultural pests using light; Collection device for collecting the agricultural pests; A conveying device for conveying the collected agricultural pests to the acquisition area of ​​the image acquisition device; Image acquisition device, used to acquire images of the agricultural pests; An image analysis device is used to identify the type and quantity of the agricultural pests to obtain a first identification result, and to perform secondary identification on the first type of agricultural pests to obtain a second identification result. A data transmission device is used to send the first identification result and / or the second identification result to a client; The image acquisition device includes: an RGB image sensor and a near-infrared spectral imaging sensor; The first type of agricultural pest includes: agricultural pests with similar appearances that are pre-defined by the first identification result.

9. An agricultural pest monitoring and forecasting device, characterized in that, Including the processor and memory; The processor is connected to the memory; The memory is used to store executable program code; The processor runs a program corresponding to the executable program code stored in the memory to perform the steps of the method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, The computer-readable storage medium stores instructions that, when executed on a computer or processor, cause the computer or processor to perform the steps of the method as described in any one of claims 1-7.

Images