Real-time greenhouse flower thrip detection and counting method based on positioning type full convolutional network

Abstract

The invention discloses a greenhouse flower thrips real-time detection and counting method based on a positioning type full convolutional network, which can accurately count and detect flower thrips on xx flower leaves in a greenhouse in real time. The TCD-Net is a unique full convolutional network structure, takes efficient ConvNeXtV2-Nano as a backbone network, extracts abundant flower thrip features in combination with lightweight Hybrid attention, and realizes accurate counting and detection by predicting a small area for each flower thrip. Experiments are carried out on a data set containing more than 470000 flower thrip marks, and results show that the TCD-Net obtains very ideal counting and detection results, and meanwhile, the TCD-Net has relatively low model complexity and operation speed far exceeding that of real-time detection. On a test set, the F1 fraction of TCD-Net reaches 89.71%, the correlation of counting results reaches 83.82%, and the counting and detection precision of the method exceeds that of an existing method. Meanwhile, the model size (15.94 M) and the theoretical calculation amount (83.92 GFLOPs) of the method are less than half of that of a two-stage target detection method, and the reasoning speed (99.49 bit / s) is more than two times of that of the two-stage target detection method.

Description

Technical Field

[0001] This invention relates to the field of greenhouse flower thrips research technology, and more specifically to a method for real-time detection and counting of greenhouse flower thrips based on a localized fully convolutional network. Background Technology

[0002] Thrips are insects belonging to the family Thripidae in the order Thysanoptera. They are tiny and reproduce rapidly. Nymphs are small, entirely yellow, brown, or black. Eggs are small, varied in shape including kidney-shaped, round, and oval, and in color including colorless, white, and yellow (Wu et al., 2017). Thrips have a diverse diet, but are primarily herbivorous. Most thrips live on flowers, inflorescences, tender branches, and young leaves, feeding on pollen, nectar, and fruits. They can damage more than 200 species of crops from 62 families, including Cucurbitaceae, Leguminosae, Brassicaceae, and Solanaceae (Kirk et al., 2021). Thrips cause serious economic losses both domestically and internationally. Thrips control is challenging for three main reasons: First, these pests are small and highly concealed, preferring to hide in flowers, tender shoots, and the undersides of leaves, making them difficult to detect. Second, they have a short life cycle, reproduce rapidly, and quickly develop resistance to chemical pesticides, making outbreaks extremely easy. Third, they have a wide host range, strong dispersal ability, and good ecological adaptability, causing serious damage to a variety of crops (Steenbergen et al., 2018). Therefore, early counting, detection, and prevention of bean thrips outbreaks are crucial.

[0003] Traditional manual pest counting methods are time-consuming and labor-intensive. In recent years, with the development of computer vision and deep learning technologies, the use of intelligent methods to automatically identify pests in images has been widely studied. The application of these methods can effectively improve the efficiency of pest monitoring and reduce survey costs. Currently, most pest counting and detection methods are developed based on deep learning target detection methods. The main improvements include optimizing the feature extraction backbone, improving the Feature Pyramid Network (FPN), improving the Region Proposal Network (RPN), and optimizing the anchor generation and selection mechanism to adapt to pest counting and detection tasks. For example, Liu et al. (2021b) improved FPN and RPN, enhancing the global feature extraction capability of FPN and the local feature perception capability of RPN, achieving a mAP of 74.24%. Wang et al. (2021a) added an attention mechanism to Faster R-CNN, enriching the features extracted by the network, achieving a mAP of 78.7%. Jiao et al. (2022a) improved the feature extraction capability of the model and the anchor generation mechanism of the detector, achieving a mAP of 77.8%. These studies demonstrate the strong benchmark performance of target detection in pest counting and detection tasks, and make effective improvements to address the problems of small pest size and complex backgrounds, promoting the application of target detection methods in expert systems for agricultural pest detection.

[0004] However, research on accurate and efficient counting methods for extremely small pests such as thrips, rice planthoppers, and whiteflies is still lacking. Small target detection has always been a challenge for target detectors, prone to missed detections and false detections. Extremely small pests have few features and low signal-to-noise ratios, making it difficult for target detectors to extract sufficient features or locate accurate target anchors. Some scholars have explored solutions to these problems. (He et al., 2020) used Faster R-CNN to achieve counting and detection of brown planthoppers. (Lee et al., 2020) also used Faster R-CNN to detect tea thrips, achieving a detection accuracy of over 70%. (Wang et al., 2021a) redesigned the RPN and reconstructed the anchor box regression mechanism, improving the detection performance of rice planthoppers in the field. (Wang et al., 2021b) fused an attention mechanism in the feature extraction stage and introduced a balanced sampling strategy into the RPN. (Huang et al., 2023) integrated attention and feature fusion techniques into YOLOv5 to achieve accurate detection of brown planthoppers. (De Cesaro et al., 2022) used Mask R-CNN to count aphids and parasitic wasps, achieving approximately 80% correlation in the results. (Li et al., 2022) proposed a two-stage method for detecting whiteflies and thrips, first using spectral features to locate pests, and then using Support Vector Machines (SVM) for identification. (Wang et al., 2023) constructed an anchorless framework and developed a dynamic detection head, achieving competitive results on two multi-class small-target pest datasets. (Dong et al., 2024) designed a multi-scale feature aggregation and dynamic perception module, achieving ideal detection performance. (Yang et al., 2024) introduced a super-resolution module and multi-level feature fusion into YOLOv8, achieving 57% mAP for extremely small pests. (Zhang et al., 2024a) innovatively proposed a rice planthopper detection method based on a fully convolutional architecture and object counting loss, achieving an F1 score of 92.36%. (Banerjee et al., 2024) designed an IoT-based onion thrips and disease detection system, effectively improving the efficiency of onion pest and disease detection. (Wuet et al., 2024) designed a pest detection and monitoring system for mango thrips, detecting mango thrips on sticky insect boards to monitor their population.

[0005] The aforementioned research provides innovative research ideas and improvement routes for the counting and detection of extremely small pests. However, existing literature lacks research on intelligent counting methods for flower thrips. For the accurate counting and detection of flower thrips, existing methods still have significant room for improvement in both detection accuracy and model running efficiency. Therefore, this paper uses flower thrips as the research object, collects thrips damage data from peace lilies cultivated in a greenhouse, and proposes a new real-time counting and detection algorithm for flower thrips, providing an efficient and reliable intelligent method for the intelligent monitoring of small pests in greenhouses. Summary of the Invention

[0006] To overcome the aforementioned deficiencies of the prior art, this invention provides a method for real-time detection and counting of greenhouse flower thrips based on a localized fully convolutional network.

[0007] The technical solution of this invention is as follows:

[0008] A real-time detection and counting method for greenhouse flower thrips based on localized fully convolutional networks includes the following steps:

[0009] (1) Data Acquisition

[0010] At random time intervals, use a smartphone to randomly photograph flower thrips on leaves, taking 2-3 photos of each location, and keeping only the clearest one;

[0011] (2) Dataset creation

[0012] After data collection, a total of 5618 images were selected to form the dataset, and all images were resized to 1280×1280 resolution. The thrips annotation was completed collaboratively by the photographers and underwent secondary verification to ensure accuracy. The entire dataset contains 47726 thrips annotations, which were divided into training, validation, and test sets in a 6:2:2 ratio.

[0013] (3) Construction of the flower thrips counting and detection network model

[0014] The backbone network is used to extract rich multi-scale feature information from the input image, and attention is used to further improve feature representation. Then, the multi-scale features are input into the feature pyramid network to improve the network's detection performance for small targets. Finally, the final prediction result is output through the output head. Unlike object detection methods, this network is a fully convolutional structure. The output head consists of four 1×1 convolutions, which reduce the number of output channels of the FPN to 1, interpolate back to the input size, and finally merge to output a single prediction result.

[0015] (4) Feature extraction backbone

[0016] We employ ConvNeXtV2 as the backbone network. ConvNeXtV2 enhances inter-channel feature competition through Global Response Normalization (GRN) layers, avoiding feature collapse and improving model expressive power. Simultaneously, it combines with the Fully Convolutional Mask Autoencoder (FCMAE) framework, utilizing sparse convolution techniques to process masked images, significantly improving pre-training efficiency. Its open-source pre-trained model provides a reliable foundation for transfer learning. We choose the nano version of ConvNeXtV2. ConvNeXtV2_nano is a lightweight version of the ConvNeXtV2 model family, offering good feature extraction capabilities while having lower parameter and computational costs.

[0017] (5) Construction of Hybrid Attention Feature Pyramid Network

[0018] A lightweight hybrid channel and spatial attention mechanism is introduced and combined with a feature pyramid network. Its focus is on improving the model's detection accuracy for small targets with only a small amount of additional computational overhead. In the network, hybrid attention is computed for all four multi-scale features (C2-C5) to fully utilize the rich feature information extracted by the backbone network. These features are then fed into a normal feature pyramid network for multi-scale feature fusion.

[0019] For the input feature f, first calculate its channel attention, then calculate its spatial attention, and finally add its residual, as shown in formula (1). Channel attention and spatial attention will be explained in detail below.

[0020] HA(f)=SA(CA(f))+f (1)

[0021] Most channel attention mechanisms perform global average pooling on the feature map, incorporating only channel information. This may offer limited performance improvement for detecting small targets because global pooling weakens the features of tiny targets. We blend local and global features in channel attention to improve its performance for small targets with lower computational overhead, as shown in the figure. The input feature map is first subjected to adaptive average pooling to obtain a local pooling result of size local size (ls). Then, the local pooling result is subjected to global average pooling to obtain the global pooling result. Local pooling focuses more on features of local regions, while global pooling focuses more on the distribution features of the entire feature map. Both the local and global pooling results are subjected to a 1D convolution for feature extraction to compute attention. The global attention is interpolated to the size of the local attention and then fused using addition. Finally, the fused result is interpolated to the input size and multiplied with the input feature map to obtain the channel attention feature map.

[0022] The implementation of spatial attention is relatively simple. We adopt the spatial attention module in the Convolutional Block Attention Module (CBAM), which has low computational overhead. First, we obtain the distribution information of spatial features in the channel dimension through average pooling and max pooling, and then we use a 2D convolution to calculate the attention.

[0023] (6) Loss function construction

[0024] The fully convolutional network is trained using object counting loss and uses the same object-level annotations as for object detection.

[0025] The object counting loss uses a center point optimization model to predict object centers, limits the prediction range of each object based on the bounding box, and continuously constrains false positives during training, as shown in Equation (2). The three sub-objectives are optimized together during training, extending the original semantic segmentation capability of the fully convolutional network to object detection and counting.

[0026]

[0027] After forward propagation, the model outputs a prediction matrix P of the same size as the input image. For each pixel i, the original output value is represented as follows. The Sigmoid activation function is applied to the original output value to activate it, converting it into a probability score. Let the probability of a thrips at pixel i after Sigmoid activation be 1; the closer this value is to 1, the higher the probability that the location is a thrips.

[0028] Define two label truth matrices, TL and TB, both the same size as the input. Matrix TL stores the center position of each pest: it assigns a value of 1 to the center of each pest and 0 to all other pixels, used to indicate the center position of each pest during training. Matrix TB records the object boundaries: it assigns a value of 0 to the bounding box pixels of each pest and 1 to other regions. This matrix is ​​designed to guide the model to distinguish the pest boundary from its surrounding area.

[0029] (7) Aphid counting and detection

[0030] Thrip counting: Thrips are counted by calculating the number of independent regions in the model's predicted result P. A connected component labeling algorithm is used to determine the number of independent regions in P, specifically implemented using the `label` method from the SciPy library. Thrip detection: Thrips localization and detection results are obtained by calculating the centroid coordinates of each independent region. First, the set of non-zero labels in the label matrix is ​​extracted, and the background is removed. Then, a two-dimensional coordinate matrix with the same size as the input is constructed, recording the row and column indices of each pixel. The label matrix and coordinate matrix are then flattened into one-dimensional arrays for vectorization operations. Histogram statistics are used to count the number of pixels for each label, and the row and column coordinates are weighted and accumulated to obtain the sum of the ordinate and abscissa coordinates of pixels within each connected region. Finally, the centroid coordinates are calculated for each label based on the centroid calculation formula.

[0031] Furthermore, the construction of the loss function specifically includes:

[0032] Localization loss

[0033] (Bearman et al. 2016) proposed a point-supervised semantic segmentation loss function, which only requires point-level annotation of the target to achieve the segmentation of the general outline of the target. We apply and integrate this loss function to form the Localization loss in the object counting loss, which can endow the model with the ability to accurately locate the target. The Localization loss optimizes the model based on the center of each object, predicts a region for each object, thereby endowing the model with the ability to locate the target. As shown in Equation (3):

[0034]

[0035] Based on the bounding box annotations, we first calculate the center point coordinates of each target and generate the ground truth matrix T of the target center points. L Given that the target center point label value is 1, be the set of coordinates inT L Where the label is one. For these coordinates with a label value of 1, let the predicted value in the model prediction result P be the position corresponding to that position. We expect the model's output value to be close to 1 at these locations. This optimization objective ensures that the model has the ability to locate each thrips. To allow the model to be trained more fully, we set a dynamic parameter λ. L =sum(T) L The more targets there are in an image, the greater their contribution to the loss value.

[0036] Boundary loss is a loss function used in deep learning.

[0037] Localization loss only optimizes the model's prediction of the center region of each object, but it doesn't indicate or constrain the extent of the predicted region. This can lead to the model becoming "lazy," resulting in a lack of constraint on the predicted region. Boundary loss, on the other hand, constrains the model's prediction range using the bounding box information, ultimately allowing the model to predict a small region around the center of each thrips. We can preload a matrix T containing all target boundary information into the Dataloader. B In this matrix, there is one and only one element with a value of 1 at the location corresponding to the target bounding box. B This allows us to indicate the boundary coordinates of each target. Then, be the set of coordinates in T B Where the label is one. For these boundary coordinates, let the predicted value in the model prediction result P corresponding to the position be... We expect the model output to be close to 0 at these locations, and the Boundary Loss is shown in Equation (4).

[0038]

[0039] This optimization objective constrains the model's prediction range, ensuring the model accurately centers the prediction regions for each target. Similarly, we set a dynamic parameter λ. B =sum(T) L -1. When an image contains more targets, the distance between them may be closer, so Boundary Loss should be given a larger weight to ensure that each target is independent of the others.

[0040] False positive loss

[0041] Localization loss and boundary loss only contribute to the model's prediction of positive samples, without allowing the model to learn the features of negative samples. Therefore, it is also necessary to combine false positive loss to train the model's ability to detect negative samples. The process is as follows: during training, find the coordinates of regions that the model incorrectly predicts as positive samples, and encourage the model to predict these regions as background, as shown in formula (5).

[0042]

[0043] The calculation process for the incorrectly predicted region is as follows: First, each independent region of the prediction result P is numbered using a connected component labeling algorithm. Then, P is compared with the ground truth matrix T of the target center point. L Element-wise multiplication yields the predicted region numbers containing the ground truth target center point. Finally, the remaining numbered regions are the incorrectly predicted regions. Let... Let P be the set of coordinates of elements in the erroneous prediction region. We expect the model's output to be close to 0 at these locations.

[0044] The technical effects and advantages of this invention are as follows:

[0045] A highly efficient thrips counting and detection model (TCD-Net) is proposed, capable of real-time and accurate counting and detection of thrips on leaves of various flowers in a greenhouse. TCD-Net is a unique fully convolutional network structure that uses the efficient ConvNeXtV2-Nano as its backbone and combines it with lightweight hybrid attention to extract rich thrips features. Accurate counting and detection are achieved by predicting a small region for each thrips. Experiments were conducted on a dataset containing over 47,000 labeled thrips. The results show that TCD-Net achieves excellent counting and detection results while maintaining low model complexity and significantly faster execution speed than real-time detection. On the test set, TCD-Net achieves an F1 score of 89.71% and a counting result relevance of 83.82%, exceeding the counting and detection accuracy of existing methods. Meanwhile, its model size (15.94M) and theoretical computational cost (83.92 GFLOPs) are less than half that of the two-stage target detection method, while its inference speed (99.49 it / s) is more than twice that of the two-stage target detection method. In summary, TCD-Net achieves higher thrips counting and detection accuracy with lower computational complexity, demonstrating its potential in the detection of extremely small pests. Attached Figure Description

[0046] Figure 1 Images of plant greenhouses;

[0047] Figure 2 The overall structure diagram of the flower thrips counting and detection network;

[0048] Figure 3 The overall network structure for the nano version of ConvNeXtV2;

[0049] Figure 4 This is a diagram showing the overall structure of the hybrid attention feature pyramid network.

[0050] Figure 5 A schematic diagram of mixed local channel attention;

[0051] Figure 6 This is a schematic diagram of spatial attention.

[0052] Figure 7 Schematic diagram of the training process of the flower thrips detection and counting model;

[0053] Figure 8 A visualization of the training process;

[0054] Figure 9 Output a visualization of the model's predictions;

[0055] Figure 10 A visual comparison chart is provided to visualize the prediction results with existing methods;

[0056] Figure 11 A visualization of TCD-Net's prediction results for flower thrips. Detailed Implementation

[0057] The technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0058] The main work of the patent is as follows.

[0059] (1) Thrip Counting and Detection Network (TCD-Net) was designed. The fully convolutional network was trained using the object counting loss (OC Loss) designed for extremely small pests, and combined with a high-performance, low-computational-load network structure to achieve accurate counting and detection of flower thrips.

[0060] (2) A lightweight feature extraction backbone, ConvNeXtV2, is adopted, and a spatial and channel attention mechanism that balances performance and efficiency is introduced into the Feature Pyramid Network (FPN). This achieves low-load real-time counting and detection of flower thrips with ideal counting and detection accuracy.

[0061] (3) A dataset of flower thrips containing 5618 images and 47726 annotations was collected. Extensive experiments and comparisons were conducted on this dataset to verify the superiority of TCD-Net in terms of detection accuracy and computational efficiency.

[0062] The specific method of this invention is as follows:

[0063] 1. Materials

[0064] 1.1 Data acquisition

[0065] Our team collected data from potted peace lilies in the plant growth chamber of Jingchu University of Technology's Jingchu Science and Technology Innovation City between July and September 2024. The plant growth chamber maintained a temperature of 25℃, humidity of 50%-70%, and light intensity of 10,000 Lux. Data collection was conducted by six students majoring in plant protection. They randomly photographed flower thrips on leaves using smartphones at random time intervals, taking 2-3 photos from each location and retaining only the clearest image. The shooting environment is as follows: Figure 1 As shown.

[0066] 1.2 Dataset

[0067] After data collection, 5618 images were selected to form the dataset, and all images were resized to 1280×1280 resolution. Thrips annotation was completed collaboratively by the photographers and underwent secondary verification to ensure accuracy. The entire dataset contains 47726 thrips annotations, which were divided into training, validation, and test sets in a 6:2:2 ratio. Specific statistics are shown in Table 1. Notably, the average pixel area of ​​the flower thrips annotation boxes in the images is only 176 pixels, representing only 0.011% of the image's pixel area. This clearly demonstrates that flower thrips are extremely small targets in the images, making accurate detection a significant challenge.

[0068]

[0069] 2. Proposed method

[0070] 2.1 Network construction

[0071] The overall structure of the flower thrips counting and detection network model is as follows: Figure 2 As shown, its module design is similar to that of a typical object detection network. The backbone network extracts rich multi-scale feature information from the input image, further enhances feature representation using attention, and then inputs the multi-scale features into the feature pyramid network to improve the network's detection performance for small targets. Finally, the output head outputs the final prediction result. Unlike object detection methods, this network uses a fully convolutional structure. The output head consists of four 1×1 convolutions, which reduce the number of output channels of the FPN to 1, interpolate back to the input size, and finally merge to output a single prediction result.

[0072] Ordinary fully convolutional networks cannot count and locate tiny objects. We enable our network to be trained with object detection labels and to count and detect tiny pests in images by using a specially designed loss function during training. The implementation process will be detailed in sections 2.2 and 2.3.

[0073] 2.1.1 Feature Extraction Backbone

[0074] The choice of feature extraction backbone plays a crucial role in the model's performance. We adopt ConvNeXtV2 as the backbone network. ConvNeXtV2 enhances inter-channel feature competition through Global Response Normalization (GRN) layers, avoiding feature collapse and improving the model's expressive power. Simultaneously, it combines with the Fully Convolutional Mask Autoencoder (FCMAE) framework, utilizing sparse convolution techniques to process masked images, significantly improving pre-training efficiency. Its open-source pre-trained model provides a reliable foundation for transfer learning. We choose the nano version of ConvNeXtV2. ConvNeXtV2_nano is a lightweight version of the ConvNeXtV2 model family, possessing good feature extraction capabilities while having lower parameter and computational costs. Its overall network structure is as follows: Figure 3 As shown in (a).

[0075] The ConvNeXtV2_nano network architecture employs a layered design, divided into four stages. Each stage contains a downsampling layer, and each stage has [2, 2, 8, 2] ConvNeXtV2 Blocks from shallow to deep, with channel counts of [80, 160, 320, 640]. Multi-scale features are crucial for object detection tasks; these four stages extract four different scale features C2 to C4 for subsequent feature fusion layers. The structure of the downsampling layer is as follows: Figure 3 As shown in (b), the feature map resolution is reduced and the number of channels is expanded. At the beginning of each stage, the feature map resolution is halved and the number of channels is doubled through a convolutional layer with a kernel size of 2 and a stride of 2. Layer Normalization is applied to ensure the stability of the feature distribution and improve the model training efficiency. The structure of the ConvNeXtV2 Block is as follows: Figure 3 As shown in (c), its fusion depth can separate convolutional, GRN layers and residual connections, and has a strong feature extraction capability.

[0076] 2.1.2 Hybrid attention feature pyramid network

[0077] Although the model is a fully convolutional structure, its goal is to accurately count and locate tiny thrips, without requiring precise segmentation of the target's contour. Therefore, we introduce a lightweight hybrid channel and spatial attention mechanism combined with a feature pyramid network. This focuses on improving the model's detection accuracy for small targets with only a small amount of additional computational overhead. In the network, hybrid attention is calculated for the four multi-scale features C2-C5 to fully utilize the rich feature information extracted by the backbone network. These are then fed into a normal feature pyramid network for multi-scale feature fusion. The overall structure is as follows: Figure 4 As shown.

[0078] For the input feature f, first calculate its channel attention, then calculate its spatial attention, and finally add its residual, as shown in formula (1). Channel attention and spatial attention will be explained in detail below.

[0079] HA(f)=SA(CA(f))+f (1)

[0080] Most channel attention mechanisms perform global average pooling on feature maps, incorporating only channel information. This may offer limited performance improvement for detecting small targets because global pooling weakens the features of tiny objects. We combine local and global features in channel attention to improve its performance for small targets with lower computational overhead, such as... Figure 5 As shown, the input feature map first undergoes adaptive average pooling to obtain a local pooling result of size local size (ls). Then, this local pooling result is subjected to global average pooling to obtain the global pooling result. Local pooling focuses more on features in local regions, while global pooling focuses more on the distribution features of the entire feature map. Both the local and global pooling results are subjected to a 1D convolution for feature extraction to calculate attention. The global attention is interpolated to the local attention size, and they are then fused using addition. Finally, the fused result is interpolated to the input size and multiplied with the input feature map to obtain the channel attention feature map.

[0081] Spatial attention is relatively simple to implement. We use the spatial attention module from the Convolutional BlockAttention Module (CBAM), which has low computational overhead, such as... Figure 6 As shown, firstly, the distribution information of spatial features is obtained through average pooling and max pooling in the channel dimension, and then a 2D convolution is used to calculate attention.

[0082] 2.2 Loss function

[0083] Simple fully convolutional networks achieve accurate segmentation of target pixels by training the model to predict the target's mask, but they lack the ability to count and detect targets. Zhang et al. noted that existing target detection methods struggle to learn the precise location and contour of small pests due to a lack of features, resulting in poor performance. This problem also exists in the thrips counting and detection task, and the smaller size of thrips presents an even greater challenge. Therefore, we use object counting loss to train a fully convolutional network, which uses the same object-level annotations as in target detection. The training process of the network model is as follows: Figure 7 As shown.

[0084] The object counting loss uses a center point optimization model to predict object centers, limits the prediction range of each object based on the bounding box, and continuously constrains false positives during training, as shown in Equation (2). The three sub-objectives are optimized together during training, extending the original semantic segmentation capability of the fully convolutional network to object detection and counting.

[0085]

[0086] After forward propagation, the model outputs a prediction matrix P of the same size as the input image. For each pixel i, the original output value is represented as follows. The Sigmoid activation function is applied to the original output value to activate it, converting it into a probability score. Let the probability of a thrips at pixel i after Sigmoid activation be 1; the closer this value is to 1, the higher the probability that the location is a thrips.

[0087] Define two label truth matrices, TL and TB, both the same size as the input. Matrix TL stores the center position of each pest: it assigns a value of 1 to the center of each pest and 0 to all other pixels, used to indicate the center position of each pest during training. Matrix TB records the object boundaries: it assigns a value of 0 to the bounding box pixels of each pest and 1 to other regions. This matrix is ​​designed to guide the model to distinguish the pest boundary from its surrounding area.

[0088] 2.2.1 Localization loss

[0089] (Bearman et al. 2016) proposed a point-supervised semantic segmentation loss function, which only requires point-level annotation of the target to achieve the segmentation of the general outline of the target. We apply and integrate this loss function to form the Localization loss in the object counting loss, which can endow the model with the ability to accurately locate the target. The Localization loss optimizes the model based on the center of each object, predicts a region for each object, thereby endowing the model with the ability to locate the target. As shown in Equation (3):

[0090]

[0091] Based on the bounding box annotations, we first calculate the center point coordinates of each target and generate the ground truth matrix T of the target center points. L Given that the target center point label value is 1, be the set of coordinates inT L Where the label is one. For these coordinates with a label value of 1, let the predicted value in the model prediction result P be the position corresponding to that position. We expect the model's output value to be close to 1 at these locations. This optimization objective ensures that the model has the ability to locate each thrips. To allow the model to be trained more fully, we set a dynamic parameter λ. L =sum(T) L The more targets there are in an image, the greater their contribution to the loss value.

[0092] 2.2.2 Boundary loss as a loss function in deep learning

[0093] Localization loss only optimizes the model's prediction of the center region of each object, but it doesn't indicate or constrain the extent of the predicted region. This can lead to the model becoming "lazy," resulting in a lack of constraint on the predicted region. Boundary loss, on the other hand, constrains the model's prediction range using the bounding box information, ultimately allowing the model to predict a small region around the center of each thrips. We can preload a matrix T containing all target boundary information into the Dataloader. B In this matrix, there is one and only one element with a value of 1 at the location corresponding to the target bounding box. B This allows us to indicate the boundary coordinates of each target. Then, be the set of coordinates in T B Where the label is one. For these boundary coordinates, let the predicted value in the model prediction result P corresponding to the position be... We expect the model output to be close to 0 at these locations, and the Boundary Loss is shown in Equation (4).

[0094]

[0095] This optimization objective constrains the model's prediction range, ensuring the model accurately centers the prediction regions for each target. Similarly, we set a dynamic parameter λ. B=sum(T) L -1. When an image contains more targets, the distance between them may be closer, so Boundary Loss should be given a larger weight to ensure that each target is independent of the others.

[0096] 2.2.3 False positive loss

[0097] Localization loss and boundary loss only contribute to the model's prediction of positive samples, without allowing the model to learn the features of negative samples. Therefore, it is also necessary to combine false positive loss to train the model's ability to detect negative samples. The process is as follows: during training, find the coordinates of regions that the model incorrectly predicts as positive samples, and encourage the model to predict these regions as background, as shown in formula (5).

[0098]

[0099] The calculation process for the incorrectly predicted regions is as follows: First, each independent region of the prediction result P is numbered using a connected component labeling algorithm. Then, P is compared with the ground truth matrix T of the target center point. L Element-wise multiplication yields the predicted region numbers containing the ground truth target center point. Finally, the remaining numbered regions are the incorrectly predicted regions. Let... Let P be the set of coordinates of elements in the erroneous prediction region. We expect the model's output to be close to 0 at these locations.

[0100] 2.3 Thrip counting and detection: Aphid counting and detection

[0101] Thrip counting is achieved by calculating the number of independent regions in the predicted result P from the model. The number of independent regions in P is determined using a connected component labeling algorithm, specifically through the `label` method in the scipy library.

[0102] Thrip detection. Thrip localization and detection results are obtained by calculating the centroid coordinates of each independent region. First, the set of non-zero labels in the label matrix is ​​extracted, and the background is removed. Then, a two-dimensional coordinate matrix with the same size as the input is constructed, recording the row and column indices of each pixel. The label matrix and coordinate matrix are then flattened into one-dimensional arrays for vectorization operations. Histogram statistics are used to count the number of pixels for each label, and the row and column coordinates are weighted and accumulated to obtain the sum of the ordinate and abscissa coordinates of pixels in each connected region. Finally, the centroid coordinates are calculated for each label based on the centroid calculation formula.

[0103] 3. Experimental results

[0104] 3.1 Implementation details

[0105] The hardware used for model training and inference is an Intel Core i9 12900K CPU with 64GB of RAM and an NVIDIA RTX 4090 24GB graphics card; the operating system is Ubuntu 22.04.1LTS with CUDA version 12.1, and the model is built based on Python 3.9 and PyTorch 2.1.2.

[0106] Model details: The backbone network of the model uses ConvNeXtV2_nano pre-trained on ImageNet 1K, with downsampling ratios of 4, 8, 16, and 32 for its C2-C5 feature maps, and channel numbers of 80, 160, 320, and 640, respectively. When calculating hybrid attention for C2-C5, the local sizes are 32, 16, 8, and 4, respectively. The kernel size of Conv1D in channel attention is 3, and the kernel size of Conv2D in spatial attention is 3. All feature maps are adjusted to 256 channels in the feature pyramid network, outputting four multi-scale features with 256 channels each. Finally, four 1×1 convolutions are used to reduce the channel number of the four multi-scale features to 1, and the samples are sampled back to the input size for merging. A sigmoid function is then used for activation to obtain the final prediction result.

[0107] Details of the methods used for comparison: We compare mainstream one-stage detectors, two-stage detectors, and DETR-based detectors. YOLO is implemented using the official open-source code, employing its Large version model. The two-stage detector and DETR-based detectors are implemented using the MMDetection framework, with the backbone network using ResNet50 pre-trained on ImageNet 1K. For the anchor-based detector, the anchor generation size is adjusted and adapted according to the target size of the rice planthopper dataset.

[0108] Training details: Random flipping was used for data augmentation during training. The batch size was 1, the optimizer was Adam, the learning rate was 1e-5, the weight decay was set to 1e-4, and all methods were trained for 60 epochs.

[0109] 3.2 Evaluation metrics

[0110] Detection accuracy. The accuracy of model localization can be evaluated by determining whether the centroid of the predicted region is within the labeled bounding box. The target detection method makes this determination based on the calculated center point of the predicted bounding box. The criteria for determining TP, FP, and FN are shown in Table 2.

[0111]

[0112] The detection accuracy of the model is evaluated using Precision, Recall, and F1 score, as shown in equations (6)-(8). Our method uses 0.5 as the confidence threshold, while the confidence threshold of the object detection method is determined by finding the value corresponding to the highest point of the F1 score on the precision-recall curve.

[0113]

[0114] Counting error. The stability of the algorithm is evaluated using mean absolute error and root mean square error. Let... and Let be the actual number of targets and the predicted number of targets in the i-th image, respectively, and calculate them as shown in Equations (9) and (10).

[0115]

[0116] R-squared(R 2 The similarity between the algorithm's counting results and the actual results is evaluated, as shown in Equation (11). 2 The value ranges from 0 to 1, and the higher the value, the more accurately the algorithm reflects the pest situation.

[0117]

[0118] set up and Let N be the actual number of targets and the predicted number of targets in the i-th image, respectively, and N be the number of images.

[0119] 3.3 Training results

[0120] We visualized the model's loss decrease during training, and the changes in counting error and accuracy on the validation set, such as... Figure 8 As shown, firstly, the model's training loss steadily decreased, and each sub-loss function was well optimized. Simultaneously, in the evaluation of each training cycle, the counting error on the validation set generally showed a decreasing trend, while the F1 score showed an increasing trend. This indicates that the model, after training, achieved the training objective of detecting and counting flower thrips in images.

[0121] We further visualized the model's predicted output using a heatmap, as shown below. Figure 9 As shown, after sufficient training, the model has the ability to detect flower thrips while remaining insensitive to the background. The model predicts a small spotted region for each flower thrips, with the prediction range limited to the size of the flower thrips' body. A connected component labeling algorithm can then be used to calculate the number of independent regions, and by calculating the center point of each region, accurate detection and counting of flower thrips can be achieved.

[0122] 3.4 Quantitative analysis

[0123] We compared TCD-Net with several existing methods widely used for pest counting. First, we compared the detection performance of these models, and the results are shown in the table below. TCD-Net's precision and recall are significantly higher than those of the one-stage object detector. YOLOv8l and YOLOv11l performed relatively weakly, with lower F1 scores and recalls than other methods, possibly due to the insufficient performance of the one-stage detector for small targets. The two-stage object detector performed relatively better on the thrips detection task. Compared to the one-stage detector, the two-stage object detector significantly improved recall, but at the cost of a significant decrease in precision, resulting in more false positives and a less than ideal F1 score. DeformableDETR achieved higher detection performance, its main advantage being higher precision, but its lower recall resulted in more false negatives. TCD-Net showed the best overall performance on both the validation and test sets, with a significantly higher F1 score than other methods, and a good balance between precision and recall.

[0124]

[0125] We further compared the counting accuracy of these methods, and the results are shown in Table x. TCD-Net once again demonstrates its advantage, with the lowest MAE and RMSE, and R... 2 The highest value indicates that the counting result is closest to the actual value, and the stability is the best. Considering the detection accuracy, models with higher detection accuracy also have higher counting accuracy, reflecting the pest situation more accurately.

[0126]

[0127]

[0128] Finally, we compared the computational complexity of these methods. The comparisons were made in four aspects: model parameter count, computational cost, training speed, and inference speed. The results are shown in Table x. Compared to the YOLO model, its detection performance is significantly higher, but its parameter count and computational complexity are lower than YOLO. Although YOLOv11's training and inference speeds are faster than TCD-Net, its detection performance lags significantly behind TCD-Net. Compared to more complex models, such as Deformable DETR and RPH-Counter, these two models have detection performance only slightly lower than TCD-Net, but their computational complexity is significantly higher than TCD-Net, especially Deformable DETR, whose inference speed does not reach real-time levels. Compared to RPH-Counter, TCD-Net's computational cost is less than half, and its inference speed is more than 1.5 times faster. In summary, TCD-Net achieves higher detection and counting accuracy while maintaining a lower computational load, and its inference speed exceeds the requirements of real-time detection.

[0129]

[0130] 3.5 Visualization

[0131] We visualized the detection count results of each method for a more intuitive comparison, such as... Figure 10 As shown, observing the detection results of Faster R-CNN and Cascade R-CNN reveals insufficient detection precision, with a significant number of false positives (FPs) in the results. YOLOv11l's detection results consistently contain substantial false negatives (FNs) and FPs, leading to high counting bias in some cases. Deformable DETR's detection results contain fewer FNs and FPs compared to one-stage and two-stage object detectors, but due to its low recall, its count is lower than the actual number. Compared to existing methods, TCD-Net has fewer FNs and FPs, and its count is closer to the true number. The visualization results are largely consistent with the results shown in the quantitative analysis, further validating TCD-Net's comprehensive advantages in thrips detection and counting tasks.

[0132] Finally, as Figure 11 As shown, we present a set of detection and counting results from TCD-Net. TCD-Net exhibits high stability, with only a small number of FN and FP errors in the detection results, providing strong algorithmic support for the intelligent monitoring and management of flower thrips.

[0133] 3.6 Ablation study

[0134] As shown in Table x, TCD-Net uses ConvNeXtV2-Nano as its backbone network and combines it with the HybridAttention mechanism to achieve ideal performance in flower thrips detection and counting. Ablation experiments show that this combination achieves optimal performance on both the validation and test sets. Compared with classic backbone networks such as ResNet-50 and Swin Transformer-Tiny, ConvNeXtV2-Nano, through its lightweight design and modern convolution optimization, can extract richer features, significantly improving detection and counting accuracy. Furthermore, HybridAttention, by fusing lightweight channel and spatial attention, enhances the model's ability to focus on key regions, further reducing false negatives and false positives.

[0135]

[0136] We further explored the impact of the backbone network and attention mechanism on the computational efficiency of TCD-Net. When using the ConvNeXtV2-Nano backbone network, TCD-Net has lower parameter and computational costs, significantly lower than ResNet-50 and SwinTransformer-Tiny, and its training and inference speeds are also faster. Notably, training speed is slower with ConvNeXtV2-Nano than with ResNet-50, while inference speed only increases slightly. This is likely due to the extensive use of depthwise separable convolution operations in ConvNeXtV2, which are less efficient on GPUs than traditional convolutions. Although the theoretical parameter and computational costs are lower, the actual running speed is affected. Its advantages include lower memory consumption and easier deployment on smaller devices.

[0137] After introducing HybridAttention, the model's computational overhead increased only slightly, while the inference speed remained at 99.49 it / s, meeting the requirements of real-time detection. Compared to other attention mechanisms, HybridAttention achieves the best balance between accuracy and efficiency, ensuring higher model performance without causing significant loss in inference speed. Overall, TCD-Net uses ConvNeXtV2-Nano as its backbone network, combined with efficient HybridAttention, to achieve ideal detection performance with relatively low computational load.

[0138]

[0139] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for real-time detection and counting of thrips in greenhouse flowers based on a localized fully convolutional network, characterized by: Includes the following steps: (1) Data Acquisition At random time intervals, use a smartphone to randomly photograph flower thrips on leaves, taking 2-3 photos of each location, and keeping only the clearest one; (2) Dataset creation After the data collection was completed, a number of images were selected to form the dataset; the labeling of thrips was completed jointly by the x photographers and was double-checked to ensure the accuracy of the labeling; the entire dataset was divided into training set, validation set and test set in a 6:2:2 ratio; (3) Construction of the flower thrips counting and detection network model The backbone network is used to extract rich multi-scale feature information from the input image, and attention is used to further improve feature representation. Then, the multi-scale features are input into the feature pyramid network to improve the network's detection performance for small targets. Finally, the final prediction result is output through the output head. Unlike target detection methods, this network is a fully convolutional structure. The output head consists of four 1×1 convolutions, which reduce the number of output channels of the FPN to 1 and interpolate back to the input size. Finally, they are combined to output a prediction result. (4) Feature extraction backbone ConvNeXtV2 is used as the backbone network. ConvNeXtV2 enhances the feature competition between channels through the Global Response Normalization (GRN) layer, avoids the feature collapse problem, and improves the model's expressive power. At the same time, it combines the Fully Convolutional Mask Autoencoder (FCMAE) framework and uses sparse convolution technology to process mask images, which significantly improves the pre-training efficiency. Its open-source pre-trained model provides a reliable foundation for transfer learning. (5) Construction of Hybrid Attention Feature Pyramid Network A lightweight channel- and spatial hybrid attention mechanism is introduced and combined with a feature pyramid network; Its focus is on improving the model's detection accuracy for small targets, with only a small amount of additional computational overhead. In the network, the four multi-scale features C2-C5 all compute hybrid attention to make full use of the rich feature information extracted by the backbone network. Then they are input into the normal feature pyramid network for multi-scale feature fusion. For the input feature f, first calculate its channel attention, then calculate its spatial attention, and finally add its residual, as shown in formula (1); the channel attention and spatial attention will be described in detail below; HA(f)=SA(CA(f))+f (1) Most channel attention mechanisms perform global average pooling on the feature map, which only includes channel information. This may offer limited improvement in the detection performance of small targets because global pooling weakens the features of tiny targets. Channel attention combines local and global features to improve its performance on tiny targets with lower computational overhead. The input feature map first undergoes adaptive average pooling to obtain a local pooling result of size local size (ls). Then, the local pooling result is subjected to global average pooling to obtain the global pooling result. Local pooling focuses more on features of local regions, while global pooling focuses more on the distribution features of the entire feature map. Both the local and global pooling results are subjected to a 1D convolution for feature extraction to compute attention. The global attention is interpolated to the size of the local attention and then fused using addition. Finally, the fused result is interpolated to the input size and multiplied with the input feature map to obtain the channel attention feature map. The implementation of spatial attention is relatively simple; it adopts the spatial attention module in the Convolutional Block Attention Module (CBAM), which has a small computational cost. First, it obtains the distribution information of spatial features in the channel dimension through average pooling and max pooling, and then uses a 2D convolution to calculate attention. (6) Loss function construction A fully convolutional network is trained using object counting loss, and it uses the same object-level annotations as that used for object detection. The object counting loss uses the center point optimization model to predict the object center, limits the prediction range of the model for each object according to the bounding box, and continuously constrains false positives during the training process, as shown in formula (2). The three sub-objectives are optimized together during training, extending the original semantic segmentation capabilities of the fully convolutional network to object detection and counting; After forward propagation, the model outputs a prediction matrix P of the same size as the input image; for each pixel i, the original output value is represented as; the Sigmoid activation function is used to activate the original output value, converting it into a probability score; let the probability of a thrips at pixel i after Sigmoid activation be 1, the closer the value is to 1, the higher the probability that the position is a thrips; Define two label truth matrices TL and TB, both of the same size as the input; matrix TL stores the center position of the pest: it assigns a value of 1 to the center of each pest and a value of 0 to the pixels at other positions, and is used to indicate the center position of each pest during training. The matrix TB records the object boundary: it assigns a value of 0 to the bounding box pixels of each pest and a value of 1 to other areas; this matrix is ​​designed to guide the model to distinguish the pest boundary from its surrounding area. (7) Aphid counting and detection Thrip counting: Thrips are counted by calculating the number of independent regions in the model prediction result P; the number of independent regions in P is obtained using a connected component labeling algorithm, specifically through the label method in the scipy library; Thrip detection: Thrips localization and detection results are obtained by calculating the centroid coordinates of each independent region; first, the set of non-zero labels in the label matrix is ​​extracted, and the background is removed; Subsequently, a two-dimensional coordinate matrix with the same size as the input is constructed to record the row and column indices of each pixel. The label matrix and coordinate matrix are then flattened into a one-dimensional array for vectorization operations. Histogram statistics are used to count the number of pixels for each label, and the row and column coordinates are weighted and accumulated to obtain the sum of the vertical and horizontal coordinates of pixels in each connected region. Finally, the center coordinates are solved for each label based on the centroid calculation formula.

2. The method for real-time detection and counting of greenhouse flower thrips based on a localized fully convolutional network according to claim 1, characterized in that: In the creation of the dataset (2), after the data collection work was completed, a total of 5,618 images were selected to form the dataset; the size of all images was adjusted to 1280×1280 resolution; the entire dataset contains 47,726 thrips annotations.

3. The method for real-time detection and counting of greenhouse flower thrips based on a localized fully convolutional network according to claim 1, characterized in that: The feature extraction backbone in step (4) is to select the nano version of ConvNeXtV2. ConvNeXtV2_nano is a lightweight version in the ConvNeXtV2 model family. While having good feature extraction capabilities, it has a low number of parameters and low computational cost.

4. The method for real-time detection and counting of greenhouse flower thrips based on a localized fully convolutional network according to claim 1, characterized in that: The construction of the loss function specifically includes: Localization loss (Bearman et al. 2016) proposed a point-supervised semantic segmentation loss function, which only requires point-level annotation of the target to achieve the segmentation of the general outline of the target; applying and integrating this loss function constitutes the Localization loss in the object counting loss, which can give the model the ability to accurately locate the target; the Localization loss optimization model is based on the center of each object, predicting a region for each object, thereby giving the model the ability to locate the target; as shown in formula (3): Based on the bounding box annotations, the center point coordinates of each target are first calculated, and the ground truth matrix T of the target center points is generated. L Given that the target center point label value is 1, Let be the set of coordinates in T L Where the label is one. For these coordinates with a label value of 1, let the predicted value in the model prediction result P be the position corresponding to that position. The goal is to ensure that the model's output value is close to 1 at these locations; this optimization objective guarantees that the model has the ability to locate each thrips. To allow the model to be trained more fully, a dynamic parameter λ is set. L =sum(T) L That is, the more targets in an image, the greater their contribution to the loss value; Boundary loss is a loss function used in deep learning. Localization loss only optimizes the model's prediction of the center region of each object, but it does not indicate or constrain the extent of the predicted region. The model may become "lazy," resulting in a lack of constraint on the predicted region. Boundary loss, on the other hand, constrains the model's prediction range using the bounding box information, ultimately leading the model to predict a small region around the center of each thrips. The dataloader can preload a matrix T containing all target boundary information. B In this matrix, there is one and only one element with a value of 1 at the location corresponding to the target bounding box; T B This allows us to indicate the boundary coordinates of each target; then, Let... bethe set of coordinates in T B Where the label is one. For these boundary coordinates, let the predicted value in the model prediction result P corresponding to the position be... It is expected that the model output will be close to 0 at these positions, and the BoundaryLoss is as shown in Equation (4); This optimization objective constrains the model's prediction range, ensuring the model accurately centers the prediction region for each target. Similarly, a dynamic parameter λ is set. B =sum(T) L -1; When there are more targets in the image, the distance between the targets may be closer, so the Boundary Loss should be given a larger weight to ensure that each target is independent of the others; False positive loss Localization loss and Boundary loss only contribute to the model's prediction of positive samples, but do not allow the model to learn the features of negative samples. Therefore, it is also necessary to combine False positive loss to train the model's ability to detect negative samples. The process is to find the coordinates of the regions that the model incorrectly predicts as positive samples during the training process and encourage the model to predict these regions as background, as shown in formula (5). The calculation process for the incorrectly predicted region is as follows: First, use the connected component labeling algorithm to number each independent region of the prediction result P; then, compare P with the ground truth matrix T of the target center point. L Element-wise multiplication yields the predicted region numbers containing the ground truth target center point; finally, the remaining numbered regions are the incorrectly predicted regions; let... Let P be the set of coordinates of elements in the erroneous prediction region, where the model's output is expected to be close to 0.

Images