Field rice planthopper detecting and counting method and device fused with visual prompt

Abstract

The invention relates to a field rice planthopper detection and counting method and device fused with visual prompt, and belongs to the technical field of rice planthopper prevention and control. According to the field rice planthopper detection and counting method fused with visual prompt, the visual prompt process of a detection network is innovated, key technologies such as data set making, model structure, training process and model reasoning are included, and a set of rice planthopper field image automatic acquisition device is designed. The method has the advantages that the workload of measuring and forecasting personnel going into the field can be reduced, low-cost and self-adaptive optimization of abnormal detection results is realized through a designed visual prompt model and a small amount of manual prompts for the error detection results, high-precision detection and counting of field rice planthoppers are realized, and the working efficiency of the field rice planthoppers is improved. The problem that an existing rice planthopper detecting and counting method is insufficient in reliability is solved.

Description

Technical Field

[0001] The invention relates to a method and device for detecting and counting rice planthoppers in a field by integrating visual prompts, and belongs to the technical field of rice planthopper prevention and control. Background Art

[0002] Rice planthoppers are one of the most serious migratory pests that harm rice. Their rapid reproduction, strong migratory patterns, and cryptic habitats make them prone to sudden and severe outbreaks. Therefore, timely and accurate monitoring of rice planthopper populations is crucial.

[0003] Traditionally, rice planthopper population monitoring relies on manual counting, which is labor-intensive and inaccurate. To address this, automated monitoring methods are currently available, such as the invention patent application number 2024103779177, entitled "A Rice Planthopper Detection and Counting Method Based on a Fully Convolutional Neural Network." While the disclosed field rice planthopper detection and counting method meets monitoring needs to a certain extent, existing methods lack the ability to adaptively optimize erroneous detection results and cannot further optimize initial detection results through manual annotation, resulting in insufficient reliability.

[0004] Therefore, it is necessary to develop a new method for detecting and counting rice planthoppers to solve the problem of insufficient reliability of existing counting methods. Summary of the Invention

[0005] The purpose of the present invention is to provide a method and device for detecting and counting rice planthoppers in the field that integrates visual cues, so as to solve the problem of insufficient reliability of existing rice planthopper detection and counting methods.

[0006] The technical solution of the present invention is: The method for detecting and counting rice planthoppers in the field by integrating visual cues is characterized in that the method comprises the following steps: (1) Data acquisition: The dataset was collected by using a rice planthopper detection and counting device to capture field images. The camera was kept parallel to the base of the rice stem, the photos were taken with an aspect ratio of 4 to 3, the shooting height was 5cm-10cm, and the shooting distance was 10cm-20cm. 3-5 photos were taken in each insect-infested area, and the clearest one was retained.

[0007] (2) Dataset preparation: All field images were uniformly preprocessed: the aspect ratio was maintained and the images were resized to 1920 pixels in height and 1440 pixels in width. These processed images served as dataset images. Half of the dataset images were manually annotated using the labelme or anylabeling annotation tools. Rectangular annotation was performed using a two-point annotation format: the upper left point (x1, y1) and the lower right point (x2, y2) define a rectangular box. All rice planthopper labels were annotated with the symbol "PH." The annotation tool automatically generated an annotation JSON file with the file name corresponding to the image name.

[0008] Due to the large number of rice planthoppers, a two-step annotation-proofreading method was adopted. First, N agricultural experts divided the work and performed the first round of annotation on the dataset images. After the first round of annotation was completed, the annotations were exchanged with each other for proofreading to eliminate the influence of subjective factors in the annotation process.

[0009] To reduce the labeling workload, we adopted a labeling method that combines automatic labeling with manual proofreading. First, we used half of the labeled data to train the Cascade R-CNN object detection model. Then, we used the Cascade R-CNN model to detect the remaining half of the dataset images. The detection threshold was set to 0.75 to ensure accuracy. Then, based on the detection results, we generated a JSON annotation file for the remaining dataset images.

[0010] The generated annotation files may have missed detections. Therefore, N agricultural experts are divided into groups to proofread them. After the first round of proofreading is completed, they exchange annotations with each other for the second round of proofreading to reduce annotation errors.

[0011] In the complexities of field applications, models inevitably experience a high incidence of missed and false positives in some images. This is particularly acute in densely populated, microscopic pest counts. Existing methods rely on a "one-shot" approach, and even when significant errors are detected, they fail to correct and optimize these anomalies at the model level. Therefore, this technology incorporates a visual cue mechanism into the model. After the model's initial output, it provides a small number of erroneous results, indicating missed and false positives. The model then extracts features based on these cues, allowing for corrective action.

[0012] Since this technology involves a visual prompt mechanism for missed samples and falsely detected samples, the network model learns the feature extraction capabilities for missed samples and falsely detected samples respectively during the training process. The training of missed samples can directly extract labeled data for training, while falsely detected samples do not have corresponding labeled data. Therefore, additional falsely detected sample annotations need to be prepared. The preparation method is as follows: first train the network model proposed in this technology for 3 rounds to obtain a non-convergent model. This model will have a large number of falsely detected samples. Based on these falsely detected samples, corresponding falsely detected labeled data are generated. The format is consistent with the labeled data in the above two steps and is randomly read during training.

[0013] After completing the above three steps, the dataset production of this technology is completed. (3) Data reading: This step is how to read dataset images and their corresponding annotation files during model training and inference. The image is read using the imread method in OpenCV, and the image is formatted as a Python Numpy array. The key is in reading the annotations, which are rectangles. The annotations identify the rectangular regions of interest for each rice planthopper. The following data is then calculated and stored in a Python dictionary as key-value pairs, with the key and value being: Counts—a numeric data type. The total number of rice planthoppers in the image is obtained by counting the number of labeled boxes. Points—a Numpy array of the same size as the image. The coordinates of the center point of each labeled box are obtained by calculating the center point of each rice planthopper. The corresponding position in the points array is set to 1, and the other positions are set to 0. boundaries - only read during model training, the data type is a Numpy array, the size is the same as the image, read the coordinate points of the annotation box, and set the value of the area corresponding to the annotation box position in the boundaries array to 1, and the other positions to 0; boxes - the data type is a list, which stores the coordinates of all rectangular boxes; bboxes - data type is tensor, randomly extract three coordinates in boxes and form a tensor with 3 rows and 4 columns, each column is a coordinate, corresponding to the coordinate points x1, y1, x2, y2. If there are less than three coordinates, fill the gaps with 0; fp_boxes - data type is list, which stores the coordinates of all the false positive sample rectangles; fp_bboxes - The data type is tensor, and three coordinates are randomly extracted from fp_boxes in the same format as bboxes. To improve model robustness and data diversity, data augmentation is performed when reading data, randomly flipping images and annotations, setting a random offset x for the point coordinates in the points array, and setting a random offset y for the box coordinates in the boundaries array.

[0014] (4) Network model structure: The overall network model structure includes three key steps: image encoding, missed detection prototype matching module, false detection prototype matching module and feature fusion module.

[0015] The input image is an RGB image, with the input size being H and W in height and width, respectively. The specific process is as follows: First, the input image is encoded by the image encoder to obtain the encoded features FE. The FE has height and width h and w in height and width, respectively, and the number of channels is d, where h = H / 8, w = W / 8, and d = 256. It is then decoded by the image decoder to obtain the initial output. The image decoder uses a 1×1 convolution to compress the number of FE channels to 1, then uses bilinear interpolation to scale up to the input size, and finally uses a sigmoid activation function for the output. In complex situations, the initial output will inevitably produce suboptimal detection and counting results, manifested as a high number of obvious missed and false detections in the visualization results. Therefore, for these anomalies, manual visual annotation can be used to further optimize and reduce detection errors. Given n (n=3 in this technology) missed detection or false detection prompts, a visual prompt is given only when the number of missed detections or false detections is greater than n, and then the prompt information is sent to the missed detection / false detection prototype matching module to obtain the missed detection prototype response map RFN and the false detection prototype response map RFP (the prototype response map will be calculated only after the prompt is input. For example, if the number of false detections is less than n, no false detection visual prompt is given, and the corresponding false detection prototype response map will not be calculated). The FE, RFN and RFP are feature fused to obtain the full response map RF, and finally RF is decoded by the image decoder to obtain the optimized output.

[0016] The following describes the image encoder, the missed prototype matching module, the false detection prototype matching module, and the feature fusion module in detail. The input image of the image encoder is an RGB image. Assume that the input size is H and width, respectively. The convolution stride of the fifth layer of the backbone network ResNet50 is changed to 1. The height and width of the feature maps of the second to fifth layers are (H / 4, W / 4), (H / 8, W / 8), (H / 16, W / 16), and (H / 16, W / 16), respectively. The features of the second to fifth layers are input into the feature pyramid for multi-scale feature extraction. The number of channels of each output of the feature pyramid is 256. The four outputs are uniformly resized to (H / 8, W / 8) through bilinear interpolation and concatenated in the channel dimension, resulting in a total of 1024 channels. Finally, a 1×1 convolution is used to reduce the channel number to 256, thus obtaining the image encoding features FE.

[0017] The structures of missed detection prototype matching module and false detection prototype matching module are as follows: Figure 3 As shown in the figure. The prototype matching module structure of missed detection and false detection is the same, but they are two independent branches in the network and the parameters are not shared. For n prompt boxes, there are two query processes: (1) First, the height and width of each prompt box are calculated to obtain a tensor with a shape of (n,2). The tensor is input into an MLP for encoding. The process is: (n,2)→(n,64)→ReLU→(n,d)→ReLU→(n,k 2 ×d), where n = 3, d = 256, k = 3, and then reshape the encoded result to (nk 2 ,d), get the shape and size query tensor Q S (2) Then, based on the given prompt box coordinates, ROI Align operation is performed on the encoding feature FE to extract n k×k size and d channel representation queries, and reshape them into (nk 2 ,d), get the representation query tensor Q A . Q S , Q A The prototype query module is fed with FE to perform a query, resulting in n prototype query features of size k×k and number of channels d. The prototype query features are used as convolution kernels to perform depth-dependent convolution on the encoded features FE, resulting in n query graphs with number of channels d. Finally, a 1×1 convolution is used to integrate the outputs to obtain the response graph R.

[0018] Prototype query module such as Figure 4 As shown. LN is layer normalization, MHA is multi-head self-attention mechanism, MLP has two linear layers, and the input dimension changes to (nk 2 ,d)→(nk 2 ,8d)→GELU→(nk 2,d), and finally reshaped into (n×d,k,k), which is the prototype query feature.

[0019] The feature fusion module includes an adaptive feature gating mechanism; The prototype matching response map needs to be fused with the encoding feature FE, but the direct addition method is not stable enough and may cause the superposition of abnormal values ​​between the two feature maps, causing adverse effects. Therefore, an adaptive feature gating mechanism is designed to perform adaptive weighted fusion of the two features with a learnable module, such as Figure 5 As shown. The two features are first added element by element, and there are two branches for feature weight calculation: (1) global average pooling is performed, and then an MLP is used for feature encoding to obtain the global encoding weight with a dimension of (1, 1, d); (2) two 1×1 convolutions are performed to obtain the spatial encoding weight with a dimension of (h, w, d). The global encoding weight is then broadcasted, added element by element to the spatial encoding weight, and activated using the Sigmoid function to obtain the adaptive gating weight Wgate. Finally, the adaptive fusion result of the two features is calculated, and the fusion method is: f fusion =f1×W gate +f2×(1-W gate For the fusion of the encoding features, missed detection response maps, and false detection response maps in the network, the encoding features are first fused with the two response maps in pairs, and then the two fusions are combined and fused again to obtain the final full response map RF. If only one of the response maps is present, only the encoding features are fused with that response map, and the subsequent fusion steps are skipped.

[0020] (5) Model training; First, the input image is passed through the image encoder and image decoder to obtain the initial output OI. During training, the annotation box information of missed samples and false positive samples is read from the annotation file in the data reading stage. The missed sample information corresponds to the bboxes in the data reading stage, and the false positive sample information corresponds to the fp_bboxes in the data reading stage. Based on this information, the missed prototype matching and false positive prototype matching stages are performed respectively to obtain the missed prototype response map RFN and the false positive prototype response map RFP. The missed prototype response map RFN is fused with the encoding feature FE through adaptive feature gating to obtain the missed detection response map RF1. The fusion result is input into the image decoder to obtain the missed detection optimization output OF1; the false detection prototype response map RFP is fused with the encoding feature FE through adaptive feature gating to obtain the false detection response map RF2. The fusion result is input into the image decoder to obtain the false detection optimization output OF2; RF1 and RF2 are fused through adaptive feature gating to obtain the final response map RF, and finally input into the image encoder to obtain the final optimized output OF. Since there are a small number of images without rice planthopper annotations or false positive samples, when the number of missed detection prompt samples is 0, the missed detection prototype matching stage is skipped, and only the false positive prototype matching stage is performed, so RF = RF2; when the number of false positive prompt samples is 0, the false positive prototype matching stage is skipped, and only the missed detection prototype matching stage is performed, so RF = RF1; when the number of samples for both is 0, there is only the initial output OI; therefore, the network model has at least one output, namely OI, and at most four, namely OI, OF1, OF2, and OF; All outputs need to be optimized using a loss function. The basic loss function is for the network output O, and the loss calculation is shown in formula (1): All outputs use the object counting loss function to calculate the loss value, the loss value of OI is used as the main loss, and the other three are used as auxiliary losses; the final loss function is shown in formula (2), where λ = 0.3 The training input images are 1920×1440 in height and width (can be changed based on actual needs), and random flipping is used for data augmentation. The backbone network uses a pre-trained ResNet50, with a batch size of 1, an Adam optimizer, a learning rate of 1e-5, and a weight decay of 1e-4, and training for at least 100 epochs.

[0021] (6) Model reasoning: Counting and detecting rice planthoppers in images: The model predicts a small spot area at the center of each detected rice planthopper target. The predicted output O is activated using a Sigmoid activation function. The value of each element in the output ranges from 0 to 1, representing the probability that the pixel is a rice planthopper. If the element value is greater than 0.5, the area is considered a rice planthopper; otherwise, it is background. Counting rice planthoppers in the image is achieved by calculating the number of independent regions in the model's prediction O. The number of independent regions in P is determined using a connected component labeling algorithm, specifically the label method in the scipy library. Detecting the location of rice planthoppers in the image is achieved by calculating the centroid coordinates of each independent region. First, the set of non-zero labels in the label matrix is ​​extracted to remove background. Next, a two-dimensional coordinate matrix with the same size as the input is constructed, recording the row and column indices of each pixel. The label matrix and coordinate matrix are flattened into one-dimensional arrays for vectorized operations. The number of pixels for each label is counted using histogram statistics, and the row and column coordinates are weighted and accumulated to obtain the sum of the vertical and horizontal coordinates of the pixels in each connected area. Finally, the center coordinates are solved for each label based on the centroid calculation formula.

[0022] Model inference is divided into two stages. First, the input image passes through the encoder and decoder to obtain the initial output. In the experiment, since we already know the true value of the label of all images, we can obtain the FN and FP information of the predicted results of each image. The prompt box information given to the model will be randomly selected from these FN and FP bounding boxes. We set the activation threshold N. When the number of FN or FP is greater than N, it means that there are many errors in the detection results. At this time, n bounding boxes are randomly extracted from the FN and FP bounding boxes as prompt inputs, activating the visual prompt mechanism and performing prototype matching. This design simulates the actual application scenario for the following two reasons: (1) In actual applications, agricultural experts will only notice and correct when there are obvious errors in the detection results. (2) The correction of agricultural experts should be low-cost and only a small number of prompts should be performed, hoping to correct as many errors as possible. In summary, in actual applications, since the ground truth is unknown, the prompt process should be manually triggered by agricultural experts, and a small number of corrections will be performed only when there are obvious errors in the detection results. The selection design of FN and FP prompt boxes in our experiment is to simulate the actual application scenario.

[0023] The rice planthopper detection and counting device includes a camera, two groups of support frames, a long guide rail, a Z-axis drive motor and a control module; the two groups of support frames are respectively equipped with lifting sliders through the Z-axis drive motor; the long guide rail is installed between the lifting sliders; a sliding seat is slidably installed on the long guide rail; the control module is installed on the sliding seat; a shooting platform is installed on the sliding seat on one side of the control module through the Y-axis drive motor; the shooting platform is equipped with a camera, a distance sensor, and a temperature and humidity sensor; the sliding seat is equipped with a drive wheel through the X-axis drive motor; and the drive wheel is in contact with the long guide rail.

[0024] The rice planthopper detection and counting device is placed in the field and can automatically collect field images. The shooting platform runs on the guide rail to achieve the purpose of automatic shooting. The device is not affected by the uneven terrain in the rice field and the constantly changing depth of water accumulation. The shooting height, shooting distance and shooting position can be adjusted according to control instructions.

[0025] The camera, distance sensor, temperature and humidity sensor can capture images of the base of rice stems and obtain the shooting distance, temperature and humidity of the field at the time of shooting.

[0026] The sensors and drivers of the control module send control instructions, process the returned information, and interact with the mobile phone APP and cloud server.

[0027] The advantages of the present invention are: This method for detecting and counting rice planthoppers in the field, integrating visual cues, innovates the visual cueing process of the detection network. It incorporates key technologies such as dataset preparation, model structure, training process, and model inference. It also designs a device for automatically collecting field images of rice planthoppers. Its advantages lie in reducing the workload for field surveyors. Furthermore, through the designed visual cue model, manual prompts are provided for erroneous detection results, enabling low-cost, adaptive optimization of abnormal detection results. This results in highly accurate field rice planthopper detection and counting, addressing the reliability issues of existing rice planthopper detection and counting methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 This is the overall network model structure diagram of the present invention; Figure 2 Schematic diagram of the structure of the image encoder of the present invention; Figure 3 Schematic diagram of the structure of the prototype matching module of the present invention; Figure 4 Schematic diagram of the prototype query module structure of the present invention; Figure 5 Schematic diagram of the adaptive feature gating structure of the present invention; Figure 6 The control process and hardware module of the automatic acquisition device of the present invention; Figure 7 This is a schematic structural diagram of a device for detecting and counting rice planthoppers according to the present invention; Figure 8 for Figure 7 Schematic diagram of the enlarged structure at A in the middle; Figure 9 for Figure 7 Schematic diagram of the enlarged structure at B in the middle; Figure 10 Photos of data collected in the field during practical application for this application.

[0029] In the figure: 1. Support frame; 2. Z-axis drive motor; 3. Lifting slider; 4. Long guide rail; 5. Sliding seat; 6. Control module; 7. Y-axis drive motor; 8. Shooting platform; 9. X-axis drive motor. DETAILED DESCRIPTION

[0030] A method for detecting and counting rice planthoppers in a field by integrating visual cues is characterized in that the method comprises the following steps: (1) Data acquisition: The dataset was collected using a rice planthopper detection and counting device to capture field images. The camera was held parallel to the base of the rice stem, with an aspect ratio of 4:3. The images were taken at a height of 5-10 cm and a distance of 10-20 cm. Three to five images were taken of each infested area, and the clearest image was retained. (2) Dataset preparation: All field images were uniformly preprocessed: the aspect ratio was maintained and the images were resized to 1920 pixels in height and 1440 pixels in width. The processed images were used as dataset images. Half of the dataset images were manually annotated using the labelme or anylabeling annotation tools. Rectangular annotation was used, using a two-point annotation format: the upper left point (x1, y1) and the lower right point (x2, y2) define a rectangular box. All rice planthopper labels were annotated with the symbol "PH." The annotation tool automatically generates an annotation JSON file with the file name corresponding to the image name. Due to the large number of rice planthoppers, a two-step annotation-proofreading method was adopted. First, N agricultural experts divided the work and performed the first round of annotation on the dataset images. After the first round of annotation was completed, the annotations were exchanged with each other for proofreading to eliminate the influence of subjective factors in the annotation process.

[0031] To reduce the labeling workload, we adopted a labeling method that combines automatic labeling with manual proofreading. First, we used half of the labeled data to train the Cascade R-CNN object detection model. Then, we used the Cascade R-CNN model to detect the remaining half of the dataset images. The detection threshold was set to 0.75 to ensure accuracy. Then, based on the detection results, we generated a JSON annotation file for the remaining dataset images.

[0032] The generated annotation files may have missed detections. Therefore, N agricultural experts are divided into groups to proofread them. After the first round of proofreading is completed, they exchange annotations with each other for the second round of proofreading to reduce annotation errors.

[0033] In the complexities of field applications, models inevitably experience a high incidence of missed and false positives in some images. This is particularly acute in densely populated, microscopic pest counts. Existing methods rely on a "one-shot" approach, and even when significant errors are detected, they fail to correct and optimize these anomalies at the model level. Therefore, this technology incorporates a visual cue mechanism into the model. After the model's initial output, it provides a small number of erroneous results, indicating missed and false positives. The model then extracts features based on these cues, allowing for corrective action.

[0034] Since this technology involves a visual cue mechanism for missed samples (False Negative, FN) and false positive samples (False Positive, FP), the network model learns the feature extraction capabilities of missed samples and false positive samples respectively during the training process. The training of missed samples can directly extract labeled data for training, while false positive samples do not have corresponding labeled data. Therefore, additional false positive sample annotations need to be prepared. The preparation method is as follows: first train the network model proposed in this technology for 3 rounds to obtain a non-convergent model. This model will have a large number of false positive samples. Based on these false positive samples, corresponding false positive annotation data is generated. The format is consistent with the labeled data in the above two steps and is randomly read during training.

[0035] After completing the above three steps, the dataset production of this technology is completed.

[0036] (3) Data reading: This step is how to read the dataset images and corresponding annotation files during model training and inference. The image is read using the imread method in OpenCV, and the read image is formatted as a Python Numpy array. The key is to read the annotations, which are marked with rectangular boxes. The box annotations reveal the rectangular region of interest (ROI) for each rice planthopper. The following required data is then calculated and stored in a Python dictionary as key-value pairs, with the key and value being: Counts—a numeric data type. The total number of rice planthoppers in the image is obtained by counting the number of labeled boxes. Points—a Numpy array of the same size as the image. The coordinates of the center point of each labeled box are obtained by calculating the center point of each rice planthopper. The corresponding position in the points array is set to 1, and the other positions are set to 0. boundaries - only read during model training, the data type is a Numpy array, the size is the same as the image, read the coordinate points of the annotation box, and set the value of the area corresponding to the annotation box position in the boundaries array to 1, and the other positions to 0; boxes - the data type is a list, which stores the coordinates of all rectangular boxes; bboxes - data type is tensor, randomly extract three coordinates in boxes and form a tensor with 3 rows and 4 columns, each column is a coordinate, corresponding to the coordinate points x1, y1, x2, y2. If there are less than three coordinates, fill the gaps with 0; fp_boxes - data type is list, which stores the coordinates of all the false positive sample rectangles; fp_bboxes - data type is tensor, randomly extract three coordinates in fp_boxes, the format is the same as bboxes.

[0037] To improve model robustness and data diversity, data augmentation is performed when reading data. Images and annotations are randomly flipped, and a random offset x is set for the point coordinates in the points array and a random offset y is set for the box coordinates in the boundaries array.

[0038] (4) Network model structure: The overall network model structure is as follows Figure 1 As shown in the figure, it includes three key steps: image encoding, missed detection prototype matching module, false detection prototype matching module and feature fusion module.

[0039] The input image is an RGB image, with the input size being H and W in height and width, respectively. The specific process is as follows: First, the input image is encoded by the image encoder to obtain the encoded features FE. The FE has height and width h and w in height and width, respectively, and the number of channels is d, where h = H / 8, w = W / 8, and d = 256. It is then decoded by the image decoder to obtain the initial output. The image decoder uses a 1×1 convolution to compress the number of FE channels to 1, then uses bilinear interpolation to scale up to the input size, and finally uses a sigmoid activation function for the output. In complex situations, the initial output will inevitably produce suboptimal detection and counting results, manifested as a high number of obvious missed and false detections in the visualization results. Therefore, for these anomalies, manual visual annotation can be used to further optimize and reduce detection errors. Given n (n=3 in this technology) missed detection or false detection prompts, a visual prompt is given only when the number of missed detections or false detections is greater than n, and then the prompt information is sent to the missed detection / false detection prototype matching module to obtain the missed detection prototype response map RFN and the false detection prototype response map RFP (the prototype response map will be calculated only after the prompt is input. For example, if the number of false detections is less than n, no false detection visual prompt is given, and the corresponding false detection prototype response map will not be calculated). The FE, RFN and RFP are feature fused to obtain the full response map RF, and finally RF is decoded by the image decoder to obtain the optimized output.

[0040] Figure 1 The two image decoders appearing in have the same structure and share parameters.

[0041] The following will introduce the image encoder, missed detection prototype matching module, false detection prototype matching module and feature fusion module in detail. Figure 2As shown in the figure, the input image is an RGB image. Assume that the height and width of the input size are H and W respectively. The convolution step size of the fifth layer of the backbone network ResNet50 is modified to 1. The height and width of the feature maps from the second to the fifth layer are (H / 4, W / 4), (H / 8, W / 8), (H / 16, W / 16) and (H / 16, W / 16), respectively. The features from the second to the fifth layer are input into the feature pyramid for multi-scale feature extraction. The number of channels of each output of the feature pyramid is 256, and its four outputs are uniformly adjusted to the size of (H / 8, W / 8) through bilinear interpolation and spliced ​​in the channel dimension, that is, the number of channels after splicing is 1024. Finally, a 1×1 convolution is used to reduce the number of channels to 256, and the image encoding feature FE is obtained.

[0042] The structures of missed detection prototype matching module and false detection prototype matching module are as follows: Figure 3 As shown in the figure. The prototype matching module structure of missed detection and false detection is the same, but they are two independent branches in the network and the parameters are not shared. For n prompt boxes, there are two query processes: (1) First, the height and width of each prompt box are calculated to obtain a tensor with a shape of (n,2). The tensor is input into an MLP for encoding. The process is: (n,2)→(n,64)→ReLU→(n,d)→ReLU→(n,k 2 ×d), where n = 3, d = 256, k = 3, and then reshape the encoded result to (nk 2 ,d), get the shape and size query tensor Q S (2) Then, based on the given prompt box coordinates, ROI Align operation is performed on the encoding feature FE to extract n k×k size and d channel representation queries, and reshape them into (nk 2 ,d), get the representation query tensor Q A . Q S , Q A The prototype query module is fed with FE to perform a query, resulting in n prototype query features of size k×k and number of channels d. The prototype query features are used as convolution kernels to perform depth-dependent convolution on the encoded features FE, resulting in n query graphs with number of channels d. Finally, a 1×1 convolution is used to integrate the outputs to obtain the response graph R.

[0043] Prototype query module such as Figure 4 As shown. LN is layer normalization, MHA is multi-head self-attention mechanism, MLP has two linear layers, and the input dimension changes to (nk 2 ,d)→(nk 2 ,8d)→GELU→(nk 2 ,d), and finally reshaped into (n×d,k,k), which is the prototype query feature.

[0044] The feature fusion module includes an adaptive feature gating mechanism; The prototype matching response map needs to be fused with the encoding feature FE, but the direct addition method is not stable enough and may cause the superposition of abnormal values ​​between the two feature maps, causing adverse effects. Therefore, an adaptive feature gating mechanism is designed to perform adaptive weighted fusion of the two features with a learnable module, such as Figure 5 As shown in Figure 2, the two features are first added element-wise, and there are two branches for feature weight calculation: (1) global average pooling is performed, and then an MLP is used to encode the features to obtain global encoding weights with a dimension of (1, 1, d); (2) two 1×1 convolutions are performed to obtain spatial encoding weights with a dimension of (h, w, d).

[0045] Then the global encoding weight is broadcasted, added element by element with the spatial encoding weight, activated by the Sigmoid function, and the adaptive gating weight Wgate is obtained. Finally, the adaptive fusion result of the two features is calculated, and the fusion method is: fusion =f1×W gate +f2×(1-W gate For the fusion of the encoding features, missed detection response maps, and false detection response maps in the network, the encoding features are first fused with the two response maps in pairs, and then the two fusions are combined and fused again to obtain the final full response map RF. If only one of the response maps is present, only the encoding features are fused with that response map, and the subsequent fusion steps are skipped.

[0046] (5) Model training; First, the input image is passed through the image encoder and image decoder to obtain the initial output OI. During training, the annotation box information of missed samples and false positive samples is read from the annotation file in the data reading stage. The missed sample information corresponds to the bboxes in the data reading stage, and the false positive sample information corresponds to the fp_bboxes in the data reading stage. Based on this information, the missed prototype matching and false positive prototype matching stages are performed respectively to obtain the missed prototype response map RFN and the false positive prototype response map RFP. The missed prototype response map RFN is fused with the encoding feature FE through adaptive feature gating to obtain the missed detection response map RF1. The fusion result is input into the image decoder to obtain the missed detection optimization output OF1; the false detection prototype response map RFP is fused with the encoding feature FE through adaptive feature gating to obtain the false detection response map RF2. The fusion result is input into the image decoder to obtain the false detection optimization output OF2; RF1 and RF2 are fused through adaptive feature gating to obtain the final response map RF, and finally input into the image encoder to obtain the final optimized output OF. Since there are a small number of images without rice planthopper annotations or false positive samples, when the number of missed detection prompt samples is 0, the missed detection prototype matching stage is skipped, and only the false positive prototype matching stage is performed, so RF = RF2; when the number of false positive prompt samples is 0, the false positive prototype matching stage is skipped, and only the missed detection prototype matching stage is performed, so RF = RF1; when the number of samples for both is 0, there is only the initial output OI; therefore, the network model has at least one output, namely OI, and at most four, namely OI, OF1, OF2, and OF; All outputs need to be optimized using a loss function. The basic loss function is for the network output O, and the loss calculation is shown in formula (1): All outputs use the object counting loss function to calculate the loss value, the loss value of OI is used as the main loss, and the other three are used as auxiliary losses; the final loss function is shown in formula (2), where λ = 0.3

[0047] The training input images are 1920×1440 in height and width (can be changed based on actual needs), and random flipping is used for data augmentation. The backbone network uses a pre-trained ResNet50, with a batch size of 1, an Adam optimizer, a learning rate of 1e-5, and a weight decay of 1e-4, and training for at least 100 epochs.

[0048] (6) Model reasoning and rice planthopper detection and counting The model predicts a small, speckled area for each detected rice planthopper target. This area is centered on the target. The predicted output, O, is activated using a sigmoid activation function. Each element in the output ranges from 0 to 1, representing the probability that the pixel is a rice planthopper. If the element value is greater than 0.5, the area is considered a rice planthopper; otherwise, it is considered background.

[0049] Counting rice planthoppers: Counting rice planthoppers in an image is achieved by calculating the number of independent regions in the model prediction result O. The number of independent regions in P is obtained using the connected component labeling algorithm, specifically, the label method in the scipy library.

[0050] Detection of rice planthoppers: The location of rice planthoppers in an image is detected by calculating the centroid coordinates of each independent region. First, the set of non-zero labels in the label matrix is ​​extracted and the background is removed. Then, a two-dimensional coordinate matrix with the same size as the input is constructed to record the row and column indices of each pixel. The label matrix and the coordinate matrix are flattened into a one-dimensional array for vectorized operations. The number of pixels for each label is counted using histogram statistics, and the row and column coordinates are weighted and accumulated to obtain the sum of the vertical and horizontal coordinates of the pixels in each connected area. Finally, the center coordinates are solved for each label based on the centroid calculation formula.

[0051] Model inference occurs in two stages. First, the input image passes through the encoder and decoder to produce the initial output. Following the aforementioned rice planthopper counting and detection steps, we can process the model output to achieve rice planthopper counting and detection. During the experiment, since the true labels for all images are known, we can obtain the predicted FN and FP information for each image based on the detection results. The tooltip information provided to the model is randomly selected from these FN and FP bounding boxes.

[0052] We set an activation threshold N. When the number of FN or FP is greater than N, it means that there are many errors in the detection results. At this time, n bounding boxes are randomly extracted from the FN and FP bounding boxes as prompt inputs, activating the visual prompt mechanism and performing prototype matching. This design simulates the actual application scenario for the following two reasons: (1) In actual applications, agricultural experts will only notice and correct when there are obvious errors in the detection results. (2) The correction of agricultural experts should be low-cost and only a small number of prompts should be performed, hoping to correct as many errors as possible. In summary, in actual applications, since the ground truth is unknown, the prompt process should be manually triggered by agricultural experts, and a small number of corrections will be performed only when there are obvious errors in the detection results. The selection of FN and FP prompt boxes in our experiment is designed to simulate actual application scenarios.

[0053] Therefore, if the number of FN and FP is less than N, the final rice planthopper detection and counting result is generated by the initial output. If the number of FN or FP is greater than or equal to N, the final rice planthopper detection and counting result is generated by the output after visual prompting.

[0054] The rice planthopper detection and counting device includes a camera, two groups of support frames, a long guide rail, a Z-axis drive motor and a control module; the two groups of support frames 1 are respectively equipped with a lifting slide 3 through a Z-axis drive motor 2; a long guide rail 4 is installed between the lifting slides 3; a sliding seat 5 is slidably installed on the long guide rail 4; a control module 6 is installed on the sliding seat 5; a shooting platform 8 is installed on the sliding seat 5 on one side of the control module 6 through a Y-axis drive motor 7; the shooting platform 8 is equipped with a camera, a distance sensor, and a temperature and humidity sensor; the sliding seat 5 is equipped with a driving wheel through an X-axis drive motor 9; the driving wheel is in contact with the long guide rail (see the attached manual). Figure 7 、 8 and 9).

[0055] The rice planthopper detection and counting device is placed in the field and can automatically capture field images. The shooting platform 8 runs on the long guide rail 4 to achieve the purpose of automatic shooting. The device is not affected by the uneven terrain in the rice field and the constantly changing depth of water accumulation. The shooting height, shooting distance and shooting position can be adjusted according to the control instructions.

[0056] The camera, distance sensor, temperature and humidity sensor can capture images of the base of rice stems and obtain the shooting distance, temperature and humidity of the field at the time of shooting.

[0057] The control module sends control instructions to the sensors and drivers, processes the returned information, and interacts with the mobile phone APP and cloud server. Figure 6 shown.

[0058] As an improvement of the present invention, the field collection module transmits the collected full amount of pictures and videos to the cloud server of the insect analysis system via 4G. The cloud server is divided into a web management server and an insect GPU image recognition inference server (Runner server). The web management server is a controller and provides a user interaction interface. The Runner server is responsible for executing the neural network model described in steps (4)(5)(6) of this application.

[0059] When field images are transmitted to the backend web management server, image / video archiving is triggered. Simultaneously, the web server sends an image recognition request to an insect image processing message queue. All Runner servers subscribed to the queue receive the request, and one idle Runner executes the insect image recognition request. Once the Runner completes the task, it submits information such as the number and species of rice planthoppers in the image to the web server, which then writes the recognition result image and information to a database.

[0060] The web management system analyzes the pest situation for each collection point and collection cycle. The analysis focuses on year-over-year and month-over-month changes in rice planthopper species and abundance over a specific period. The system also collects information such as time, air temperature, ambient temperature, humidity, and light intensity, allowing users to dynamically track and analyze trends in total rice planthopper populations based on any of these conditions. When multiple collection points are present in the system, regional analysis of rice planthopper species, abundance, and trends can be performed.

[0061] The hardware must include a GPU. For model inference, we recommend a single RTX 2080Ti GPU or higher. For model training, we recommend a single RTX 4090 GPU or higher. The system memory must be at least 32GB. The operating system can be Windows or Linux. The programming language is Python 3.9, and the deep learning framework is PyTorch 2.0.

[0062] In order to verify the feasibility of this application, the applicant actually used: The data set was collected using a rice planthopper detection and counting device from June to September 2023 and July to September 2024 in rice fields planted by farmers in three townships under Jingzhou City, Hubei Province (see the attached manual for photos of the collected data). Figure 10 ) was then applied according to the scheme of this application.

[0063] This method for detecting and counting rice planthoppers in the field, integrating visual cues, innovates the visual cueing process of the detection network. It incorporates key technologies such as dataset preparation, model structure, training process, and model inference. It also designs a device for automatically collecting field images of rice planthoppers. Its advantages lie in reducing the workload for field surveyors. Furthermore, through the designed visual cue model, manual prompts are provided for erroneous detection results, enabling low-cost, adaptive optimization of abnormal detection results. This results in highly accurate field rice planthopper detection and counting, addressing the reliability issues of existing rice planthopper detection and counting methods.

Claims

1. A method for detecting and counting rice planthoppers in fields using visual cues, characterized by: The method for detecting and counting rice planthoppers in the field comprises the following steps: (1) Data acquisition: The dataset was collected using a rice planthopper detection and counting device to capture field images. The camera was held parallel to the base of the rice stem, with an aspect ratio of 4:

3. The images were taken at a height of 5-10 cm and a distance of 10-20 cm. Three to five images were taken of each infested area, and the clearest image was retained. (2) Dataset preparation: All field images were preprocessed uniformly: the aspect ratio of the photos remained unchanged, and the photos were resized to 1920 pixels in height and 1440 pixels in width. The processed photos were used as dataset images. Half of the dataset images were manually annotated using the labelme or anylabeling annotation tools. Rectangular annotation was used in a two-point annotation format, where the upper left point (x1, y1) and the lower right point (x2, y2) define a rectangular box. All rice planthopper labels were annotated with PH. The annotation tool automatically generated an annotation json file with the file name corresponding to the image name. (3) Data reading: This step is how to read dataset images and corresponding annotation files during model training and inference. The imread method in OpenCV is used to read images in Python Numpy array format. The key is reading annotations, which are rectangular boxes. The box annotations reveal the rectangular regions of interest for each rice planthopper. The following required data is then calculated and stored in a Python dictionary as key-value pairs. To improve model robustness and data diversity, data augmentation is performed when reading data. Images and annotations are randomly flipped, and a random offset x is set for the point coordinates in the points array and a random offset y is set for the box coordinates in the boundaries array. (4) Network model structure: The overall network model structure includes three key steps: image encoding, missed detection prototype matching module, false detection prototype matching module and feature fusion module; The input image is an RGB image. Assume that the input size is H and width are W respectively. The specific process is as follows: First, the input image is encoded by the image encoder to obtain the encoded feature FE. The height and width of FE are h and w respectively, and the number of channels is d, where h = H / 8, w = W / 8, and d = 256; then it is decoded by the image decoder to obtain the initial output; the image decoder uses 1×1 convolution to compress the number of FE channels to 1, and then uses bilinear interpolation to enlarge it to the input size, and finally uses the Sigmoid activation function to output; in the face of complex situations, the initial output will inevitably produce insufficient detection and calculation. The specific manifestation is that there are many obvious missed detections and false detections in the visualization results. Therefore, for these abnormal situations, manual visual annotation prompts can be used for further optimization to reduce the detection error. N missed detection or false detection prompts are given. Visual prompts are only given when the number of missed detections or false detections is greater than n. Then the prompt information is sent to the missed detection / false detection prototype matching module to obtain the missed detection prototype response map RFN and the false detection prototype response map RFP. The FE, RFN and RFP are feature fused to obtain the full response map RF. Finally, RF is decoded by the image decoder to obtain the optimized output. (5) Model training; First, the input image is passed through the image encoder and image decoder to obtain the initial output OI. During training, the annotation box information of the missed detection samples and the false detection samples is read from the annotation file in the data reading stage. The missed detection sample information corresponds to the bboxes in the data reading stage, and the false detection sample information corresponds to the fp_bboxes in the data reading stage. Based on this information, the missed detection prototype matching and false detection prototype matching stages are performed respectively to obtain the missed detection prototype response map RFN and the false detection prototype response map RFP. The missed detection prototype response map RFN is adaptively gated and fused with the encoding feature FE to obtain the missed detection response map RF1. The fusion result is input into the image decoder to obtain the missed detection optimization output OF1. The false detection prototype response map RFP is adaptively gated and fused with the encoding feature FE to obtain the false detection response map RF2. The fusion result is input into the image decoder to obtain the false detection optimization output OF2. RF1 and RF2 are adaptively gated to obtain the final response map RF, which is then input into the image encoder to obtain the final optimized output OF; Since there are a small number of images without rice planthopper annotations or false positive samples, when the number of missed detection prompt samples is 0, the missed detection prototype matching stage is skipped, and only the false positive prototype matching stage is performed, so RF = RF2; when the number of false detection prompt samples is 0, the false positive prototype matching stage is skipped, and only the missed detection prototype matching stage is performed, so RF = RF1; when the number of samples of both is 0, there is only the initial output OI; therefore, the network model has at least one output, namely OI, and at most four, namely OI, OF1, OF2, and OF; All outputs need to be optimized using a loss function. The basic loss function is for the network output O, and the loss calculation is shown in formula (1): All outputs use the object counting loss function to calculate the loss value, the loss value of OI is used as the main loss, and the other three are used as auxiliary losses; the final loss function is shown in formula (2), where λ = 0.3 The training input images are 1920×1440 in height and width (can be changed according to actual conditions), and random flipping is used for data augmentation. The backbone network uses a pre-trained ResNet50, with a batch size of 1, an Adam optimizer, a learning rate of 1e-5, and a weight decay of 1e-4. Training is performed for at least 100 epochs. (6) Model reasoning and rice planthopper detection and counting The model predicts a small spot area for the detected rice planthopper target, which is located at the center of the rice planthopper target. The predicted output O is activated using the Sigmoid activation function. The value of each position element ranges from 0 to 1, representing the probability that the pixel is a rice planthopper. If the element value is greater than 0.5, the area is determined to be a rice planthopper, otherwise it is background. Rice planthopper counting: The rice planthopper count in the image is achieved by calculating the number of independent regions in the model prediction result O. The number of independent regions in P is obtained using the connected component labeling algorithm, specifically, the label method in the scipy library. Detection of rice planthoppers: The location of rice planthoppers in an image is detected by calculating the centroid coordinates of each independent region. First, the set of non-zero labels in the label matrix is ​​extracted to remove the background. Then, a two-dimensional coordinate matrix with the same size as the input is constructed, recording the row and column indices of each pixel point. The label matrix and coordinate matrix are flattened into a one-dimensional array for vectorization. The number of pixels for each label is counted using histogram statistics, and the row and column coordinates are weighted and accumulated to obtain the sum of the vertical and horizontal coordinates of the pixels in each connected area. Finally, the center coordinates are solved for each label based on the centroid calculation formula. Model inference is divided into two stages. First, the input image is passed through the encoder and decoder to obtain the initial output. According to the above steps for rice planthopper counting and detection, we can process the model output to achieve rice planthopper counting and detection. During the experiment, since we already know the true value of the labels of all images, we can obtain the FN and FP information of the predicted results of each image based on the detection results; the prompt box information given to the model will be randomly selected from these FN and FP bounding boxes; we set the activation threshold N. When the number of FN or FP is greater than N, it means that there are many errors in the detection results. At this time, n bounding boxes are randomly extracted from the FN and FP bounding boxes as prompt inputs, activating the visual prompt mechanism and performing prototype matching; this design simulates the actual application scenario for the following two reasons: (1) In actual applications, agricultural experts will only notice and correct when there are obvious errors in the detection results; (2) The correction of agricultural experts should be low-cost, and only a small amount of prompts should be performed, hoping to correct as many errors as possible; In summary, in actual applications, since the ground truth is unknown, the prompt process should be manually triggered by agricultural experts, and a small amount of correction will be performed only when there are obvious errors in the detection results; and the selection design of FN and FP prompt boxes in our experiment is to simulate the actual application scenario; Therefore, if the number of FN and FP is less than N, the final rice planthopper detection and counting result is generated by the initial output; if the number of FN or FP is greater than or equal to N, the final rice planthopper detection and counting result is generated by the output after visual prompting.

2. The method for detecting and counting rice planthoppers in fields using visual cues according to claim 1, characterized in that: The rice planthopper detection and counting device comprises a camera, a support frame (1), a long guide rail (4), a Z-axis drive motor (2) and a control module (6); the two groups of support frames (1) are respectively provided with lifting sliders (3) through the Z-axis drive motor (2); a long guide rail (4) is provided between the lifting sliders (3); a sliding seat (5) is slidably provided on the long guide rail (4); the control module (6) is provided on the sliding seat (5); a shooting platform (8) is provided on the sliding seat (5) on one side of the control module (6) through the Y-axis drive motor (7); the shooting platform (8) is provided with a camera, a distance sensor, and a temperature and humidity sensor; a driving wheel is provided on the sliding seat (5) through the X-axis drive motor (9); and the driving wheel is in contact with the long guide rail.

3. The method for detecting and counting rice planthoppers in fields using visual cues according to claim 1, wherein: In the step (2) of data set preparation, the data set images are annotated as follows: Due to the large number of rice planthoppers, a two-step annotation-proofreading method was adopted. First, N agricultural experts divided the work and performed the first round of annotation on the dataset images. After the first round of annotation was completed, the annotations were exchanged and proofread to eliminate the influence of subjective factors in the annotation process. To reduce the annotation workload, we used a combination of automatic annotation and manual proofreading. First, we used the labeled half of the data to train the Cascade R-CNN object detection model. The Cascade R-CNN model was then used to detect the remaining half of the dataset images. The detection threshold was set to 0.75 to ensure accuracy. Based on the detection results, we generated a JSON annotation file for the remaining dataset images. The generated annotation files may have omissions, so N agricultural experts are divided into groups to proofread them. After the first round of proofreading, they exchange annotations with each other for a second round of proofreading to reduce annotation errors. In complex field applications, the model will inevitably experience a high number of missed and false detections in some images. This is particularly serious in densely populated, tiny pest counting scenarios. This technology introduces a visual prompt mechanism into the model, which provides a small amount of prompts for missed and false detections after the model's initial output. The model then extracts features based on these prompts, thereby correcting these errors. Since this technology involves a visual cue mechanism for missed and falsely detected samples, the network model learns the feature extraction capabilities for missed and falsely detected samples during training. Missed samples can be trained directly using labeled data, while falsely detected samples do not have corresponding labeled data. Therefore, additional falsely detected sample annotations are required. The preparation method is as follows: First, train the network model proposed in this technology for three rounds to obtain a non-convergent model. This model will have a large number of falsely detected samples. Based on these falsely detected samples, corresponding falsely detected labeled data is generated. The format is consistent with the labeled data in the above two steps and is randomly read during training. After completing the above three steps, the dataset production of this technology is completed.

4. The method for detecting and counting rice planthoppers in fields using visual cues according to claim 1, wherein: The keys and contents in the data reading of step (3) are respectively: Counts—a numeric data type. The total number of rice planthoppers in the image is obtained by counting the number of labeled boxes. Points—a Numpy array of the same size as the image. The coordinates of the center point of each labeled box are obtained by calculating the center point of each rice planthopper. The corresponding position in the points array is set to 1, and the other positions are set to 0. boundaries - only read during model training, the data type is a Numpy array, the size is the same as the image, read the coordinate points of the annotation box, and set the value of the area corresponding to the annotation box position in the boundaries array to 1, and the other positions to 0; boxes - the data type is a list, which stores the coordinates of all rectangular boxes; bboxes - data type is tensor, randomly extract three coordinates in boxes and form a tensor with 3 rows and 4 columns, each column is a coordinate, corresponding to the coordinate points x1, y1, x2, y2. If there are less than three coordinates, fill the gaps with 0; fp_boxes - data type is list, which stores the coordinates of all the false positive sample rectangles; fp_bboxes - data type is tensor, randomly extract three coordinates in fp_boxes, the format is the same as bboxes.

5. The method for detecting and counting rice planthoppers in fields using visual cues according to claim 1, characterized in that: The features of the image encoder, missed detection prototype matching module, false detection prototype matching module and feature fusion module in the network model structure of step (4) are as follows: The input image of the image encoder is an RGB image. Assume that the input size is H and W respectively. The convolution step size of the fifth layer of the backbone network ResNet50 is modified to 1. The height and width of the feature maps of the second to fifth layers are (H / 4, W / 4), (H / 8, W / 8), (H / 16, W / 16) and (H / 16, W / 16) respectively. The features of the second to fifth layers are input into the feature pyramid for multi-scale feature extraction. The number of channels of each output of the feature pyramid is 256, and its four outputs are uniformly adjusted to the size of (H / 8, W / 8) through bilinear interpolation and spliced ​​in the channel dimension, that is, the number of channels after splicing is 1024. Finally, a 1×1 convolution is used to reduce the number of channels to 256, and the image encoding feature FE is obtained; The structures of the missed detection prototype matching module and the false detection prototype matching module are the same, but they are two independent branches in the network and their parameters are not shared. For n prompt boxes, there are two query processes: (1) First, the height and width of each prompt box are calculated to obtain a tensor with a shape of (n, 2), and the tensor is input into an MLP for encoding. The process is: (n, 2) → (n, 64) → ReLU → (n, d) → ReLU → (n, k2 × d), where n = 3, d = 256, k = 3, and then the encoding result is reshaped and flattened to (nk2, d) to obtain the shape and size query tensor QS; (2) Then, based on the given prompt box coordinates, the ROIAlign operation is performed on the encoded feature FE to extract n representation queries with a size of k × k and a number of channels of d, and they are reshaped and flattened to (nk2, d) to obtain the representation query tensor QA; Input QS, QA, and FE into the prototype query module for query, and finally obtain n prototype query features of size k×k and number of channels d; use the prototype query features as convolution kernels, perform depth-correlated convolution on the encoding features FE, and obtain n query graphs with number of channels d. Finally, use a 1×1 convolution to integrate the outputs to obtain the response graph R; The prototype query module’s LN stands for layer normalization, MHA stands for multi-head self-attention mechanism, and MLP has two linear layers. The input dimension changes from (nk2, d) → (nk2, 8d) → GELU → (nk2, d), and is finally reshaped to (n × d, k, k), which is the prototype query feature. feature The fusion module includes an adaptive feature gating mechanism; The prototype matching response map needs to be fused with the encoding feature FE, but the direct addition method is not stable enough and may cause the superposition of outliers between the two feature maps, causing adverse effects. Therefore, an adaptive feature gating mechanism is designed to perform adaptive weighted fusion of the two features with a learnable module. The two features are first added element by element, and there are two feature weight calculation branches: (1) global average pooling is performed, and then an MLP is used for feature encoding to obtain the global encoding weight with a dimension of (1, 1, d); (2) two 1×1 convolutions are performed to obtain the spatial encoding weight with a dimension of (h, w, d); then the global encoding weight is broadcasted, added element by element to the spatial encoding weight, and activated using the Sigmoid function to obtain the adaptive gating weight Wgate; finally, the adaptive fusion result of the two features is calculated, and the fusion method is: f fusion =f1×W gate +f2×(1-W gate ); For the fusion of the coding features, missed detection response graphs, and false detection response graphs in the network, the coding features are first fused with the two response graphs in pairs, and then the two fusions are combined and fused again to obtain the final full response graph RF; if there is only one response graph, only the coding features are fused with the response graph, and the subsequent fusion steps are skipped.

Images