Crop disease and pest identification method and system

Abstract

The invention relates to the field of agriculture, in particular to a crop disease and pest recognition method and system, and the method comprises the steps: obtaining an image of a crop, carrying out the preprocessing of the image, extracting features, obtaining a multi-scale feature map, and enabling the multi-scale feature map to serve as the input of an encoder; determining the position of a reference point in each feature graph input by each layer of the encoder, for each deformable attention head, inputting the feature of the reference point into a linear layer to obtain the offset of a sampling point, taking the sampling point and the reference point as the nodes of the graph, determining the weight of an edge based on the offset of the sampling point and the reference point, carrying out one-time graph convolution on the graph, and carrying out one-time graph convolution; obtaining a reference point feature corresponding to the deformable attention head based on a result of the graph convolution; and inputting the characteristic pattern output by the encoder into a decoder, and obtaining a crop disease and insect pest identification result by using a query vector of the decoder.

Description

Technical Field

[0001] The present invention relates to the field of agriculture, and in particular to a method and system for identifying crop pests and diseases. Background Art

[0002] The occurrence of pests and diseases is one of the important factors threatening crop yield and quality. If the spread of pests and diseases cannot be identified and controlled in a timely and accurate manner, it will not only cause serious economic losses, but also affect the stability of the ecosystem, increase the use of pesticides, and further affect human health and environmental safety. In traditional agricultural management, the prevention and control of pests and diseases usually relies on farmers' experience and manual detection. However, manual detection is inefficient, especially in large-scale farmland, which is time-consuming and labor-intensive to check each piece of land. In addition, due to the wide variety and complex morphology of crop pests and diseases, manual detection is prone to misjudgment and omission, resulting in untimely prevention and control measures or excessive use of pesticides.

[0003] In recent years, with the rapid development of artificial intelligence, image recognition technology and big data technology, agricultural informatization and intelligence have become important means to solve the problem of crop pest and disease identification. Pest and disease identification technology based on computer vision relies on high-resolution image data and uses deep learning and other algorithms to accurately identify pest and disease types. This automated pest and disease detection method can not only greatly improve detection efficiency, but also improve recognition accuracy and reduce dependence on human resources, thereby providing strong technical support for precision agriculture. Most pest and disease identification systems have high requirements for environmental conditions and are greatly affected by factors such as light and weather, resulting in low recognition rates in complex field environments. In addition, traditional recognition algorithms have limited recognition accuracy for small targets, and it is difficult to identify pests and diseases in large images taken by drones. Summary of the invention

[0004] In order to solve the above problems, in a first aspect of the present invention, the present invention provides a method for identifying crop pests and diseases, the method comprising the following steps: Acquire an image of a crop, extract features from the image after preprocessing to obtain a multi-scale feature map, and use the multi-scale feature map as an input of an encoder; Determine the reference point position in each feature map of each layer of the encoder input, for each deformable attention head, input the reference point feature into the linear layer to obtain the offset of the sampling point, use the sampling point and the reference point as nodes of the graph, and determine the weight of the edge based on the offset of the sampling point and the reference point, perform a graph convolution on the graph, and obtain the reference point feature corresponding to the deformable attention head based on the result of the graph convolution; The feature map output by the encoder is input into the decoder, and the query vector of the decoder is used to obtain the crop pest and disease recognition result.

[0005] Preferably, the weight of the edge is determined based on the offset between the sampling point and the reference point, specifically: Calculate the sum of the length and width of the feature map where the sampling point and the reference point are located, as well as the distance between the sampling point and the reference point on the feature map, calculate the ratio of the distance to the sum, and use the ratio as the weight of the edge.

[0006] Preferably, the reference point features corresponding to the deformable attention head are obtained based on the result of the graph convolution, specifically: The features of the reference points output by the graph convolution are input into the linear layer to obtain the weights of the sampling points, the weights are normalized, and the reference point features corresponding to the deformable attention head are calculated by the features of the sampling points output by the graph convolution and the normalized weights; or, the reference point features output by the graph convolution are used as the reference point features corresponding to the deformable attention head.

[0007] Preferably, the features of the sampling points output by the graph convolutional network and the normalized weights are used to calculate the reference point features corresponding to the deformable attention head, specifically: The normalized weights are used to weight the features of each sampling point output by the graph convolutional network and sum them up to obtain the reference point features corresponding to the deformable attention head.

[0008] Preferably, after obtaining the reference point features corresponding to the deformable attention head based on the result of the graph convolution, the method further includes: The reference point features output by different deformable attention heads of the same reference point are fused to obtain the features of the reference point on a single input feature map; The features of the reference points at the same position or number in all input feature maps are fused to obtain the features of the reference points in the output feature map.

[0009] In a second aspect of the present invention, the present invention provides a crop pest identification system, the system comprising the following modules: A feature extraction module is used to obtain an image of the crop, extract features from the image after preprocessing to obtain a multi-scale feature map, and use the multi-scale feature map as an input of the encoder; An encoding module, used to determine the reference point position in each feature map of each layer of the encoder input, for each deformable attention head, input the reference point feature into the linear layer to obtain the offset of the sampling point, use the sampling point and the reference point as nodes of the graph, and determine the weight of the edge based on the offset of the sampling point and the reference point, perform a graph convolution on the graph, and obtain the reference point feature corresponding to the deformable attention head based on the result of the graph convolution; The decoding and recognition module is used to input the feature map output by the encoder into the decoder, and use the query vector of the decoder to obtain the crop pest and disease recognition result.

[0010] Preferably, the weight of the edge is determined based on the offset between the sampling point and the reference point, specifically: Calculate the sum of the length and width of the feature map where the sampling point and the reference point are located, as well as the distance between the sampling point and the reference point on the feature map, calculate the ratio of the distance to the sum, and use the ratio as the weight of the edge.

[0011] Preferably, the reference point features corresponding to the deformable attention head are obtained based on the result of the graph convolution, specifically: The features of the reference points output by the graph convolution are input into the linear layer to obtain the weights of the sampling points, the weights are normalized, and the reference point features corresponding to the deformable attention head are calculated by the features of the sampling points output by the graph convolution and the normalized weights; or, the reference point features output by the graph convolution are used as the reference point features corresponding to the deformable attention head.

[0012] Preferably, the features of the sampling points output by the graph convolutional network and the normalized weights are used to calculate the reference point features corresponding to the deformable attention head, specifically: The normalized weights are used to weight the features of each sampling point output by the graph convolutional network and sum them up to obtain the reference point features corresponding to the deformable attention head.

[0013] Preferably, after obtaining the reference point features corresponding to the deformable attention head based on the result of the graph convolution, the method further includes: The reference point features output by different deformable attention heads of the same reference point are fused to obtain the features of the reference point on a single input feature map; The features of the reference points at the same position or number in all input feature maps are fused to obtain the features of the reference points in the output feature map.

[0014] In addition, the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method described in the first aspect is implemented.

[0015] In view of the problem of low recognition accuracy of small targets in the prior art, the present invention improves the deformable attention in the Deformable DETR model. When calculating the deformable attention of the reference point, a graph structure of the reference point is first constructed, and then graph convolution is used to fuse the information of the reference point and the information of the sampling point, thereby avoiding the problem of only using the reference point features for both the offset and the weight in the deformable attention calculation, improving the expressiveness of the model, and achieving better recognition effect for small targets. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a flow chart of Embodiment 1; Figure 2 It is a schematic diagram of the structure of a graph; Figure 3 This is a schematic diagram of the recognition results; Figure 4 This is a structural diagram of the second embodiment. DETAILED DESCRIPTION

[0017] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0018] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0019] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0020] In a first aspect, the present invention provides a method for identifying crop pests and diseases, such as Figure 1 As shown, the method comprises the following steps: S1, obtaining an image of a crop, extracting features from the image after preprocessing to obtain a multi-scale feature map, and using the multi-scale feature map as an input of an encoder; Images of crops are captured by an imaging device, preferably by a drone. After the captured images are acquired, the images are preprocessed, wherein the preprocessing includes but is not limited to: image denoising, color adjustment, brightness adjustment, contrast enhancement, image scaling and normalization. After obtaining the preprocessed images, feature extraction is performed to obtain feature maps of multiple scales, and the feature extraction methods include but are not limited to using convolution extraction, using backbone network extraction, etc. When identifying pests and diseases of crops, the recognition includes two parts, encoding by an encoder and decoding by a decoder. In one embodiment, the recognition model preferably uses a Deformable DETR model, which has better recognition of small targets.

[0021] S2, determining the reference point position in each feature map of each layer of the encoder input, for each deformable attention head, inputting the reference point feature into the linear layer to obtain the offset of the sampling point, taking the sampling point and the reference point as nodes of the graph, and determining the weight of the edge based on the offset of the sampling point and the reference point, performing a graph convolution on the graph, and obtaining the reference point feature corresponding to the deformable attention head based on the result of the graph convolution; There are multiple identical layers in the encoder. Each layer inputs feature maps of multiple scales and outputs feature maps of multiple scales. Each layer uses multi-scale deformable multi-head attention for calculation. In the multi-scale deformable multi-head attention calculation, the reference point is determined on the input feature map. The reference point position of the input feature map can be obtained through get_reference_points or gen_encoder_output_proposals of Deformable DETR.

[0022] In multi-scale deformable multi-head attention, each reference point will have multiple deformable attention heads. For each deformable attention head, the features of the reference point are input into the linear layer to obtain the offset of the sampling point, so that multiple sampling points corresponding to the reference point can be obtained. In Deformable DETR, the offset and the weight of each sampling point are obtained by inputting the features of the reference point into the Linear layer. However, relying solely on the features of the reference points cannot fully reflect the content relevance between the sampling points. This will result in inaccurate weight distribution and inability to accurately distinguish between important and minor sampling points. Moreover, relying solely on the features of the reference points cannot capture the complex spatial relationship between the sampling points and the reference points.

[0023] After using the linear layer to obtain the offset of the sampling point, the present invention uses the sampling point and the reference point as the nodes of the graph, and uses the offset of the sampling point and the reference point to determine the weight of the edge, and performs a graph convolution operation on the graph corresponding to the reference point, so that after the features of the reference point and the sampling point are fused, the reference point features of the deformable attention head are no longer obtained solely by relying on the reference point or the sampling point. The nodes of the graph of the reference point are the reference point or the sampling point, and the edges of the graph are the distances between the reference point and the sampling point. Figure 2 The process of obtaining the graph based on the reference points and sampling points is shown. Figure 2 In the figure, D1-D4 are the distances from the sampling points to the reference points, and W1-W4 are the weights of the edges in the graph.

[0024] Since it is a multi-scale feature map, even if the distances between the sampling points and the reference points are the same, the distances of the features in the multi-scale feature map are not the same. In an optional embodiment, the weight of the edge is determined based on the offset of the sampling points and the reference points, specifically: Calculate the sum of the length and width of the feature graph where the sampling point and the reference point are located, as well as the distance between the sampling point and the reference point on the feature graph, calculate the ratio of the distance to the sum, and use the ratio as the weight of the edge. After obtaining the weights of all edges in the graph, normalize the edge weights. For example, if the length and width of a feature graph are 100 and 100 respectively, the sum of the length and width is 200. If the distance between the sampling point and the reference point on the feature graph is 2, the ratio is 0.01. Assuming the normalized value is 0.4, the weight of the edge in the graph is 0.4.

[0025] After the graph convolution, the features of the reference point are integrated with the features of the sampling point, and the features of the sampling point are also integrated with the features of the reference point. In an optional embodiment, the reference point features corresponding to the deformable attention head are obtained based on the result of the graph convolution, specifically: The features of the reference points output by the graph convolution are input into the linear layer to obtain the weights of the sampling points, the weights are normalized, and the reference point features corresponding to the deformable attention head are calculated by the features of the sampling points output by the graph convolution and the normalized weights; or, the reference point features output by the graph convolution are used as the reference point features corresponding to the deformable attention head.

[0026] In the first solution, the weight calculation method in the deformable attention head is still used, that is, the features of the reference point are used to obtain the weight of each sampling point, and then the weight of the sampling point is normalized. The sampling point features output by the graph convolution and the normalized weight calculation are used to obtain the reference point features corresponding to the deformable attention head. Compared with the original deformable attention, although the weight calculation method also uses the features of the reference point, the features of the reference point are integrated with the features of the sampling point through graph convolution, which avoids the problem of relying solely on the reference point to calculate the weight and ignoring the features of the sampling point; in addition, the features of the sampling point no longer use the original features of the sampling point, but use the features of the sampling point after graph convolution, which can improve the expression ability of the model.

[0027] In the second scheme, the features of the reference point after convolution are directly used as the reference point features corresponding to the deformable attention head. Similarly, the features of the reference point fuse the features of the sampling point, and reduce the weights and the process of calculating the output of the deformable attention head using the weights, which can achieve higher processing speed than the first scheme.

[0028] In an optional embodiment, the features of the sampling points output by the graph convolutional network and the normalized weights are used to calculate the reference point features corresponding to the deformable attention head, specifically: The normalized weights are used to weight the features of each sampling point output by the graph convolutional network and sum them up to obtain the reference point features corresponding to the deformable attention head. For each deformable attention head, there will be multiple sampling points, each sampling point will have a weight, the sampling point features and the sampling point weights are multiplied, and then summed to obtain the reference point features corresponding to the deformable attention head.

[0029] In the above process, when there are multiple deformable attention heads, the deformable attention heads of the reference point are output for feature fusion. The feature fusion method includes but is not limited to averaging, etc., to obtain a multi-head deformable attention result corresponding to the reference point. In one embodiment, after obtaining the reference point feature corresponding to the deformable attention head based on the result of the graph convolution, it also includes: The reference point features output by different deformable attention heads of the same reference point are fused to obtain the features of the reference point on a single input feature map; The features of the reference points at the same position or number in all input feature maps are fused to obtain the features of the reference points in the output feature map.

[0030] In an optional embodiment, each layer of the encoder is a multi-scale deformable multi-head attention, and the reference point features output by different deformable attention heads of the same reference point are fused to obtain the features of the reference point on a single input feature map. This process obtains a deformable multi-head attention on one scale. When there are multiple scales, the deformable multi-head attentions of the reference points with the same number on each scale are fused to obtain the multi-scale deformable multi-head attention of the reference points, so that the multi-scale feature map output by each layer of the encoder can be obtained. The focus of the present invention is on the calculation of the reference point on a deformable attention head. In one embodiment, the calculation method of each deformable attention head in the Deformable DETR model is replaced by the deformable attention head calculation method of the present invention.

[0031] S3, input the feature map output by the encoder into the decoder, and use the query vector of the decoder to obtain the crop pest and disease recognition result.

[0032] After obtaining the feature map output by the encoder, the feature map is input into the decoder, wherein the decoder preferably uses the Deformable DETR decoder to obtain the recognition result of crop pests and diseases through the query vector (Object Queries). Figure 3 The results of pest and disease identification on crop images taken by drones are shown.

[0033] In a second aspect, the present invention provides a crop pest identification system, such as Figure 4 As shown, the system includes the following modules: A feature extraction module is used to obtain an image of the crop, extract features from the image after preprocessing to obtain a multi-scale feature map, and use the multi-scale feature map as an input of the encoder; An encoding module, used to determine the reference point position in each feature map of each layer of the encoder input, for each deformable attention head, input the reference point feature into the linear layer to obtain the offset of the sampling point, use the sampling point and the reference point as nodes of the graph, and determine the weight of the edge based on the offset of the sampling point and the reference point, perform a graph convolution on the graph, and obtain the reference point feature corresponding to the deformable attention head based on the result of the graph convolution; The decoding and recognition module is used to input the feature map output by the encoder into the decoder, and use the query vector of the decoder to obtain the crop pest and disease recognition result.

[0034] Preferably, the weight of the edge is determined based on the offset between the sampling point and the reference point, specifically: Calculate the sum of the length and width of the feature map where the sampling point and the reference point are located, as well as the distance between the sampling point and the reference point on the feature map, calculate the ratio of the distance to the sum, and use the ratio as the weight of the edge.

[0035] Preferably, the reference point features corresponding to the deformable attention head are obtained based on the result of the graph convolution, specifically: The features of the reference points output by the graph convolution are input into the linear layer to obtain the weights of the sampling points, the weights are normalized, and the reference point features corresponding to the deformable attention head are calculated by the features of the sampling points output by the graph convolution and the normalized weights; or, the reference point features output by the graph convolution are used as the reference point features corresponding to the deformable attention head.

[0036] Preferably, the features of the sampling points output by the graph convolutional network and the normalized weights are used to calculate the reference point features corresponding to the deformable attention head, specifically: The normalized weights are used to weight the features of each sampling point output by the graph convolutional network and sum them up to obtain the reference point features corresponding to the deformable attention head.

[0037] Preferably, after obtaining the reference point features corresponding to the deformable attention head based on the result of the graph convolution, the method further includes: The reference point features output by different deformable attention heads of the same reference point are fused to obtain the features of the reference point on a single input feature map; The features of the reference points at the same position or number in all input feature maps are fused to obtain the features of the reference points in the output feature map.

[0038] In a third aspect, the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method described in the first aspect.

[0039] In a fourth aspect, this embodiment provides a computer program product, including a computer program, which, when executed by a processor, implements the method described in the first aspect.

[0040] All or part of the steps of the above-mentioned method embodiments can be completed by hardware related to program instructions. The above-mentioned program can be stored in a readable memory. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the above-mentioned memory (storage medium) includes: read-only memory (ROM), random access memory (RAM), flash memory, hard disk, solid state drive, magnetic tape, floppy disk, optical disc and any combination thereof.

[0041] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processing unit of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

Claims

1. A method for identifying crop pests and diseases, characterized in that: The method comprises the following steps: Acquire an image of a crop, extract features from the image after preprocessing to obtain a multi-scale feature map, and use the multi-scale feature map as an input of an encoder; Determine the reference point position in each feature map of each layer of the encoder input, for each deformable attention head, input the reference point feature into the linear layer to obtain the offset of the sampling point, use the sampling point and the reference point as nodes of the graph, and determine the weight of the edge based on the offset of the sampling point and the reference point, perform a graph convolution on the graph, and obtain the reference point feature corresponding to the deformable attention head based on the result of the graph convolution; The feature map output by the encoder is input into the decoder, and the query vector of the decoder is used to obtain the crop pest and disease recognition result.

2. The method according to claim 1, characterized in that The weight of the edge is determined based on the offset between the sampling point and the reference point, specifically: Calculate the sum of the length and width of the feature map where the sampling point and the reference point are located, as well as the distance between the sampling point and the reference point on the feature map, calculate the ratio of the distance to the sum, and use the ratio as the weight of the edge.

3. The method according to claim 1, characterized in that The reference point features corresponding to the deformable attention head are obtained based on the result of the graph convolution, specifically: The features of the reference points output by the graph convolution are input into the linear layer to obtain the weights of the sampling points, the weights are normalized, and the reference point features corresponding to the deformable attention head are calculated by the features of the sampling points output by the graph convolution and the normalized weights; or, the reference point features output by the graph convolution are used as the reference point features corresponding to the deformable attention head.

4. The method according to claim 3, characterized in that The features of the sampling points output by the graph convolutional network and the normalized weights are used to calculate the reference point features corresponding to the deformable attention head, specifically: The normalized weights are used to weight the features of each sampling point output by the graph convolutional network and sum them up to obtain the reference point features corresponding to the deformable attention head.

5. The method according to claim 1, characterized in that After obtaining the reference point features corresponding to the deformable attention head based on the result of the graph convolution, the method further includes: The reference point features output by different deformable attention heads of the same reference point are fused to obtain the features of the reference point on a single input feature map; The features of the reference points at the same position or number in all input feature maps are fused to obtain the features of the reference points in the output feature map.

6. A crop pest identification system, characterized in that: The system includes the following modules: A feature extraction module is used to obtain an image of the crop, extract features from the image after preprocessing to obtain a multi-scale feature map, and use the multi-scale feature map as an input of the encoder; An encoding module, used to determine the reference point position in each feature map of each layer of the encoder input, for each deformable attention head, input the reference point feature into the linear layer to obtain the offset of the sampling point, use the sampling point and the reference point as nodes of the graph, and determine the weight of the edge based on the offset of the sampling point and the reference point, perform a graph convolution on the graph, and obtain the reference point feature corresponding to the deformable attention head based on the result of the graph convolution; The decoding and recognition module is used to input the feature map output by the encoder into the decoder, and use the query vector of the decoder to obtain the crop pest and disease recognition result.

7. The system according to claim 6, characterized in that The weight of the edge is determined based on the offset between the sampling point and the reference point, specifically: Calculate the sum of the length and width of the feature map where the sampling point and the reference point are located, as well as the distance between the sampling point and the reference point on the feature map, calculate the ratio of the distance to the sum, and use the ratio as the weight of the edge.

8. The system according to claim 6, characterized in that The reference point features corresponding to the deformable attention head are obtained based on the result of the graph convolution, specifically: The features of the reference points output by the graph convolution are input into the linear layer to obtain the weights of the sampling points, the weights are normalized, and the reference point features corresponding to the deformable attention head are calculated by the features of the sampling points output by the graph convolution and the normalized weights; or, the reference point features output by the graph convolution are used as the reference point features corresponding to the deformable attention head.

9. The system according to claim 6, characterized in that After obtaining the reference point features corresponding to the deformable attention head based on the result of the graph convolution, the method further includes: The reference point features output by different deformable attention heads of the same reference point are fused to obtain the features of the reference point on a single input feature map; The features of the reference points at the same position or number in all input feature maps are fused to obtain the features of the reference points in the output feature map.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the computer program implements the method according to any one of claims 1 to 5.

Images