Deep learning-based poplar pest image recognition method and system

Abstract

The invention discloses a deep learning-based poplar pest image recognition method and system, and the method comprises the steps: obtaining poplar pest domain knowledge, carrying out the knowledge distillation enhancement, constructing a poplar pest knowledge graph, determining a poplar variety, constructing a poplar pest ecological deep recognition model, and carrying out the recognition of a poplar pest image. Inputting a to-be-identified poplar pest ecological picture into the poplar pest ecological depth identification model to obtain a first poplar pest ecological feature, processing the multi-band pest picture to obtain a secondary pest feature, performing feature crossing to obtain a second poplar pest ecological feature, determining a spatial-temporal feature, and identifying the poplar pest ecological depth identification model; and inputting the spatial-temporal characteristics, the poplar variety and the second poplar pest ecological characteristics into the poplar pest knowledge graph to obtain a poplar pest prediction result. The method not only can improve the efficiency and accuracy of poplar pest image recognition, but also has good interpretability, and can be directly applied to a poplar pest image recognition system.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image recognition, in particular to a poplar pest image recognition method and system based on deep learning. BACKGROUND

[0002] As an important fast-growing tree species in the sustainable development of forestry, poplar is widely used in wood production, ecological protection and other fields, and its healthy growth is directly related to the stability of forestry economy and the balance of ecological environment. However, the frequent outbreak of poplar pests often leads to hindered growth of trees, decreased wood quality, and even large-scale death, causing huge losses to the forestry industry. Therefore, quickly and accurately identifying the species and damage degree of poplar pests is of great significance for timely taking targeted prevention measures and reducing disaster losses.

[0003] The existing poplar pest identification method mainly relies on a single visible light RGB image and a general deep model, which has the following defects: first, the information dimension is single, lacking multi-band data such as near-infrared and ultraviolet, resulting in a high early-stage egg mass and larva stage missing detection rate; second, prior knowledge such as poplar varieties, symbiotic organisms, diseases, and spatio-temporal distribution is not systematically introduced, and the model has poor migration performance in complex outdoor scenes; finally, there is a lack of knowledge distillation and atlas reasoning mechanism, making it difficult to distinguish between "similar pests" or "rare regional types" in fine granularity. Therefore, the present application proposes a poplar pest image recognition method and system based on deep learning, which enhances the field knowledge by knowledge distillation to build a poplar pest knowledge graph, fuses the ecological characteristics of pests, symbiotic organisms and poplar diseases, and combines multi-band processing and spatio-temporal alignment technology to effectively overcome the defects of traditional technology. This method not only overcomes the limitations of traditional identification technology, but also provides strong technical support for early warning and scientific prevention of forestry pests, which has important practical significance for improving the intelligent level of forestry management. SUMMARY

[0004] The purpose of the present application is to provide a poplar pest image recognition method and system based on deep learning.

[0005] To achieve the above purpose, the present application is implemented according to the following technical solutions:

[0006] The present application comprises the following steps:

[0007] Obtain poplar pest field knowledge for knowledge distillation enhancement and build a poplar pest knowledge graph;

[0008] Obtain poplar images and compare them with a standard poplar image library to determine the poplar variety, and obtain annotated poplar pest ecological pictures to build a poplar pest ecological deep recognition model; the poplar pest ecological pictures include pest pictures, symbiotic organism pictures and poplar disease pictures;

[0009] inputting a to-be-identified poplar pest ecological picture into the poplar pest ecology deep recognition model to obtain first poplar pest ecological features; the first poplar pest ecological features include main pest features, symbiotic organism features, and poplar disease features;

[0010] processing a multi-band pest picture by using a band difference saliency map algorithm to obtain secondary pest features, and performing feature intersection on the secondary pest features and the first poplar pest ecological features to obtain second poplar pest ecological features;

[0011] performing spatio-temporal alignment on the to-be-identified poplar pest ecological picture and shooting data to determine spatio-temporal features, inputting the spatio-temporal features, the poplar variety, and the second poplar pest ecological features into the poplar pest knowledge graph to obtain a poplar pest prediction result.

[0012] Further, the method for constructing the poplar pest knowledge graph comprises:

[0013] obtaining poplar pest field knowledge to construct a structured knowledge base from expert experience, literature, image annotation, and a text database; the poplar pest field knowledge includes pest morphological features, pest growth stage features, symbiotic organism features, poplar disease features, and growth environment features;

[0014] training a teacher model by using the structured knowledge base, establishing a relationship between features learned by the teacher model and the knowledge graph based on a knowledge graph alignment loss function, and migrating an output of the teacher model to a student model for knowledge distillation; the knowledge distillation includes soft label distillation, feature layer distillation, and knowledge graph embedding distillation; the soft label distillation uses soft labels output by the teacher model to guide the student model; the feature layer distillation constructs a mapping relationship between an intermediate feature map of the teacher model and an intermediate feature map of the student model; and the knowledge graph embedding distillation constructs a mapping relationship between a knowledge graph embedding vector of the teacher model and a feature map of the student model;

[0015] the knowledge graph alignment loss function expression is:

[0016]

[0017] wherein is a knowledge graph alignment loss function, M is a field knowledge weight matrix, and is related to feature weight corresponding to poplar pest field knowledge, is a KL divergence, used to measure a probability distribution difference between outputs of the teacher model and the student model, is a softened probability distribution of the teacher model, is a softened probability distribution of the student model, T is an original prediction vector output by the teacher model, S is an original prediction vector output by the student model, η1 is a temperature hyperparameter, F T is an intermediate layer feature map of the teacher model, F SThe student model corresponds to a layer feature map;

[0018] According to the knowledge distillation enhanced poplar pest field knowledge, a poplar pest knowledge graph is constructed;

[0019] The feature information of the newly annotated poplar pest ecological picture is extracted to randomly form a plurality of candidate triples, the similarity of the corresponding candidate triples and the existing triples in the knowledge graph is calculated, when the similarity is higher than the similarity threshold, the weight of the corresponding relationship is updated, and when there is no corresponding triple, it is added as a new relationship;

[0020] A pre-trained relation extraction model is used to process new text to extract a plurality of candidate triples, entity linking technology is used to link the candidate triples to the nodes in the knowledge graph, the knowledge graph where the node confidence is greater than the confidence threshold is updated, and the expression is:

[0021]

[0022] Among them is the edge weight of node i to the neighbor node j at t+1, is the neighbor node set of node i, is the node confidence, and φ(r kj ), φ(r mj ) are the relationship type weights, and λ is the self-enhancement coefficient.

[0023] Further, the method for determining the poplar variety comprises:

[0024] Color correction and distortion correction are performed on the poplar image to obtain a corrected poplar image, and the corrected poplar image is segmented into a poplar trunk image, a poplar leaf image and a poplar branch image according to the parts, and the image features of the poplar trunk image, the poplar leaf image and the poplar branch image are extracted to obtain trunk features, leaf features and branch features;

[0025] The cosine similarity of the trunk features, the leaf features and the branch features with the corresponding part features of the same variety poplar in the standard poplar library is calculated to obtain part similarity, the part season weight is determined according to the identification season, and the variety similarity is calculated by weighting according to the part similarity and the part season weight; the part season weight is a fixed value set in advance according to the growth law of poplar leaves;

[0026] The standard poplar variety corresponding to the highest value of the variety similarity is taken as the poplar variety to be identified.

[0027] Further, the method for obtaining the first poplar pest ecological feature comprises:

[0028] The annotated pest graph, symbiotic organism graph and poplar disease graph are used to construct a poplar pest ecological deep recognition model; the poplar pest ecological deep recognition model comprises an input layer, a biological layer, a plant layer, an enhancement layer, a full connection layer and an output layer; the biological layer and the plant layer are designed in parallel;

[0029] The biological layer uses ResNet to extract pest graph features and symbiotic organism graph features, and obtains initial pest extraction features and initial symbiotic organism extraction features by processing the graph features through the biological attention mechanism of the biological channel, and the expression is as follows:

[0030]

[0031] Wherein, BA(F i ) is the attention of the feature F i , F i includes the feature tensor F v of the pest image and the feature tensor F s of the symbiotic organism image, is a nonlinear mapping about the biological feature F i , and W g is a learnable biological feature convolution kernel;

[0032] The plant layer uses UNet to extract multi-scale disease features of the poplar disease graph, and obtains poplar disease extraction features by processing the multi-scale disease features through the plant attention mechanism of the plant channel, and the expression is as follows:

[0033]

[0034] Wherein, PA(F p ) is the attention of the multi-scale disease feature F p , is a multi-scale disease perception branch of the feature F p , and multi-scale α i includes average pooling, maximum pooling, hole convolution and Gabor filtering, is an α i scale channel convolution kernel;

[0035] The enhancement layer is embedded with a pest prototype memory bank, and the initial pest extraction features and the initial symbiotic organism extraction features input are processed through a prototype contrast loss function, so as to enlarge the distance between similar pests and obtain pest extraction features and symbiotic organism extraction features; the pest prototype memory bank stores cross-scale prototype vectors of each type of pest, including microscopic prototype vectors, mesoscopic prototype vectors and macroscopic prototype vectors; and the prototype contrast loss function expression is as follows:

[0036]

[0037] Wherein sim is the prototype-to-prototype loss function, sim s sim is the feature similarity score at the current scale s, sim s sim is the feature similarity score at the current scale s, η2 is the temperature parameter, f s is the feature representation of the current sample at scale s, is the prototype vector of the same class y at scale s, is the prototype vector of the different class y at scale s, m is the boundary margin;

[0038] The fully connected layer is connected to the plant layer and the enhancement layer respectively, and extracts the feature of the poplar disease, the feature of the pest and the feature of the symbiotic organism, performs feature fusion and compression, and decodes the main pest feature, the symbiotic organism feature and the poplar disease feature.

[0039] The to-be-recognized poplar pest ecological picture is input into the poplar pest ecological deep recognition model to obtain the first poplar pest ecological feature.

[0040] Further, the method for obtaining the second poplar pest ecological feature comprises:

[0041] The near-infrared pest image, the ultraviolet pest image and the visible light pest image are acquired for spatio-temporal alignment, and the spatio-temporally aligned near-infrared pest image, ultraviolet pest image and visible light pest image are subjected to band difference saliency processing to obtain a band difference saliency map, and the expression is:

[0042]

[0043] wherein, BDSM is the band difference saliency map, α is the enhancement coefficient, NIR is the near-infrared pest image, UV is the ultraviolet pest image, RGB is the visible light pest image, ∈ is a numerical stability constant, I is the original pest image, ω k is the kth stage larva feature weight parameter, μ k is the kth stage larva feature Gaussian center, σ k is the kth stage larva feature Gaussian standard deviation;

[0044] The band difference saliency map is subjected to feature extraction to obtain a secondary pest feature, the secondary pest feature and the main pest feature in the first poplar pest ecological feature are subjected to feature cross to obtain a pest feature, and the second poplar pest ecological feature is composed of the pest feature, the symbiotic organism feature and the poplar disease feature; the secondary pest feature comprises growth stage, body number, body color transmittance and motion trajectory entropy.

[0045] Further, the method for obtaining the poplar pest prediction result comprises:

[0046] The ecological picture of the poplar pest is spatiotemporally aligned with the shooting data to obtain shooting time and shooting area, and the climate environment is determined according to the shooting time and the shooting area, and the spatiotemporal feature is composed of the climate environment, the shooting time and the shooting area;

[0047] The spatiotemporal feature, the poplar variety and the second poplar pest ecological feature are input into the poplar pest knowledge graph to obtain a poplar pest prediction result; the poplar pest prediction result includes a pest variety, a pest growth stage ratio and a prediction probability.

[0048] In a second aspect, a poplar pest image recognition system based on deep learning comprises:

[0049] The knowledge graph module is used for obtaining poplar pest field knowledge for knowledge distillation enhancement, and constructing a poplar pest knowledge graph; and the spatiotemporal feature, the poplar variety and the second poplar pest ecological feature are input into the poplar pest knowledge graph to obtain a poplar pest prediction result;

[0050] The tree species recognition module is used for comparing a poplar image with a standard poplar image library to determine a poplar variety;

[0051] The deep recognition module is used for constructing a poplar pest ecological deep recognition model by using labeled poplar pest ecological pictures; and a first poplar pest ecological feature is obtained by inputting a to-be-recognized poplar pest ecological picture into the poplar pest ecological deep recognition model;

[0052] The feature deepening module is used for processing a multi-band pest image by using a band difference saliency algorithm to obtain a secondary pest feature, and performing feature intersection on the secondary pest feature and the first poplar pest ecological feature to obtain a second poplar pest ecological feature; and the spatiotemporal feature is obtained by spatiotemporally aligning the to-be-recognized poplar pest ecological picture with shooting data.

[0053] The poplar pest image recognition method and system based on deep learning have the following technical effects compared with the prior art:

[0054] The poplar pest image recognition method and system based on deep learning have the following technical effects compared with the prior art:

[0055] The poplar pest image recognition method and system based on deep learning have the following technical effects compared with the prior art: BRIEF DESCRIPTION OF DRAWINGS

[0056] Figure 1 This is a flowchart illustrating the steps of the deep learning-based poplar pest image recognition method of the present invention. Detailed Implementation

[0057] The present invention will be further described below through specific embodiments. The illustrative embodiments and descriptions herein are used to explain the present invention, but are not intended to limit the present invention.

[0058] The present invention provides a deep learning-based image recognition method and system for poplar pests, comprising the following steps:

[0059] like Figure 1 As shown, this embodiment includes the following steps:

[0060] Knowledge distillation and enhancement are performed on knowledge of poplar pests to construct a knowledge graph of poplar pests.

[0061] Poplar tree images are obtained and compared with a standard poplar tree image library to determine poplar tree species. Annotated ecological images of poplar tree pests are obtained to construct a deep ecological identification model of poplar tree pests. The ecological images of poplar tree pests include images of pests, images of symbiotic organisms, and images of poplar tree diseases.

[0062] The ecological images of poplar pests to be identified are input into the deep ecological identification model of poplar pests to obtain the first ecological characteristics of poplar pests; the first ecological characteristics of poplar pests include the characteristics of major pests, the characteristics of symbiotic organisms and the characteristics of poplar diseases.

[0063] The band difference significance map algorithm is used to process the multi-band pest map to obtain secondary pest features. The secondary pest features are then combined with the first poplar pest ecological features to obtain the second poplar pest ecological features.

[0064] The spatiotemporal features of the ecological images of poplar pests to be identified are determined by spatiotemporal alignment with the shooting data. The spatiotemporal features, the poplar species, and the second ecological features of poplar pests are then input into the poplar pest knowledge graph to obtain the poplar pest prediction results.

[0065] In this embodiment, the method for constructing a knowledge graph of poplar pests includes:

[0066] A structured knowledge base is constructed by acquiring knowledge about poplar pests from expert experience, literature, image annotation, and text databases. The knowledge about poplar pests includes the morphological characteristics of pests, the characteristics of pest growth stages, the characteristics of symbiotic organisms, the characteristics of poplar diseases, and the characteristics of the growth environment.

[0067] The teacher model is trained by using a structured knowledge base, a knowledge graph alignment loss function is used to establish the relationship between the features learned by the teacher model and the knowledge graph, and the output of the teacher model is migrated to the student model for knowledge distillation; the knowledge distillation includes soft label distillation, feature layer distillation, and knowledge graph embedding distillation; the soft label distillation uses the soft label output by the teacher model to guide the student model; the feature layer distillation constructs a mapping relationship between the intermediate feature maps of the teacher model and the intermediate feature maps of the student model; and the knowledge graph embedding distillation constructs a mapping relationship between the knowledge graph embedding vector of the teacher model and the feature map of the student model.

[0068] The knowledge graph alignment loss function expression is:

[0069]

[0070] Wherein is the knowledge graph alignment loss function, M is a domain knowledge weight matrix, and is related to the feature weight corresponding to the poplar pest domain knowledge, is the KL divergence, which is used to measure the difference between the probability distributions output by the teacher model and the student model, is the soft probability distribution of the teacher model, is the soft probability distribution of the student model, T is the original prediction vector output by the teacher model, S is the original prediction vector output by the student model, η1 is a temperature hyperparameter, F T is the intermediate layer feature map of the teacher model, F S is the corresponding layer feature map of the student model.

[0071] According to the poplar pest domain knowledge enhanced by knowledge distillation, a poplar pest knowledge graph is constructed.

[0072] Feature information of newly labeled poplar pest ecological pictures is extracted to randomly form multiple candidate triplets, the similarity of the corresponding candidate triplets and existing triplets in the knowledge graph is calculated, when the similarity is higher than a similarity threshold, the weight of the corresponding relationship is updated, and when there is no corresponding triplet, it is added as a new relationship.

[0073] A pre-trained relation extraction model is used to process new text to extract multiple candidate triplets, entity linking technology is used to link the candidate triplets to nodes in the knowledge graph, the knowledge graph where the node confidence is greater than a confidence threshold is updated, and the expression is:

[0074]

[0075] Wherein is the edge weight of node i to the neighbor node j at t+1, is the neighbor node set of node i, is the node confidence, φ(rkj ), φ(r mj ) is a relationship type weight, and λ is a self-enhancement coefficient;

[0076] In the actual evaluation, the poplar pest field knowledge includes pest morphological characteristics (appearance / texture / size / color / posture / motion), pest growth stage characteristics (morphological evolution law of egg / larva / pupa / adult), symbiotic organism characteristics (visual identification / symbiotic type / spatial co-location / dimension ratio / interaction frequency of specific parasites and pathogenic bacteria), poplar disease characteristics (disease site / disease texture / disease shape / leaf discoloration pattern / disease reaction), and growth environment characteristics (parasitic poplar varieties / geographical distribution / seasonal adaptability), and the corresponding characteristic weights of the poplar pest field knowledge are 0.35 / 0.25 / 0.15 / 0.15 / 0.10, and the poplar pest field knowledge is distilled;

[0077] The node types of the poplar pest knowledge graph include pests, growth stages, symbiotic organisms, disease manifestations, poplar varieties, geographical regions, and climate environments, and the edge relationships include pest-symbiotic organism edges, pest-poplar variety edges, pest-disease manifestation edges, pest-growth stage edges, pest-geographical region edges, pest-climate environment edges, disease manifestation-poplar variety edges, geographical region-climate environment edges, geographical region-poplar variety edges, and poplar variety-climate environment edges;

[0078] The geographical information, time information, growth stage information, symbiotic organism information, and disease manifestation information of the newly annotated poplar pest ecological pictures are extracted, a plurality of candidate triplets are randomly formed, the similarity threshold is 0.85, the similarity of the triplets is calculated, and the relationship is updated. The relationship weight is updated;

[0079] The pre-trained relationship extraction model architecture is BERT-EP+GRN+BiLSTM, the BERT-EP base model is BERT-base-Chinese, the GRN gate relationship network controls the input of text information through a relationship gate vector and a ReLU activation function, and the BiLSTM bidirectional LSTM decoding layer captures long-distance dependency relationships to output candidate triplets and probability distributions.

[0080] In the embodiment, the method for determining the poplar variety comprises:

[0081] The poplar image is subjected to color correction and distortion correction to obtain a corrected poplar image, the corrected poplar image is segmented into a poplar trunk image, a poplar leaf image, and a poplar branch image according to parts, and image features of the poplar trunk image, the poplar leaf image, and the poplar branch image are extracted to obtain trunk features, leaf features, and branch features, respectively;

[0082] The cosine similarity of the tree trunk feature, the tree leaf feature and the tree branch feature with the corresponding part feature of the same variety poplar in the standard poplar library is obtained to obtain a part similarity, a part seasonal weight is determined according to a recognition season, and a variety similarity is calculated by weighting the part similarity and the part seasonal weight; the part seasonal weight is a fixed value set in advance, and is set according to the growth rule of poplar leaves;

[0083] The standard poplar variety corresponding to the highest variety similarity value is taken as the poplar variety to be identified;

[0084] In actual evaluation, the Retinex theory is used for color correction, and the improved Brown-Conrady model is used for distortion correction to obtain a corrected poplar image;

[0085] The Mask R-CNN is used to divide the corrected poplar image to obtain a poplar trunk image, and the Hu moment invariant feature, the LBP-TOP and the minimum circumscribed rectangle are used to extract the shape, the texture and the size of the poplar trunk image to form the trunk feature;

[0086] The SLIC superpixel clustering is used to divide the corrected poplar image to obtain a poplar leaf image, and the HSV histogram, the GLCM co-occurrence matrix, the Zernike matrix and the leaf count / area are used to extract the color, the texture, the shape and the density of the poplar leaf image to form the leaf feature;

[0087] The U-Net is used to divide the corrected poplar image to obtain a poplar branch image, and the skeleton fractal dimension and the branch diameter distribution histogram are used to extract the shape and the size of the poplar branch image to form the branch feature;

[0088] The fixed value of the part seasonal weight is shown in Table 1:

[0089] Table 1 Reference table of part seasonal weight

[0090] Season Trunk weight Leaf weight Branch weight Adjustment basis Spring 0.2 0.6 0.2 New leaf emergence period Summer 0.3 0.5 0.2 Leafy canopy period Autumn 0.4 0.4 0.2 Leaf color change period Winter 0.5 0.1 0.4 Leaf fall dormancy period

[0091] Taking the variety determination of the poplar variety to be identified as an example, the part similarities of the identified poplar variety with three candidate poplar varieties are determined to be European poplar (tree trunk similarity 0.92 / tree leaf similarity 0.88 / tree branch similarity 0.85), white poplar (tree trunk similarity 0.78 / tree leaf similarity 0.91 / tree branch similarity 0.82) and Xinjiang poplar (tree trunk similarity 0.85 / tree leaf similarity 0.76 / tree branch similarity 0.88), the part seasonal weights are determined to be 0.4, 0.4 and 0.2 respectively according to the recognition season (autumn), and the variety similarities are calculated by weighting to be 0.89, 0.84 and 0.81 respectively, and the poplar variety to be identified is determined to be European poplar 107.

[0092] In the embodiment, the method for obtaining the ecological characteristics of the first poplar pest comprises the following steps:

[0093] The poplar pest ecological deep recognition model is constructed by using the annotated pest image, the symbiotic organism image and the poplar disease image; the poplar pest ecological deep recognition model comprises an input layer, a biological layer, a plant layer, an enhancement layer, a full connection layer and an output layer; the biological layer and the plant layer are designed in parallel;

[0094] The biological layer adopts ResNet to extract the pest image features and the symbiotic organism image features, and processes the image features through the biological attention mechanism of the biological channel to obtain initial pest extraction features and initial symbiotic organism extraction features, and the expression is as follows:

[0095]

[0096] Wherein, BA(F i ) is the attention of the feature F i , F i includes the feature tensor F v of the pest image and the feature tensor F s of the symbiotic organism image, is a nonlinear mapping about the biological feature F i , and W g is a learnable biological feature convolution kernel;

[0097] The plant layer adopts UNet to extract the multi-scale disease features of the poplar disease image, and processes the multi-scale disease features through the plant attention mechanism of the plant channel to obtain poplar disease extraction features, and the expression is as follows:

[0098]

[0099] Wherein, PA(F p ) is the attention of the multi-scale disease feature F p , is a multi-scale disease perception branch of the feature F p , and multi-scale α i includes average pooling, maximum pooling, hole convolution and Gabor filtering, is an α i scale channel convolution kernel;

[0100] The enhancement layer is embedded with a pest prototype memory bank, and the initial pest extraction features and the initial symbiotic organism extraction features are processed through a prototype contrast loss function to enlarge the distance between similar pests to obtain the pest extraction features and the symbiotic organism extraction features; the pest prototype memory bank stores cross-scale prototype vectors of each type of pest, including microscopic prototype vectors, mesoscopic prototype vectors and macroscopic prototype vectors; and the prototype contrast loss function expression is as follows:

[0101]

[0102] wherein is the prototype contrast loss function, sim s is the feature similarity score at the current scale s, sim s is the feature similarity score at the current scale s, η2 is the temperature parameter, f s is the feature representation of the current sample at scale s, is the prototype vector of the same class y at scale s, is the prototype vector of the different class y at scale s, m is the margin;

[0103] The fully connected layer is connected with the plant layer and the enhancement layer respectively, and the feature fusion and compression are performed on the poplar disease extraction feature, the pest extraction feature and the symbiotic organism extraction feature, and the main pest feature, the symbiotic organism feature and the poplar disease feature are decoded and output.

[0104] The to-be-recognized poplar pest ecological picture is input into the poplar pest ecological deep recognition model to obtain the first poplar pest ecological feature.

[0105] In the actual evaluation, in the enhancement layer, the micro prototype vector (texture / line) is extracted by the local binary pattern+HOG, the meso prototype vector (shape / color) is extracted by the ResNet34 block 3 feature, the macro prototype vector (number scale / space distribution) is extracted by the VisionTransformer global feature, and a pest prototype memory bank is formed, the temperature parameter η2=2, and the margin m=0.8.

[0106] In the fully connected layer, the poplar disease extraction feature is decoded by the disease feature branch to obtain the poplar disease feature (disease site / texture roughness / color abnormal value / disease spot shape / diffusion speed), the pest extraction feature is decoded by the pest feature branch to obtain the main pest feature (shape size / color histogram / line direction / texture entropy / number scale / space density), and the symbiotic organism extraction feature is decoded by the symbiotic organism feature branch to obtain the symbiotic organism feature (symbiotic type / parasitic intensity / space co-position degree / size ratio / motion trend).

[0107] The poplar pest ecological deep recognition model adjusts the recognition accuracy through a hybrid loss function composed of cross-entropy loss, prototype contrast loss and auxiliary segmentation loss.

[0108] In the embodiment, the method for obtaining the second poplar pest ecological feature comprises:

[0109] The near-infrared pest image, the ultraviolet pest image and the visible light pest image are acquired for spatio-temporal alignment, and the spatio-temporal aligned near-infrared pest image, the spatio-temporal aligned ultraviolet pest image and the spatio-temporal aligned visible light pest image are subjected to band difference saliency processing to obtain a band difference saliency map, and the expression is:

[0110]

[0111] wherein the BDSM is the band difference saliency map, the α is an enhancement coefficient, the NIR is the near-infrared pest image, the UV is the ultraviolet pest image, the RGB is the visible light pest image, the ∈ is a numerical stability constant, the I is an original pest image, the ω k is a k-stage larva characteristic weight parameter, the μ k is a k-stage larva characteristic Gaussian center, the σ k is a k-stage larva characteristic Gaussian standard deviation;

[0112] The secondary pest characteristics are extracted from the band difference saliency map to obtain secondary pest characteristics, the secondary pest characteristics and the main pest characteristics in the first poplar pest ecological characteristics are subjected to feature cross to obtain pest characteristics, and the second poplar pest ecological characteristics are composed of the pest characteristics, the symbiotic organism characteristics and the poplar disease characteristics; the secondary pest characteristics include growth stages, body segment numbers, body color transmittance and motion trajectory entropy;

[0113] In actual evaluation, the enhancement coefficient α is taken as 2.5, the numerical stability constant ∈ is taken as 10 -7 , the band difference saliency is processed to obtain the band difference saliency map, the morphological opening operation + area analysis is used to process the band difference saliency map to obtain the growth stages, the ridge line detection + local maximum value is used to process the band difference saliency map to obtain the body segment number, the NIR channel mean value is used to process the band difference saliency map to obtain the body color transmittance, the time sequence frame difference information entropy is used to process the band difference saliency map to obtain the motion trajectory entropy, and the standard reference table 2 of each growth stage is determined:

[0114] Table 2: Reference table of pest growth stage

[0115]

[0116] The specific method for cross-features of the secondary pest characteristics and the main pest characteristics in the first poplar pest ecological characteristics to obtain the pest characteristics is that the same characteristics (growth stage, body color, texture, motion characteristic) of the secondary pest characteristics and the main pest characteristics are subjected to weighted fusion, and the different characteristics of the secondary pest characteristics and the main pest characteristics are subjected to feature splicing, wherein the weighted weights of the main pest characteristics and the secondary pest characteristics are β1 and β2 respectively, and β1 + β2 = 1, γ = 1.5 is an adjustment factor, and conf secondary is a secondary feature confidence;

[0117] The feature intersection is performed to obtain pest features, including basic features (shape size / color), texture features (line / texture), behavior features (number scale / density), development features (growth stage / body segment), physiological features (body color / transmittance), and motion features (trajectory entropy), and the second poplar pest ecological features are composed of the pest features, symbiotic organism features, and poplar disease features.

[0118] In the embodiment, the method for obtaining the poplar pest prediction result comprises:

[0119] The spatio-temporal alignment is performed on the poplar pest ecological picture and the shooting data to obtain a shooting time and a shooting area, the climate environment is determined according to the shooting time and the shooting area, and the spatio-temporal features are composed of the climate environment, the shooting time, and the shooting area;

[0120] The spatio-temporal features, the poplar variety, and the second poplar pest ecological features are input into the poplar pest knowledge graph to obtain the poplar pest prediction result; the poplar pest prediction result comprises a pest variety, a pest growth stage proportion, and a prediction probability;

[0121] In the actual evaluation, taking the poplar pest image recognition in a certain place as an example, the EXIF metadata of the poplar pest ecological picture is extracted, the shooting season (July 15, summer) is determined based on the astronomical algorithm, the GPS coordinates of the poplar pest ecological picture are extracted, the ecological area (North China Plain deciduous forest area) is determined according to the H3 geographical grid ecological zoning, the climate is searched according to the shooting season and the ecological area, the spatio-temporal features are composed of the climate search result (daily mean temperature 28.5℃, relative humidity 65%, rainfall 15.2mm, sunshine duration 8.2h), the spatio-temporal features, the corresponding poplar variety (Populus tremula 107), and the second poplar pest ecological features are input into the poplar pest knowledge graph to obtain the poplar pest prediction candidate (Anoplophora chinensis / 0.92, Apriona germari / 0.41, and Holcus pectineus / 0.63), the prediction candidate with the highest prediction probability (Anoplophora chinensis) is taken as the prediction result, and the corresponding pest growth stage proportion and prediction probability are output, and the pest control means is determined according to the poplar pest prediction result, the poplar variety, the spatio-temporal features, and the second poplar pest ecological features.

[0122] The spatio-temporal features and the corresponding poplar variety (Populus tremula 107), the second poplar pest ecological features are input into the poplar pest knowledge graph to obtain the poplar pest prediction candidate (Anoplophora chinensis / 0.92, Apriona germari / 0.41, and Holcus pectineus / 0.63), the prediction candidate with the highest prediction probability (Anoplophora chinensis) is taken as the prediction result, and the corresponding pest growth stage proportion and prediction probability are output, and the pest control means is determined according to the poplar pest prediction result, the poplar variety, the spatio-temporal features, and the second poplar pest ecological features.

[0123] In a second aspect, a poplar pest image recognition system based on deep learning comprises:

[0124] The knowledge graph module is used to obtain the poplar pest field knowledge for knowledge distillation enhancement, and construct a poplar pest knowledge graph; and the spatio-temporal features, the poplar variety, and the second poplar pest ecological features are input into the poplar pest knowledge graph to obtain the poplar pest prediction result.

[0125] A tree species identification module is configured to compare a poplar image with a standard poplar image library to determine a poplar variety;

[0126] A depth identification module is configured to obtain labeled poplar pest ecological pictures to construct a poplar pest ecological depth identification model, and input a to-be-identified poplar pest ecological picture into the poplar pest ecological depth identification model to obtain first poplar pest ecological features;

[0127] A feature deepening module is configured to process multi-band pest images by using a band difference saliency map algorithm to obtain secondary pest features, cross the secondary pest features with the first poplar pest ecological features to obtain second poplar pest ecological features, and perform spatio-temporal alignment on the to-be-identified poplar pest ecological picture and shooting data to determine spatio-temporal features.

[0128] The above description is merely preferred embodiments of the present application, and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A deep learning-based image recognition method for poplar pests, characterized in that, Includes the following steps: S1. Acquire knowledge in the field of poplar pests, perform knowledge distillation and enhancement, and construct a knowledge graph of poplar pests; S2. Obtain poplar images and compare them with a standard poplar image library to determine the poplar species. Obtain labeled poplar pest ecological images to construct a deep ecological identification model for poplar pests. The poplar pest ecological images include pest images, symbiotic organism images, and poplar disease images. S3. Input the ecological image of poplar pests to be identified into the deep ecological identification model of poplar pests to obtain the first ecological characteristics of poplar pests; the first ecological characteristics of poplar pests include the characteristics of major pests, the characteristics of symbiotic organisms and the characteristics of poplar diseases; S4. Use the band difference significance map algorithm to process the multi-band pest map to obtain secondary pest features, and perform feature cross-interaction between the secondary pest features and the first poplar pest ecological features to obtain the second poplar pest ecological features. S5. The ecological images of poplar pests to be identified are spatiotemporally aligned with the shooting data to determine the spatiotemporal features. The spatiotemporal features, the poplar species, and the second ecological features of poplar pests are input into the poplar pest knowledge graph to obtain the poplar pest prediction results.

2. The method for recognizing poplar pest images based on deep learning according to claim 1, characterized in that, The method for constructing a knowledge graph of poplar pests includes: A structured knowledge base is constructed by acquiring knowledge about poplar pests from expert experience, literature, image annotation, and text databases. The knowledge about poplar pests includes the morphological characteristics of pests, the characteristics of pest growth stages, the characteristics of symbiotic organisms, the characteristics of poplar diseases, and the characteristics of the growth environment. A structured knowledge base is used to train a teacher model. A knowledge graph alignment loss function is used to establish the relationship between the features learned by the teacher model and the knowledge graph. The output of the teacher model is then transferred to the student model for knowledge distillation. The knowledge distillation includes soft label distillation, feature layer distillation, and knowledge graph embedding distillation. The soft label distillation uses the soft labels output by the teacher model to guide the student model. The feature layer distillation constructs a mapping relationship between the intermediate feature maps of the teacher model and the intermediate feature maps of the student model. The knowledge graph embedding distillation constructs a mapping relationship between the knowledge graph embedding vectors of the teacher model and the feature maps of the student model. The expression for the knowledge graph alignment loss function is as follows: in Here, M represents the knowledge graph alignment loss function, and M is the domain knowledge weight matrix, which is related to the feature weights corresponding to the domain knowledge of poplar pests. KL divergence is used to measure the difference in probability distributions between the teacher model and the student model. The softened probability distribution for the teacher model. Let T be the softening probability distribution of the student model, T be the original prediction vector output by the teacher model, S be the original prediction vector output by the student model, η1 be the temperature hyperparameter, and F be the temperature hyperparameter. T F represents the intermediate layer feature map of the teacher model. S The feature map of the corresponding layer in the student model; A knowledge graph of poplar pests is constructed based on knowledge distillation-enhanced domain knowledge of poplar pests. Feature information of newly labeled poplar pest ecological images is extracted and randomly formed into multiple candidate triples. The similarity between the corresponding candidate triples and the existing triples in the knowledge graph is calculated. When the similarity is higher than the similarity threshold, the weight of the corresponding relation is updated. When there is no corresponding triple, it is added as a new relation. A pre-trained relation extraction model is used to process new text and extract multiple candidate triples. Entity linking technology is used to link the candidate triples to nodes in the knowledge graph. The knowledge graph is updated for nodes with confidence scores greater than a confidence threshold. The weights of neighboring edges are adjusted based on the node confidence scores. The expression is as follows: in Let be the edge weight from node i to neighboring node j at time t+1. Let be the set of neighboring nodes of node i. For node confidence, φ(r) kj ), φ(r) mj ) represents the relation type weight, and λ is the self-enhancement coefficient.

3. The method for recognizing poplar pest images based on deep learning according to claim 1, characterized in that, The method for determining poplar varieties includes: Color correction and distortion correction are performed on poplar images to obtain corrected poplar images. The corrected poplar images are then segmented into poplar trunk images, poplar leaf images, and poplar branch images according to their parts. Image features of the poplar trunk images, poplar leaf images, and poplar branch images are extracted respectively to obtain trunk features, leaf features, and branch features. The cosine similarity between the trunk features, leaf features, and branch features and the corresponding features of the same poplar species in the standard poplar image library is used to obtain the part similarity. The seasonal weight of the part is determined according to the identification season. The species similarity is calculated by weighting the part similarity and the seasonal weight of the part. The seasonal weight of the part is a fixed value set in advance and is set according to the growth pattern of poplar leaves. The standard poplar variety with the highest similarity value is selected as the poplar variety to be identified.

4. The poplar pest image recognition method based on deep learning according to claim 1, characterized in that, The method for obtaining the ecological characteristics of the first poplar pest includes: A deep ecological identification model for poplar pests was constructed using labeled pest images, symbiotic organism images, and poplar disease images. The deep ecological identification model for poplar pests includes an input layer, a biological layer, a plant layer, an enhancement layer, a fully connected layer, and an output layer. The biological layer and the plant layer are designed in parallel. The biological layer uses ResNet to extract pest and symbiotic image features. The initial pest and symbiotic image features are obtained by processing the image features through the biological attention mechanism of the biological channel. The expression is: Among them, BA(F) i ) is a feature F i attention, F i Including the feature tensor F of the pest image v The feature tensor F of the symbiotic image s , For biometrics F i nonlinear mapping, W g Learnable biometric convolution kernels; The plant layer uses UNet to extract multi-scale disease features from poplar disease images. The plant attention mechanism in the plant channel is then used to process these multi-scale disease features to obtain the extracted poplar disease features. The expression is as follows: Where PA(F) p ) represents multi-scale disease characteristics F p attention, For feature F p Multiscale disease perception branch, multiscale α i This includes average pooling, max pooling, dilated convolution, and Gabor filtering. For α i Scale-channel convolution kernel; The enhancement layer embeds a pest prototype memory library. It processes the initial pest extraction features and initial symbiotic extraction features using a prototype contrast loss function to increase the distance between approximate pest classes, thereby obtaining the pest extraction features and symbiotic extraction features. The pest prototype memory library stores cross-scale prototype vectors for each pest class, including microscopic, mesoscopic, and macroscopic prototype vectors. The prototype contrast loss function expression is as follows: in For the prototype contrastive loss function, sim s For the feature similarity score at the current scale s, sim s' The feature similarity score for traversing scale s', where η2 is the temperature parameter, and f s This represents the feature representation of the current sample at scale s. Let y be the prototype vector of the same category at scale s. Let be the prototype vectors of different categories y at scale s, and m be the boundary margin; The fully connected layer connects the plant layer and the reinforcement layer respectively, performs feature fusion and compression on the extracted features of poplar diseases, pests and symbiotic organisms, and performs fully connected feature decoding to output the main pest features, symbiotic organism features and poplar disease features; The ecological images of poplar pests to be identified are input into the poplar pest ecological depth recognition model to obtain the first ecological features of poplar pests.

5. The method for recognizing poplar pest images based on deep learning according to claim 1, characterized in that, The method for obtaining the ecological characteristics of the second poplar pest includes: Near-infrared, ultraviolet, and visible light images of pests are obtained and spatiotemporally aligned. The aligned images are then subjected to band difference significance processing to obtain a band difference significance map, expressed as: Where BDSM is the band difference significance map, α is the enhancement coefficient, NIR is the near-infrared pest map, UV is the ultraviolet pest map, RGB is the visible light pest map, ∈ is the numerical stability constant, I is the original pest map, and ω k μ is the feature weight parameter for the k-th stage larva. k For the characteristic Gaussian center of the k-th stage larva, σ k The standard deviation of the Gaussian characteristics of the k-th stage larvae; Feature extraction is performed on the band difference significance map to obtain secondary pest features. The secondary pest features are then cross-referenced with the primary pest features in the first poplar pest ecological features to obtain pest features. The second poplar pest ecological features are composed of pest features, symbiotic organism features, and poplar disease features. The secondary pest features include growth stage, number of body segments, body color transmittance, and movement trajectory entropy.

6. The method for recognizing poplar pest images based on deep learning according to claim 1, characterized in that, The method for obtaining poplar pest prediction results includes: By aligning the ecological images of poplar pests with the shooting data in time and space, the shooting time and shooting area are obtained. The climate environment is determined based on the shooting time and shooting area. The spatiotemporal characteristics are composed of the climate environment, shooting time and shooting area. The spatiotemporal characteristics, poplar species, and ecological characteristics of second poplar pests are input into the poplar pest knowledge graph to obtain poplar pest prediction results; the poplar pest prediction results include pest species, pest growth stage ratio, and prediction probability. Taking the image recognition of poplar pests in a certain location as an example, the EXIF ​​metadata of the poplar pest ecological image is extracted and the shooting season (July 15, summer) is determined based on the astronomical algorithm. The GPS coordinates of the poplar pest ecological image are extracted and the ecological region (North China Plain deciduous forest area) is determined according to the H3 geographic grid ecological zoning. Climate is retrieved according to the shooting season and ecological region. Based on the climate retrieval results (average daily temperature 28.5℃, relative humidity 65%, rainfall 15.2mm, sunshine duration 8.2h), the spatiotemporal features are composed of the shooting season, ecological region and climate retrieval results. The spatiotemporal characteristics and corresponding poplar varieties (European and American poplar 107), along with the ecological characteristics of second poplar pests, are input into the poplar pest knowledge graph to obtain poplar pest prediction candidates: Asian longhorn beetle / 0.92, mulberry longhorn beetle / 0.41, and poplar stem weevil / 0.

63. The Asian longhorn beetle with the highest prediction probability is selected as the prediction result, and the corresponding pest growth stage ratio and prediction probability are output. Based on the poplar pest prediction results, poplar varieties, spatiotemporal characteristics, and second poplar pest ecological characteristics, pest control methods are determined.

7. A deep learning-based poplar pest image recognition system, used to perform the method according to any one of claims 1-6, characterized in that, include: Knowledge Graph Module: Used to acquire knowledge in the field of poplar pests, perform knowledge distillation and enhancement, and construct a knowledge graph of poplar pests; This is used to input the spatiotemporal features, the poplar species, and the second poplar pest ecological features into the poplar pest knowledge graph to obtain poplar pest prediction results; Tree species identification module: used to acquire poplar images and compare them with a standard poplar image library to determine the poplar species; The deep recognition module is used to acquire labeled ecological images of poplar pests and construct a deep ecological recognition model for poplar pests; it is also used to input the ecological images of poplar pests to be identified into the deep ecological recognition model for poplar pests to obtain the first ecological features of poplar pests. Feature enhancement module: used to process multi-band pest images using the band difference significance map algorithm to obtain secondary pest features, and to perform feature cross-referencing between the secondary pest features and the first poplar pest ecological features to obtain the second poplar pest ecological features; used to perform spatiotemporal alignment between the poplar pest ecological image to be identified and the shooting data to determine spatiotemporal features.

Images