A method for identifying pests

Abstract

The present invention discloses a pest identification method, comprising: obtaining an image of a pest to be identified; inputting the image of the pest to be identified into a trained pest identification model; extracting image features of the image of the pest to be identified through a feature extraction layer; using the image features of the image of the pest to be identified as a head entity of a pest knowledge fact; if the head entity of the pest knowledge fact is a head entity in a pre-constructed pest knowledge graph, constructing the head entity of the pest knowledge fact and its dependency relationship as a query vector, and querying a tail entity in the pre-constructed pest knowledge graph as a recognition result; otherwise, using the head entity of the pest knowledge fact and its corresponding dependency relationship as input features of the pest identification model, and outputting a recognition result of the image of the pest to be identified through forward propagation; when processing a new category of pests, the present invention can use the relationship in the knowledge graph to infer the tail entity to reduce the misjudgment rate.

Description

Technical Field

[0001] The invention relates to the field of pest identification, in particular to a pest identification method. Background Art

[0002] In traditional pest recognition models, the training data set mainly relies on manually annotated pest image samples. The usual process is to collect a large number of pest image samples, and then have domain experts manually annotate these samples to form a training set. During the model training process, image recognition algorithms such as convolutional neural networks (CNNs) are used to extract and classify images. Based on this training method, the recognition model can gradually learn the external features of different pests, thereby classifying the input images and determining the types of pests in the images. In addition, some existing pest recognition technologies have also begun to try to introduce prior knowledge, such as using knowledge bases or simple rule systems to introduce expert experience into the model to improve the model's recognition ability. However, the knowledge system of this type of method is relatively static and cannot adapt to the complex and changeable pest image features. It is also difficult to dynamically optimize its reasoning ability through model learning.

[0003] The Chinese patent document with patent publication number CN113673340B discloses a pest species image recognition method and system, which introduces a spatial attention module and a channel attention network to more accurately locate pest images and interact across channels, and makes the pest recognition model pay more attention to effective channels, thereby reducing the impact of image background on pest classification accuracy, improving classification accuracy, and achieving accurate identification of pest species, making contributions to the field of agricultural plant protection;

[0004] However, the existing technology requires a lot of manual participation in the annotation of large-scale pest image datasets, which is time-consuming and error-prone, seriously affecting the training efficiency and recognition effect of the model; traditional recognition models are too dependent on image features and cannot fully utilize the prior knowledge in the pest knowledge graph, resulting in poor performance of the model when processing complex images and semantic information. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention provides a pest identification method, which gradually generates clean prediction labels by automatically annotating and generating a first pseudo-random prediction label and a second pseudo-random prediction label through multiple iterative optimization, thereby solving the technical problems raised in the background technology.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0007] A pest identification method is based on a visual and knowledge multimodal large model algorithm, wherein the algorithm is used in a pest identification model to identify pests; the identification method comprises:

[0008] Acquire an image of pests to be identified;

[0009] Inputting the pest image to be identified into the trained pest identification model;

[0010] Extracting image features of the pest image to be identified through a feature extraction layer;

[0011] The image features of the pest image to be identified are used as the head entity of the pest knowledge fact;

[0012] If the head entity of the pest knowledge fact exists in the pre-constructed pest knowledge graph, the head entity and its dependencies are constructed as a query vector, and the tail entity is queried in the pre-constructed pest knowledge graph as the recognition result;

[0013] Otherwise, the head entity and its dependencies are used as input features of the pest recognition model, and the recognition results of the pest images to be identified are output through forward propagation.

[0014] In some embodiments, the training process of the pest identification model includes:

[0015] S1. Obtain a pest image training set, where the training samples of the pest image training set are pest image samples with clean prediction labels;

[0016] S2. training a knowledge graph model using a pest image training set; wherein the knowledge graph model is embedded with a pre-built pest knowledge graph;

[0017] S3: If the training reaches the convergence condition, stop the training and export the knowledge graph model with the current iteration model parameters as the pest recognition model.

[0018] In some embodiments, obtaining a pest image training set includes:

[0019] S1-1. Obtain several unlabeled pest image samples and construct them into an initial image set;

[0020] S1-2. Receive pest image samples of the initial image set as input features of the large model, and output a first pseudo-random prediction label;

[0021] The expression for outputting the first pseudo-random prediction label is:

[0022] represents the first pseudo-random predicted label of the qth pest image sample in the initial image set, represents the activation function, represents the weight matrix of the large model, represents the image features of the qth pest image sample in the initial image set, represents the bias term of the large model;

[0023] S1-3. Constructing the pest image samples with the first pseudo-random predicted labels into a pseudo-random annotation set;

[0024] S1-4. Self-train a small model through a pseudo-random annotation set to obtain clean prediction labels;

[0025] S1-5. Select pest image samples corresponding to clean prediction labels and construct a pest image training set.

[0026] In some embodiments, a small model is self-trained by a pseudo-random annotation set, including:

[0027] S1-4-1. Receive a pest image sample of a pseudo-random annotation set as an input feature of the small model, and output a second pseudo-random prediction label;

[0028] The expression for outputting the second pseudo-random prediction label is:

[0029] represents the second pseudo-random predicted label of the qth pest image sample in the pseudo-random annotation set, represents the weight matrix of the small model, represents the image features of the qth pest image sample in the pseudo-random annotation set, represents the bias term of the small model;

[0030] S1-4-2. Encode the first pseudo-random predicted label to generate a first pseudo-random true label;

[0031] S1-4-3. Calculate the classification loss of the second pseudo-random predicted label and the first pseudo-random true label;

[0032] The expression of classification loss is:

[0033] represents the classification loss, n represents the number of pest image samples, k represents the total number of category labels, represents the first pseudo-random true label of the i-th sample, representing the predicted probability that sample i belongs to category label j, Represents the second pseudo-random prediction label of the i-th sample, representing the predicted probability that the sample i output by the small model belongs to the category label j. Represents the logarithm of the second pseudo-random predicted label of the i-th sample, which is used to calculate the cross entropy loss;

[0034] S1-4-4. According to the preset small loss strategy, determine whether the classification loss is lower than the preset loss value;

[0035] The expression for determining whether the classification loss is lower than the preset loss value is:

[0036] Represents the preset loss value, which indicates the maximum allowed classification loss;

[0037] S1-4-5. Select a second pseudo-random prediction label corresponding to a classification loss lower than a preset loss value and determine it as a clean prediction label.

[0038] In some embodiments, training a knowledge graph model using a pest image training set includes:

[0039] S2-1. Define the head entity of the pest knowledge graph as the input feature of the knowledge graph model. Define the tail entity as the output label of the knowledge graph model. Define the relationship as the dependency between the head entity and the tail entity in the knowledge graph model to construct the real pest knowledge facts;

[0040] The expression for constructing the real pest knowledge fact is:

[0041] T represents the real pest knowledge fact, representing a relation entity in the pest knowledge graph. Represents the head entity, Indicates dependency, representing the association between the head entity and the tail entity. Represents the tail entity, representing the object connected to the head entity through a dependency relationship;

[0042] S2-2. Receive pest image samples in the pest image training set as input features of the knowledge graph model, and obtain feature vectors of the pest image samples by image feature extraction;

[0043] S2-3. Given the feature vector of the pest image sample as the head entity of the pest knowledge fact, based on the head entity and the dependency relationship, output the predicted tail entity to obtain the predicted pest knowledge fact;

[0044] The expression of the predicted pest knowledge fact is:

[0045]

[0046] represents the predicted pest knowledge facts, A feature vector representing a given pest image sample; represents the tail entity predicted by the knowledge graph model, Represents the weight matrix of the knowledge graph model, It is the bias term of the knowledge graph model;

[0047] S2-4. Calculate the matching degree between the predicted pest knowledge facts and the real pest knowledge facts;

[0048] The expression for calculating the matching degree between the predicted pest knowledge facts and the real pest knowledge facts is:

[0049]

[0050] represents the matching degree between the predicted pest knowledge facts and the real pest knowledge facts, The norm between the feature vector and the tail entity vector representing a given pest image sample;

[0051] S2-5. Calculate the pest knowledge fact loss based on the matching degree;

[0052] The expression for calculating the pest knowledge fact loss is:

[0053]

[0054] Indicates the factual loss of pest knowledge, Represents negative sample pest knowledge facts The matching degree between the real pest knowledge facts, represents the prediction hyperparameter, represents a non-negative sign;

[0055] S2-6. Perform back propagation and gradually adjust the model parameters to minimize the pest knowledge fact loss.

[0056] In some of these embodiments, performing back propagation includes:

[0057] S2-6-1, obtaining the pest knowledge fact loss of the current iteration, and calculating the gradient of the pest knowledge fact loss of the current iteration relative to the model parameters;

[0058] S2-6-2, according to the gradient of the model parameters, back propagate it to obtain the updated model parameters

[0059] S2-6-3, according to the updated model parameters, perform the forward propagation of the next iteration and output the predicted tail entity.

[0060] In some of the embodiments, the convergence condition includes: the pest knowledge fact loss reaches a preset loss threshold or the training rounds reach a preset maximum number of training rounds.

[0061] In some of the embodiments, searching for a tail entity in a pre-built pest knowledge graph as a recognition result according to a query vector includes:

[0062] Obtain pest knowledge facts (h, r, x), where h is the acquired head entity and r is the dependency relationship between the acquired head entity and the tail entity x to be identified;

[0063] In the pre-built pest knowledge graph, query whether there is an acquired head entity h;

[0064] If it exists, construct the obtained head entity h and dependency relationship r into the query vector q;

[0065] The query vector is used to query the pre-built pest knowledge graph, and the determined tail entity is selected as the recognition result of the query based on the head entity and dependency relationship.

[0066] The present invention provides a pest identification method, which realizes automatic label generation through multiple iterative optimization of the first pseudo-random prediction label and the second pseudo-random prediction label generated by automatic annotation, reduces manual intervention and time cost, and greatly improves the efficiency of building a high-quality training set; in the case of insufficient labels or incomplete annotations, it can rely on the background information provided by the knowledge graph to enhance the reasoning ability of the model, thereby improving the accuracy of pest identification. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 It is a schematic diagram of the identification method flow of the present invention;

[0068] Figure 2 A flowchart for constructing a pest image training set of the present invention;

[0069] Figure 3 A schematic diagram of the process of obtaining clean prediction labels of the present invention;

[0070] Figure 4 A schematic diagram of the training process of the pest identification model of the present invention; DETAILED DESCRIPTION

[0071] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0072] Example 1: Please refer to Figure 1 The present invention provides a pest identification method, based on a visual and knowledge multimodal large model algorithm, wherein the algorithm is used for a pest identification model to identify pests; the identification method comprises:

[0073] Acquire an image of pests to be identified;

[0074] Inputting the pest image to be identified into the trained pest identification model;

[0075] Extracting image features of the pest image to be identified through a feature extraction layer;

[0076] The image features of the pest image to be identified are used as the head entity of the pest knowledge fact;

[0077] If the head entity of the pest knowledge fact exists in the pre-constructed pest knowledge graph, the head entity and its dependencies are constructed as a query vector, and the tail entity is queried in the pre-constructed pest knowledge graph as the recognition result;

[0078] Otherwise, the head entity and its dependencies are used as input features of the pest recognition model, and the recognition results of the pest images to be identified are output through forward propagation.

[0079] The triples refer to the knowledge facts in the pest knowledge graph, which include a head entity, a tail entity, and a dependency relationship between the head entity and the tail entity that represents a certain association between them.

[0080] The pest identification method of this embodiment, by combining image features with a pre-built pest knowledge graph, can utilize dependencies in the knowledge graph (such as "lives in") to assist the pest identification process, which helps to more accurately determine the pest species; for the head entity existing in the pest knowledge graph, the associated tail entity can be quickly obtained directly through the query vector, thereby speeding up the recognition speed and providing clearer results, and through the additional information provided by the knowledge graph, the recognition result of the pest recognition model reduces to a certain extent the possible misidentification that may occur when the recognition is based solely on image features.

[0081] Example 2: See Figures 2 to 4 The technical solution of this embodiment 2 is different from that of embodiment 1 in that the training process of the pest identification model in embodiment 1 is disclosed, and the training process includes:

[0082] S1. Obtain a pest image training set, where the training samples of the pest image training set are pest image samples with clean prediction labels;

[0083] The difference between this embodiment and the traditional technology is that the target labels of the traditional pest image training set are all formed by manual annotation, which is time-consuming and labor-intensive, while the target labels of this embodiment are clean prediction labels output by the model, replacing manual annotation.

[0084] S2. training a knowledge graph model using a pest image training set; wherein the knowledge graph model is embedded with a pre-built pest knowledge graph;

[0085] Specifically, the knowledge graph model of this embodiment constructs the expert prior knowledge in the field of pest identification into a pest identification knowledge graph, and embeds the pest identification knowledge graph into a large model through triple knowledge facts to serve as a training model for the pest image training set, namely, a knowledge graph model.

[0086] S3: If the training reaches the convergence condition, stop the training and export the knowledge graph model with the current iteration model parameters as the pest recognition model.

[0087] Flow S1 is a process of how to perform automatic standardization in this embodiment, including:

[0088] S1-1. Obtain several unlabeled pest image samples and construct them into an initial image set;

[0089] Specifically, a dataset containing multiple pest images is obtained from actual pest image acquisition equipment, pest research laboratories or field collection. The pest image samples in the dataset are constructed into an initial image set according to certain rules (such as file format, acquisition date, etc.). In other words, the initial image set is used as the input feature of the actual dataset, not the input feature of the training set. Therefore, the pest image samples do not contain any manual annotations.

[0090] S1-2. Receive pest image samples of the initial image set as input features of the large model, and output a first pseudo-random prediction label;

[0091] The expression for outputting the first pseudo-random prediction label is:

[0092] represents the first pseudo-random predicted label of the qth pest image sample in the initial image set, represents the activation function, represents the weight matrix of the large model, represents the image features of the qth pest image sample in the initial image set, represents the bias term of the large model;

[0093] The image features of the pest image samples in the input initial image set are processed by the weight matrix and bias term of the large model and then by the activation function to obtain the first pseudo-random prediction label.

[0094] Specifically, an initial set of images without any manual annotations is input into the large model, and a preliminary predicted first pseudo-random prediction label is output. Because the large model performs preliminary reasoning and annotation based on a large amount of data from a wide range of fields, the large model has a limited understanding of pest image samples that do not contain any manual annotations, which results in the output first pseudo-random prediction label being random to a certain extent, the so-called "pseudo-random", which may contain erroneous or inaccurate annotations. But from another perspective, although the first pseudo-random prediction label is random to a certain extent, because the large model inputs pest image samples, the large model will still use the input features of the pest image samples during the reasoning process. This means that the first pseudo-random prediction labels are not completely random, and these first pseudo-random prediction labels reflect certain characteristics of the pest images to a certain extent.

[0095] S1-3. Constructing the pest image samples with the first pseudo-random predicted labels into a pseudo-random annotation set;

[0096] That is, a data set with the first pseudo-random label as the input feature (non-training sample) is obtained;

[0097] S1-4. Self-train a small model through a pseudo-random annotation set to obtain clean prediction labels;

[0098] Specifically, process S1-4 includes:

[0099] S1-4-1. Receive a pest image sample of a pseudo-random annotation set as an input feature of the small model, and output a second pseudo-random prediction label;

[0100] The expression for outputting the second pseudo-random prediction label is:

[0101] represents the second pseudo-random predicted label of the qth pest image sample in the pseudo-random annotation set, represents the weight matrix of the small model, represents the image features of the qth pest image sample in the pseudo-random annotation set, represents the bias term of the small model;

[0102] The image features of the pest image samples in the input pseudo-random annotation set are processed by the weight matrix and bias term of the small model and then by the activation function to obtain the second pseudo-random prediction label.

[0103] Although the large model has limited understanding, it still has some relevance, which provides a basis for subsequent small model training, allowing the small model to be further optimized on the pseudo-random annotation set to generate a more accurate second pseudo-random prediction label. The samples of the pseudo-random annotation set are pest images that have undergone the first screening. The first screening is to output the first pseudo-random prediction label through the large model for pest image samples that do not contain any manual annotations, and match the corresponding pest image samples according to the first pseudo-random prediction label, thereby achieving the first screening of pest image samples.

[0104] Specifically, the small model receives pest image samples in the pseudo-random annotation set as input features and deeply extracts features from the samples. Since these image samples have generated preliminary first pseudo-random prediction labels through the large model, the small model can process the input features through further forward propagation and generate more accurate second pseudo-random prediction labels.

[0105] S1-4-2. Encode the first pseudo-random predicted label to generate a first pseudo-random true label;

[0106] The second pseudo-random true label exists as a label target label, and in this embodiment is actually still a predicted value. Therefore, it is defined as the first pseudo-random true label in order to improve the property of the label as a target label.

[0107] Specifically, after the small model generates the second pseudo-random prediction label, it needs to encode the first pseudo-random prediction label and convert it into a real label that meets the classification requirements. By encoding the output second pseudo-random prediction label, a real label that is easy to calculate with the second pseudo-random prediction label is obtained, so that the model can better perform subsequent loss calculations and ultimately optimize the accuracy of the model.

[0108] S1-4-3. Calculate the classification loss of the second pseudo-random predicted label and the first pseudo-random true label;

[0109] The expression of classification loss is:

[0110] represents the classification loss, n represents the number of pest image samples, k represents the total number of category labels, represents the first pseudo-random true label of the i-th sample, representing the predicted probability that sample i belongs to category label j, Represents the second pseudo-random prediction label of the i-th sample, representing the predicted probability that the sample i output by the small model belongs to the category label j. Represents the logarithm of the second pseudo-random predicted label of the i-th sample, which is used to calculate the cross entropy loss;

[0111] Specifically, the classification loss between the second pseudo-random predicted label and the first pseudo-random true label after encoding is calculated, which represents the predicted difference between the predicted value output by the large model and the "true label" output by the small model; wherein, the classification loss function can adopt the cross entropy loss function or other loss functions suitable for multi-label classification problems, and measure the accuracy of the prediction result of the small model by calculating the difference between the predicted value and the true label.

[0112] S1-4-4. According to the preset small loss strategy, determine whether the classification loss is lower than the preset loss value;

[0113] The expression for determining whether the classification loss is lower than the preset loss value is:

[0114] Represents the preset loss value, which indicates the maximum allowed classification loss. When , it means that the prediction of the current model is accurate enough, and the second pseudo-random prediction label can be determined as a clean prediction label;

[0115] In this implementation, the small model refers to a lightweight pest recognition model, which is smaller in complexity and size than the large model, but can be refined in learning and optimization in a specific task (pest recognition). The task of the small model is to learn and refine pest features from the pseudo-random annotation set, and perform more accurate classification and annotation based on the large model. It may be better able to cope with pest-specific fields or specific types of data, such as specific images in pest recognition, through a large number of example training. Since the first pseudo-random prediction label generated by the large model may contain more noise, the small model can filter out the noise and generate a more accurate second pseudo-random prediction label through further learning;

[0116] The pseudo-random annotation set is input into the small model. Through the self-training of the small model, it is equivalent to using the small model to perform noise distillation and purification. The small model is used to extract features of the pest image samples corresponding to the first pseudo-random prediction label to generate a more accurate second pseudo-random prediction label. Based on the small loss strategy, the clean prediction label is screened out, thereby improving the accuracy of subsequent prediction results.

[0117] The small loss strategy is that after the small model generates the second pseudo-random prediction label, the second pseudo-random prediction label is matched with the first pseudo-random prediction label, the second pseudo-random prediction label with lower classification loss is screened out, and then the second pseudo-random prediction label is used as the clean prediction label.

[0118] S1-4-5. Select a second pseudo-random prediction label corresponding to a classification loss lower than a preset loss value and determine it as a clean prediction label.

[0119] Specifically, when the preset loss condition is met, the second pseudo-random prediction label with lower classification loss is selected as the clean prediction label. This process is based on the small loss screening strategy, which ensures the accuracy of the clean prediction label by ensuring a smaller loss value. These clean prediction labels will be used as target labels for subsequent model optimization and training.

[0120] S1-5. Select pest image samples corresponding to clean prediction labels to construct a pest image training set;

[0121] Specifically, pest image samples corresponding to clean prediction labels are selected as training samples of the pest image training set, that is, a high-quality pest image training set is constructed based on clean training samples, in which the training samples can better represent the actual pest classification situation, and thus can be used to further optimize the large model.

[0122] It should be noted that the clean prediction label is not the manually annotated target label in traditional technology, but the second pseudo-random prediction label generated by the small model based on the first pseudo-random prediction label. The second prediction label is generated by forward propagation of the small model on the pseudo-random image set.

[0123] Process S2 is a process of how to embed the pest knowledge graph into the big model and perform training based on the embedded big model (knowledge graph model) in this embodiment, including:

[0124] S2-1. Define the head entity of the pest knowledge graph as the input feature of the knowledge graph model. Define the tail entity as the output label of the knowledge graph model. Define the relationship as the dependency between the head entity and the tail entity in the knowledge graph model to construct the real pest knowledge facts;

[0125] The expression for constructing the real pest knowledge fact is:

[0126] T represents the real pest knowledge fact, representing a relation entity in the pest knowledge graph. Indicates a head entity, such as the name of a pest, such as "tea looper", Indicates dependency relationships, representing the association between the head entity and the tail entity, such as "living in"; Represents the tail entity, representing the object connected to the head entity through a dependency relationship, such as "tea tree" as a host plant.

[0127] Specifically, when training the knowledge graph model, the pre-built pest knowledge graph is constructed by domain experts based on existing pest classification knowledge and experience, and contains known real pest knowledge facts (head entity, relationship, tail entity). The real pest knowledge facts reflect the dependencies between known pest entities (for example, pest name, characteristics, host plant, etc.).

[0128] S2-2. Receive pest image samples in the pest image training set as input features of the knowledge graph model, and obtain feature vectors of the pest image samples by image feature extraction;

[0129] Specifically, features are extracted from the input pest image training set through an image feature extraction network (such as a convolutional neural network) to generate an image feature vector. The image feature vector will be used as input for further training of the knowledge graph model.

[0130] S2-3. Given the feature vector of the pest image sample as the head entity of the pest knowledge fact, based on the head entity and the dependency relationship, output the predicted tail entity to obtain the predicted pest knowledge fact;

[0131] The expression of the predicted pest knowledge fact is:

[0132]

[0133] represents the predicted pest knowledge facts,

[0134] A feature vector representing a given pest image sample; represents the tail entity predicted by the knowledge graph model, Represents the weight matrix of the knowledge graph model, It is the bias term of the knowledge graph model;

[0135] Specifically, in the knowledge graph model, the extracted image feature vector is used as the head entity of the pest knowledge fact, and the predicted tail entity is generated by combining the dependency relationship in the knowledge graph (such as biological characteristics, pest control, etc.). The predicted tail entity is derived through forward propagation based on the head entity and the dependency relationship; finally, the head entity, dependency relationship and predicted tail entity are combined into the predicted pest knowledge fact.

[0136] S2-4. Calculate the matching degree between the predicted pest knowledge facts and the real pest knowledge facts;

[0137] Specifically, the accuracy of the prediction result is measured by calculating the matching degree between the predicted pest knowledge facts (head entity, relationship, tail entity) and the real pest knowledge facts. The matching degree can be calculated by cosine similarity or other methods to measure the similarity of embedded vectors.

[0138] The expression for calculating the matching degree between the predicted pest knowledge facts and the real pest knowledge facts is:

[0139]

[0140] represents the matching degree between the predicted pest knowledge facts and the real pest knowledge facts, The norm between the feature vector and the tail entity vector representing a given pest image sample;

[0141] S2-5. Calculate the pest knowledge fact loss based on the matching degree;

[0142] The expression for calculating the pest knowledge fact loss is:

[0143]

[0144] Indicates the factual loss of pest knowledge, Represents negative sample pest knowledge facts The matching degree between the real pest knowledge facts, represents the prediction hyperparameter, Represents a non-negative sign.

[0145] In the pest knowledge fact loss function, negative samples refer to samples that are different from positive samples and do not conform to the current relationship or entity. They are used to help the model distinguish. Negative samples can be generated by replacing an element (head entity, tail entity or relationship) in the real pest knowledge fact. The prediction results of the model should be closer to the positive samples and away from the negative samples.

[0146] Specifically, the higher the matching degree, the closer the predicted pest knowledge facts are to the real pest knowledge facts. Based on the matching degree, the pest knowledge fact loss function is calculated. The loss function can measure the accuracy of the prediction results based on the error. Common loss functions include cross entropy loss, square error, etc. The difference between the predicted pest knowledge facts and the real pest knowledge facts is evaluated through the loss function.

[0147] S2-6. Perform back propagation and gradually adjust the model parameters to minimize the pest knowledge fact loss.

[0148] Specifically, after calculating the loss, the loss is fed back to each layer of the model through the back propagation algorithm, and the parameters of the model are adjusted to reduce the loss. Through iterative training, the model is gradually optimized to make its predicted tail entity more accurate.

[0149] Process S2-6 includes:

[0150] S2-6-1, obtaining the pest knowledge fact loss of the current iteration, and calculating the gradient of the pest knowledge fact loss of the current iteration relative to the model parameters;

[0151] S2-6-2, according to the gradient of the model parameters, back propagate it to obtain the updated model parameters

[0152] S2-6-3, according to the updated model parameters, perform the forward propagation of the next iteration and output the predicted tail entity.

[0153] By obtaining the pest knowledge fact loss of the current iteration, the difference between the predicted tail entity output by the model in the current round and the actual pest knowledge fact is evaluated. By calculating the gradient of the pest knowledge fact loss relative to the model parameters, the gradient is back-propagated, and the model parameters are updated. This successive optimization process ensures that the model can further reduce the prediction error in each round; with each iteration of optimization, the knowledge graph model gradually grasps the dependencies in the pest knowledge graph. After several iterations of training, the model can more accurately identify the pest images to be identified and output the predicted tail entity.

[0154] Convergence conditions include: the pest knowledge fact loss reaches a preset loss threshold or the training rounds reach a preset maximum number of training rounds.

[0155] The preset loss threshold refers to the knowledge graph model being considered to be trained to the pest identification model round, and the pest knowledge fact loss reaches a preset loss threshold. That is, when the knowledge graph model reaches a sufficiently accurate prediction state, the training is stopped in time to avoid the problem of model overfitting caused by overtraining. In other words, by limiting the maximum training rounds, the computational cost of model training can be effectively controlled to avoid excessive resource consumption.

[0156] From the first aspect, the pest recognition model based on the pest knowledge graph no longer relies on the traditional manual labeling method by introducing automatic labeling into the training process of the pest recognition model.

[0157] Specifically, traditional manual labeling is inevitably prone to human errors or subjective differences, while the first pseudo-random prediction label and the second pseudo-random prediction label generated by automatic labeling are optimized through multiple iterations, which can not only effectively reduce the deviations and errors that may occur in manual labeling, but also improve the accuracy of labels by using the learning ability of the model. Through the self-learning process of the small model, clean prediction labels are gradually generated, reducing label noise.

[0158] The first pseudo-random prediction label is generated by the large model, and the second pseudo-random prediction label is generated by the small model. The small loss strategy is then used for screening, and the labels are dynamically optimized using the self-training method. When the initial model effect is not ideal, more accurate labels can be generated through a self-correcting iterative process, thereby realizing automatic label generation, reducing manual intervention and time costs, and greatly improving the efficiency of building high-quality training sets.

[0159] Secondly, traditional training methods often rely on a large amount of data and labels, while this embodiment embeds the prior knowledge of domain experts in the form of triple knowledge facts into the model through a pre-constructed pest knowledge graph. This enables the model to better capture the dependency relationship between pests and their related features, host plants, etc. with the help of the structured information embedded in the knowledge graph during the training process, reducing the dependence on training data, and can rely on the background information provided by the knowledge graph in the case of insufficient or incomplete labels to enhance the reasoning ability of the model; and this embodiment calculates the matching degree between the predicted pest knowledge facts and the real pest knowledge facts, and continuously optimizes the model parameters through the pest knowledge fact loss function, so that the model can gradually learn the dependency relationship in the knowledge graph. This process can effectively improve the recognition accuracy of the model for pest images, especially when dealing with new categories of pests, the model can use the relationship in the knowledge graph to infer the tail entity (i.e., the recognition result), thereby reducing the misjudgment rate.

[0160] In this embodiment, the specific process steps of searching for the tail entity as the recognition result in the pre-built pest knowledge graph according to the query vector are:

[0161] Obtain pest knowledge facts (h, r, x), where h is the acquired head entity and r is the dependency relationship between the acquired head entity and the tail entity x to be identified;

[0162] In the pre-built pest knowledge graph, query whether there is an acquired head entity h;

[0163] If it exists, construct the obtained head entity h and dependency relationship r into the query vector q;

[0164] The query vector is used to query the pre-built pest knowledge graph, and the determined tail entity is selected as the recognition result of the query based on the head entity and dependency relationship.

[0165] For example, in a pest identification task, where:

[0166] h (the head entity obtained) = Tea Looper

[0167] r (dependency) = living in

[0168] x (tail entity) = x (location to be queried)

[0169] The query steps are:

[0170] Determine whether the tea looper exists as a head entity in the knowledge graph.

[0171] Construct (Tea looper, lives in) as the query vector.

[0172] Use the query vector to determine the location defined by the dependency relationship in the knowledge graph, and finally return "tea tree" as the tail entity.

[0173] The above embodiments may be implemented in whole or in part by software, hardware, firmware or any other combination thereof. When implemented using software, the above embodiments may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., infrared, wireless, microwave, etc.) means.

[0174] The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., DVD ), or semiconductor media. The semiconductor media may be a solid state drive.

[0175] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division of a waterway underwater terrain change analysis system and method. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0176] The above description is only a specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. A pest identification method, based on a visual and knowledge multimodal large model algorithm, wherein the algorithm is used for pest identification model to identify pests, characterized in that: The identification method comprises: Acquire an image of pests to be identified; Inputting the to-be-identified pest image into a trained pest identification model; Extracting image features of the pest image to be identified through a feature extraction layer; Using the image features of the to-be-identified pest image as the head entity of the pest knowledge fact; If the head entity of the pest knowledge fact exists in the pre-constructed pest knowledge graph, the head entity and its dependency relationship are constructed as a query vector, and the tail entity is queried in the pre-constructed pest knowledge graph as a recognition result; Otherwise, the head entity and its dependency relationship are used as input features of the pest recognition model, and the recognition result of the to-be-recognized pest image is output through forward propagation; The training process of the pest identification model includes: S1. Obtain a pest image training set, wherein the training samples of the pest image training set are pest image samples with clean prediction labels; S2. training a knowledge graph model using a pest image training set; wherein the knowledge graph model is embedded with a pre-built pest knowledge graph; S3, if the training reaches the convergence condition, stop the training and derive the knowledge graph model with the current iteration model parameters as the pest recognition model; The step of obtaining a pest image training set comprises: S1-1. Obtain several unlabeled pest image samples and construct them into an initial image set; S1-2. Receiving pest image samples of the initial image set as input features of the large model, and outputting a first pseudo-random prediction label; The expression for outputting the first pseudo-random prediction label is: ; represents the first pseudo-random predicted label of the qth pest image sample in the initial image set, represents the activation function, represents the weight matrix of the large model, represents the image features of the qth pest image sample in the initial image set, represents the bias term of the large model; S1-3. Constructing the pest image samples with the first pseudo-random predicted labels into a pseudo-random annotation set; S1-4. The clean prediction label is obtained by self-training a small model through a pseudo-random annotation set; S1-5. Select pest image samples corresponding to the clean prediction labels to construct the pest image training set.

2. A pest identification method according to claim 1, characterized in that: The self-training small model through the pseudo-random annotation set includes: S1-4-1. Receive the pest image sample of the pseudo-random annotation set as the input feature of the small model, and output a second pseudo-random prediction label; The expression for outputting the second pseudo-random prediction label is: ; represents the second pseudo-random predicted label of the qth pest image sample in the pseudo-random annotation set, represents the weight matrix of the small model, represents the image features of the qth pest image sample in the pseudo-random annotation set, represents the bias term of the small model; S1-4-2. Encode the first pseudo-random predicted label to generate a first pseudo-random true label; S1-4-3. Calculate the classification loss of the second pseudo-random predicted label and the first pseudo-random true label; The expression of classification loss is: ; represents the classification loss, n represents the number of pest image samples, k represents the total number of category labels, represents the first pseudo-random true label of the i-th sample, representing the predicted probability that sample i belongs to category label j, Represents the second pseudo-random prediction label of the i-th sample, representing the predicted probability that the sample i output by the small model belongs to the category label j. Represents the logarithm of the second pseudo-random predicted label of the i-th sample, which is used to calculate the cross entropy loss; S1-4-4. According to the preset small loss strategy, determine whether the classification loss is lower than the preset loss value; The expression for determining whether the classification loss is lower than the preset loss value is: ; Represents the preset loss value, which indicates the maximum allowed classification loss; S1-4-5. Select a second pseudo-random prediction label corresponding to a classification loss lower than a preset loss value and determine it as the clean prediction label.

3. A pest identification method according to claim 1, characterized in that: The method of using the pest image training set to train the knowledge graph model includes: S2-1. Define the head entity of the pest knowledge graph as the input feature of the knowledge graph model, the tail entity as the output label of the knowledge graph model, and the relationship as the dependency between the head entity and the tail entity in the knowledge graph model to construct real pest knowledge facts; The expression for constructing the real pest knowledge fact is: ; T represents the real pest knowledge fact, representing a relation entity in the pest knowledge graph, h represents the head entity, r represents the dependency relationship, representing the association between the head entity and the tail entity, and t represents the tail entity, representing the object connected to the head entity through the dependency relationship; S2-2. Receive pest image samples in the pest image training set as input features of the knowledge graph model, and obtain feature vectors of pest image samples by image feature extraction; S2-3. Given the feature vector of the pest image sample as the head entity of the pest knowledge fact, based on the head entity and the dependency relationship, output the predicted tail entity to obtain the predicted pest knowledge fact; The expression of the predicted pest knowledge fact is: ； ； represents the predicted pest knowledge facts, A feature vector representing a given sample of the pest image; represents the tail entity predicted by the knowledge graph model, Represents the weight matrix of the knowledge graph model, It is the bias term of the knowledge graph model; S2-4. Calculate the matching degree between the predicted pest knowledge facts and the real pest knowledge facts; The expression for the matching degree between the calculated predicted pest knowledge facts and the real pest knowledge facts is: ； represents the matching degree between the predicted pest knowledge facts and the real pest knowledge facts, Representing the norm between the feature vector and the tail entity vector of the given pest image sample; S2-5. Calculate the pest knowledge fact loss based on the matching degree; The expression for calculating the pest knowledge fact loss is: ； Indicates the factual loss of pest knowledge, Represents negative sample pest knowledge facts The matching degree between the real pest knowledge facts, represents the prediction hyperparameter, represents a non-negative sign; S2-6. Perform back propagation and gradually adjust the model parameters to minimize the pest knowledge fact loss.

4. A pest identification method according to claim 3, characterized in that: The back propagation is performed, including: S2-6-1, obtaining the pest knowledge fact loss of the current iteration, and calculating the gradient of the pest knowledge fact loss of the current iteration relative to the model parameters; S2-6-2, according to the gradient of the model parameters, back propagate it to obtain the updated model parameters S2-6-3, according to the updated model parameters, perform the forward propagation of the next iteration and output the predicted tail entity.

5. A pest identification method according to claim 4, characterized in that: Convergence conditions include: the pest knowledge fact loss reaches a preset loss threshold or the training rounds reach a preset maximum number of training rounds.

6. A pest identification method according to claim 5, characterized in that: According to the query vector, searching for the tail entity in the pre-built pest knowledge graph as a recognition result includes: Obtain pest knowledge facts (h, r, x), where h is the acquired head entity and r is the dependency relationship between the acquired head entity and the tail entity x to be identified; In the pre-built pest knowledge graph, query whether there is an acquired head entity h; If it exists, construct the obtained head entity h and dependency relationship r into the query vector q; The query vector is used to query in the pre-built pest knowledge graph, and the determined tail entity is selected as the recognition result of the query according to the head entity and the dependency relationship.

Images