A pest identification method based on multimodal self-supervised Transformer architecture

Abstract

This invention discloses a pest recognition method based on a multimodal self-supervised Transformer architecture. This method extracts image and text features based on the construction of an image encoder and a text encoder for application in pest recognition. The method comprises the following steps: constructing an image encoder and training a ViT model modified by PSM using contrastive learning to extract image features; constructing a text encoder and processing natural language descriptions of pests based on the Albert model to output text features; and fusing the obtained image and text features to obtain final pest features for classification. By utilizing self-supervised learning, fine-grained recognition, and multimodal information to improve the Transformer architecture, the method not only reduces the model's reliance on data volume but also improves the performance of fine-grained pest recognition. Compared with existing image recognition methods, this method focuses on subdivided regions of pest images, achieving high recognition accuracy, even with small sample datasets.

Description

Technical Field

[0001] The present invention relates to the technical field of fine-grained pest identification, and in particular to a pest identification method based on a multimodal self-supervised Transformer architecture. Background Art

[0002] Agricultural pests severely impact agricultural production and crop storage. Preventing agricultural pests requires accurate identification of pest species and the implementation of targeted control measures. Deep learning-based image recognition methods have demonstrated great potential in smart agriculture, particularly in pest identification. However, most current pest identification technologies require high-quality pest datasets manually labeled by agricultural experts, which is costly and impractical in real life. Therefore, new pest identification technologies that can adapt to small sample sizes and low-quality pest data have become a hot topic of research.

[0003] Currently, the mainstream method for image classification based on large datasets is deep convolutional neural networks (CNNs). However, these methods suffer from a reliance on labeled information. To address this challenge, image recognition methods based on transfer learning have achieved significant results on small sample datasets. However, these methods require that the data in the source and target domains be as similar as possible, which often makes them difficult to achieve for fine-grained recognition. Summary of the Invention

[0004] The purpose of the present invention is to overcome the shortcomings of the existing technology. To achieve the above purpose, a pest identification method based on a multimodal self-supervised Transformer architecture is adopted to solve the problems raised in the above background technology.

[0005] A pest recognition method based on a multimodal self-supervised Transformer architecture is proposed. The method extracts image and text features based on the construction of an image encoder and a text encoder and applies them to pest recognition. The method includes the following steps:

[0006] Step 1: Build an image encoder and train the ViT model improved by PSM based on contrastive learning to extract image features.

[0007] Step 2: Build a text encoder to process the natural language description of pests based on the Albert model and output text features;

[0008] Step 3: Fuse the obtained image features and text features to obtain the final pest features and classify them.

[0009] As a further solution of the present invention: the specific steps in step 1 include:

[0010] Acquire pest images;

[0011] Perform two different random transformations on the input pest image and generate a set of view sets V, which contains two global views and and several local views of smaller resolutions, all of which are passed through the student network, while the global view is passed through the teacher network;

[0012] Among the student networks Based on the teacher network The output of the two networks is trained with the same structure and different parameters. s and θ t express;

[0013] For the input image x, the two networks output the probability distribution over N dimensions, denoted as P s and P t The probability P is obtained by normalizing the output of the network g using the softmax function, and the expression is:

[0014]

[0015] Where N is the probability distribution P s The dimension of n is P s The dimension of T s >0, is the temperature coefficient used to control the sharpness of the output distribution, setting a fixed teacher network g θ , we minimize the cross entropy loss (the parameter

[0016] t

[0017] Number θ s ) to learn to match these distributions. The expression is:

[0018]

[0019] Among them, H(a,b)=-, parameter θ s It is obtained by minimizing the above formula through stochastic gradient descent;

[0020] The ViT model based on the improved PSM is used to extract regional features with subtle differences, and the last Transformer layer is used as the input of the feature layer to utilize attention information;

[0021] Assuming that the ViT model has M self-attention heads, the hidden features of the last layer input are recorded as:

[0022]

[0023] The attention weights of the above layers are:

[0024]

[0025]

[0026] The attention weights of previous layers are integrated by recursively performing matrix multiplication on the original attention weights of all layers:

[0027]

[0028] Then according to w final Among the M different attention heads in, select the maximum value W1,W2,…,W M The index of , and use these positions as the index of the model to extract a L-1 The corresponding mark in

[0029] Finally, the selected token is concatenated with the classification token as the input of the last Transformer Layer, expressed as:

[0030]

[0031] As a further solution of the present invention: the specific steps in step 2 include:

[0032] The Albert model of Transformer encoder with GELU nonlinearity is used to encode text information;

[0033] Denote the vocabulary size as V, the vocabulary embedding size as E, and the hidden layer size as H. We improve the Albert model based on factored embedding parameterization and cross-layer parameter sharing.

[0034] The Albert model uses factorization of the embedding parameters, projects them into a low-dimensional embedding space of size E, and then projects them into the latent space. By factorization, the embedding parameters are reduced from O(V*H) to O(V*H+E*H), where E<<H, reducing the number of parameters. At the same time, the Albert model shares all parameters across layers, further reducing the number of parameters.

[0035] As a further solution of the present invention: the specific steps in step three include:

[0036] Feed the pest image into the image encoder to obtain image features of size K;

[0037] Then the corresponding text description information is sent to the text encoder to obtain a text feature of size M;

[0038] The obtained image features and text features are then concatenated to obtain pest features of size K+M, which are sent to the linear layer for classification as the final predicted category.

[0039] Compared with the prior art, the present invention has the following technical effects:

[0040] The aforementioned technical solution, which leverages self-supervised learning, fine-grained recognition, and multimodal information to improve the Transformer architecture, not only reduces the model's reliance on data volume but also improves fine-grained pest recognition performance. Compared to existing image recognition methods, this method can focus on specific areas of pest images, achieving high recognition accuracy even with small sample data sets. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The specific embodiments of the present invention are described in detail below with reference to the accompanying drawings:

[0042] Figure 1 A schematic diagram of the steps of the pest identification method disclosed in the present application;

[0043] Figure 2 A schematic diagram of the architecture of the pest identification method according to the embodiment disclosed in this application;

[0044] Figure 3 This is a schematic diagram of an image encoder according to an embodiment disclosed in this application. DETAILED DESCRIPTION

[0045] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0046] Please refer to Figure 1 In an embodiment of the present invention, a pest identification method based on a multimodal self-supervised Transformer architecture is provided. The method extracts image features and text features based on constructing an image encoder and a text encoder for application in pest identification, and includes the following steps:

[0047] like Figure 2 The figure shows the architecture of the pest recognition method. It uses the joint multimodal information of images and natural language descriptions, extracts image features and text features using image encoders and text encoders respectively, and merges the two features to provide a classification strategy for pest image classification.

[0048] Step 1: Build an image encoder and train the ViT model improved by PSM based on contrastive learning to extract image features. The specific steps include:

[0049] Acquire pest images;

[0050] like Figure 3 As shown, the figure is a schematic diagram of an image encoder;

[0051] Perform two different random transformations on the input pest image and generate a set of view sets V, which contains two global views and and several local views of smaller resolutions, all of which are passed through the student network, while the global view is passed through the teacher network;

[0052] Among the student networks Based on the teacher network The output of the two networks is trained with the same structure and different parameters. s and θ t express;

[0053] For the input image x, the two networks output the probability distribution over N dimensions, denoted as P s and P t The probability P is obtained by normalizing the output of the network g using the softmax function, and the expression is:

[0054]

[0055] Where N is the probability distribution P s The dimension of n is P s The dimension of T s >0, is the temperature coefficient used to control the sharpness of the output distribution, setting a fixed teacher network We minimize the cross entropy loss (the parameters θ of the student network s ) to learn to match these distributions. The expression is:

[0056]

[0057] Among them, H(a,b)=-, parameter θ s It is obtained by minimizing the above formula through stochastic gradient descent;

[0058] In order to better extract regional features with subtle differences, the student and teacher networks adopt the ViT model improved by PSM, and use the last Transformer layer as the input of the feature layer to fully utilize the attention information.

[0059] Assuming that the ViT model has M self-attention heads, the hidden features of the last layer input are recorded as:

[0060]

[0061] The attention weights of the above layers are:

[0062]

[0063]

[0064] The attention weights of previous layers are integrated by recursively performing matrix multiplication on the original attention weights of all layers:

[0065]

[0066] Compared with the single layer original attention weight w L-1 In comparison, w final is a better choice for selecting the discriminative region because it captures how information is passed from the input layer to the embedding of higher layers.

[0067] Then according to w final Among the M different attention heads in, select the maximum value W1,W2,…,W M The index of , and use these positions as the index of the model to extract a L-1 The corresponding mark in

[0068] Finally, the selected token is concatenated with the classification token as the input of the last Transformer Layer, expressed as:

[0069]

[0070] Step 2: Build a text encoder to process the natural language description of pests based on the Albert model and output text features. The specific steps include:

[0071] Albert is used to encode text information. The backbone of the Albert architecture is similar to Bert, both using a Transformer encoder with GELU nonlinearity. The vocabulary size is denoted as V, the vocabulary embedding size is denoted as E, and the hidden layer size is denoted as H. Albert has made improvements in two aspects: factorized embedding parameterization and cross-layer parameter sharing. Albert uses factorization of embedding parameters. Instead of directly projecting one-hot vectors into a latent space of size H, it first projects them into a low-dimensional embedding space of size E, and then projects them into the latent space. Through this factorization, the embedding parameters are reduced from O(V*H) to O(V*H+E*H), where E<<H, thereby reducing the number of parameters. On the other hand, Albert shares all parameters across layers, further reducing the number of parameters.

[0072] Step 3: Fuse the obtained image features and text features to obtain the final pest features and classify them. The specific steps include:

[0073] The pest image is fed into the image encoder to obtain image features of size K=768;

[0074] Then the corresponding text description information is sent to the text encoder to obtain a text feature of size M = 768;

[0075] The obtained image features and text features are then concatenated to obtain pest features of size K+M=1536, which are sent to the linear layer for classification as the final predicted category.

[0076] Although the embodiments of the present invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and variations may be made to these embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the appended claims and their equivalents, and all should be included within the scope of protection of the present invention.

Claims

1. A pest identification method based on a multimodal self-supervised Transformer architecture, characterized by: Based on the construction of image encoder and text encoder, image features and text features are extracted and applied to pest identification, including the following steps: Step 1: Build an image encoder, train the PSM-modified ViT model based on contrastive learning, and extract image features. The specific steps include: Acquire pest images; Perform two different random transformations on the input pest image and generate a set of view sets V, which contains two global views and , and several local views of smaller resolutions, and all views are passed through the student network, while the global view is passed through the teacher network; Among the student networks Based on the teacher network The output of the training is obtained. The structure of the two networks is the same, but the parameters are different. and express; For the input image , the two network outputs The probability distribution over dimensions is expressed as and ; Probability The network is constructed by using the softmax function The output of is normalized and the expression is: in, is the probability distribution Dimensions, for The first few dimensions, >0, is the temperature coefficient used to control the sharpness of the output distribution, setting a fixed teacher network , we minimize the cross entropy loss, the parameters of the student network , to learn to match these distributions; the expression is: in, log ,parameter It is obtained by minimizing the above formula through stochastic gradient descent; The ViT model based on the improved PSM is used to extract regional features with subtle differences, and the last Transformer layer is used as the input of the feature layer to utilize attention information; Assuming that the ViT model has M self-attention heads, the hidden features of the last layer input are recorded as: ； The attention weights of the above layers are: The attention weights of previous layers are integrated by recursively performing matrix multiplication on the original attention weights of all layers: Based on Among the M different attention heads in , select the maximum value , ,…, , and use these positions as the index of the model to extract The corresponding mark in Finally, the selected token is concatenated with the classification token as the input of the last Transformer Layer, expressed as: ； Step 2: Build a text encoder to process the natural language description of pests based on the Albert model and output text features; Step 3: Fuse the obtained image features and text features to obtain the final pest features and classify them. The specific steps include: Feed the pest image into the image encoder to obtain a size of Image features; Then send the corresponding text description information to the text encoder to get the size of Text features of Then concatenate the obtained image features and text features to obtain a size of The pest features are sent to the linear layer for classification as the final predicted category.

2. The pest identification method based on a multimodal self-supervised Transformer architecture according to claim 1, characterized in that: The specific steps in step 2 include: The Albert model of Transformer encoder with GELU nonlinearity is used to encode text information; Denote the vocabulary size as , the vocabulary embedding size is expressed as , the hidden layer size is expressed as , the Albert model is improved based on factor embedding parameterization and cross-layer parameter sharing; The Albert model uses a factorization of the embedding parameters, projecting them onto a The low-dimensional embedding space is then projected into the hidden space, and the embedding parameters are factorized from Reduced to ,in , reducing the number of parameters. At the same time, the Albert model shares all parameters across layers, reducing the number of parameters again.

Images