An agricultural pest and disease image-text retrieval method, system and electronic device

Abstract

The application discloses an agricultural pest and disease image-text retrieval method and system and electronic equipment, and relates to the technical field of electric digital data processing. The application extracts the features of the data to be retrieved by using a feature extraction model, constructs a cross-modal feature fusion network, determines the matching information of the feature sequence of the data to be retrieved in the data retrieval library by using the cross-modal feature fusion network, and takes the matching information as the cross-modal retrieval result, so that the image-text retrieval or the text-image retrieval of the pest and disease is better realized, and the efficiency and speed of the agricultural information retrieval are improved.

Description

Technical Field

[0001] This invention relates to the field of electronic digital data processing technology, and in particular to a method, system, and electronic device for image and text retrieval of agricultural pests and diseases. Background Technology

[0002] Cross-modal retrieval is a popular research direction in artificial intelligence, aiming to achieve mutual correspondence and retrieval of data in different modalities such as text, voice, video, and images. That is, using data in one modality to query data in another modality. However, due to the "semantic gap" between different modalities, it is not easy to achieve this cross-modal retrieval. It is necessary to find the potential associations between them and complete the matching.

[0003] At present, cross-modal retrieval based on deep learning mainly involves extracting features from the original data using a pre-trained object detector, mapping the extracted features to a common space, and using constraints such as loss functions to reduce the difference between cross-modal data and high-level semantic features. Then, the sample with the highest similarity is retrieved as a match using methods such as cosine similarity. However, in the field of agricultural pests and diseases, a pre-trained object detector is not required, and its training and inference require more computing resources.

[0004] Currently, various information technologies are widely used in the field of agricultural informatization, such as text classification, intelligent question answering, and image-based pest and disease detection. However, the application of cross-modal technologies has not yet been widely adopted. Current pest and disease detection methods mostly involve processing image data first, identifying pest and disease categories through images, and then retrieving relevant text information, which is inefficient. Finding a suitable cross-modal retrieval technology that allows users to retrieve matching text by inputting crop pest and disease images, or directly retrieve images of pests and diseases using sentences, can effectively improve farmers' planting techniques and pest and disease prevention methods, promote information dissemination, and improve retrieval efficiency.

[0005] Therefore, developing a method for cross-modal retrieval in the field of agricultural pests and diseases has high application value. Summary of the Invention

[0006] To address the aforementioned problems in the existing technology, this invention provides a method, system, and electronic device for image and text retrieval of agricultural pests and diseases.

[0007] To achieve the above objectives, the present invention provides the following solution:

[0008] A method for image and text retrieval of agricultural pests and diseases, including:

[0009] Obtain the data to be retrieved; the data to be retrieved is image data or text data of agricultural pests and diseases;

[0010] The features of the data to be retrieved are extracted using a feature extraction model to obtain a feature sequence of the data to be retrieved.

[0011] Construct a cross-modal feature fusion network;

[0012] The cross-modal feature fusion network is used to determine the matching information of the feature sequence of the data to be retrieved in the data retrieval database, and the matching information is used as the cross-modal retrieval result.

[0013] Optionally, a feature extraction model is used to extract features from the data to be retrieved, resulting in a feature sequence of the data to be retrieved, including:

[0014] When the data to be retrieved is image data, an image encoder is used to extract the features of the image data, and the image sequence features are obtained by combining the pixel feature extraction method.

[0015] When the data to be retrieved is text data, a text encoder is used to extract the features of the text data, and a dictionary information fusion method is combined to obtain text sequence features.

[0016] Optionally, the construction process of the image encoder includes:

[0017] A ResNet-50 network with channel attention was used as the original feature extractor, and the non-sequential features output by the original feature extractor were processed into sequential features using a pixel feature extraction method to construct an image feature extraction model.

[0018] An image feature extraction model is trained on an image classification task using image sample data, and the trained image feature extraction model is used as the image encoder.

[0019] Optionally, the construction process of the text encoder includes:

[0020] The Chinese RoBERTa model was used to extract character-level features from the text sample data;

[0021] A vocabulary list in the field of pests and diseases is collected and added to the word segmenter. During the text segmentation preprocessing, words appearing in the vocabulary list in the text sample data are used as character-level tokens and input into the Chinese RoBERTa model. Using the collected text, the Chinese RoBERTa model is adjusted on the text classification task to generate embedding vectors that meet the set requirements for the newly added words in the vocabulary, thus obtaining the adjusted Chinese RoBERTa model. The adjusted Chinese RoBERTa model is then used as the text encoder.

[0022] Optionally, before the text data is input into the text encoder, markers "" are added to the beginning and end of the text data respectively. <cls>"and mark" <seq>"; where the marked text data is:

[0023] S2 = " <cls> ”+S1+" <seq>”;

[0024] In the formula, S2 is the marked text data, and S1 is the original data.

[0025] Optionally, the process of constructing a cross-modal feature fusion network includes:

[0026] Obtain image sample features and text sample features, and combine the image sample features and text sample features to obtain the input sample sequence;

[0027] Train an Embedding layer of length 2, and then add the Embedding layer to the image sample features and text sample features respectively;

[0028] Construct an encoder structure for a self-attention transform network, input the input sample sequence into the encoder structure, and obtain the output result;

[0029] The text sequence features and image sequence features of the output results are respectively processed by attention pooling to obtain one-dimensional representations of the text sequence features and one-dimensional representations of the image sequence features. Then, the categories of the one-dimensional representations of the text sequence features and the one-dimensional representations of the image sequence features are respectively obtained by multi-class prediction of nonlinear changes of FC-Relu-FC combination.

[0030] Extract the joint embedding features from the output results, and use FC-Sigmoid combination to predict the similarity between images and text in the input sample sequence;

[0031] Matching information is determined based on the similarity and preset value, and cross-modal retrieval results are determined based on the matching information.

[0032] Optionally, the loss function used during the training of the cross-modal feature fusion network is:

[0033] Loss = L1 + λL2;

[0034] The first part of the loss function L1 is:

[0035]

[0036] In the formula, m represents the number of the first sample, and y p This indicates that the p-th sample matches the label. This represents the prediction result for the similarity of the p-th sample;

[0037] The second part of the loss function L2 is:

[0038]

[0039] In the formula, nums represents the second sample size, 9 represents the 9 types of pests and diseases, and s represents the s-th type of pest or disease. This represents the label category of the q-th image sample. This represents the category label of the q-th text sample. This represents the probability of the image sample predicted by the softmax function. This represents the probability of a text sample predicted by the softmax function.

[0040] Furthermore, the present invention provides an agricultural pest and disease image-text retrieval system, the system being used to implement the above-mentioned agricultural pest and disease image-text retrieval method; the system includes:

[0041] The data acquisition module is used to acquire the data to be retrieved; the data to be retrieved is image data or text data of agricultural pests and diseases.

[0042] The feature extraction module is used to extract features from the data to be retrieved using a feature extraction model, thereby obtaining a feature sequence of the data to be retrieved.

[0043] The network building module is used to construct cross-modal feature fusion networks;

[0044] The image and text retrieval module is used to determine the matching information of the feature sequence of the data to be retrieved in the data retrieval database using the cross-modal feature fusion network, and to use the matching information as the cross-modal retrieval result.

[0045] Furthermore, the present invention also provides an electronic device comprising:

[0046] Memory, used to store computer programs;

[0047] A processor, connected to the memory, is used to retrieve and execute the computer program to implement the above-described method for image and text retrieval of agricultural pests and diseases.

[0048] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0049] This invention employs a feature extraction model to extract features from the data to be retrieved, and constructs a cross-modal feature fusion network. The cross-modal feature fusion network is used to determine the matching information of the feature sequence of the data to be retrieved in the data retrieval database, and the matching information is used as the cross-modal retrieval result. This better enables text-to-image or image-to-text search for pests and diseases, thereby improving the efficiency and speed of agricultural information retrieval. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 A flowchart of the agricultural pest and disease image retrieval method provided by the present invention;

[0052] Figure 2 This is a diagram illustrating the implementation architecture of an agricultural pest and disease image retrieval system provided in an embodiment of the present invention.

[0053] Figure 3 This is a flowchart illustrating the process of word segmentation and training in RoBERTa by fusing dictionary information, as provided in an embodiment of the present invention; wherein, Figure 3 (a) is the flowchart of the word segmentation process of the original text encoder. Figure 3 (b) is a flowchart of the word segmentation process after adding the dictionary;

[0054] Figure 4 A flowchart for retrieving text provided in an embodiment of the present invention. Detailed Implementation

[0055] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0056] The purpose of this invention is to provide a method, system, and electronic device for image and text retrieval of agricultural pests and diseases, which can improve the efficiency and speed of agricultural information retrieval.

[0057] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0058] like Figure 1 As shown, the agricultural pest and disease image retrieval method provided by the present invention includes:

[0059] Step 100: Obtain the data to be retrieved. The data to be retrieved is either image data or text data about agricultural pests and diseases.

[0060] Step 101: Use a feature extraction model to extract features from the data to be retrieved, obtaining a feature sequence of the data to be retrieved. Specifically, when the data to be retrieved is image data, an image encoder is used to extract features from the image data, and this is combined with a pixel feature extraction method to obtain image sequence features. When the data to be retrieved is text data, a text encoder is used to extract features from the text data, and this is combined with a dictionary information fusion method to obtain text sequence features.

[0061] In practical applications, the construction process of an image encoder includes:

[0062] (1) A ResNet-50 network with channel attention was used as the original feature extractor, and the non-sequence features output by the original feature extractor were processed into sequence features using the pixel feature extraction method to construct the image feature extraction model.

[0063] (2) Use image sample data to train an image feature extraction model on an image classification task, and use the trained image feature extraction model as an image encoder.

[0064] B. The process of building a text encoder includes:

[0065] (1) The Chinese RoBERTa model is used to extract character-level features from the text sample data.

[0066] (2) Collect a vocabulary list in the field of pests and diseases and add it to the word segmenter. During the text segmentation preprocessing, the words that appear in the vocabulary list in the text sample data are used as character-level tokens and input into the Chinese RoBERTa model. Using the collected text, the Chinese RoBERTa model is adjusted on the text classification task to generate embedding vectors that meet the set requirements for the newly added words in the vocabulary list. The adjusted Chinese RoBERTa model is then used as the text encoder.

[0067] Step 102: Construct a cross-modal feature fusion network.

[0068] In practical applications, the process of constructing a cross-modal feature fusion network includes:

[0069] (1) Obtain image sample features and text sample features, and combine image sample features and text sample features to obtain input sample sequence.

[0070] (2) Train an Embedding layer of length 2, and add the Embedding layer to the image sample features and text sample features respectively.

[0071] (3) Construct the encoder structure of the self-attention transformation network, input the input sample sequence into the encoder structure, and obtain the output result.

[0072] (4) The text sequence features and the image sequence features of the output results are respectively processed by attention pooling to obtain one-dimensional representations of the text sequence features and one-dimensional representations of the image sequence features. Then, the categories of the one-dimensional representations of the text sequence features and the categories of the one-dimensional representations of the image sequence features are respectively obtained by multi-class prediction of nonlinear changes of FC-Relu-FC combination.

[0073] (5) Take the joint embedding features from the output results and predict the similarity between images and text in the input sample sequence by combining FC-Sigmoid.

[0074] (6) Determine matching information based on similarity and preset values, and determine cross-modal retrieval results based on matching information.

[0075] Furthermore, to improve retrieval accuracy, the loss function used during the training of the cross-modal feature fusion network can be:

[0076] Loss = L1 + λL2;

[0077] The first part of the loss function L1 is:

[0078]

[0079] In the formula, m represents the number of the first sample, and y p This indicates that the p-th sample matches the label. This represents the prediction result for the similarity of the p-th sample;

[0080] The second part of the loss function L2 is:

[0081]

[0082] In the formula, nums represents the second sample size, 9 represents the 9 types of pests and diseases, and s represents the s-th type of pest or disease. This represents the label category of the q-th image sample. This represents the category label of the q-th text sample. This represents the probability of the image sample predicted by the softmax function. This represents the probability of a text sample predicted by the softmax function.

[0083] Step 103: Use a cross-modal feature fusion network to determine the matching information of the feature sequence of the data to be retrieved in the data retrieval database, and use the matching information as the cross-modal retrieval result.

[0084] The following two embodiments illustrate the specific application process of the agricultural pest and disease image retrieval method provided by the present invention.

[0085] Example 1

[0086] In this embodiment, the overall architecture of the agricultural pest and disease image retrieval method is as follows: Figure 2 As shown, the implementation process is as follows:

[0087] Step 1: Construct a cross-modal dataset of pests and diseases and expand the dataset through data augmentation. Use a ResNet-50 network model with channel attention and a Chinese RoBERTa pre-trained model as the feature extraction network for images and text. Combine pixel feature extraction and dictionary information fusion methods to extract sequence features of images and text respectively.

[0088] A) The specific steps for constructing the cross-modal dataset of pests and diseases in step one are as follows:

[0089] Training and testing datasets were constructed using image-text pairs. The images primarily consisted of pictures of crop pest and disease symptoms, while the text consisted of relevant text related to these pests and diseases. Data augmentation was used to expand the dataset. All matching image-text pairs were used as positive samples, and mismatched negative samples were randomly constructed between different types of pest and disease data, maintaining an equal number of positive and negative samples.

[0090] B) The specific steps for extracting text and image feature vectors using the pre-trained text encoder and image encoder in step one are as follows:

[0091] 1) Construct a pre-trained text encoder and use a pre-trained Chinese RoBERTa model to extract character-level features from the text. Next, collect a professional vocabulary in the field of pests and diseases and add it to the word segmenter. During text segmentation preprocessing, words appearing in the vocabulary in the text are used as character-level tokens input into the model. The collected text is used to fine-tune the RoBERTa model on a text classification task to generate appropriate embedding vectors for newly added words in the dictionary. The model is then saved as a text encoder.

[0092] Furthermore, before the text input text encoder, the text S1 needs to be processed as follows:

[0093] S2 = " <cls> ”+S1+" <seq>(4)

[0094] Add two markers at the beginning and end of the text: <cls>"and" <seq>The text features are extracted using a text encoder and represented as follows:

[0095] V = {'[ismh]', v1, v2, ..., v n ,'[seq]'} (5)

[0096] In the formula, v1, v2, ..., v n Given n text data points, the original BERT CLS is represented as ISMH after feature fusion.

[0097] 2) Construct a pre-trained image encoder: The image encoder uses a ResNet-50 network with added channel attention as the original feature extractor, and uses pixel feature extraction to process the non-sequential features output by the model into sequential features. The collected images are used to train the ResNet model as the image encoder on the image classification task.

[0098] Furthermore, the channel attention and pixel feature extraction are specifically as follows:

[0099] Channel attention, following each residual connection in the ResNet-50 network, performs feature compression on the input feature map of dimension (D×H×W) along the dimension of spatial features D, transforming each two-dimensional feature map (H×W) into a real number using the following formula:

[0100]

[0101] Where, d i Z represents the feature map of the i-th channel. i H represents the result of the feature map calculation for the i-th channel, where H and W both represent the number of channels.

[0102] After obtaining the real-valued representation of each channel feature, the real values ​​are mapped through a non-linear transformation of (FC-Tanh-FC). Finally, a Sigmoid (.) operation is used to map the real values ​​to the range (0~1), which serves as the attention weight set for each channel of the original feature map. Each two-dimensional feature map of the original feature map in D dimension is multiplied by the corresponding weight in the attention weight set to obtain a new feature map with added channel attention.

[0103] The feature map output by the ResNet network has dimensions [2048, 3, 3]. Treating the last two dimensions of the feature map as 9 pixels, we extract 2048-dimensional features for each pixel, concatenating these 9 sets. The resulting image feature dimensions are represented as [9, 2048], denoted as U = {u1, u2, ..., u9}. u1, u2, ..., u9 are the 9 processed image features.

[0104] Step 2: Construct a cross-modal feature fusion network based on a Transformer encoder structure to fuse image and text features. Image and text features are subspaced using a mapping matrix, then concatenated and input into the feature fusion network. A new joint embedding vector is generated within the feature fusion network to learn image-text matching features. A fully connected layer predicts the similarity to determine if the current input sample matches, returning the matched sample. Furthermore, the multi-dimensional features of the image and text features after the feature fusion network are constrained into one-dimensional features using attention pooling, and their categories are predicted using linear layers. This serves as a subtask to constrain model training, allowing the model to perceive more differences between input samples through classification.

[0105] The specific steps for constructing the cross-modal feature fusion network in step two are as follows:

[0106] The encoder part of a self-attention transform network (Transformer) is used as the cross-modal feature fusion network. Specifically, the mapped image feature sequence U and text feature sequence V are combined into the input sequence f according to the following rules:

[0107] f = {'[ismh]', v1, v2, ..., v n ,'[seq]',u1,u2,...,u9} (7)

[0108] Simultaneously, train an Embedding layer of length 2, and associate the first and second embeddings with image features v respectively. i and text features u i Addition is used to distinguish data from different modalities.

[0109] Next, the encoder structure of the Transformer is constructed, taking the concatenated image and text features as the input to the model, and the output is represented as...

[0110]

[0111] In the formula, This indicates the output result corresponding to *.

[0112] Finally, the text sequence features of the output results will be... and image sequence features Its one-dimensional representation is obtained through attention pooling. and Then, the category is predicted separately using multi-class classification based on the nonlinear variations of FC-ReLU-FC combinations. The joint embedding features of the location are used to predict the similarity between the input image and the text through FC-Sigmoid combination, and values ​​greater than 0.5 and less than 0.5 are used as matching and non-matching conditions, respectively.

[0113] When training the model, the loss function used to constrain the model is shown in Equations (1) to (3).

[0114] Step 3: Train a cross-modal feature fusion network on the pest and disease image-text pair dataset, incorporate an alerting process during training, and collect pest and disease-related text and images as a data retrieval library.

[0115] Furthermore, the steps in step three for training the network model and creating the data retrieval library are as follows:

[0116] The entire model, including a text encoder, an image encoder, and a cross-modal feature fusion network, is trained using the collected image and text dataset. A short text with the paradigm T, "This is a picture of a XX pest," is added to the text training data, where XX represents the name of the pest. In the model's text-based image retrieval inference, the user enters the name of the pest, which is placed in the XX position of T. The entire text is then used as the retrieval query, allowing users to retrieve relevant images by directly inputting the pest's name. A subset of images and text from the training and testing sets is collected as the model's data retrieval library, and features from this library are extracted and stored beforehand using the image and text encoders.

[0117] Step 4: When using the trained model for cross-modal retrieval, extract the serialized features of the input image (or text) using its corresponding feature extraction method. Then, use the cross-modal feature fusion network to determine whether the features match the text (or image) in the data retrieval database, and output the matching information as the cross-modal retrieval result.

[0118] Furthermore, the step in step four, using the model to perform cross-modal matching on the input data, is as follows:

[0119] When an image (or text) is input, an image encoder (or text encoder) is used to obtain its serialized features. These features are then combined with pre-extracted text (or image) from a data retrieval database and passed through a cross-modal feature fusion network to obtain the matching result, which is returned to the requester.

[0120] Example 2

[0121] In this implementation, a pre-trained model for image and text feature extraction is pre-trained. A ResNet50 network with channel attention is used as the backbone for image feature extraction, combined with pixel feature extraction methods to obtain image representations. The Chinese RoBERTa model, combined with dictionary information fusion, is used to extract text sequence representations. After subspace mapping through a fully connected layer, the resulting concatenations are fed into a cross-modal feature fusion network. The joint embedding features are then used to predict similarity through a fully connected layer to determine a match. Hints are added to better enable text-based image search for pest and disease information. The trained model can retrieve matching data from a retrieval database for another modality from the input data as output. The entire implementation framework is as follows: Figure 2 As shown, specifically:

[0122] Step 1: Use a ResNet-50 model with added channel attention as an image encoder, and extract the pixel channel features of the image from the output feature map.

[0123] In this process, channel attention is added after each residual block of the Resnet-50 network. The input feature map with dimension (D×H×W) is compressed in the dimension of spatial feature D, and the two-dimensional feature map (H×W) of each channel is transformed into a real number by formula (6).

[0124] After obtaining the real-valued representation of each channel dimension feature, the real values ​​are mapped through a non-linear transformation of (FC-ReLU-FC). Finally, a Sigmoid(·) operation is used to map the real values ​​to the range (0~1), which serves as the attention weight set for each channel of the original feature map. The D-dimensional feature map of the original feature map is scaled with the corresponding weights in the attention weight set, that is, the weights after the Sigmoid(·) operation are weighted onto the features of each channel, which serves as a new feature map with channel attention added, and then added to the original features.

[0125] The pixel feature extraction method represents an image that, after removing the classification layer and adding a (3×3) AdaptiveAvgPool2d layer ResNet-50 network for feature extraction and multiple downsampling, is finally represented as a (2048×3×3) feature map. Because the deep learning network fully learns the channel features, and subsequent feature fusion requires processing the image into a sequence of features, the 2048-dimensional spatial features corresponding to each pixel in the (3×3) dimension are extracted from the feature map and concatenated in the order (1,1), (1,2), (1,3), (2,1), (2,2), (2,3), (3,1), (3,2), (3,3), processing the image features into a (9×2048) sequence of features. Adding channel attention and extracting its channel-dimensional features as image serialization representation significantly improves the quality of image features. The image encoder is pre-trained on the obtained image dataset using an image classification task, and the trained checkpoint is saved as the image encoder for this method.

[0126] The process of using the Chinese RoBERTa model as a text encoder and fusing domain dictionary information is as follows:

[0127] Since the Chinese RoBERTa model uses characters as tokens, and the pre-training data for the Chinese RoBERTa model may not contain much data in this domain, splitting words in the pest and disease domain into character-level representations is not accurate. Therefore, domain-specific nouns can be added to the word segmenter of the Chinese RoBERTa model, directly treating nouns as tokens, and then fine-tuned on the text data of this domain to allow the model to learn more accurate noun representations.

[0128] like Figure 3 As shown, the process of segmenting and training to obtain new tensors involves: collecting specialized terms in the field of pests and diseases, including various scientific names, common symptoms, commonly used equipment, and colloquialisms, and adding them to the segmentation table of the word segmenter in text preprocessing. During word segmentation, specialized terms in the text are treated as character-level tokens, and the embedding layer of the Chinese RoBERTa model is expanded to randomly generate new vector representations for each specialized term. The Chinese RoBERTa model is then fine-tuned using a text classification task on the collected pests and diseases text dataset to make it more accurate in representing the random vector representations of the expanded specialized terms.

[0129] We created a tokenizer and a Chinese RoBERTa model using BertTokinzer and BertModel provided by Huggging Face, and collected domain-specific terms using the API: Tokenizer.add_tokens(list).

[0130] Adding nouns to the word segmentation library allows the segmenter to avoid splitting nouns already present in the list within the text. Simultaneously, a new initialization vector representation is added to the embedding layer of the Chinese RoBERTa model for the newly added nouns, which can be achieved by calling the API. Specifically, `bert.resize_token_embedding(len(tokenize))` automatically creates random vectors for the new words added to the segmenter. Then, the Chinese RoBERTa model is fine-tuned on collected pest and disease domain texts using a text classification task to obtain a more accurate representation of the domain nouns added to the segmenter. Finally, the Chinese RoBERTa model is saved as a text encoder.

[0131] Before extracting text features using a text encoder, add two markers, '[cls]' and '[seq]', before and after the text respectively as classification features and segmentation vectors. Set the text length max_len to 50; text exceeding or falling short of 50 characters needs to be truncated and padded. The extracted features are represented as {'[cls]', v1, v2, ..., v 48 ,'[seq]'}, where each feature has 768 dimensions.

[0132] Step 2: Construct and preprocess a cross-modal image-text matching dataset for pests and diseases. This involves collecting images and text related to various pests and diseases through methods such as web crawling, book scanning, and field photography. The images primarily depict the symptoms of a particular pest or disease, while the text provides various descriptions related to that pest or disease, including a brief overview, symptoms, severity, and control measures. First, the image modality data is expanded using methods such as flipping and random cropping. The text modality data is expanded using methods such as synonym replacement, random insertion, and random deletion. During the construction of the (image-text pair) training set, matching methods are used, for example, for a certain category of image and text data: in, Indicates a set of images. This represents a text set. Then, by... and The matching methods constitute the dataset. As positive samples for cross-modal retrieval matching, mismatched image-sentence pairs are randomly shuffled as negative samples, while maintaining the same number of positive and negative samples to ensure the model learns the correct discrimination method. This data augmentation method can expand the training data and enhance the model's generalization ability.

[0133] Incorporating Prompt Engineering during model training aims to improve the accuracy of text-to-image search during the model's inference phase. The specific steps are as follows:

[0134] Add a text paradigm T to the training and test sets: "This is a picture of XX pest / disease," where XX represents the name of the pest / disease. In the text-based image retrieval process of the model inference, the user inputs the name of the pest / disease, which is placed in the XX position of T. Then, the entire text is used as the retrieval query, allowing direct retrieval of relevant images by inputting the name of the pest / disease.

[0135] Step 3: The collected image-text pairs are fed into the image and text encoders respectively to obtain their representations. The image representation is represented as U = {u1, u2, ..., u9}, with dimensions [9, 2048]. The text representation is represented as V = {'[cls]', v1, v2, ..., v 48 ,'[seq]'}, with dimensions [48,768].

[0136] Step 4: The extracted image and text features are subspaced by training two subspace matrices. The steps are to train two fully connected layers, and the image and text features are mapped to a common subspace through the fully connected layers, and the features in U and V are mapped to a length of 1024.

[0137] Step 5: Map the image features U i and text features V i The input sequence is combined according to the following rules as shown in formula (7).

[0138] A cross-modal feature fusion network based on the encoder structure of Transformer is constructed. Cross-modal attention between image pixels and language is learned through the multi-head self-attention of the encoder part of Transformer. The combined input sequence is passed through the encoder part, and the result is shown in Equation (8).

[0139] Furthermore, the text sequence features of the output results are... and image sequence features Its one-dimensional representation is obtained through attention pooling. and Then, predict its category using multi-class classification with FC-ReLU-FC combinations. The joint embedding features of location are used to predict the similarity between the input image and text by combining FC-Sigmoid features, thereby determining whether the input image and text match.

[0140] Attention pooling is achieved by learning the set of attention weights for each sequence through a fully connected layer and Softmax(·), denoted as AttU and AttV. Each sequence feature in the image feature U and text feature V is multiplied by its corresponding attention weight in AttU and AttV, and then summed across sequences to obtain the one-dimensional feature representations u and v for the image and text, respectively.

[0141]

[0142] Multi-class classification yields classification results for u and v. and

[0143] The model training is constrained by the loss functions in formulas (1) to (3), where λ in Loss is a hyperparameter, which can be referenced as 0.001.

[0144] Step 5: After training is complete, use as follows Figure 4 The process shown completes image-to-text retrieval or text-to-image retrieval:

[0145] A data retrieval database that collects images and texts of agricultural pests and diseases can selectively select texts from the database, including introductions to pests and diseases, symptoms, damage, and treatment methods. Tensor representations of the data are obtained in advance through a pre-trained model and stored in a .mat file. This saves time on feature extraction during the inference stage and improves retrieval efficiency.

[0146] When an image (or text) is input, an image encoder (or text encoder) is used to obtain its serialized features. These features are then combined with pre-extracted text (or images) from the data retrieval database and passed through a cross-modal feature fusion network to obtain the matching result, which is returned to the requester. The user inputs the name of the pest or disease they wish to retrieve. The complete text is obtained by placing the pest or disease name into the XX position of paradigm T, and then input into the model to match it with images in the retrieval database.

[0147] Based on the above description, this invention utilizes a ResNet50 network with channel attention to extract pixel channel features from images, and employs a Chinese RoBERTa model combined with dictionary information fusion to extract text sequence features. After subspace mapping via a matrix, both features are fed into a cross-modal feature fusion network. The joint embedded features of the output are used to calculate their similarity and determine a match. Hints are added to better enable text-based image search for pests and diseases. The trained model can then use the input data to identify and match another modality in the search database as the output.

[0148] Furthermore, this invention provides an agricultural pest and disease image-text retrieval system, which is used to implement the above-mentioned agricultural pest and disease image-text retrieval method. The system includes:

[0149] The data acquisition module is used to acquire the data to be retrieved. The data to be retrieved can be image data or text data of agricultural pests and diseases.

[0150] The feature extraction module is used to extract features from the data to be retrieved using a feature extraction model, thereby obtaining a feature sequence of the data to be retrieved.

[0151] The network building module is used to build cross-modal feature fusion networks.

[0152] The image and text retrieval module is used to determine the matching information of the feature sequence of the data to be retrieved in the data retrieval database using a cross-modal feature fusion network, and to use the matching information as the cross-modal retrieval result.

[0153] Furthermore, the present invention also provides an electronic device comprising: a memory and a processor.

[0154] Memory is used to store computer programs.

[0155] The processor is connected to the memory to retrieve and execute computer programs to implement the above-described method for image and text retrieval of agricultural pests and diseases.

[0156] Furthermore, when the computer program in the aforementioned memory is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory, random access memory, magnetic disks, or optical disks.

[0157] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0158] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.< / seq> < / cls> < / seq> < / cls> < / seq> < / cls> < / seq> < / cls>

Claims

1. A method for image and text retrieval of agricultural pests and diseases, characterized in that, include: Retrieve the data to be searched; The data to be retrieved is image data or text data of agricultural pests and diseases. The features of the data to be retrieved are extracted using a feature extraction model to obtain a feature sequence of the data to be retrieved. Construct a cross-modal feature fusion network; The cross-modal feature fusion network is used to determine the matching information of the feature sequence of the data to be retrieved in the data retrieval database, and the matching information is used as the cross-modal retrieval result; The process of constructing a cross-modal feature fusion network includes: Obtain image sample features and text sample features, and combine the image sample features and text sample features to obtain the input sample sequence; An embedding layer of length 2 is trained, and the embedding layer is added to the image sample features and text sample features respectively to obtain the concatenated sequence; Construct an encoder structure for a self-attention transform network, input the concatenated sequence into the encoder structure, and obtain the output result; the output result is represented as... =' ', , ,..., ,' ', ,..., ; - Indicates the output result Text sequence features, - Indicates the output result Image sequence features, It is obtained by feature fusion after adding the marker cls at the beginning of the text. A marker added to the end of the text. Indicates and The corresponding output results; The text sequence features and image sequence features of the output results are respectively processed by attention pooling to obtain one-dimensional representations of the text sequence features and one-dimensional representations of the image sequence features. Then, the categories of the one-dimensional representations of the text sequence features and the one-dimensional representations of the image sequence features are respectively obtained by multi-class prediction of nonlinear changes of FC-Relu-FC combination. Take the position in the output result as The joint embedding features are used to predict the similarity between images and text in the input sample sequence through FC-Sigmoid combination; the joint embedding features are generated by a feature fusion network. Matching information is determined based on the similarity and preset value, and cross-modal retrieval results are determined based on the matching information; The loss function used in the training of the cross-modal feature fusion network is: Loss=L 1 +λL 2； The first part of the loss function L 1 is: L 1 =- ； In the formula, m Indicates the first sample size. Indicates the first p Each sample matches the label. Indicates the first p The prediction results of sample similarity; Part Two: Loss Function L 2 is: L 2 =- ； In the formula, nums This indicates the second sample size, and 9 represents the 9 categories of pests and diseases. s This represents the s-th type of pest or disease. Indicates the first q The label category of each image sample. Indicates the first q Category labels for each text sample This represents the probability of the image sample predicted by the softmax function. This represents the probability of a text sample predicted by the softmax function.

2. The method for image and text retrieval of agricultural diseases and pests according to claim 1, characterized in that, The features of the data to be retrieved are extracted using a feature extraction model to obtain a feature sequence of the data to be retrieved, including: When the data to be retrieved is image data, an image encoder is used to extract the features of the image data, and the image sequence features are obtained by combining the pixel feature extraction method. When the data to be retrieved is text data, a text encoder is used to extract the features of the text data, and a dictionary information fusion method is combined to obtain text sequence features.

3. The method for image and text retrieval of agricultural diseases and pests according to claim 2, characterized in that, The construction process of the image encoder includes: A ResNet-50 network with channel attention was used as the original feature extractor, and the non-sequential features output by the original feature extractor were processed into sequential features using a pixel feature extraction method to construct an image feature extraction model. An image feature extraction model is trained on an image classification task using image sample data, and the trained image feature extraction model is used as the image encoder.

4. The method for image and text retrieval of agricultural diseases and pests according to claim 2, characterized in that, The construction process of the text encoder includes: The Chinese RoBERTa model was used to extract character-level features from the text sample data; A vocabulary list in the field of pests and diseases is collected and added to the word segmenter. During the text segmentation preprocessing, words appearing in the vocabulary list in the text sample data are used as character-level tokens and input into the Chinese RoBERTa model. Using the collected text, the Chinese RoBERTa model is adjusted on the text classification task to generate embedding vectors that meet the set requirements for the newly added words in the vocabulary, thus obtaining the adjusted Chinese RoBERTa model. The adjusted Chinese RoBERTa model is then used as the text encoder.

5. The method for image and text retrieval of agricultural diseases and pests according to claim 2, characterized in that, Before the text data is input into the text encoder, markers "" are added to the beginning and end of the text data respectively. <cls>"and mark" <seq> "; where the marked text data is:< / seq> < / cls> S2 = " <cls> ” + S1 + " <seq> ”；< / seq> < / cls> In the formula, S2 is the marked text data, and S1 is the original data.

6. An agricultural pest and disease image-text retrieval system, characterized in that, The system is used to implement the agricultural pest and disease image retrieval method as described in any one of claims 1-5; the system includes: The data acquisition module is used to acquire the data to be retrieved; the data to be retrieved is image data or text data of agricultural pests and diseases. The feature extraction module is used to extract features from the data to be retrieved using a feature extraction model, thereby obtaining a feature sequence of the data to be retrieved. The network building module is used to construct cross-modal feature fusion networks; The image and text retrieval module is used to determine the matching information of the feature sequence of the data to be retrieved in the data retrieval database using the cross-modal feature fusion network, and to use the matching information as the cross-modal retrieval result.

7. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, connected to the memory, is used to retrieve and execute the computer program to implement the agricultural pest and disease image retrieval method as described in any one of claims 1-5.

Images