A method for intelligent identification of cordyceps based on VIT

Abstract

The application discloses a kind of winter worm summer intelligent identification method based on VIT, comprising the following steps: S1. build cordyceps identification model;S2. user inputs winter worm summer image needing to be identified into cordyceps identification model;S3. cordyceps identification model is analyzed to the image of input, and the corresponding matching image and text data are output;S4. generate identification report;The cordyceps identification model specifically includes contrast learning module, word processing module, multi-modal encoding module, multi-modal decoding module.The application combines image and text information, realizes the deep fusion of image and text by contrast learning, multi-modal encoding and decoding etc., detailed description about winter worm summer can be generated by this network architecture, not only has category information, but also has the characteristic information about worm grass, compared with traditional identification method has higher identification precision and explainability.

Description

Technical Field

[0001] This invention relates to the field of machine vision technology, specifically to a VIT-based intelligent identification method for Cordyceps sinensis. Background Technology

[0002] The identification of wild and cultivated Cordyceps sinensis is a crucial issue in the Cordyceps industry, particularly valuable for protecting consumer rights and maintaining market order. Currently, Cordyceps identification relies primarily on manual methods, visually observing morphological characteristics and classifying grades by origin and weight. Overall, this experience-based method suffers from drawbacks such as low efficiency, low accuracy, over-reliance on personal experience, and inability to be applied on a large scale.

[0003] Currently, there is no mature, comprehensive, and efficient mechanism, method, or technology for identifying the quality of Cordyceps sinensis. For users, identifying different qualities of Cordyceps sinensis remains a significant challenge. In recent years, with the rapid development of various disciplines and computer technology, the use of various instruments and analytical methods has been continuously updated and improved. As a result, some new technologies for identifying traditional Chinese medicine have emerged, providing a solid technical and theoretical foundation for the identification of Cordyceps sinensis. Among these, identification based on the morphological characteristics of Cordyceps sinensis remains a commonly used method for quality identification.

[0004] The significant differences in appearance and details of Cordyceps sinensis at different growth stages and under varying environmental conditions make authentication exceptionally difficult. Traditional identification methods rely heavily on the experience and knowledge of professionals, which are not only susceptible to subjective influences but also inefficient and costly. Therefore, developing intelligent authentication technologies is urgently needed to address the major challenges faced by consumers and the industry in authenticating Cordyceps sinensis. Summary of the Invention

[0005] To address the above problems, this invention provides a VIT-based intelligent identification method for Cordyceps sinensis, comprising the following steps: S1. Construct a model for identifying Cordyceps sinensis; S2. The user inputs the image of the Cordyceps sinensis to be identified into the Cordyceps sinensis identification model; S3. The Cordyceps sinensis identification model analyzes the input image and outputs the corresponding matching image and text data; S4. Generate an identification report; The Cordyceps sinensis identification model specifically includes: a contrast learning module, a word processing module, a multimodal encoding module, and a multimodal decoding module; The contrastive learning module specifically involves: acquiring images of Cordyceps sinensis and related text information from multiple different perspectives, and using the CLIP model to encode the image and text data into the same vector space for comparison through contrastive learning; The word processing module specifically employs BPE to fuse image and text data and perform sequence encoding. The multimodal coding module specifically involves fusing and aligning the image and its corresponding text information through masked language modeling, multi-sentence understanding, and image-text matching. The multimodal decoding module is specifically designed to be trained by masked language modeling and to generate corresponding matching image and text data using beam search.

[0006] Furthermore, the contrastive learning module specifically includes the following sub-modules: Transformer encoding sub-module, VIT encoding sub-module, multi-view feature fusion sub-module, and contrastive learning sub-module; The Transformer encoding submodule introduces positional encoding to extract features and model global information from the input image, and to mark the relative / absolute positional information of the sequence. The VIT encoding submodule divides the image into a series of small blocks, and converts the small blocks into vectors through linear transformation, forming a sequence that is then passed to the Transformer encoding submodule for processing. The multi-view feature fusion submodule uses matrix multiplication to calculate the pixel correlation degree on feature maps from different perspectives. The contrastive learning submodule sets positive samples, anchor samples, and negative samples. By narrowing the distance between positive samples and anchor samples and increasing the distance between anchor samples and negative samples, similar text image pairs are brought closer in the feature space, while dissimilar text image pairs are kept further apart in the feature space.

[0007] Furthermore, the sequence encoding in the word processing module specifically includes: special markers, word embedding information, image feature information, location information, and category information; Among them, there are multiple special markers used to represent the overall features of the image; word embedding information is used to represent the global semantic information of the text and is fused with the semantic information of individual words; image feature information is used to add image features in the contrast learning module; position information is used to represent the position of the text and image; and category information is used to distinguish different sentences.

[0008] Furthermore, the formula for calculating the pixel correlation degree on feature maps from different perspectives in the multi-view feature fusion submodule is as follows: ; In the formula, s jiThis represents the correlation between position i in one viewpoint feature map and position j in another viewpoint feature map; N represents the number of pixels; P and Q are two feature maps from different viewpoints generated by convolution, where {P,Q}∈R C×N .

[0009] Furthermore, it also includes a dynamic queue. By setting up a dynamic queue, the feature vectors of the current training samples are input into the queue during the training of the contrastive learning module, and the corresponding number of feature vectors are removed from the queue. The dynamic queue is used to decouple the number of training samples from the number of negative samples.

[0010] Furthermore, the multimodal coding module specifically includes the following steps: The relationship between images and text is analyzed by modeling and learning using mask language. A single sequence of input text is randomly masked with a preset probability and replaced with a special sequence mask. The replaced words are restored using the image information and contextual text information input from the word processing module. Multi-sentence understanding is used to analyze and understand multiple sets of text representations of a single image, and to predict the multi-class probabilities of image-text pairs. Mutually matching image-text pairs are set as positive samples, while image-text pairs with high similarity but no match, selected from the contrastive learning module, are set as negative samples. The joint features of the image-text pairs are then calculated and analyzed.

[0011] Furthermore, the formula for calculating the joint features of image-text pairs in the multimodal coding module is: L ite =L mlm +L msc +L itm ; In the formula, L mlm Loss function for modeling masked languages, , Represents masked text. This represents the predicted probability of the mask word. This indicates that the input text is one-hot encoded, with a truth value of 1, and H represents the cross-entropy function; L msc For loss functions of multi-sentence understanding, T represents a combination sequence of multiple text sets. This represents the probability of predicting an image and multiple text pairs. H represents a label-based vector where positively correlated pairs have a probability of 1 and negatively correlated pairs have a probability of 0, and H represents the cross-entropy loss function. L itm This represents the similarity value between the image and the text. T represents the text to be judged. This represents the similarity value between image and text pairs. This is a k+1 dimensional vector based on the number of samples, where the probability of a positively correlated pair is 1, and the probability of a negatively correlated pair is 0. H represents the number of negative samples, and H represents the cross-entropy loss function.

[0012] Furthermore, the masked language modeling in the multimodal decoding module specifically generates the text description corresponding to the image by calculating the probability distribution of the combination of the image and its text description. The calculation formula is as follows: ; In the formula, Represents a text sequence. This indicates the probability of predicting the current token based on the previous token sequence. and All parameters are preset. =0.8, =0.2, This represents the similarity value of the image-text matching calculated in the multimodal coding module.

[0013] This invention provides a VIT-based intelligent identification method for Cordyceps sinensis, which has the following beneficial effects: This invention combines image and text information, and achieves deep fusion of images and text through methods such as contrastive learning, multimodal encoding and decoding. This network architecture can generate detailed descriptions of Cordyceps sinensis, including not only category information but also feature information about Cordyceps sinensis, which has higher recognition accuracy and interpretability compared to traditional recognition methods. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0015] Figure 1 A schematic diagram of the network architecture of the Cordyceps identification model provided by this invention; Figure 2 This is a schematic diagram of the network architecture of the VIT encoding submodule provided by the present invention; Figure 3 This is a schematic diagram of multi-view feature fusion provided by the present invention; Figure 4 This is a schematic diagram of multimodal feature fusion provided by the present invention. Detailed Implementation

[0016] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0017] The following detailed description of the implementation method of the present invention is in conjunction with the accompanying drawings. The description is only a partial embodiment and not all embodiments. For clarity, representations and descriptions unrelated to the present invention are omitted in the drawings and description.

[0018] To provide a clearer understanding of the technical features, objectives, and beneficial effects of this invention, the following detailed description of the technical solution is provided. Obviously, the described embodiments are only a portion of the embodiments of this invention, not all of them, and should not be construed as limiting the scope of implementation of this invention. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without inventive effort are within the protection scope of this invention.

[0019] This invention provides a VIT-based intelligent identification method for Cordyceps sinensis, comprising the following steps: S1. Constructing a Cordyceps sinensis identification model; S2. The user inputs an image of Cordyceps sinensis to be identified into the Cordyceps sinensis identification model; S3. The Cordyceps sinensis identification model analyzes the input image and outputs corresponding matching image and text data; S4. Generating an identification report.

[0020] The Cordyceps sinensis identification model specifically includes four modules: contrastive learning, word processing, multimodal encoder, and multimodal decoder. Figure 1 As shown.

[0021] The contrastive learning module consists of two image encoders, one text encoder, a multi-view feature fusion module, and a dynamic queue. It is trained using an image-text contrastive loss task to align the feature representations of images and text, obtaining the overall features of the image. CLIP is employed to encode images and text into the same vector space through contrastive learning, enabling the model to learn the correspondence between image content and corresponding text descriptions. By matching an input image of Cordyceps sinensis with a large amount of text descriptions, accurate textual descriptions are generated to determine the category of Cordyceps sinensis.

[0022] The CLIP model employs a dual-tower structure with Transformer and VIT encoding submodules, and achieves a deeper understanding of the relationship between image and text representations by aligning their representations. Images and text are represented as points in a shared vector space, ensuring that they have similar representations within the same vector space.

[0023] The Transformer encoding submodule is responsible for feature extraction and global information modeling of the input image. Since it lacks the ability to learn sequence order, position encoding is introduced into the Transformer encoding submodule to mark the relative / absolute position information of the sequence, thereby enabling the model to learn sequence order. ; ,in It indicates the first The dimension of each positional word vector. Indicates the first One word, This represents the dimension of the word vector. The input word vector is added to its positional encoding vector before being fed into the encoder layers.

[0024] The Transformer encoding submodule uses a self-attention mechanism to compute a correlation score between the representation of each position in the input sequence and the representations of all other positions, representing the dependency between any two positions in the input sequence. First, for each position in the input sequence... Calculate three different linear transformations to obtain the query vector. Key vector Sum value vector : ;in, , is a learnable weight matrix. Then, the query vector is calculated. and all key vectors The attention weights are obtained by taking the dot product between the two values ​​and applying the softmax function. : ,here This is a scaling factor used to prevent the gradient vanishing problem in the softmax function. Finally, based on the attention weights... value vector We perform a weighted summation to obtain the output of the self-attention mechanism. : .

[0025] To enable the model to learn the residual relationship between input and output more effectively and to accelerate the training process, the Transformer encoding submodule uses residual connections and layer normalization after the self-attention mechanism. Specifically, the output of the self-attention mechanism... With input Add them together and then perform layer normalization: Here, LN represents the layer normalization operation. After residual connections and layer normalization, the output is fed into a feed-forward network (FFN). A feed-forward network typically consists of two linear transformations and an activation function (such as ReLU): Where W1 and W2 are weight matrices, and b1 and b2 are bias vectors. The output of the feedforward neural network is then processed through residual connections and layer normalization to obtain the final encoder output.

[0026] The VIT encoding submodule segments the image into a series of small blocks, transforms these blocks into vectors using a linear transformation, and then passes this sequence to the Transformer encoding submodule for processing. Figure 2 As shown. Patch embedding is a crucial preprocessing step, responsible for transforming the input image into a sequence format that the model can process. First, the input image is segmented into a series of fixed-size patches. Assume the input image size is... ,in and It is the height and width of the image. This refers to the number of channels (usually 3, corresponding to the three RGB channels), and the size of each patch is... Then, the image will be segmented into There are 10 patches, and the size of each patch is 1. It was flattened into a shape with a length of The first one-dimensional vector is then transformed into a fixed-dimensional embedding vector through a linear transformation (i.e., a fully connected layer, or embedding layer). The dimension of this embedding vector is typically a hyperparameter of the model design, denoted as . Let the first The flattened vector of each patch is The weight matrix of the linear transformation is The bias vector is So, the first Embedding vector of each patch It can be calculated using the following formula: ,here This is the result after patch embedding, which will be used as input to the Transformer model.

[0027] For each patch, in addition to the patch embeddings described above, a corresponding position embedding vector is added. There are several ways to generate position embeddings; a common method is to use a learnable embedding matrix. Let the position embedding matrix be... Each row corresponds to a position embedding vector. Therefore, the first... The final embedding representation of each patch It can be calculated using the following formula: ,here, The position embedding matrix represents the first... Line, i.e., the first The position embedding vectors of each patch are then used. Finally, the embedding representations of all patches are combined into a sequence, which serves as input to the Transformer coding submodule. Through this patch embedding process, ViT successfully transforms images into a sequence format that the Transformer coding submodule can process, laying the foundation for subsequent processing such as self-attention mechanisms.

[0028] The multi-view feature fusion submodule calculates the pixel correlation degree on feature maps from different viewpoints through matrix multiplication, and this correlation degree can be used as a weight vector for another viewpoint map: ;in Used to measure the first feature in a view feature map The first position and the second viewpoint feature map The relevance of each position. Indicates the number of pixels. and The feature maps generated by convolution from two perspectives, correspondingly... The higher the similarity between pixels at two locations, the stronger the correlation between them.

[0029] like Figure 3 As shown, in order to reduce the generation of feature maps and , To reduce computational cost, the feature maps are first convolved to compress the channels. and This represents the width and height of the feature map, and then the feature map is changed accordingly. and the size of feature map B to and transpose matrix and Multiply and compute the attention map using a softmax layer. ,implement Attention map Matrix multiplication to generate feature maps Finally, let B and The final output is obtained by performing pixel-by-pixel addition. as follows: .here, It is initialized to 0, and it will gradually be assigned more weights through learning. Indicates the th in the low-level feature map Each channel location This represents the first feature in the top-level feature map. Each channel position. Final output. It is a weighted sum of feature maps from multiple perspectives, which can effectively preserve the features of each perspective.

[0030] The contrastive learning submodule aims to make similar text image pairs closer in the feature space and dissimilar text image pairs farther apart by narrowing the distance between positive and anchor samples and increasing the distance between anchor and negative samples. For a given... Given a pair of image and text, the text and image are input into a text encoder and an image encoder, respectively, to extract the text feature vector. and image feature vectors For each image-text pair, cosine similarity is calculated, and softmax is used to calculate the text-to-image and image-to-text cross-entropy losses: ; ;in It is a learnable temperature parameter. and One-hot encoding is performed based on the labels, where the probability of a positively correlated pair is 1, and the probability of a negatively correlated pair is 0. This represents the cross-entropy loss function.

[0031] The contrastive learning module also includes a dynamic queue. By setting up a dynamic queue, the feature vectors of the current training samples are input into the queue during the training of the contrastive learning module, and a corresponding number of feature vectors are removed from the queue. The dynamic queue decouples the number of training samples from the number of negative samples, prevents model overfitting, and improves training efficiency.

[0032] The word processing module uses BPE to fuse image and text data and perform sequence encoding. Each sequence is prefaced by two special markers, [img] block [Image] can be added optionally to represent pixel-level information of the image. block[] is used to represent the overall features of an image. The number of blocks depends on the number of input images. Each image feature corresponds to a special label, and the end label of each sentence is denoted as ([end]). The feature representation of sequence encoding is obtained by adding four parts: word embedding information, image feature information, position information, and category information. The value of word embedding information is automatically learned during model training and is used to represent the global semantic information of the text and is fused with the semantic information of individual words. Image feature information is used to add image features in the image-text comparison learning module. Position information is used to represent position information. Category information is a learned embedding used to distinguish different sentences.

[0033] The multimodal encoding module is based on the Transformer Encoder architecture and is trained using three tasks: masked language modeling, multi-sentence understanding, and image-text matching. This enhances the model's overall understanding of image information and fuses and aligns the image with its corresponding text. The complete training objective of the multimodal encoding module (image-text encoder) is as follows: .

[0034] Fusing multimodal features, such as Figure 4 As shown, its input X is a matrix composed of the image feature vector and the text feature vector, that is: ;in yes dimensional image feature vectors yes dimensional text feature vectors, yes An input vector of dimension . Performing a self-attention operation yields the following calculation formula: .

[0035] It is a block matrix, so the self-attention operation can be divided into the following four blocks and performed simultaneously: 1) In and Self-attention operations between image feature vectors are self-attention operations between image feature vectors; 2) In and Self-attention operations between text and images simulate the cross-attention operation between text and images; 3) In and Self-attention operations between text feature vectors refer to self-attention operations between text feature vectors; and Self-attention operations between images and text are cross-attention operations between images and text.

[0036] Masked language modeling (MLM) aims to learn the relationship between images and text, achieving fine-grained alignment of image and text information. It involves randomly masking individual sequences of the input text with a certain probability and replacing them with a special sequence [mask]. The replaced words are then reconstructed using image information and contextual text information from the word processing module. Simultaneously, in the later stages of model training, entire words of the input text are randomly masked with the same probability for training purposes, allowing the model to better learn image information. The loss function for training the MLM task is: Among them, set This represents the masked text. This represents the model's predicted probability for the mask word. This indicates that the input text is one-hot encoded, and the corresponding true value is 1. This represents the cross-entropy function.

[0037] Multi-sentence comprehension (MSC) refers to the ability of a model to learn multiple text representations of a single image from different perspectives or angles. MSC tasks enhance the multimodal encoder's understanding of images and improve its robustness. The input to the MSC task consists of multiple text representations and corresponding image information, specially labeled. The final output features are used as joint features for image-text pairs. A fully connected layer is then applied to the output of these joint features to predict the multi-class probabilities of the image and multiple text pairs. The loss function for the MSC task is: T represents a combination sequence of multiple text sets. This represents the probability of predicting an image and multiple text pairs. Let H represent a multi-vector based on labels, where the probability of a positively correlated pair is 1 and the probability of a negatively correlated pair is 0, and H represents the cross-entropy loss function.

[0038] Image-text matching (ITM) aims to determine the relevance of a pair of images and text, aligning information from these two modalities at a coarser level. During training, matching image-text pairs are designated as positive samples, while a text-image comparison learning task is used to filter out highly similar but mismatched image-text pairs as negative samples. These are then labeled with special tags. The final output features are used as the joint features of the image-text pair. A fully connected layer and a sigmoid function are then applied to the output of these joint features to predict the similarity value between the image and text pairs. Litm represents the similarity value of the image-text matching. T represents the text to be judged. This represents the similarity value between image and text pairs. This is a k+1 dimensional vector based on the number of samples, where the probability of a positively correlated pair is 1, and the probability of a negatively correlated pair is 0. H represents the number of negative samples, and H represents the cross-entropy loss function.

[0039] The multimodal decoder module consists of a Transformer Decoder block, which uses a masked language modeling task to generate textual descriptions of cordyceps categories. It is trained using the masked language modeling task and generates text using a beam search method, thus completing the image-to-text generation task.

[0040] Causal language modeling (CLM) aims to generate textual descriptions of a given image. Multimodal decoders are based on the unidirectional structure of the Transformer Decoder, where each token can only see information about tokens preceding it, not those following it. The training objective of the CLM task is to predict the current token based on all previous token information, selecting the probability of word combinations by calculating probability distributions. The loss function for the CLM task is: ;in This represents the input text sequence. This indicates that the module predicts the probability of the current token based on the previously input tokens. , and All are preset adjustable parameters. This represents the similarity value calculated by the ITM task in the encoder module.

[0041] The purpose of beam search is to find the optimal sequence output by the model. Its core idea is to maintain a candidate sequence set of size k and continuously expand these sequences until the final goal is reached or the search ends. When the search space is large, it can effectively find an approximate optimal solution, while maintaining relatively fast search speed. Beam search favors shorter results over longer ones. Through length normalization, it can effectively balance the scores of sequences of different lengths, generating more reasonable longer sequences, making the scoring fairer and more accurate, and improving the generation quality. The score is calculated as follows: ;in Represents the model's state The predicted score, Indicates the sequence length. Set to 4, Set to 0.7, It is a hyperparameter used to adjust the effect of length on the score.

[0042] After inputting the cordyceps image into the model, it first enters the image-text comparison module to output the cordyceps image feature information, then undergoes information fusion through the word processing module, and is then input into the multimodal decoder module for text generation. Finally, a beam search algorithm is used to filter out multiple sets of matching text. The generated multiple sets of text are then fed into the multimodal encoder module to calculate similarity values ​​with the image. The text with the highest similarity value is selected for reinforcement training. The KL divergence between the original model and the reinforcement training model is calculated to prevent over-optimization. ,in This represents the input text sequence. This represents the probability of a module predicting a token based on self-supervised training. This indicates that the pre-trained module predicts the probability of the token. , , =0.9, and These are preset, adjustable parameters. This represents the similarity value calculated for the ITM task in the multimodal encoder module.

[0043] This invention combines image and text information, and achieves deep fusion of images and text through methods such as contrastive learning, multimodal encoding and decoding. This network architecture can generate detailed descriptions of Cordyceps sinensis, including not only category information but also feature information about Cordyceps sinensis, which has higher recognition accuracy and interpretability compared to traditional recognition methods.

[0044] This invention selects features from multiple perspectives for image encoding and fuses features from different perspectives. By combining information from different perspectives, the model can obtain richer structural information about Cordyceps, which helps improve the model's recognition accuracy. At the same time, multi-view feature extraction can help the model learn more generalized feature representations, enabling it to maintain good recognition performance when faced with new and unseen perspectives.

[0045] This invention employs strategies such as dynamic queues and bundle search to improve training efficiency and the quality of cordyceps category description generation, thereby balancing model performance and computational resource consumption.

[0046] In this invention, through modular design, the network structure is divided into modules such as contrastive learning, word processing, multimodal encoder and multimodal decoder. Each module is responsible for different functions, making the design of the whole system reasonable and easy to understand.

[0047] The above description is merely a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the forms disclosed herein and should not be construed as excluding other embodiments. It can be used in various other combinations, modifications, and environments, and can be altered within the scope of the concept described herein through the above teachings or related technologies or knowledge. Modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention should be within the protection scope of the appended claims.

Claims

1. A VIT-based intelligent identification method for Cordyceps sinensis, characterized in that, Includes the following steps: S1. Construct a model for identifying Cordyceps sinensis; S2. The user inputs the image of the Cordyceps sinensis to be identified into the Cordyceps sinensis identification model; S3. The Cordyceps sinensis identification model analyzes the input image and outputs the corresponding matching image and text data; S4. Generate an identification report; The Cordyceps sinensis identification model specifically includes: a contrast learning module, a word processing module, a multimodal encoding module, and a multimodal decoding module; The contrastive learning module specifically involves: acquiring images of Cordyceps sinensis and related text information from multiple different perspectives, and using the CLIP model to encode the image and text data into the same vector space for comparison through contrastive learning; The word processing module specifically employs BPE to fuse image and text data and perform sequence encoding. The multimodal coding module specifically involves fusing and aligning the image and its corresponding text information through masked language modeling, multi-sentence understanding, and image-text matching. The multimodal decoding module is specifically designed to be trained by masked language modeling and to generate corresponding matching image and text data using beam search.

2. The intelligent identification method for Cordyceps sinensis based on VIT according to claim 1, characterized in that, The contrastive learning module specifically includes the following sub-modules: Transformer encoding submodule, VIT encoding submodule, multi-view feature fusion submodule, contrastive learning submodule; The Transformer encoding submodule introduces positional encoding to extract features and model global information from the input image, and to mark the relative / absolute positional information of the sequence. The VIT encoding submodule divides the image into a series of small blocks, and converts the small blocks into vectors through linear transformation, forming a sequence that is then passed to the Transformer encoding submodule for processing. The multi-view feature fusion submodule uses matrix multiplication to calculate the pixel correlation degree on feature maps from different perspectives. The contrastive learning submodule sets positive samples, anchor samples, and negative samples. By narrowing the distance between positive samples and anchor samples and increasing the distance between anchor samples and negative samples, similar text image pairs are brought closer in the feature space, while dissimilar text image pairs are kept further apart in the feature space.

3. The intelligent identification method for Cordyceps sinensis based on VIT according to claim 2, characterized in that, The sequence encoding in the word processing module specifically includes: special markers, word embedding information, image feature information, location information, and category information; Among them, there are multiple special markers used to represent the overall features of the image; word embedding information is used to represent the global semantic information of the text and is fused with the semantic information of individual words; image feature information is used to add image features in the contrast learning module; location information is used to represent the location of the text and image; and category information is used to distinguish different sentences.

4. The intelligent identification method for Cordyceps sinensis based on VIT according to claim 2, characterized in that, The formula for calculating the pixel correlation degree on feature maps from different perspectives in the multi-view feature fusion submodule is as follows: ； In the formula, s ji This represents the correlation between position i in one viewpoint feature map and position j in another viewpoint feature map; N represents the number of pixels; P and Q are two feature maps from different viewpoints generated by convolution, where {P,Q}∈R C×N .

5. The intelligent identification method for Cordyceps sinensis based on VIT according to claim 2, characterized in that, It also includes a dynamic queue, which, when training the contrastive learning module, inputs the feature vector of the current training sample into the queue and removes the corresponding number of feature vectors from the queue, thereby decoupling the number of training samples from the number of negative samples.

6. The intelligent identification method for Cordyceps sinensis based on VIT according to claim 1, characterized in that, The multimodal coding module specifically includes the following steps: The relationship between images and text is analyzed by modeling and learning using mask language. A single sequence of input text is randomly masked with a preset probability and replaced with a special sequence mask. The replaced words are restored using the image information and contextual text information input from the word processing module. Multi-sentence understanding is used to analyze and understand multiple sets of text representations of a single image, and to predict the multi-class probabilities of image-text pairs. Mutually matching image-text pairs are set as positive samples, while image-text pairs with high similarity but no match, selected from the contrastive learning module, are set as negative samples. The joint features of the image-text pairs are then calculated and analyzed.

7. The intelligent identification method for Cordyceps sinensis based on VIT according to claim 6, characterized in that, The formula for calculating the joint features of image-text pairs in the multimodal coding module is: L ite =L mlm +L msc +L itm ; In the formula, L mlm Loss function for modeling masked languages, , Represents masked text. This represents the predicted probability of the mask word. This indicates that the input text is one-hot encoded, with a truth value of 1, and H represents the cross-entropy function; L msc For loss functions of multi-sentence understanding, T represents a combination sequence of multiple text sets. This represents the probability of predicting an image and multiple text pairs. H represents a label-based vector where positively correlated pairs have a probability of 1 and negatively correlated pairs have a probability of 0, and H represents the cross-entropy loss function. L itm This represents the similarity value between the image and the text. T represents the text to be judged. This represents the similarity value between image and text pairs. This is a k+1 dimensional vector based on the number of samples, where the probability of a positively correlated pair is 1, and the probability of a negatively correlated pair is 0. H represents the number of negative samples, and H represents the cross-entropy loss function.

8. The intelligent identification method for Cordyceps sinensis based on VIT according to claim 7, characterized in that, The masked language modeling in the multimodal decoding module specifically generates the text description corresponding to the image by calculating the probability distribution of the combination between the image and its text description. The calculation formula is as follows: ； In the formula, Represents a text sequence. This indicates the probability of predicting the current token based on the previous token sequence. and All parameters are preset. =0.8, =0.2, This represents the similarity value of the image-text matching calculated in the multimodal coding module.

Images