Multi-modal optimization feature-based herbal plant small sample image classification system and generation method
Through the herbal plant small sample image classification system with multimodal optimization characteristics, the text and image encoder combined with the prompt module and output layer is used to solve the problem of sparse samples and low accuracy in the herbal plant image classification, and efficient herbal plant classification is achieved.
Patent Information
- Application Number
- CN202510507328.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-22
- Publication Date
- 2025-08-05
AI Technical Summary
Herbal plant image classification faces the problems of sparse samples, high manual labeling costs, low accuracy of model classification and insufficient performance of single modal models. Especially when collecting herbal data sets is difficult, it is difficult to achieve efficient classification.
A small sample image classification system for herbal plants based on multimodal optimization features is adopted, including text encoder, image encoder, prompt module and output layer. Multi-Head Attention, linear layer, batch normalization and Clip position embedding modules are used to improve feature extraction and classification accuracy through prompt learning and L2 regularization.
The accuracy of herbal plant image classification was significantly improved under small sample conditions, reaching 88.27%, which was better than the existing methods, and an increase of 2.17% to 6.34%.
Smart Images
Figure SMS_1 
Figure SMS_2 
Figure QLYQS_1
Abstract
Description
Technical Field
[0001] The present invention relates to the fields of computer vision, small-sample herbal plant image classification, and deep learning. Specifically, it relates to a method for classifying small-sample herbal plant images using multimodal optimization features, aiming to fully integrate image features with text features to effectively improve classification accuracy. Background Art
[0002] Few-shot learning is one of the many branches of meta-learning. Few-shot image classification involves performing relevant classification tasks with a small number of training categories. Compared to traditional image classification, few-shot image classification achieves relatively high accuracy while requiring fewer samples, offering promising development prospects for the refined classification of herbal plants.
[0003] The classic herbal plant classification method relies on professionals proficient in herbal classification knowledge to classify these herbs. However, this is extremely time-consuming and labor-intensive. These various factors have limited the development and inheritance of herbal classification technology. Therefore, the development of mechanized herbal classification technology is urgent. With the advancement of natural language processing technology, deep learning has driven progress in image classification and has been applied to this field. Existing classification methods can be divided into text unimodality, visual unimodality, and multimodality. Text unimodality is focused solely on the textual modality and utilizes textual features for classification; visual unimodality is focused solely on the visual modality and utilizes image features for classification; and multimodality utilizes information from multiple modalities, including text and vision, for classification. A typical approach is contrastive language-image pre-training (Clip), which simultaneously extracts text and image features. Language-image models, as an effective model in deep learning, have achieved significant breakthroughs in computer vision.
[0004] Although remarkable progress has been made in the field of herbal plant image classification, it still faces huge challenges. The main problems are: (1) Manual labeling of herbal plants is very costly and samples are very scarce; (2) When the number of samples is small, it is very difficult for the model to perform classification tasks and the accuracy is not high; (3) Different herbal samples vary greatly (for example, the flowers, stems, leaves, roots and fruits of different plants may all be different herbs), which increases the difficulty of model classification; (4) The accuracy of single-modality models in solving classification tasks is not high; (5) It is difficult to collect herbal datasets. Summary of the Invention
[0005] In order to solve the problems existing in the field of herbal plant image classification and further improve the classification accuracy, this embodiment proposes a method for classifying small sample images of herbal plants based on multimodal optimization features.
[0006] In order to achieve the above object, the present invention adopts the following technical solutions:
[0007] The embodiment of the present invention provides a system for classifying small sample images of herbal plants based on multimodal optimization features, comprising:
[0008] It consists of a text encoder, an image encoder, a prompt module and an output layer.
[0009] The purpose of the text encoder is to learn the optimal text feature information using prompt learning technology. It consists of Multi-HeadAttention, linear layer, batch normalization and position embedding module encapsulated in Clip.
[0010] The purpose of the image encoder is to learn the feature information of the input image using different model architectures. It consists of a Multi-HeadAttention, a linear layer, a batch normalization, and a position embedding module encapsulated in Clip.
[0011] The purpose of the prompt module is to extract the image feature information output by the image encoder and the text feature information output by the text encoder, map their feature information to the mapping layer under the same dimension through the softmax layer operation, combine them with the originally frozen text feature information through the position encoding operation, automatically generate new text feature information, and finally select the most appropriate text embedding. It consists of a position encoding module, batch normalization and a Clip-encapsulated mapping layer module.
[0012] The output layer is used to output the probabilities of different categories. It consists of a fully connected layer. In order to avoid overfitting and improve classification accuracy, the present invention uses L2 regularization in the output layer to optimize the system.
[0013] The present invention is implemented and comprises the following steps:
[0014] (1) A development platform for implementing a small sample image classification system for herbal plants based on multimodal optimization features was established. The hardware platform of the present invention is a server based on an i5-13600k CPU and an NVIDIA RTX4060Ti GAMING SLIM 16GB. The server has 16GB of video memory and 64GB of internal memory. The software platform is an Ubuntu 18.04 operating system with an operating environment of CUDA 11.3.0, Pytorch 1.10.2, and Python 3.8.
[0015] (2) Herbal plant image data division and reading. The private small sample herbal plant dataset AID was selected (including 4400 images in 44 categories, of which 3080 were used for training, 440 for verification, and 880 for validation). The input image size was 224×224, and the batch size was 16.
[0016] (3) Construct a classification system for herbal plant small sample images based on multimodal optimization features. The constructed classification system is as described above.
[0017] (4) Training and testing of the classification system. The present invention uses 70% of the samples for training, 10% for validation, and 20% for testing. During the training process, in order to improve the convergence speed and convergence ability of the model, the present invention uses a gradient descent learning rate to further optimize the system.
[0018] (5) Evaluation of the herbal plant classification system. To verify the performance of the classification system, the present invention used accuracy and error rates on the test set for evaluation. The present invention also compared the system with several classic methods in this field on the same AID dataset. The names of the various methods are shown in Table 1. The specific comparison results are listed in Table 2.
[0019] Table 1 Comparison method name
[0020]
[0021] Table 2 Evaluation of fusion results
[0022] Methods MaPLe CoOp CoCoOp VPT KNN Support Vector Machine ZeroShot-Clip MmFO(Our) Accuracy 86.10 81.93 52.70 78.20 69.75 67.93 2.2 88.27 BRIEF DESCRIPTION OF THE DRAWINGS
[0023] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0024] Figure 1 It is an overall block diagram of the system of the present invention;
[0025] Figure 2 is a schematic diagram of a text encoder module in the system of the present invention;
[0026] Figure 3 is a schematic diagram of an image encoder module in the system of the present invention;
[0027] Figure 4 Schematic diagram of the self-attention mechanism in the system of the present invention;
[0028] Figure 5 is a flow chart of the system generation method of the present invention;
[0029] Figure 6 is the heat map obtained by the system of the present invention on the AID dataset;
[0030] Figure 7 This is a bar chart comparing the classification results of the system of the present invention with those of other methods on the AID dataset; DETAILED DESCRIPTION
[0031] The present invention is further described in detail below with reference to the accompanying drawings and embodiments.
[0032] Figure 1 This is the overall block diagram of the system of the present invention. The system consists of a text and image input layer, a text and image encoder, a prompt module, a mapping connection layer, and an output layer.
[0033] Figure 2 It is a schematic diagram of the text encoder module in the system of the present invention. The text encoder module consists of an input encoder, a multi-head attention layer (Multi-HeadAttention), a regularization layer, a forward propagation and an output layer. It can extract text feature information and convert it into the form of a Tensor to lay the foundation for subsequent work. Among them, forward propagation is the process of data passing from the input layer through the hidden layer and finally reaching the output layer. It is mainly through continuous adjustment of weights and biases that the neural network can learn the characteristics of the input data and make correct predictions; the multi-head attention layer is mainly used to capture the dependencies between different positions in the input sequence, as well as the relationship between queries and key values. By learning different attention weights of multiple heads, the outputs of multiple heads are spliced or weighted summed to obtain the final representation, so that the model can better capture the complex relationships in the input sequence and improve the representation and generalization capabilities of the model.
[0034] Figure 3 It is a schematic diagram of the image encoder module in the system of the present invention. The image encoder module consists of an input encoder, a regularization layer, a multi-head attention layer (Multi-Head Attention), a multi-layer perceptron (MultilayerPerceptron abbreviated as: MLP) and an output layer. It can extract image feature information and convert it into the form of a Tensor tensor to prepare for subsequent work. Among them, MLP is composed of one or more hidden layers (i.e., intermediate layers). Each hidden layer contains multiple neurons, and each neuron is connected to all neurons in the previous layer. Each connection has a corresponding weight, and the connection between neurons is processed by an activation function. MLP can learn and represent more complex nonlinear function relationships through a combination of multiple hidden layers. The purpose is to adjust the connection weights between neurons through the back-propagation algorithm to minimize the loss function of the model. The calculation formula is as follows:
[0035] f(x)=G(b (2) +W (2) (s(b (1) +W(1) x)))Where G represents Softmax, W represents the weight on the corresponding connection, s represents the activation function, and b represents the bias.
[0036] Figure 4 : This is a schematic diagram of the self-attention mechanism module in the system of the present invention. The self-attention mechanism is a mechanism for subsequence data processing. It can capture the dependencies between any positions in the sequence data without being restricted by the length of the sequence. In the self-attention mechanism, each input element can be associated with other elements in the sequence by calculating the corresponding attention score. These attention scores represent the degree of association between each element and other elements, which can help the model better understand the interaction between the elements in the sequence. By taking a weighted average of the attention scores, a new representation of each element can be obtained, which contains the association information between it and other elements. The calculation formula is as follows:
[0037]
[0038] Where Q, K, and V represent the queries, keys, and values of the input sequence, respectively.
[0039] Figure 5 The flowchart of the system generation method of the present invention is divided into five steps: (1) building a system development platform; (2) dividing and reading the herbal plant data set; (3) constructing the herbal plant classification system; (4) training and testing the classification system; and (5) evaluating the classification system.
[0040] Figure 6 This is a heat map obtained by the system of the present invention on the AID dataset. In order to verify the performance of the system of the present invention, this example performs a heat map visualization display on the AID dataset. The x and y axes of the heat map represent the image features and text feature information respectively, and the z axis represents the similarity score information of the image feature information and the text feature information. The color band on the right side of the image represents the similarity score from top to bottom. The upper end of the color band (the lighter the color) indicates a high similarity score, and the lower end of the color band (the darker the color) indicates a low similarity score. Because the AID dataset is a small sample herbal plant dataset in a wild scene, it is different from the conventional high-definition herbal plant dataset under laboratory conditions. It has a large noise impact, resulting in low performance of conventional models. The herbal plant small sample image classification system based on multimodal optimization features of the present invention can perform classification tasks well, further illustrating the performance of this classification system.
[0041] Figure 7This is a bar chart comparing the classification results of the system of the present invention with those of other methods on the AID dataset. To validate the performance of the system of the present invention, this example was evaluated on the AID dataset. The overall accuracy of the present invention reached 88.27%, exceeding that of similar classical methods. The present classification method achieved a 2.17% improvement in overall accuracy over the best-performing MaPLe method compared on this dataset, and a 6.34% improvement over the baseline CoOp method. This comparison demonstrates the effectiveness of the present classification method.
Claims
1. The present invention is implemented by comprising the following steps: (1) Build a development platform for implementing a small sample image classification system of herbal plants based on multimodal optimization features; the hardware platform of the present invention is a server based on i5-13600k CPU and NVIDIA RTX4060Ti GAMING SLIM 16G; the server has a video memory of 16G and a memory of 64G; the software platform is an Ubuntu 18.04 operating system, and has an operating environment of CUDA 11.3.0, Pytorch 1.10.2, and Python 3.
8. (2) Herbal plant image data division and reading: The private small sample herbal plant dataset AID was selected (including 4400 images in 44 categories, of which 3080 were used for training, 440 for verification, and 880 for validation), and the input image size was 224×224, and the batch size was 16. (3) Construct a small sample image classification system of herbal plants based on multimodal optimization features; the constructed classification system is as described above. (4) Training and testing of the classification system; the present invention uses 70% of the samples for training, 10% of the samples for verification, and 20% of the samples for testing; during the training process, in order to improve the convergence speed and convergence ability of the model, the present invention uses a gradient descent learning rate to further optimize the system. (5) Evaluate the herbal plant classification system. To verify the performance of the classification system, the present invention uses the accuracy and error rate on the test set for evaluation. The present invention is also compared with some classic methods in this direction on the same AID dataset. The names of various methods are shown in Table 1. The specific comparison results are listed in Table 2. Table 1 Comparison method name Table 2 Evaluation of fusion results