Multi-mode Longjing tea leaf intelligent grading method integrating five-dimensional evaluation scale

Abstract

The invention discloses a multi-mode Longjing tea leaf intelligent grading method integrating five-dimensional evaluation scales. The method comprises the following steps: collecting Longjing tea leaf samples; the method comprises the following steps: collecting an appearance picture data set, a soup color picture data set, a leaf bottom picture data set, an aroma data set and a taste data set of Longjing tea leaf samples, performing preprocessing, and dividing the data sets into a training set, a test set and a verification set; constructing a multi-modal Longjing tea leaf grading model, and training, testing and verifying the multi-modal Longjing tea leaf grading model through the training set, the test set and the verification set to obtain an optimal multi-modal Longjing tea leaf grading model; and grading the Longjing tea leaves through the optimal multi-mode Longjing tea leaf grading model. According to the method, the intelligent grading model is constructed by using the digital feature vectors through a multi-modal fusion and contrast learning method, and finally, a visual green tea quality evaluation system is formed, so that the aims of improving the evaluation consistency and reducing human subjective errors are achieved.

Description

Technical Field

[0001] The present invention relates to the field of tea quality assessment, and in particular to a multimodal Longjing tea intelligent grading method integrating five-dimensional evaluation scales. Background Art

[0002] In the vast tea market, where whole-leaf tea is often sold according to its grade, accurate and consistent grading is crucial. Longjing green tea, renowned for its quality, is categorized into different grades based on appearance, aroma, flavor, tea soup color, and leaf characteristics. These grades influence consumer purchasing choices and significantly impact the economic value of tea, highlighting the importance of precise quality assessment.

[0003] Traditionally, tea grading relies primarily on manual sensory evaluation, a process prone to subjectivity and inconsistency. While expert tea tasters play an important role in maintaining quality standards, the shortage of skilled evaluators and the inherent limitations of sensory analysis highlight the need for more objective, efficient, and reliable grading methods.

[0004] Recent advances in artificial intelligence (AI) offer promising solutions for data classification, and this technology is widely used in a variety of fields, including drug discovery, intelligent diagnosis, and personalized agriculture. Studies have investigated machine learning algorithms and computer vision systems to study specific tea characteristics and tea polyphenol-based drug mechanisms. However, these methods often focus on isolated features rather than the holistic evaluation used by reviewers in sensory evaluation. A Chinese patent with publication number CN118797413A discloses a method for comprehensive evaluation of tea grades based on deep learning coupled with a multimodal fusion sensor for sensory quality. This invention is relatively comprehensive in terms of overall evaluation, including color, aroma, taste, and shape, and uses an electronic nose, electronic eye, electronic tongue, and near-infrared spectroscopy to collect tea information. However, compared to tea evaluation, this invention still lacks the evaluation of the two dimensions of leaf base and tea soup color, which is an important evaluation link for the value of tea. Summary of the Invention

[0005] The purpose of the present invention is to provide a multimodal Longjing tea intelligent grading method integrating five-dimensional evaluation scales to solve the problems raised in the above background technology.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] A multimodal Longjing tea intelligent grading method integrating five-dimensional evaluation scales, including:

[0008] S1. Collect various Longjing tea samples of different prices and grades;

[0009] S2. Collect a dataset of Longjing tea samples' appearance images, tea color images, tea leaf bottom images, aroma data, and flavor data. Preprocess the datasets separately, and divide the datasets into a training set, a test set, and a validation set.

[0010] S3. Build a multimodal Longjing tea grading model, train, test, and validate the multimodal Longjing tea grading model using a training set, a test set, and a validation set to obtain the optimal multimodal Longjing tea grading model;

[0011] S4. Grade Longjing tea using the optimal multimodal Longjing tea grading model.

[0012] Furthermore, the S2 includes:

[0013] Collecting appearance image datasets: We took images of tea samples before brewing and enhanced the data using data augmentation methods.

[0014] Collect tea soup color image datasets and tea leaf bottom image datasets: Brew tea samples, remove the tea leaves from the tea soup, and then take pictures of the tea soup color and tea leaf bottom images;

[0015] Collecting aroma datasets: extracting volatile compounds from brewed tea samples and identifying the types and concentrations of volatile compounds;

[0016] Collecting a flavor dataset: Identifying the types and concentrations of metabolites in brewed tea.

[0017] Furthermore, the multimodal Longjing tea grading model includes an input module, a deep convolution module, a contrastive learning module, an integrated learning module and an output module;

[0018] The input module pre-processes the input image, pre-processes the chemical analysis data, and divides and directs the input data downstream;

[0019] The deep convolutional module first uses the SimCLR self-supervised visual representation learning framework to pre-train on unlabeled data, then migrates to the ResNet convolutional model to classify the tea appearance images based on seven sub-comment dimensions, and converts the tea appearance images into comments and scores based on linear mapping;

[0020] The contrastive learning module uses CLIP combined with a buffer built based on a few-shot dataset to perform zero-shot learning prediction on the tea image category, and builds a multimodal ensemble learning supervisor based on this CLIP;

[0021] The integrated learning module uses a multimodal supervisor to dynamically adjust the training process of XGBoost and MLP, and uses the adjusted trained models XGBoost and TCSMLP to perform regression prediction on aroma data, flavor data, soup color data, and leaf bottom data, where TCSMLP is the MLP trained by CLIP supervision;

[0022] The output module linearly weights the five human evaluation scale scores, realizes text visualization of the tea quality evaluation process, and outputs the tea quality evaluation results.

[0023] Furthermore, the SimCLR self-supervised visual representation learning framework of the multimodal Longjing tea grading model learns better visual representations through contrastive learning. Defined as:

[0024]

[0025] Among them, exp represents the natural exponential function with e as the base, which is used to scale the similarity value to the positive range; sim represents the similarity calculation function, z i is the vector obtained by L2 normalization of a single sample. L2 means L2 norm normalization of the feature vector, that is, dividing each vector by its own Euclidean norm to normalize its modulus to 1, thereby projecting all vectors onto the unit sphere, which is convenient for subsequent calculation of cosine similarity; j is with z i Feature representation of paired positive samples, z k Indicates that it is used with z i The kth sample feature among all compared samples (including positive and negative samples), τ is the temperature coefficient, which is used to control the sharpness of the probability distribution. When τ is smaller, the similarity difference is amplified, and the model pays more attention to those negative samples with average similarity but misclassified. When τ is larger, the distribution is smoother, and the model is less sensitive to similarity differences. N is the total number of samples. The similarity calculation function sim is specifically defined as follows:

[0026]

[0027] Among them, sim(z i ,z j ) represents sample z i With sample z j The cosine similarity between i and zj are the feature vectors of the two samples respectively; z i *z j represents their dot product; ‖z i ‖ and ‖z j‖ are the L2 norms of the two vectors, which are used for normalization to ensure that the similarity value range is between [-1,1].

[0028] Furthermore, the ResNet convolutional model of the multimodal Longjing tea grading model includes a core structure residual block and a residual connection operation, wherein the output formula of each core structure residual block is:

[0029] h l+1 =h l +σ1(W l *h l +b l )

[0030] Among them, h l represents the input of the lth layer, h l+1 Represents the transformation after convolution and activation function, W l is the weight of the lth layer, σ1 is the ReLU activation function, b l is bias;

[0031] The cross entropy loss function of the objective function of the ResNet convolutional model for:

[0032]

[0033] Where N is the number of samples, C is the number of categories, and y i,c is the true label of the i-th sample in the c-th category, p i,c It is the predicted probability that the i-th sample belongs to the c-th class after the output layer is activated by the Softmax function. The last layer of the ResNet convolutional model outputs the classification probability through the fully connected layer, and the cross entropy loss drives the parameter update.

[0034] Furthermore, the MLP part of the TCSMLP of the multimodal Longjing tea grading model is a feedforward neural network consisting of an input layer, a hidden layer, and an output layer. The calculation formula of each hidden layer is:

[0035] h (l) =σ2(W (l) h (l-1) +b (l) )

[0036] Among them, l represents the lth layer in the network, usually numbered from 1, h (l-1) represents the input of the l-1 layer, W (l) and b (l) are the weight and bias of the lth layer, σ2 is the ReLU activation function, and for the input layer, h (l-1)As the model input, for the output layer, the regression task directly calculates the weights, biases and hidden layer outputs to obtain the final output without going through the activation function;

[0037] The MLP objective function of the TCSMLP of the multimodal Longjing tea grading model is designed based on the mean square error, and the specific formula is:

[0038]

[0039] in, Represents the mean square error (MSE) loss function used by the MLP network in the TCSMLP module, which is used to measure the square error between the model prediction value and the true value. n is the total number of samples, and y i is the true value of the i-th sample, is the model’s predicted value for the i-th sample, represents the square of the prediction error of the sample, is the normalization coefficient used to calculate the average squared error of all samples.

[0040] Furthermore, the XGBoost of the multimodal Longjing tea grading model is an ensemble learning method based on gradient boosting, and its objective function for:

[0041]

[0042] Among them, loss is the loss function, Ω(f k ) is the regularization term, f k represents the kth decision tree, K is the number of decision trees, n is the number of samples, y i is the true sample label of the i-th sample, is the model prediction value of the i-th sample;

[0043] The regularization term is used to control model complexity and prevent overfitting. It is specifically defined as:

[0044]

[0045] Among them, T is the number of leaf nodes in the tree, γ penalizes the number of leaf nodes, ω is the weight of the leaf node, and λ controls the weight amplitude.

[0046] Furthermore, the working process of the multimodal Longjing tea grading model includes:

[0047] Step 1: Construct a multimodal ensemble learning supervisor, including: building a cache based on the collected small sample image-text dataset, using the pre-trained CLIP VisualEncoder module to embed the image data of each category, and constructing the cache index as shown in the formula:

[0048] Key=VisualEncoder(I k )∈R NK×C

[0049] Among them, Key represents the key vector in the image buffer, VisualEncoder is the visual encoder module in the CLIP model, which is used to extract features from image samples, R represents the real number domain, indicating that the value of Key is in the real number range, and I k is the image input of the kth category or sample, N represents the number of categories, the number of rows NK represents a total of N categories × K image samples per category, and a total of NK samples; C represents the embedding feature dimension output by the visual encoder, that is, the vector dimension after each image is encoded;

[0050] Use OneHot to perform unique heat encoding on the text label corresponding to each category image, and construct the cache key value as shown below:

[0051] Vdlue=OneHot(L k )∈R NK×N

[0052] Among them, Value represents the value vector in the image buffer, OneHot represents the one-hot encoding module, which is used to encode the category label. k is the text label or class label corresponding to the k-th image sample, N represents the number of categories, K represents the number of images of each category participating in the buffer construction, and the encoding result is a matrix of dimension NK×N;

[0053] Query = VisualEncoder(I)∈R 1×C

[0054] Where Query represents the query vector obtained after the image to be identified is extracted by the visual encoder, I represents the input image code; 1 indicates that there is only one input image sample at present, and C represents the embedding feature dimension output by the visual encoder, so the final output is an embedding vector with a dimension of 1×C;

[0055] For the image to be predicted, the image to be predicted is input into VisualEncoder to obtain the corresponding embedded coding vector as the query value; the similarity between the cache index and the query value is calculated using the following formula, and the key value is taken from the cache for decoding based on the similarity:

[0056]

[0057] Among them, Similarity represents the similarity between the query vector and the key vector, β is the scaling factor used to adjust the range of the similarity function, and Key TIndicates the transposition of Key; Query*Key T is the dot product similarity; therefore, the dimension of the final output Similarity is 1×N×K;

[0058] Based on the decoding results, the classifier in the pre-trained CLIP model is weighted and combined to make zero-shot predictions on the image data as the corresponding labels of the ensemble learning supervisor:

[0059] Logits=α*Similarity*Value+Query*W T

[0060] Among them, Logits represents the final predicted output vector, which is used to judge the corresponding label in supervised learning. α is the regularization coefficient, which is used to balance the contribution of similarity-weighted labels and the original CLIP classifier. T It is the weight transposed matrix of the CLIP model classifier, which is used to map the query to the prediction score of each category;

[0061] Step 2. The input module preprocesses the tea appearance image, specifically including: cropping, filtering, and channel normalization of the tea appearance image, cropping, normalization, color feature extraction, and texture feature extraction of the tea soup image and leaf bottom image; preprocessing the aroma dataset and flavor dataset, specifically including cleaning, screening, filling, and standardization of the dataset; randomly pairing the tea appearance image dataset, tea soup image dataset, leaf bottom image dataset, aroma dataset, and flavor dataset according to categories in a rectangular distribution; and importing the data pairs into the downstream sub-model for processing according to different modes, specifically inputting the appearance image dataset into Resnet, the leaf bottom image dataset, aroma dataset, and flavor dataset into TCSMLP, and the tea soup image dataset into XGBoost.

[0062] Step 3: SimCLR in the deep convolutional module first performs self-supervised pre-training on a dataset of various tea appearance comment dimensions. It then transfers the learned representations to ResNet-18 to classify tea appearance images into seven sub-comment dimensions: straightness, smoothness, tenderness, greenness, glossiness, color uniformity, and completeness. It then performs a linear conversion from comment vectors to scores based on a comment-score mapping table.

[0063] Step 4: The integrated learning module performs regression fitting on the soup color and leaf bottom appearance image data, aroma data, and flavor data. Specifically, this includes establishing a dynamic adjustment mechanism for the regularization term of network training based on the positive and negative feedback of the multimodal supervisor labels during the training process to optimize the training of the MLP to obtain the TCSMLP. The dynamic adjustment mechanism is defined as follows:

[0064] Let the total objective function of the TCSMLP model be where is the task loss, which is the mean squared error used in MLP regression, is the regularization term, λ(t) is the dynamic adjustment coefficient, which is controlled by the feedback signal of the supervisor; the construction method of the dynamic coefficient is defined as follows:

[0065]

[0066] where, λ0 represents the initial regularization coefficient, γ represents the feedback sensitivity coefficient, M represents the number of regularization coefficient influence factors, β m represents the weight of the m-th influence factor, F m (t) represents the feedback function of the m-th influence factor at time step t;

[0067] When m = 1, the definition of the feedback function F m (t) is as follows:

[0068]

[0069] When m = 2, the definition of the feedback function F m (t) is as follows:

[0070]

[0071] where, represents the value of the loss function of TCSMLP at time step t, sign is the standard signal function, Label supervisor represents the label recorded by the supervisor, Label pred represents the label currently output by the model, sign' is the signal function based on boolean values; when increases or the output of TCSMLP is inconsistent with the label recorded by the supervisor, F m (t) takes a positive value to enhance regularization, otherwise F m (t) takes a negative value to reduce regularization;

[0072] Based on the TCSMLP trained by optimization, regression prediction is performed on aroma data and taste data, and regression prediction is performed on soup color picture data based on XGBoost;

[0073] Step 5, the output module performs linear weighting on the model results of five human evaluation scales. The scoring ratios of appearance, soup color, aroma, taste and leaf bottom are p1, p2, p{3}, p4 and p5 respectively, 0.20 < p1 < 0.30, 0.10 < p2 < 0.20, 0.20 < p3 < 0.30, 0.20 < p4 < 0.35, 0.05 < p5 < 0.15, a + b + c + d + e = 1, as shown in the following formula:

[0074] Result=p1*Appearance+p2*SoupColor+p3*Aroma+p4*Taste+p5*InfusedLeaf

[0075] Among them, Result is the comprehensive score of tea, Appearance is the appearance score, SoupColor is the soup color score, Aroma is the aroma score, Taste is the taste score, and InfusedLeaf is the leaf bottom score.

[0076] Furthermore, in step 5, the correspondence between the total score and the grade of the tea is: C<88≤B<90≤A.

[0077] Furthermore, the correspondence between the scores of each single dimension and the quality level in step 5 is:

[0078] Appearance: C<88≤B<91≤A;

[0079] Aroma: C<88≤B<92≤A;

[0080] Taste: C<88≤B<91≤A;

[0081] Soup color: C≤87<B≤89<A;

[0082] Leaf bottom: C≤87<B≤89<A.

[0083] Compared with the prior art, the present invention has the following beneficial effects:

[0084] 1) Comprehensive Machine Tea Evaluation: This invention, for the first time, expands the scope of machine tea evaluation to include five key dimensions: appearance, aroma, flavor, tea color, and tea leaves. Compared to previous methods, this broad approach provides a more comprehensive and objective assessment of tea quality.

[0085] 2) More Detailed Appearance Dimensions: This invention also introduces a more refined evaluation of tea appearance, breaking it down into seven sub-dimensions: straightness, smoothness, tenderness, greenness, glossiness, color uniformity, and integrity. Previously, such a detailed assessment of tea appearance was unavailable.

[0086] 3) High performance across multiple tea quality dimensions: The present invention achieves 92% accuracy in classifying tea samples based on multiple dimensions (appearance, flavor, leaf base, aroma, and tea color), demonstrating that the AI-driven system is highly effective in capturing subtle nuances in tea quality.

[0087] 4) Combined use of multimodal and multimodal systems: The CLIP (Contrastive Language-Image Pre-training) model was used to supervise the training of each sub-model of the hybrid expert system to ensure the interpretability of Longjing tea (green) quality grading, while the multimodal-CLIP model was used to ensure the classification performance of tea quality grades. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] Figure 1 Workflow diagram for the digitalization of appearance, soup color, aroma, taste, and leaf bottom data and model training.

[0089] Figure 2 Schematic diagram of data collection for appearance, tea soup color, aroma, flavor, and tea leaf; a is a schematic diagram of the camera equipment for region of interest extraction; b is the tea leaf and tea soup color of three standard grades A, B, and C, including special-grade, second-grade, and first-grade tea from Longwu Tea Town; c is a schematic diagram of metabolite extraction;

[0090] Figure 3 This is an image of the appearance of the standard sample and a schematic diagram of the collection of leaf bottom soup color.

[0091] Figure 4 Figure a in the middle is a diagram of fine-tuning the ResNet-18 model. Figure 4 In the figure, bh are the training result graphs of seven sub-dimensions constructed using the fine-tuned ResNet-18 model: accuracy and loss curve graphs, b is the accuracy curve graph of smoothness, c is the accuracy curve graph of greenness, d is the accuracy curve graph of tenderness, e is the accuracy curve graph of glossiness, f is the accuracy curve graph of color uniformity, g is the accuracy curve graph of integrity, h is the loss curve of the accuracy curve graph of straightness, i is the loss curve graph of smoothness, j is the loss curve graph of greenness, k is the loss curve graph of tenderness, l is the loss curve graph of glossiness, m is the loss curve graph of color uniformity, n is the loss curve graph of integrity, o is the loss curve graph of straightness, and p is the confusion matrix of tea appearance classification after adding up the scores of each dimension.

[0092] Figure 5 This is a confusion matrix diagram of the seven sub-dimensions of tea appearance and the corresponding classification result measurement diagram. In the figure, a represents smoothness, b represents greenness, c represents tenderness, d represents glossiness, e represents color uniformity, f represents integrity, g represents linearity, h represents smoothness, i represents greenness, j represents tenderness, k represents glossiness, l represents color uniformity, m represents integrity, and n represents linearity.

[0093] Figure 6The training results of Tangse Yedi obtained by training with the XGBoost model, in the figure a is a schematic diagram of the Xgboost model; n is the training loss curve of Tangse, c is the test loss curve of Tangse, d is the three-class ROC curve of Tangse, e is the confusion matrix of Tangse, f is the training loss curve of Yedi, g is the test loss curve of Yedi, h is the three-class ROC curve of Yedi, and i is the confusion matrix of Yedi.

[0094] Figure 7 Schematic diagram of PCA and OPLS-DA analysis of aroma GC-MS data. Figure (a) shows the PCA analysis of aroma data from three different tea grades, groups ABC (QC represents a mixed sample of equal mass). Figure (b) shows the OPLS-DA analysis of aroma data from groups AB (c), BC (BC), and AC (d).

[0095] Figure 8 Schematic diagram of PCA and OPLS-DA analysis of metabolomics data derived from flavor. Figure (a) shows the PCA analysis of flavor data for three different grades of tea, groups ABC, where QC represents a mixed sample of equal mass. Figure (b) shows the OPLS-DA analysis of flavor data for groups AB, c shows the OPLS-DA analysis of flavor data for groups BC, and d shows the OPLS-DA analysis of flavor data for groups AC.

[0096] Figure 9 This is a schematic diagram of the CLIP supervisor principle. In the figure, a is comparative pre-training; b is creating a dataset classifier from labeled text; c is used for zero-probability prediction; d is the Tip-CLIP pipeline.

[0097] Figure 10Figure 2 shows the metabolome, GCMS, leaf bottom training results of TCSMLP (MLP after CLIP supervised training) and the comprehensive rating results of tea combined with five dimensions. In the figure, a is the MLP structure diagram of metabolomics, b is the metabolomics training loss curve diagram, c is the metabolomics three-classification ROC diagram, and d is the metabolomics confusion matrix diagram; e is the MLP structure diagram of GC-MS, f is the GC-MS training loss curve diagram, g is the GC-MS three-classification ROC diagram, and h is the GC-MS confusion matrix diagram; i is the MLP structure diagram of leaf bottom, j is the leaf bottom training loss curve diagram, k is the leaf bottom three-classification ROC diagram, and l is the leaf bottom confusion matrix diagram; m is the tea stacking picture using CLIP The ROC curve of the three categories is obtained by using CLIP as image data and metabolome, GCMS, soup color, and leaf bottom as text data; n is the confusion matrix obtained by using CLIP to train the picture of tea pile as image data and metabolome, GCMS, soup color, and leaf bottom as text data; o is the ROC graph of the three categories with Resnet trained on the picture of tea pile and TCSMLP trained on the metabolome, GCMS, and leaf bottom, and XGBoost trained on soup color, and weighted classification based on the five data dimensions; p is the confusion matrix with Resnet trained on the picture of tea pile and TCSMLP trained on the metabolome, GCMS, and leaf bottom, and XGBoost trained on soup color, and weighted classification based on the five data dimensions. DETAILED DESCRIPTION

[0098] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0099] 1. Experimental Materials

[0100] We collected 38 varieties of Longjing green tea, each with three replicates, for a total of 114 samples, sourced from five suppliers in Hangzhou, Zhejiang Province. Details of pricing and origin are provided in Table 1. Tea samples were categorized into three different quality grades based on manual sensory evaluation, details of which are provided in the following paragraphs. All samples were purchased directly from retail markets, packaged in aluminum foil sealed bags, and stored at 4°C until analysis.

[0101] 2. Sensory evaluation

[0102] Tea sensory evaluation and grading

[0103] Tea experts conducted a sensory evaluation based on the GBT23666-2018 green tea quality standard, evaluating appearance, aroma, flavor, tea soup color, and tea leaves after brewing. A weighted overall score was calculated. Each sample (3 grams) was brewed in 150ml of freshly boiled water for four minutes. The following attributes were evaluated: appearance, tea soup color, aroma, flavor, and tea leaves. The scores for appearance, tea soup color, aroma, flavor, and tea leaves accounted for 25%, 10%, 25%, 30%, and 10%, respectively.

[0104] Appearance: Evaluate the shape, tenderness, color, breakage and cleanliness of the dry tea.

[0105] Tea soup color: Evaluate the color type, chroma, brightness, depth and turbidity of the tea soup.

[0106] Aroma: Evaluate the aroma type, concentration, purity and persistence of the tea.

[0107] Taste: Evaluate the concentration, thickness, mellowness, astringency, purity and freshness of the tea soup.

[0108] Leaf bottom: Evaluate the tenderness, color, brightness, depth and uniformity of the leaf bottom (including uniformity of tenderness and color).

[0109] 3. Appearance dimension data collection and preprocessing

[0110] Tea photography

[0111] We use a custom computer vision system to capture images of tea samples, such as Figure 1 Shown in center (a). The system was surrounded by a black opaque acrylic sheet, simulating a darkroom environment. It was equipped with a Sony ILCE-7M2 digital SLR camera with a 50mm prime lens, positioned 350mm from the sample. Three LED light strips provided stable white light. Five grams of tea leaves were placed on a 400×400mm platform for each sample. Raw images were saved in JPEG format using RGB color mode. All images used consistent camera settings: M mode, aperture f / 4.5, shutter speed 1 / 160, ISO 100, auto white balance, zero exposure compensation, wide-area focus, single-point focus mode, and multi / center metering mode. 200 images were captured for each sample.

[0112] The original image resolution is 6000 × 4000 pixels. Unlike typical fruit and vegetable images (with clear object-background boundaries), tea images are composed of stacked leaves, so accurate segmentation is not required. Image preprocessing is performed by extracting a 224 × 224 pixel region of interest (ROI) from the original image. The extracted region is centered, as shown in the following example. Figure 2This simple extraction technique effectively captures the color, shape, and texture of tea leaves without the need for complex segmentation algorithms.

[0113] from Figure 2 As can be seen in Figure a, tea images of varying quality exhibit similar visual characteristics, posing a challenge to image-based classification algorithms. High-quality tea leaves typically have a complete, uniform shape and color, with few impurities, while low-quality tea leaves are often curled, shrunken, discolored, and contain visible impurities, including broken leaves.

[0114] Each sample is collected with approximately 200 images, with 75% of the dataset used for training and validation and 25% for testing. To enhance the dataset, data augmentation techniques were used, including image rotation (90° and 180°) and flipping (vertical and horizontal), such as Figure 3 This process increases the dataset by six times, enhances the generalization ability of the model, and reduces the risk of overfitting. Table 1 provides detailed information about the tea image dataset and the data augmentation method.

[0115] 4. Leaf base and soup color dimension data collection and preprocessing: texture and color feature extraction

[0116] Before photographing the color of the tea soup and leaf bottom, we followed the methods of the GB / T23776-2018 Tea Sensory Evaluation Standard. Before photographing, the leaf bottom was separated from the tea soup. The evaluation of the tea soup color was carried out after the first brewing. The 38 tea samples were divided into three batches, each containing 12 to 13 samples. 3 grams of tea leaves were taken from each sample, placed in an evaluation cup, 150 ml of boiling water was added, brewed for 4 minutes and covered. Subsequently, the tea soup was poured into the evaluation bowl and left to cool for 25 minutes until no water vapor was visible. The tea soup was placed in a white porcelain bowl to suppress the transmission of light, and the image was captured using a photographic system. Due to the limited number of samples, only one image of the tea soup color was taken for each sample, and a total of 38 images were obtained. For the tea soup image dataset, considering that the color of the tea soup and the amount of tea leaves are key features of the dataset, it was necessary to exclude factors that affect these features, such as the teacup and the tabletop. For each tea soup image, we first used a 1600px×1600px rectangle to crop the original image from its center. The resulting crop was then divided into 3×3 equal-distance regions. Five images, namely the top, bottom, left, right, and center images, were selected as the tea soup image dataset, thus expanding the single-category tea soup color dataset from 38 to 190 images. The same approach was applied to the leaf bottom image dataset.

[0117] Similarly, the leaf bottom was placed in a white porcelain bowl, and the shooting parameters were the same as when shooting the shape of stacked tea leaves. One leaf bottom image was taken for each sample, and a total of 38 images were obtained. The tea soup was subjected to color feature extraction, and a total of 9 features were extracted, including R red value, G green value, B blue value, H hue (Hue), S saturation (Saturation), V brightness (Value), L* brightness (Lightness), a* red-green axis (a*, red to green), and b* yellow-blue axis (b*, yellow to blue). The leaf bottom was subjected to color feature and texture feature extraction. In addition to the above 9 color features, contrast, dissimilarity, homogeneity, correlation and angular second-order moment (4 directions 0°, 45°, 90°, 135°) were also extracted. Preprocessing process:

[0118] When extracting the color features of the soup and leaf base, we use the OpenCV library's cvtColor function to convert the image to a specific color space, such as HSV or Lab. We then use the split function to calculate color features based on the pixels of the entire image. When extracting the morphological features of the soup and leaf base, we first convert each image to grayscale space, then map the grayscale levels from 0-255 to 0-16, thereby compressing the dimensionality of the grayscale co-occurrence matrix. Morphological features are then calculated based on the grayscale co-occurrence matrix at the four directional offsets.

[0119] 5. Aroma data collection and preprocessing

[0120] Volatile compounds were extracted from tea samples (0.5 g). The samples were sealed in 20 mL glass vials, and 10 μL of ethyl decanoate (internal standard, concentration: 10 mg / L) and 5 mL of boiling deionized water were added sequentially. After equilibration for 5 minutes, the vials were placed in a 60°C water bath. The gaseous volatiles were adsorbed for 60 minutes using a StableFlex 2 cm fiber coated with divinylbenzene / chlorinated polystyrene / polydimethylsiloxane (50 / 30 μm, Supelco, Bellefonte, PA, USA). The volatiles were desorbed by heating at 250°C for 5 minutes in a GC-MS injector. Volatile analysis was performed using an Agilent 7890 gas chromatograph coupled to an Agilent HP 5977MSD ion trap mass spectrometer (Wilmington, DE, USA) equipped with an HP-5MS capillary column (30 m × 250 μm × 0.25 μm). Gas chromatography conditions were as follows: inlet temperature of 250°C, carrier gas of high-purity helium (99.999%) at a flow rate of 1.0 mL / min (splitless mode). The temperature program was as follows: 40°C for 2 minutes, ramped to 85°C at 2°C / min and held for 2 minutes; ramped to 180°C at 2.5°C / min and held for 2 minutes; and ramped to 230°C at 10°C / min and held for 2 minutes. Mass spectrometry (MS) analysis was performed in electron impact (EI) mode at 70 eV, with a mass scan range of 40–400 m / z and an ion source temperature of 230°C.

[0121] Identification of volatile compounds was performed using the NIST mass spectrum search program (NIST Library 20) and a GC-MS data analysis program developed in Microsoft Excel. GC peaks were identified by matching mass spectra and retention indices. Semiquantitative concentrations of volatile compounds were calculated by comparing the total ion current response of the mass spectrometer with the response of the internal standard.

[0122] 6. Taste data collection and processing

[0123] To extract metabolites, 1 gram of sample was mixed with 50 grams of boiling water (100°C), shaken thoroughly, and heated in a 100°C water bath for 30 minutes. The mixture was shaken every 10 minutes, repeated twice. The mixture was then centrifuged at 8,000 rpm for 10 minutes, and the supernatant was removed. Next, the mixture was centrifuged at 12,000 rpm (13,800 × g) at 4°C for 15 minutes, and the supernatant was collected and transferred to an injection vial for analysis. For quality control (QC), equal amounts of supernatant from each sample were combined and used for machine testing.

[0124] LC-MS / MS analysis was performed using a UHPLC system (Vanquish, Thermo Fisher Scientific) equipped with a UPLCHSS T3 column (2.1 × 100 mm, 1.8 μm) coupled to an Orbitrap Exploris 120 mass spectrometer (Thermo Fisher Scientific). The mobile phase consisted of 5 mmol / L ammonium acetate and 5 mmol / L acetic acid in water (A), and acetonitrile (B). The autosampler was maintained at 4°C, and the injection volume was 2 μL. The Orbitrap Exploris 120 mass spectrometer was operated in signal-dependent acquisition mode, controlled by Xcalibur software (Thermo Fisher). The electrospray ionization (ESI) source conditions were as follows: shield gas flow rate 50 Arb, auxiliary gas flow rate 15 Arb, capillary temperature 320°C, full scan resolution 60,000, MS / MS resolution 15,000, collision energy 10 / 30 / 60 NCE mode, and spray voltage 3.8 kV (positive ion mode) or 3.4 kV (negative ion mode).

[0125] The raw data were converted to mzXML format using ProteoWizard, and peak detection, extraction, alignment, and integration were performed using an in-house program based on XCMS (R). Metabolite annotation was performed using an in-house MS2 database (BiotreeDB) with an annotation threshold of 0.3.

[0126] 7. Statistical analysis

[0127] All experiments were performed in triplicate. Multivariate analyses were performed using SIMCA 13.0 (Umetrics AB, Umea, Sweden). Statistical analysis was performed using the IBM Statistical Package (version 25.0, SPSS Inc., Chicago, USA) using one-way analysis of variance (ANOVA) and significance tests (P < 0.05). Figures were generated using Origin (version 2018, OriginLabCorp., Northampton, USA).

[0128] PCA (Principal Component Analysis) is a commonly used dimensionality reduction method. It transforms a set of correlated variables into a set of smaller, uncorrelated variables, called principal components. The first principal component explains the most variance, and subsequent principal components are ranked from largest to smallest. PCA reduces the dimensionality of the data while preserving the maximum amount of information possible. Before PCA is performed, the data is normalized to mitigate the influence of outliers.

[0129] Orthogonal partial least squares discriminant analysis (OPLS-DA) is a statistical method used to distinguish and explain categorical differences in multivariate datasets. OPLS-DA enhances model interpretability and predictive accuracy by decomposing predictive variables into orthogonal components that are both class-dependent and class-independent. It is widely used in metabolomics, genomics, and related fields to identify biomarkers and categorical features. Compared to PLS-DA, OPLS-DA is more effective in removing class-independent noise, thereby improving model robustness and explanatory power.

[0130] 8. Training model structure and design

[0131] A multimodal Longjing tea grading model

[0132] 8.1. Overall model structure

[0133] The present invention provides a multimodal Longjing tea grading model integrating five human evaluation scales, the model comprising:

[0134] Input module: Responsible for receiving and preprocessing multimodal data, including appearance images, soup color images, leaf bottom images, aroma data and taste data, and performing operations such as data enhancement, standardization and segmentation.

[0135] Deep Convolution Module: Based on the improved ResNet convolutional neural network, it extracts multi-dimensional features of appearance images and converts the features into corresponding scores through linear mapping.

[0136] Integrated learning module: Improved XGBoost and TCSMLP (CLIP-supervised MLP) are used to perform regression prediction on aroma, taste, soup color and leaf bottom data to achieve quantitative evaluation of tea quality.

[0137] Contrastive learning module: uses the improved CLIP model to predict the category of tea images, and combines it with an integrated learning supervision mechanism to improve the model's generalization ability.

[0138] Output module: linearly weights the scores of each evaluation dimension, outputs the final tea quality grade, and visually displays the evaluation results.

[0139] 8.2. Workflow

[0140] The workflow of the multimodal Longjing tea grading model of the present invention includes:

[0141] 1. Data Collection and Preprocessing: Longjing tea samples of varying prices and grades were collected, and image data on appearance, tea color, and leaf bottoms, as well as data on volatile compounds and metabolites, were obtained. Image data was enhanced, cropped, filtered, and normalized, while chemical analysis data was cleaned, filtered, padded, and standardized.

[0142] 2. Model training:

[0143] Deep convolution module training: The ResNet network is used to extract multi-dimensional features of the appearance image, and automatic scoring is achieved through a comment-score mapping mechanism.

[0144] Integrated learning module training:

[0145] XGBoost is used for regression prediction of soup color data.

[0146] TCSMLP training:

[0147] ① Pre-train TCSMLP through the CLIP supervision mechanism to build a multimodal supervisor.

[0148] ②Generate image embedding based on CLIP visual encoder (VisualEncoder) and store category features through the cache:

[0149] Key=VisualEncoder(I k )∈R NK×C

[0150] ③Use OneHot encoding to process text category labels:

[0151] Value=OneHot(L k )∈R NK×N

[0152] ④Calculate the similarity score of CLIP supervision:

[0153]

[0154] ⑤ Combine CLIP classifier and TCSMLP for final prediction:

[0155] Logits=α*Similarity*Value+Query*W T

[0156] ⑥Finally, TCSMLP is used to perform regression prediction on aroma, flavor, and leaf bottom data, and the loss function is optimized to improve the prediction accuracy.

[0157] 8.3 Reasoning and Grading:

[0158] 1. Process the multimodal data of the input tea samples and perform feature extraction and score calculation through each sub-model;

[0159] 2. Use linear weighting to calculate the comprehensive score of tea and determine the grade of tea based on the preset scoring criteria;

[0160] 3. Generate final evaluation results, including numerical scores and grading recommendations

[0161] 8.4. Mathematical description of each module

[0162] ResNet residual block calculation formula:

[0163] h l+1 =h l +σ(W l *h l +b l )

[0164] Among them, h l represents the input of the lth layer, F(h l, W l ) represents the transformation after convolution and activation function, W l is the weight of the lth layer, σ is the activation function, b l is the bias.

[0165] TCSMLP calculation formula (CLIP supervised MLP):

[0166] h (l) =σ(W (l) h (l-1) +b (l) )

[0167] Among them, h (l) represents the input of the lth layer, W (l) and b (l) are the weight and bias of the lth layer respectively, and σ is the activation function.

[0168] XGBoost loss function:

[0169]

[0170] Among them, loss is the loss function, Ω(f k ) is the regularization term, K is the number of decision trees, n is the number of samples, y i is the true label of the sample, is the model prediction value;

[0171] Final score calculation:

[0172] Result=0.25*Appearance+0.1*SoupColor+0.25*Aroma+0.3*Taste+0.1*InfusedLeaf

[0173] 8.5. Tea Grading Standards

[0174] The correspondence between the total score and grade: C<88≤B<90≤A.

[0175] Scoring criteria for each dimension:

[0176] Appearance: C<88≤B<91≤A;

[0177] Aroma: C<88≤B<92≤A;

[0178] Taste: C<88≤B<91≤A;

[0179] Soup color: C≤87<B≤89<A;

[0180] Leaf bottom: C≤87<B≤89<A.

[0181] The present invention realizes the intelligent grading of Longjing tea through multimodal data fusion, CLIP supervised learning, deep learning and ensemble learning technology, improves the accuracy and generalization ability of grading, and provides an efficient and accurate method for tea quality assessment.

[0182] All experiments were conducted on a computer running Windows 10, equipped with an 8-core i7-9750H CPU (2.6GHz), 16GB of DDR4 memory, and an NVIDIA GeForce GTX 1060 GPU (6GB of video memory, CUDA 10.2.89). The CNN model was deployed using the PyTorch framework (version 1.10.2) based on Python 3.6.3 to ensure stability. Libraries used for image preprocessing, feature extraction, and machine learning included py-opencv 3.4.2, numpy 1.19.5, and scikit-learn 0.24.2. For the GC-MS and metabolomics datasets, the regression models converged after 200 epochs of training, while the classification model for the image dataset converged after 10 epochs of training. The parameters of the best-performing model on the training set were saved for testing. Each training experiment was repeated five times with random initialization to minimize error, and the average results are reported.

[0183] 8.6 Detailed introduction of each module

[0184] 8.6.1 ResNet

[0185] ResNet (residual network) is a deep neural network architecture proposed by Kaiming He et al. at Microsoft Research Asia in 2015. By introducing residual connections, ResNet solves the problems of vanishing and exploding gradients during deep network training, effectively increasing the depth of the network and improving its performance. The core concept of ResNet is to add shortcut connections, allowing input information to bypass intermediate layers and pass directly to subsequent layers. This mechanism ensures that even if the deep network fails to learn useful features, the model can still rely on input information from earlier layers, avoiding network performance degradation. Residual connections enable the network to only learn the residual (that is, the difference between the input and output), simplifying the learning process.

[0186] ResNet consists of multiple residual blocks connected in series. Each block has two paths: an identity mapping path, which passes the input information directly to the output; and a residual mapping path, which transforms the input before merging it with the output of the identity mapping. This structure facilitates the flow of gradients during backpropagation, thereby alleviating the problem of vanishing or exploding gradients.

[0187] In the present invention, ResNet and its extended models are selected to classify tea morphology data to verify the effectiveness of the residual structure. The architecture used is ResNet-18, which includes an input layer (224×224×3), a convolutional layer (7×7, 64 filters, step size 2, output size 112×112), a maximum pooling layer (3×3, step size 2, output size 56×56), and a convolutional layer with gradually increasing number of filters (64, 128, 256, 512 filters), and the output size gradually decreases. This is followed by an average pooling layer and a fully connected layer with an output size of 1×3 for classification. The floating operations of this model are approximately 1.8×10 9 .

[0188] The ResNet-18 model was pre-trained on the ImageNet-1K dataset. After loading the pre-trained parameters, the model was fine-tuned on the stacked tea image dataset, which consists of 5,822 training images and 1,941 test images. The training hyperparameters were set as follows: learning rate = 5e-5, batch size = 128, and number of training epochs = 10. A dynamic learning rate adjustment strategy was employed during training, with a learning rate decay parameter set. The learning rate was decayed when the number of training epochs reached a preset threshold, effectively ensuring model convergence. To increase the size of the training set and improve the model's classification accuracy and generalization, data augmentation techniques such as 90° rotation, vertical and horizontal flipping, and 180° rotation were employed, increasing the training set size fivefold.

[0189] After training each classification model for evaluating tea appearance, the parameters associated with the best classification accuracy on the validation set were saved. These models were then used to perform inference on individual tea appearance images, generating a sequence of evaluated descriptors for each image. This process was applied to the entire tea appearance image dataset, resulting in multiple descriptor sequence matrices. To establish the conversion between tea appearance descriptors and appearance scores, a pre-built database containing descriptors and their corresponding scores for each category was used to facilitate the conversion of descriptor matrices into score sequences. Specifically, seven evaluation dimensions were used to stack tea appearance: straightness, smoothness, tenderness, greenness, luster, color uniformity, and completeness. Each dimension included multiple categories, for a total of 46 categories. Therefore, each processed stacked tea image generated a 1×46 descriptor sequence vector. The complete stacked tea appearance dataset contains 7,763 images, resulting in a 7,763×46 matrix. Using the descriptor-score database, this matrix was converted to a 7,763×1 score sequence through matrix operations, with each entry representing the appearance score of the corresponding image.

[0190] 8.6.2 Multilayer Perceptron (MLP)

[0191] A multilayer perceptron (MLP) is a basic artificial neural network (ANN) architecture consisting of multiple layers: an input layer, one or more hidden layers, and an output layer. The MLP is a feedforward network, meaning that information flows unidirectionally from input to output, and all layers are fully connected. By combining multiple layers of neurons with nonlinear activation functions, the MLP is able to learn complex relationships between input and output.

[0192] The learning process proceeds through forward propagation and backpropagation. During forward propagation, the input signal is passed through the hidden layers to generate the model's predictions. Backpropagation calculates the gradient of the loss function with respect to the model parameters and updates the weights. Through repeated forward and backpropagation, the model adjusts its weights and biases, gradually improving its predictions toward the true value.

[0193] In the present invention, a customized MLP structure was selected as the regression model for GC-MS and metabolomics data to evaluate its effectiveness. For metabolomics data, the architecture includes an input layer (18 features), three hidden layers (256, 128, 64 nodes) and an output layer. The architecture of the GC-MS model includes an input layer (312 features), three hidden layers (209, 140, 94 nodes) and an output layer. For the appearance of scattered tea leaves, the architecture includes an input layer (20 features), three hidden layers (256, 128, 64 nodes) and an output layer. The training parameters of the two models are set as follows: num_epochs = 200, learning rate = 0.01 (metabolomics) or 0.001 (GC-MS), weight_decay = 0.003, batch size = 16. To ensure robustness, a five-fold cross validation was applied.

[0194] 8.6.3 XGBoost

[0195] XGBoost (eXtreme Gradient Boosting) is a high-performance gradient boosting framework that improves prediction accuracy by iteratively building decision trees. The goal of each tree is to correct the errors of the previous tree. XGBoost also incorporates optimization techniques such as regularization to prevent overfitting.

[0196] The key parameters of XGBoost include:

[0197] max_depth: The maximum depth of the tree (increasing the depth will increase the complexity and the risk of overfitting).

[0198] eta (learning rate): controls the step size of weight updates.

[0199] num_round: The number of rounds of boosting.

[0200] In this experiment, the parameters are set as: max_depth = 3, eta = 0.1, num_round = 100, and the loss function is regression.

[0201] 8.6.4Tip Clip and TCSMLP

[0202] The CLIP (Contrastive Language-Image Pre-Training) model is a multimodal pre-trained neural network released by OpenAI in 2021. It represents an effective and easily transferable method for learning through natural language supervision. The basic CLIP model is pre-trained on a large scale of 400 million text-image pairs. The core idea of ​​the model is to use a large amount of text and image paired data to learn the alignment between images and text. The model consists of two main components: a text encoder (Text Encoder), which converts text into a low-dimensional vector representation; and an image encoder (Image Encoder), which converts images into similar vector representations. In the prediction stage, the CLIP model generates predictions by calculating the cosine similarity between text and image vectors, making CLIP particularly suitable for zero-shot learning and few-shot learning tasks ( Figure 9 (a–c).

[0203] For zero-shot classification of a tea quality grading dataset (including appearance images, GCMS data, metabolomics data, and leaf bottom images), feature engineering is used to convert the latter four types of data into feature vectors of specific dimensions. These are then combined with the appearance image output by the image encoder to obtain a new encoding vector I1. The text encoder outputs text encoding vectors T1, T2, and T3 for special grade tea, first-grade tea, and second-grade tea. The cosine similarity between I1 and T1, T2, and T3 is calculated, and the category corresponding to the highest text encoding vector is used as the category of the current combined input data.

[0204] The key issue is that the basic CLIP model is pre-trained on 400 million text-image pairs and may not have learned domain-specific data, such as tea quality grades, which results in low classification accuracy. To further improve the transfer performance of CLIP in downstream tasks, while considering the cost of time and space resources, this paper introduces a few-shot image classification method, Tip Clip, which improves the accuracy of CLIP without the need for additional downstream training. This method uses CLIP to build a cache model (Cache Model) to store classification knowledge from downstream training data in a non-training manner, in this case, tea quality grading knowledge. Subsequently, during testing, more reliable classification results are obtained by linearly summing the prediction results of the cache model and the basic CLIP model.

[0205] In this method, the comparative language-image pre-training (CLIP) model is used to train existing text (leaf bottom, metabolomics, GC-MS) and image (tea appearance images) datasets. However, since its pre-training is performed on general data, it may not obtain professional representations for tea quality grades, resulting in reduced classification accuracy. To enhance the migration performance of CLIP and address time and space limitations, a small number of sample image classification method Tip-CLIP is proposed. This method improves the accuracy of CLIP without the need for additional downstream training. Tip-CLIP uses CLIP to create a cached model ( Figure 9 In (d), the classification knowledge obtained from downstream training data (especially knowledge about tea grading) is retained in a non-training manner. During testing, the classification results are obtained by a linear combination of the predictions of the cached model and the CLIP base model, resulting in more reliable classification results. After enhancing the CLIP model with tea grading knowledge, the improved Tip-CLIP model performed supervised regression tasks on metabolomics, GC-MS and leaf bottom datasets. The Tip-CLIP supervised multilayer perceptron (TCSMLP) was developed under CLIP supervision and significantly improved the fitting performance, R 2 The value exceeds that of the original MLP model.

[0206] 9. Results and Analysis

[0207] (1) Longjing Green Tea Evaluation Results

[0208] We conducted a standardized sensory evaluation of 38 Longjing green tea samples according to the GBT 23776-201824 sensory evaluation standard for tea. Comments and scores for each evaluation criterion were recorded, and a weighted sum was calculated to determine the overall score, ranging from 85 to 92 (Table 1). Most samples exhibited the typical smooth, straight appearance of Longjing tea; however, six samples were over-roasted, affecting their fresh aroma and resulting in a lower aroma score. Market preferences, processing methods, and manufacturer philosophy influence the degree of roasting. While high-temperature roasting was more common in the past, most samples exhibited a medium roast. In addition to a fresh aroma, some samples also exhibited floral notes, which enhanced the aroma score. A total of ten samples exhibited floral notes, a relatively high proportion. Factors such as variety, leaf tenderness, and processing method significantly influence grading. Longjing tea is evaluated based on appearance, tea soup color, aroma, flavor, and leaf base, and is categorized into two main categories: external and internal quality. Tea picked in early spring is generally fresher and commands a higher price.

[0209] Ideally, appearance and internal quality should be consistent, provided proper processing is performed. However, due to processing variations, tea with excellent appearance can sometimes have lower internal quality grades. GBT 23776-2018 stipulates that appearance accounts for 25% and internal quality accounts for 75% (including 30% for flavor, 25% for aroma, 10% for liquor color, and 10% for tea leaves), with flavor having the greatest impact on the overall score. According to the West Lake Longjing Group Standard TXHLJ001-202125, samples are classified as second-grade, first-grade, premium, and special-grade. In this evaluation, due to overroasting, the aroma score of the intended premium-grade standard sample dropped to 88, resulting in an overall score lower than that of the premium sample. Premium-grade samples should score slightly higher than premium-grade samples in all aspects. Based on expert recommendation, three tea samples were selected as the benchmark (second-grade raw tea with a score of 4.8, first-grade raw tea with a score of 3.31, and premium raw tea with a score of 3.21). A total score below 88 is considered second-grade, 88 to 90 is considered first-grade, and 90 or above is considered premium. Of the 38 samples, 16 were rated as excellent (Class A), 16 as first-class (Class B), and 6 as second-class (Class C).

[0210] Table 1 Sensory evaluation of Longjing green tea: Appearance

[0211]

[0212]

[0213] Table 2 Sensory evaluation of Longjing green tea: soup color

[0214]

[0215]

[0216] Table 3 Sensory evaluation of Longjing green tea: aroma

[0217]

[0218]

[0219] Table 4 Sensory evaluation of Longjing green tea: leaf bottom

[0220]

[0221]

[0222] Table 5 Sensory evaluation of Longjing green tea: taste

[0223]

[0224]

[0225] Table 6 Sensory evaluation of Longjing green tea: total score and rating

[0226]

[0227]

[0228]

[0229] 8.2 Construction of Tea Quality Recognition Model Based on Appearance Dimension

[0230] This method combines visual images of tea leaves with reviews from human sensory evaluations, calculating appearance scores by correlating the model-generated reviews with the scores. The accuracy of the model classification was evaluated by establishing a sensory evaluation scoring range (C < 88 ≤ B < 91 ≤ A), resulting in 12 A-grade teas, 16 B-grade teas, and 10 C-grade teas.

[0231] This method combines visual images with human sensory evaluation reviews. Appearance scores are derived from model-generated reviews and a rating scale. Preliminary testing showed that a review-scoring model that excludes deductions and bonuses provided a better fit, so only a seven-dimensional sub-model was retained for training. Tea appearance is evaluated along seven dimensions: straightness, smoothness, tenderness, greenness, glossiness, color uniformity, and wholeness. The camera only captures the front of the leaves, omitting the assessment of flatness. For each dimension, a sub-model generates an evaluation score that matches the rating scale. Scores for all dimensions are summed to determine the overall appearance score.

[0232] The total number of sample appearance images is 7,763. After data augmentation, the data volume has increased sixfold, with 75% used for training and augmentation and 25% for testing.

[0233] Using a fine-tuned ResNet-18 convolutional neural network (CNN) model ( Figure 4 (a) Classify tea samples based on visual images. Accuracy and loss plots for seven dimensions ( Figure 4 The results of the CNN (Figure 2) show that after seven training rounds, the model converged and exhibited high accuracy and low loss on both the training and test sets. The ResNet-18 model achieved a test accuracy greater than 0.85 in almost all dimensions, with the highest accuracy for the green dimension (0.976) and the lowest accuracy for the tenderness dimension (0.863). The final tea appearance score was derived from the seven sub-dimension models. The overall classification accuracy was 83.77%, as shown in the confusion matrix ( Figure 4 As shown in (i).

[0234] Table 7 Appearance dimension review score correspondence table

[0235]

[0236]

[0237] 8.3 Construction of Tea Quality Recognition Model Based on the Dimensions of Infused Leaf and Broth Color

[0238] The infused leaf and broth color contribute 10% to the tea score. The infused leaf is evaluated by tenderness, color, and uniformity, similar to the appearance of dry tea. Since it is difficult to identify the edge of the infused leaf, only color and texture features are extracted. For the broth color, sensory evaluation focuses on color type, chroma, brightness, and clarity, and color feature extraction is adopted.

[0239] For the classification of the infused leaf, the scoring range (C ≤ 87 < B ≤ 89 < A) yields 17 grade A samples, 12 grade B samples, and 9 grade C samples. A similar procedure is adopted for the scoring range of the broth color (C ≤ 87 < B ≤ 89 < A), resulting in 15 grade A samples, 16 grade B samples, and 7 grade C samples. One image is obtained for each category, and data set augmentation increases the number of images in each dimension to 190. Data preprocessing follows the same steps as for the tea appearance. The data set is expanded by cropping, and the number of infused leaf and broth color images increases from 38 to 190, including the color and texture features of the infused leaf images and the color features of the broth color images.

[0240] XGBoost is used for the classification of the broth color. Figure 6 In a, it is the schematic diagram of the structure of XGBoost. r, R 2 , logRMSE is shown in Table 1, R 2 is 0.9969, with a high fitting degree. The training loss curve and test loss curve of the broth color ([[]] Figure 6 in b - c) show that after training about 150 rounds, the model converges and shows high accuracy and low loss on both the training set and the test set. The ROC curve ([[]] Figure 6 in d) shows that the classification of broth color category C is better than other categories (AUC 0.81), and the confusion matrix shows ([[]] Figure 6 in e) that the accuracy of the single - dimension classification of the broth color is 70.05%.

[0241] Therefore, we replace MLP with TCSMLP for classification. TCSCLIP is used for the classification of the infused leaf, R

[0241] is 0.9996, and the structure of the MLP part in its model is as shown in [[[]] Figure 10 2 in i. Figure 10 In j and k are the training loss curve and the three - classification ROC curve respectively, indicating that the model converges after training about 150 rounds. Figure 10 In l is its confusion matrix, and the accuracy of the single - dimension classification of the infused leaf reaches 81.58%. Figure 10

[0242] Table 8 Regression Index Table of Training Models for Each Dimension

[0243]

[0244] 8.4 Construction of Tea Quality Identification Model Based on Aroma Dimension:

[0245] This method analyzes the volatile compounds produced by brewing 38 Longjing tea samples (three replicates each) in boiling water and classifies them using SPME-GC-MS. Approximately 300 aroma compounds, including aldehydes, ketones, alcohols, terpenes, acids, benzenes, cycloalkanes, aromatics, and amino acid derivatives, were identified by retention time and the NIST 20 database (Supplementary Data 2). Principal component analysis (PCA), an unsupervised statistical method, was used for 298 variables to identify differences and associations between variables and samples. The contribution rates of PC1 and PC2 were 6.9% and 6.3%, respectively, indicating that PCA has limited ability to distinguish tea grades based on aroma ( Figure 7 However, orthogonal partial least squares discriminant analysis (OPLS-DA) successfully distinguished the tea grades ( Figure 7 (b–d).

[0246] The human reviewers' ratings of tea aroma were used as labels to input into the model. A regression strategy was used to output the model's ratings for the aroma dimension. By defining the ABC score interval for the human reviewers' ratings of the flavor dimension (C<88≤B<92≤A, with 14 A, 16 B, and 8 C), a classification for the flavor dimension was obtained.

[0247] GC-MS data were classified using TCSCLIP. 2 is 0.9998, and the structure of the MLP part of the model is as follows Figure 10 As shown in e. Figure 10 Figures f and g are the training loss curve and the three-class ROC curve, respectively, indicating that the model converged after about 150 rounds of training. At the same time, the AUCs for the three classifications are close to 1, indicating high classification accuracy. Figure 10 In the figure h is the confusion matrix, and the accuracy of single-dimensional flavor classification reaches 99.12%.

[0248] 8.5 Construction of Tea Quality Identification Model Based on Taste Dimension

[0249] A non-targeted metabolomics approach using liquid chromatography-mass spectrometry (LC-MS) technology was used to digitize sensory evaluation and analyze taste compounds in tea soup. Metabolomics is widely used to analyze metabolic profiles and associate chemical components with the sensory properties of tea. In this invention, metabolic profiles were generated from 38 Longjing green tea samples, which were divided into three quality grades. After removing bias and normalizing missing values, 12,287 ions in positive mode and 9,648 ions in negative mode were retained (Supplementary Data 3). This method identified 718 metabolites in positive mode and 265 metabolites in negative mode as candidate differential metabolites (Supplementary Data 4). These samples were evaluated using PCA and OPLS-DA methods. PCA results showed that the first and second principal components explained 21.5% and 11.9% of the variance, respectively, but failed to effectively distinguish tea grades. However, OPLS-DA showed good clustering effect (Supplementary Data Figure 4 c–e), can effectively distinguish between Class C, Class B and Class A.

[0250] Due to the complexity and comprehensiveness of tea evaluation, this method uses a regression strategy to solve the classification problem in this overall dimension. When considering the label selection of each group of metabolomics data, this method does not use the type of tea or the specific grade as the label input model as other previous literature, but instead uses the manual review score of the tea in terms of taste as the label input model. Naturally, the output of the model becomes the model's score for the tea, and it is not the same tea classification label as other traditional methods. By delineating the ABC grade score interval of the manual review score under the taste dimension (C<88≤B<91≤A, A: 14, B: 18, C: 6), the classification under the single dimension of taste is obtained.

[0251] A regression model was established to output flavor scores for tea samples of different grades. Metabolomics can measure a large number of metabolites, with 983 identified in each sample through MS / MS alone. When 100 metabolites were input into the MLP model, the model overfitted, resulting in an accuracy of only approximately 30%. Therefore, this method employed a strategy to reduce the number of input model dimensions. Reducing the number of dimensions to 18 significantly improved the model's fit, reaching 99%. Therefore, this method selected the 18 substances with the highest MS / MS identification scores for input into the model.

[0252] Figure 10 Figure a is a schematic diagram of the structure of the MLP part used in metabolomics. After training the entire dataset, the training loss curve, three-class ROC curve and confusion matrix were obtained. 2 The increase in value,

[0253] The same data obtained from metabolomics were classified using TCSCLIP. 2 is 0.9999, and the structure of the MLP part of the model is as follows Figure 10 As shown in a. Figure 10 Figures b and c are the training loss curve and the three-class ROC curve, respectively, indicating that the model converged after about 150 rounds of training. At the same time, the AUCs for the three categories are close to 1, indicating high classification accuracy. Figure 10 The d in the middle is its confusion matrix, and the accuracy of single-dimensional taste classification reaches 97.37%.

[0254] Comprehensive quality grading of tea based on artificial intelligence

[0255] After creating evaluation models for tea appearance, aroma, flavor, tea color, and tea leaf, the best-performing method was selected as a sub-model for the overall evaluation. Each sub-model generated a score for its corresponding dimension, which was used to calculate the final overall score. ResNet was used for training the appearance dimension, while TCSMLP was applied to the flavor, aroma, and tea leaf datasets. XGBoost was used for the tea leaf color dataset due to its superior performance.

[0256] To address the data volume inconsistencies between datasets and to objectively evaluate the accuracy and effectiveness of these methods in practical applications, tea color, tea leaf, GC-MS, and metabolomics data were randomly paired with corresponding appearance image data, with equal pairing probability for each tea type. This ensured that each image was paired with the corresponding GC-MS and metabolomics data for accurate evaluation.

[0257] The predicted scores for each evaluation dimension—appearance (25%), tea color (10%), aroma (25%), flavor (30%), and tea base (10%)—are weighted and summed to calculate the final tea score. Based on the total score, tea grades are categorized as C<88≤B<90≤A. The number of grades is 16 for A, 16 for B, and 6 for C, resulting in 3,303 samples for A, 3,218 for B, and 1,242 for C, respectively.

[0258] The ROC curve and confusion matrix of the multimodal system Long-Tea (10o, p in the figure) show that the system has a classification accuracy of 92.00%, which can accurately classify tea grades.

[0259] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A multimodal Longjing tea intelligent grading method integrating five-dimensional evaluation scales, characterized by: include: S1. Collect various Longjing tea samples of different prices and grades; S2. Collect a dataset of Longjing tea samples' appearance images, tea color images, tea leaf bottom images, aroma data, and flavor data. Preprocess the datasets separately, and divide the datasets into a training set, a test set, and a validation set. S3. Build a multimodal Longjing tea grading model, train, test, and validate the multimodal Longjing tea grading model using a training set, a test set, and a validation set to obtain the optimal multimodal Longjing tea grading model; S4. Grade Longjing tea using the optimal multimodal Longjing tea grading model.

2. The multimodal Longjing tea intelligent grading method integrating five-dimensional evaluation scale according to claim 1 is characterized in that: The S2 includes: Collecting appearance image datasets: We took images of tea samples before brewing and enhanced the data using data augmentation methods. Collect tea soup color image datasets and tea leaf bottom image datasets: Brew tea samples, remove the tea leaves from the tea soup, and then take pictures of the tea soup color and tea leaf bottom images; Collecting aroma datasets: extracting volatile compounds from brewed tea samples and identifying the types and concentrations of volatile compounds; Collecting a flavor dataset: Identifying the types and concentrations of metabolites in brewed tea.

3. The multimodal Longjing tea intelligent grading method integrating five-dimensional evaluation scale according to claim 1 is characterized in that: The multimodal Longjing tea grading model includes an input module, a deep convolution module, a contrastive learning module, an integrated learning module and an output module; The input module pre-processes the input image, pre-processes the chemical analysis data, and divides and directs the input data downstream; The deep convolutional module first uses the SimCLR self-supervised representation learning framework to pre-train on unlabeled data, then migrates to the ResNet convolutional model to classify tea appearance images based on seven sub-comment dimensions, and converts the tea appearance images into comments and scores based on linear mapping; The contrastive learning module uses CLIP combined with a buffer built based on a few-shot dataset to perform zero-shot learning prediction on the tea image category, and builds a multimodal ensemble learning supervisor based on this CLIP; The integrated learning module uses a multimodal supervisor to dynamically adjust the training process of XGBoost and MLP, and uses the adjusted trained models XGBoost and TCSMLP to perform regression prediction on aroma data, flavor data, soup color data, and leaf bottom data, where TCSMLP is the MLP trained by CLIP supervision; The output module linearly weights the five human evaluation scale scores, realizes text visualization of the tea quality evaluation process, and outputs the tea quality evaluation results.

4. The multimodal Longjing tea intelligent grading method integrating five-dimensional evaluation scale according to claim 3 is characterized in that: The SimCLR self-supervised visual representation learning framework of the multimodal Longjing tea grading model learns better visual representations through contrastive learning. Defined as: Among them, exp represents the natural exponential function with e as the base, which is used to scale the similarity value to the positive range; sim represents the similarity calculation function, z i is the vector obtained by L2 normalization of a single sample. L2 means L2 norm normalization of the feature vector, that is, dividing each vector by its own Euclidean norm to normalize its modulus to 1, thereby projecting all vectors onto the unit sphere, which is convenient for subsequent calculation of cosine similarity; j is with z i Feature representation of paired positive samples, z k Indicates that it is used with z i The kth sample feature among all the samples to be compared, τ is the temperature coefficient, which is used to control the sharpness of the probability distribution. When τ is smaller, the similarity difference is amplified, and the model pays more attention to those negative samples with average similarity but misclassified. When τ is larger, the distribution is smoother, and the model is less sensitive to similarity differences. N is the total number of samples. The similarity calculation function sim is specifically defined as follows: Among them, sim(z i ,z j ) represents sample z i With sample z j The cosine similarity between i and zj are the feature vectors of the two samples respectively; z i *z j represents their dot product; ‖z i ‖ and ‖z j ‖ are the L2 norms of the two vectors, which are used for normalization to ensure that the similarity value range is between [-1,1].

5. The multimodal Longjing tea intelligent grading method integrating five-dimensional evaluation scale according to claim 3 is characterized in that: The ResNet convolutional model of the multimodal Longjing tea grading model includes a core structure residual block and a residual connection operation, wherein the output formula of each core structure residual block is: h l+1 =h l +σ1(W l *h l +b l ) Among them, h l represents the input of the lth layer, h l+1 Represents the transformation after convolution and activation function, W l is the weight of the lth layer, σ1 is the ReLU activation function, b l is bias; The cross entropy loss function of the objective function of the ResNet convolutional model for: Where N is the number of samples, C is the number of categories, and y i,c is the true label of the i-th sample in the c-th category, p i,c It is the predicted probability that the i-th sample belongs to the c-th class after the output layer is activated by the Softmax function. The last layer of the ResNet convolutional model outputs the classification probability through the fully connected layer, and the cross entropy loss drives the parameter update.

6. The multimodal Longjing tea intelligent grading method integrating five-dimensional evaluation scale according to claim 3 is characterized in that: The MLP part of the TCSMLP of the multimodal Longjing tea grading model is a feedforward neural network consisting of an input layer, a hidden layer, and an output layer. The calculation formula of each hidden layer is: h (l) =σ2(W (l) h (l-1) +b (l) ) Among them, l represents the lth layer in the network, usually numbered from 1, h (l-1) represents the input of the l-1 layer, W (l) and b (l) are the weight and bias of the lth layer, σ2 is the ReLU activation function, and for the input layer, h (l-1) As the model input, for the output layer, the regression task directly calculates the weights, biases and hidden layer outputs to obtain the final output without going through the activation function; The MLP objective function of the TCSMLP of the multimodal Longjing tea grading model is designed based on the mean square error, and the specific formula is: in, Represents the mean square error loss function used by the MLP network in TCSMLP, which is used to measure the square error between the model prediction value and the true value. n is the total number of samples, y i is the true value of the i-th sample, is the model’s predicted value for the i-th sample, represents the square of the prediction error of the sample, is the normalization coefficient used to calculate the average squared error of all samples.

7. The multimodal Longjing tea intelligent grading method integrating five-dimensional evaluation scale according to claim 3 is characterized in that: The XGBoost of the multimodal Longjing tea grading model is an ensemble learning method based on gradient boosting, and its objective function is for: Among them, loss is the loss function, Ω(f k ) is the regularization term, f k represents the kth decision tree, K is the number of decision trees, n is the number of samples, y i is the true sample label of the i-th sample, is the model prediction value of the i-th sample; The regularization term is used to control model complexity and prevent overfitting. It is specifically defined as: Among them, T is the number of leaf nodes in the tree, γ penalizes the number of leaf nodes, ω is the weight of the leaf node, and λ controls the weight amplitude.

8. The multimodal Longjing tea intelligent grading method integrating five-dimensional evaluation scales according to claim 3 is characterized in that: The working process of the multimodal Longjing tea grading model includes: Step 1: Construct a multimodal ensemble learning supervisor, including: building a cache based on the collected small sample image-text dataset, using the pre-trained CLIP VisualEncoder module to embed the image data of each category, and constructing the cache index as shown in the formula: Key=VisualEncoder(I k )∈R NK×C Among them, Key represents the key vector in the image buffer, VisualEncoder is the visual encoder module in the CLIP model, which is used to extract features from image samples, R represents the real number domain, indicating that the value of Key is in the real number range, and I k is the image input of the kth category or sample, N represents the number of categories, the number of rows NK represents a total of N categories × K image samples per category, and a total of NK samples; C represents the embedding feature dimension output by the visual encoder, that is, the vector dimension after each image is encoded; Use OneHot to perform unique heat encoding on the text label corresponding to each category image, and construct the cache key value as shown below: Value=OneHot(L k )∈R NK×N Among them, Value represents the value vector in the image buffer, OneHot represents the one-hot encoding module, which is used to encode the category label. k is the text label or class label corresponding to the k-th image sample, N represents the number of categories, K represents the number of images of each category participating in the buffer construction, and the encoding result is a matrix of dimension NK×N; Query=VisualEncoder(I)∈R 1×C Where Query represents the query vector obtained after the image to be identified is extracted by the visual encoder, I represents the input image code; 1 indicates that there is only one input image sample at present, and C represents the embedding feature dimension output by the visual encoder, so the final output is an embedding vector with a dimension of 1×C; For the image to be predicted, the image to be predicted is input into VisualEncoder to obtain the corresponding embedded coding vector as the query value; the similarity between the cache index and the query value is calculated using the following formula, and the key value is taken from the cache for decoding based on the similarity: Among them, Similarity represents the similarity between the query vector and the key vector, β is the scaling factor used to adjust the range of the similarity function, and Key T Indicates the transposition of Key; Query*Key T is the dot product similarity; therefore, the dimension of the final output Similarity is 1×N×K; Based on the decoding results, the classifier in the pre-trained CLIP model is weighted and combined to make zero-shot predictions on the image data as the corresponding labels of the ensemble learning supervisor: Logits=α*Similarity*Value+Query*W T Among them, Logits represents the final predicted output vector, which is used to judge the corresponding label in supervised learning. α is the regularization coefficient, which is used to balance the contribution of similarity-weighted labels and the original CLIP classifier. T It is the weight transposed matrix of the CLIP model classifier, which is used to map the query to the prediction score of each category; Step 2. The input module preprocesses the tea appearance image, specifically including: cropping, filtering, and channel normalization of the tea appearance image, cropping, normalization, color feature extraction, and texture feature extraction of the tea soup image and leaf bottom image; preprocessing the aroma dataset and flavor dataset, specifically including cleaning, screening, filling, and standardization of the dataset; randomly pairing the tea appearance image dataset, tea soup image dataset, leaf bottom image dataset, aroma dataset, and flavor dataset according to categories in a rectangular distribution; and importing the data pairs into the downstream sub-model for processing according to different modes, specifically inputting the appearance image dataset into Resnet, the leaf bottom image dataset, aroma dataset, and flavor dataset into TCSMLP, and the tea soup image dataset into XGBoost. Step 3: SimCLR in the deep convolutional module first performs self-supervised pre-training on a dataset of various tea appearance comment dimensions. It then transfers the learned representations to ResNet-18 to classify tea appearance images into seven sub-comment dimensions: straightness, smoothness, tenderness, greenness, glossiness, color uniformity, and completeness. It then performs a linear conversion from comment vectors to scores based on a comment-score mapping table. Step 4: The integrated learning module performs regression fitting on the soup color and infused leaf appearance picture data, aroma data, and taste data, specifically including establishing a dynamic adjustment mechanism for the regularization term of network training based on the positive and negative feedback of the multi-modal supervisor labels during the training process, so as to optimize the training of the MLP to obtain the TCSMLP; the dynamic adjustment mechanism is defined as follows: Assume that the overall objective function of the TCSMLP model is in The task loss is the mean square error used for MLP regression, is the regularization term, λ(t) is the dynamic adjustment coefficient, which is controlled by the feedback signal of the supervisor; the definition of the dynamic coefficient is as follows: Among them, λ0 represents the initial regularization coefficient, γ represents the feedback sensitivity coefficient, M represents the number of factors affecting the regularization coefficient, β m Indicates the weight of the mth influencing factor, F m (t) represents the feedback function of the mth influencing factor at time step t; When m=1, the feedback function F m (t) is defined as follows: When m=2, the feedback function F m (t) is defined as follows: in, Represents the loss function value of TCSMLP at time step t, sign is the standard signal function, Label supervisor Indicates the label recorded by the supervisor, Label pred Indicates the label of the current output of the model, sign' is a signal function based on Boolean values; when When the TCSMLP output is inconsistent with the supervisor record label, F m (t) takes a positive value to enhance regularization, otherwise F m (t) takes a negative value to reduce regularization; Perform regression prediction on the aroma data and taste data based on the TCSMLP that has completed training through optimization, and perform regression prediction on the soup color picture data based on XGBoost; Step 5: The output module performs linear weighting on the model results of the five human evaluation scales. The scoring ratios of appearance, soup color, aroma, taste, and infused leaf are p1, p2, p3, p4, and p5 respectively, where 0.20 < p1 < 0.30, 0.10 < p2 < 0.20, 0.20 < p3 < 0.30, 0.20 < p4 < 0.35, 0.05 < p5 < 0.15, and a + b + c + d + e = 1, as shown in the following formula: Result = p1 * Appearance + p2 * SoupColor + p3 * Aroma + p4 * Taste + p5 * InfusedLeaf Among them, Result is the comprehensive score of the tea, Appearance is the appearance score, SoupColor is the soup color score, Aroma is the aroma score, Taste is the taste score, and InfusedLeaf is the infused leaf score.

9. The multimodal Longjing tea intelligent grading method integrating five-dimensional evaluation scales according to claim 8, characterized in that: The corresponding relationship between the total score of the tea and the grade in Step 5 is: C < 88 ≤ B < 90 ≤ A.

10. The multimodal Longjing tea intelligent grading method integrating five-dimensional evaluation scales according to claim 8, characterized in that: The corresponding relationship between the score - quality grade of each single dimension in Step 5: Appearance: C < 88 ≤ B < 91 ≤ A; Aroma: C < 88 ≤ B < 92 ≤ A; Taste: C < 88 ≤ B < 91 ≤ A; Soup color: C ≤ 87 < B ≤ 89 < A; Infused leaf: C ≤ 87 < B ≤ 89 < A.

Images