A multi-modal longjing tea intelligent grading method integrating five-dimensional evaluation scales

Abstract

The application discloses a kind of multi-modal longjing tea intelligent grading methods of integrating five-dimensional review scale, comprising: collecting longjing tea sample;Collect the appearance picture dataset of longjing tea sample, soup color picture dataset, leaf bottom picture dataset, aroma dataset and taste dataset, and carry out pre-processing, data set is divided into training set, test set and verification set;Build multi-modal longjing tea grading model, through training set, test set, verification set, training, testing, verifying to multi-modal longjing tea grading model, obtain the optimal multi-modal longjing tea grading model;Through the optimal multi-modal longjing tea grading model, longjing tea is graded.The application uses digital feature vector, constructs intelligent grading model by multi-modal fusion and contrast learning method, finally forms visual green tea quality evaluation system, achieves the purpose of improving review consistency, reducing human subjective error.

Description

Technical Field

[0001] This invention relates to the field of tea quality assessment, specifically to a multimodal intelligent grading method for Longjing tea that integrates five-dimensional evaluation criteria. Background Technology

[0002] In the vast tea market, whole-leaf tea is often sold according to its grade, making accurate and consistent grading crucial. Longjing green tea, renowned for its quality, is graded based on appearance, aroma, taste, liquor color, and leaf characteristics. These grades influence consumer purchasing choices and significantly impact the economic value of the tea, highlighting the importance of precise quality assessment.

[0003] Traditionally, tea grading has relied primarily on human sensory evaluation, a process prone to subjectivity and inconsistency. While tea experts play a vital role in maintaining quality standards, the shortage of skilled assessors and the inherent limitations of sensory analysis underscore the need for more objective, efficient, and reliable grading methods.

[0004] Recent advances in artificial intelligence (AI) have provided promising solutions for data classification, a technology widely applied in various fields, including drug discovery, intelligent diagnostics, and personalized agriculture. Research has investigated machine learning algorithms and computer vision systems to study specific tea characteristics and the mechanisms of action of tea polyphenols. However, these methods often focus on isolated features rather than the holistic evaluation used by experts in sensory assessment. Chinese patent CN118797413A discloses a method for comprehensive tea grading based on deep learning coupled with multimodal sensory quality fusion sensors. This invention is relatively comprehensive in its overall evaluation, encompassing color, aroma, flavor, and shape, and uses electronic nose, electronic eye, electronic tongue, and near-infrared spectroscopy to collect tea information. However, compared to traditional tea evaluation, this invention still lacks the evaluation of the tea leaf appearance and liquor color, missing crucial evaluation elements for the value of tea. Summary of the Invention

[0005] The purpose of this invention is to provide a multimodal intelligent grading method for Longjing tea that integrates five-dimensional evaluation criteria, in order to solve the problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] A multimodal intelligent grading method for Longjing tea integrating five-dimensional evaluation criteria includes:

[0008] S1. Collect samples of Longjing tea with different prices and grades;

[0009] S2. Collect the appearance image dataset, liquor color image dataset, leaf bottom image dataset, aroma dataset, and taste dataset of Longjing tea samples. Preprocess the appearance image dataset, liquor color image dataset, leaf bottom image dataset, aroma dataset, and taste dataset respectively, and divide the dataset into training set, test set, and validation set.

[0010] S3. Construct a multimodal Longjing tea grading model. Train, test, and validate the multimodal Longjing tea grading model using training, testing, and validation sets to obtain the optimal multimodal Longjing tea grading model.

[0011] S4. Grade Longjing tea using the optimal multimodal Longjing tea grading model.

[0012] Further, S2 includes:

[0013] Collect an appearance image dataset: Take images of tea samples before brewing and use data augmentation methods to enhance the data;

[0014] Collect images of tea liquor color and tea leaf base: Brew tea samples, remove the tea leaves from the tea liquor after brewing, and then take images of tea liquor color and tea leaf base.

[0015] Aroma dataset collection: Extract volatile compounds from brewed tea samples and identify the types and concentrations of volatile compounds;

[0016] Collect a flavor dataset: Identify the types and concentrations of metabolites from brewed tea.

[0017] Furthermore, the multimodal Longjing tea grading model includes an input module, a deep convolution module, a contrastive learning module, an ensemble learning module, and an output module;

[0018] The input module preprocesses the input image, preprocesses the chemical analysis data, and segments and guides the input data downstream.

[0019] The deep convolutional module first uses the SimCLR self-supervised visual representation learning framework to pre-train the delabeled data, and then transfers it to the ResNet convolutional model to classify the tea shape image in 7 sub-comment dimensions, and performs comment-rating transformation on the tea shape image based on linear mapping.

[0020] The contrastive learning module uses CLIP, which combines a cache built based on a few-shot dataset, to perform zero-shot learning prediction of tea image categories, and builds a multimodal ensemble learning supervisor based on this CLIP.

[0021] The integrated learning module uses a multimodal supervisor to dynamically adjust the training process of XGBoost and MLP, and uses the adjusted and trained models XGBoost and TCSMLP to perform regression prediction on aroma data, taste data, soup color data and leaf residue data. TCSMLP is an MLP trained under CLIP supervision.

[0022] The output module performs linear weighting on the scores of five human evaluation criteria, visualizes the tea quality evaluation process in text, and outputs the tea quality evaluation results.

[0023] Furthermore, the SimCLR self-supervised visual representation learning framework of the multimodal Longjing tea grading model learns better visual representations through contrastive learning, and its contrastive loss... Defined as:

[0024]

[0025] Where exp represents the natural exponential function with base e, used to scale similarity values ​​to the positive range; sim represents the similarity calculation function, z i This is a vector obtained by L2 normalizing a single sample. L2 represents L2 norm normalization of the feature vectors, which means dividing each vector by its own Euclidean norm to normalize its magnitude to 1, thereby projecting all vectors onto a unit sphere, facilitating subsequent calculation of cosine similarity; z j Is with z i The feature representation of paired positive samples, z k Indicates the use of z i The similarity calculation function sim is defined as follows: τ is the temperature coefficient used to control the sharpness of the probability distribution among all samples being compared (including positive and negative samples). When τ is smaller, the similarity difference is amplified, and the model pays more attention to negative samples that are misclassified but have only moderate similarity. When τ is larger, the distribution is smoother, and the model's sensitivity to similarity differences decreases. N is the total number of samples.

[0026]

[0027] Where, sim(z) i ,z j ) represents sample z i With sample z j Cosine similarity between them; z i z and zj are the feature vectors of two samples, respectively; z i *z j Represent their dot product; ||z i ‖ and ‖z j‖ are the L2 norms of the two vectors, used for normalization to ensure that the similarity values ​​are in the range of [-1, 1].

[0028] Furthermore, the ResNet convolutional model of the multimodal Longjing tea grading model includes core structure residual blocks and residual connection operations, wherein the output formula of each core structure residual block is:

[0029] h l+1 =h l +σ1(W l *h l +b l )

[0030] Among them, h l h represents the input of the l-th layer. l+1 W represents the transformation after convolution and activation function. l Here, σ1 is the weight of the l-th layer, and b is the ReLU activation function. l It is a bias;

[0031] The cross-entropy loss function of the objective function of the ResNet convolutional model for:

[0032]

[0033] Where N is the number of samples, C is the number of classes, and y i,c p is the true label of the i-th sample in class c. i,c It is the predicted probability that the i-th sample belongs to the c-th class after the output layer is activated by the Softmax function. The last layer of the ResNet convolutional model outputs the classification probability through a fully connected layer, and the cross-entropy loss drives the parameter update.

[0034] Furthermore, the MLP portion of the TCSMLP in the multimodal Longjing tea grading model is a feedforward neural network, consisting of an input layer, hidden layers, and an output layer. The calculation formula for each hidden layer is as follows:

[0035] h (l) =σ2(W (l) h (l-1) +b (l) )

[0036] Where l represents the l-th layer in the network, usually numbered starting from 1, h (l-1) W represents the input of the (l-1)th layer. (l) and b (l) These are the weights and biases of the l-th layer, respectively, and σ² is the ReLU activation function. For the input layer, h... (l-1)For the model input, and for the output layer, the regression task directly calculates the weights, biases and hidden layer outputs to obtain the final output without going through an activation function;

[0037] The MLP objective function of the TCSMLP for the multimodal Longjing tea grading model is designed based on mean squared error, and the specific formula is as follows:

[0038]

[0039] in, This indicates the mean squared error (MSE) loss function used by the MLP network in the TCSMLP module, which measures the squared error between the model's predicted values ​​and the true values. n is the total number of samples, and y... i It is the true value of the i-th sample. It is the model's predicted value for the i-th sample. This represents the squared prediction error for that sample. It is the normalization coefficient, used to calculate the average squared error value of all samples.

[0040] Furthermore, the XGBoost method of the multimodal Longjing tea grading model is an ensemble learning method based on gradient boosting, and its objective function is... for:

[0041]

[0042] Where loss is the loss function, Ω(f k ) is the regularization term, f k This represents the k-th decision tree, where K is the number of decision trees, n is the number of samples, and y i It is the true label of the i-th sample. It is the model prediction value of the i-th sample;

[0043] Regularization terms are used to control model complexity and prevent overfitting. They are specifically defined as follows:

[0044]

[0045] Where T is the number of leaf nodes in the tree, γ is the number of leaf nodes penalized, ω is the weight of the leaf nodes, and λ controls the magnitude of the weights.

[0046] Furthermore, the working process of the multimodal Longjing tea grading model includes:

[0047] Step 1: Construct a multimodal ensemble learning supervisor, including: building a cache based on a collected small sample image-text dataset; using the VisualEncoder module of the pre-trained CLIP to embed and encode the image data for each category; and constructing the cache index as shown in the formula:

[0048] Key = VisualEncoder(I k )∈R NK×C

[0049] Where Key represents the key vector in the image buffer, VisualEncoder is the visual encoder module in the CLIP model, used for feature extraction from image samples, R represents the real number field, indicating that the value of Key is in the real number range, and I k It is the image input of the k-th category or sample, where N represents the number of categories, NK represents the total number of N categories × K image samples per category, and there are NK samples in total; C represents the embedding feature dimension of the visual encoder output, that is, the vector dimension of each image after encoding;

[0050] OneHot is used to perform one-hot encoding on the text labels corresponding to each category of image, and the cache key-value pair is constructed as shown in the following formula:

[0051] Vdlue=OneHot(L k )∈R NK×N

[0052] Where Value represents the value vector in the image buffer, OneHot represents the one-hot encoding module used to encode the category labels one-hot, and L k It is the text label or class label corresponding to the k-th image sample, where N represents the number of categories, K represents the number of images in each category that participate in the construction of the cache, and the encoding result is a matrix of dimension NK×N;

[0053] Query = VisualEncoder(I)∈R 1×C

[0054] Where Query represents the query vector obtained after the image to be recognized is extracted by the visual encoder, I represents the input image encoding; 1 indicates that there is only one input image sample at present, and C represents the embedding feature dimension of the visual encoder output. Therefore, the final output is an embedding vector with a dimension of 1×C.

[0055] For the image to be predicted, input the image into VisualEncoder to obtain the corresponding embedded encoding vector as the query value; calculate the similarity between the cache index and the query value using the following formula, and then retrieve the key value from the cache for decoding based on the similarity:

[0056]

[0057] Where Similarity represents the similarity between the query vector and the key vector, β is a scaling factor used to adjust the range of variation of the similarity function, and Key TIndicates the transpose of the key; Query*Key T The similarity is calculated using dot product; therefore, the final output similarity has a dimension of 1×N×K.

[0058] Based on the decoding results, the weighted combination of the classifier in the pre-trained CLIP model is used to perform zero-shot predictions on the image data, which serve as the labels for the ensemble learning supervisor.

[0059] Logits=α*Similarity*Value+Query*W T

[0060] Where Logits represents the final predicted output vector, used for label judgment in supervised learning, and α is the regularization coefficient used to balance the contributions of similarity-weighted labels and the original CLIP classifier; W T It is the weight transpose matrix of the CLIP model classifier, used to map the query to the predicted scores of each category;

[0061] Step 2: The input module preprocesses the tea appearance images, specifically including: cropping, filtering, and channel normalization of the tea appearance images; cropping, normalizing, extracting color features, and extracting texture features from the liquor color and leaf bottom images; preprocessing the aroma and flavor datasets, specifically including cleaning, filtering, filling, and standardization of the datasets; randomly pairing the tea appearance image dataset, liquor color image dataset, leaf bottom image dataset, aroma dataset, and flavor dataset according to their categories, following a rectangular distribution; and importing the data pairs into downstream sub-models for processing based on their different modalities, specifically inputting the appearance image dataset into ResNet, the leaf bottom image dataset, aroma dataset, and flavor dataset into TCSMLP, and the liquor color image dataset into XGBoost.

[0062] Step 3: SimCLR in the deep convolution module first performs self-supervised pre-training on the dataset of tea appearance evaluation dimensions, and then transfers the learned representations to ResNet-18 to classify the tea appearance images according to seven sub-evaluation dimensions, specifically including straightness, smoothness, tenderness, greenness, gloss, color uniformity, and integrity; and performs linear transformation from evaluation vectors to scores based on the evaluation-score mapping table.

[0063] Step 4: The ensemble learning module performs regression fitting on the soup color and leaf appearance image data, aroma data, and taste data. Specifically, this includes establishing a dynamic adjustment mechanism for the regularization term of the network training based on the positive and negative feedback of the multimodal supervisor labels during the training process, thereby optimizing the training of the MLP to obtain the TCSMLP. The dynamic adjustment mechanism is defined as follows:

[0064] Let the total objective function of the TCSMLP model be where is the task loss, which is the mean squared error used in MLP regression, is the regularization term, λ(t) is the dynamic adjustment coefficient, which is controlled by the feedback signal of the supervisor; the construction method of the dynamic coefficient is defined as follows:

[0065]

[0066] where, λ0 represents the initial regularization coefficient, γ represents the feedback sensitivity coefficient, M represents the number of regularization coefficient influence factors, β m represents the weight of the m-th influence factor, F m (t) represents the feedback function of the m-th influence factor at time step t;

[0067] When m = 1, the definition of the feedback function F m (t) is as follows:

[0068]

[0069] When m = 2, the definition of the feedback function F m (t) is as follows:

[0070]

[0071] where, represents the value of the loss function of TCSMLP at time step t, sign is the standard signal function, Label supervisor represents the label recorded by the supervisor, Label pred represents the label currently output by the model, sign' is the signal function based on boolean values; when increases or the output of TCSMLP is inconsistent with the label recorded by the supervisor, F m (t) takes a positive value to enhance regularization, otherwise F m (t) takes a negative value to reduce regularization;

[0072] Based on the TCSMLP trained by optimization, regression prediction is performed on aroma data and taste data, and regression prediction is performed on soup color picture data based on XGBoost;

[0073] Step 5: The output module performs linear weighting on the model results of five human evaluation scales. The scoring ratios of appearance, soup color, aroma, taste, and leaf bottom are p1, p2, p3, p4, and p5 respectively, 0.20 < p1 < 0.30, 0.10 < p2 < 0.20, 0.20 < p3 < 0.30, 0.20 < p4 < 0.35, 0.5 < p5 < 0.15, a + b + c + d + e = 1, as shown in the following formula:

[0074] Result=p1*Appearance+p2*SoupColor+p3*Aroma+p4*Taste+p5*InfusedLeaf

[0075] Among them, Result is the overall score of tea, Appearance is the appearance score, Soup Color is the soup color score, Aroma is the aroma score, Taste is the taste score, and Infused Leaf is the infused leaf score.

[0076] Furthermore, in step 5, the total score of the tea corresponds to the grade as follows: C<88≤B<90≤A.

[0077] Furthermore, the score-quality level correspondence for each single dimension in step 5 is as follows:

[0078] Appearance: C < 88 ≤ B < 91 ≤ A;

[0079] Aroma: C < 88 ≤ B < 92 ≤ A;

[0080] Taste: C < 88 ≤ B < 91 ≤ A;

[0081] Soup color: C≤87<B≤89<A;

[0082] Leaf base: C≤87<B≤89<A.

[0083] Compared with the prior art, the beneficial effects of the present invention are:

[0084] 1) Comprehensive Scope of Machine Tea Evaluation: This invention expands the scope of machine tea evaluation to five key dimensions for the first time: appearance, aroma, taste, liquor color, and infused leaf appearance. Compared with previous methods, this broad approach provides a more comprehensive and objective assessment of tea quality.

[0085] 2) More refined appearance dimensions: This invention also introduces a more refined evaluation of tea appearance, refining it into seven different sub-dimensions: straightness, smoothness, tenderness, greenness, gloss, color uniformity, and integrity. Previously, the appearance of tea had not been evaluated in such detail.

[0086] 3) High performance across multiple tea quality dimensions: The accuracy of classifying tea samples based on multiple dimensions (appearance, taste, leaf residue, aroma, and liquor color) is as high as 92%, indicating that the AI-driven system is very effective in capturing subtle differences in tea quality.

[0087] 4) Combined use of multimodal and multimodal systems: The CLIP (contrastive language-image pre-trained) model is used to supervise the training of each sub-model of the hybrid expert system to ensure the interpretability of Longjing tea (green) quality grading, while the multimodal-CLIP model is used to ensure the classification performance of tea quality grades. Attached Figure Description

[0088] Figure 1 A flowchart illustrating the workflow for digitizing data on appearance, soup color, aroma, taste, and leaf residue, as well as for model training.

[0089] Figure 2 Schematic diagram of data collection for appearance, liquor color, aroma, taste, and infused leaf; a) schematic diagram of the shooting equipment; extraction of the region of interest; b) the infused leaf and liquor color of tea leaves with three grade standards (A, B, and C), from the special grade, second grade, and first grade tea leaves of Longwu Tea Town; c) schematic diagram of metabolite extraction.

[0090] Figure 3 Images of the standard sample and a schematic diagram of the infusion color collected from the leaf base.

[0091] Figure 4 In the middle, 'a' represents the fine-tuned ResNet-18 model. Figure 4 The graphs in the table below represent the training results for seven sub-dimensions using a fine-tuned ResNet-18 model: bh represents the accuracy and loss curves; b represents the accuracy curve for smoothness; c represents the accuracy curve for greenness; d represents the accuracy curve for tenderness; e represents the accuracy curve for gloss; f represents the accuracy curve for color uniformity; g represents the accuracy curve for integrity; h represents the loss curve for straightness; i represents the loss curve for smoothness; j represents the loss curve for greenness; k represents the loss curve for tenderness; l represents the loss curve for gloss; m represents the loss curve for color uniformity; n represents the loss curve for integrity; o represents the loss curve for straightness; and p represents the tea appearance classification confusion matrix after summing the scores for each dimension.

[0092] Figure 5 The diagram shows the confusion matrix for the seven sub-dimensions of tea appearance and the corresponding classification result metric. In the diagram, a represents smoothness, b represents greenness, c represents tenderness, d represents gloss, e represents color uniformity, f represents integrity, g represents straightness, h represents smoothness, i represents greenness, j represents tenderness, k represents gloss, l represents color uniformity, m represents integrity, and n represents straightness.

[0093] Figure 6The image shows the training results of the leaf base model (e.g., the XGBoost model), where a is a schematic diagram of the XGBoost model; n is the training loss curve of the leaf base model; c is the test loss curve of the leaf base model; d is the three-class ROC curve of the leaf base model; e is the confusion matrix of the leaf base model; f is the training loss curve of the leaf base model; g is the test loss curve of the leaf base model; h is the three-class ROC curve of the leaf base model; and i is the confusion matrix of the leaf base model.

[0094] Figure 7 This is a schematic diagram of PCA and OPLS-DA analysis of GC-MS data derived from aroma. In the figure, a) shows the PCA analysis of aroma data for three different grades of tea (A, B, and C), where QC represents the equal-mass mixed sample group of the three groups. b) shows the OPLS-DA analysis of aroma data for tea groups A and B. c) shows the OPLS-DA analysis of aroma data for tea groups B and C. d) shows the OPLS-DA analysis of aroma data for tea groups A and C.

[0095] Figure 8 Schematic diagram of PCA and OPLS-DA analysis of metabolomics data derived from taste. In the figure, a) is the PCA analysis of taste data of different grades of tea in groups A, B, and C, where QC represents the equal mass mixed sample group of groups A, B, and C; b) is the OPLS-DA analysis of taste data of tea in groups A and B; c) is the OPLS-DA analysis of taste data of tea in groups B and C; and d) is the OPLS-DA analysis of taste data of tea in groups A and C.

[0096] Figure 9 This is a schematic diagram of the CLIP supervisor principle. In the diagram, a represents contrastive pre-training; b represents creating a dataset classifier from labeled text; c represents zero-probability prediction; and d represents the Tip-CLIP pipeline.

[0097] Figure 10The figures show the metabolomics, GC-MS, and leaf substrate training results of TCSMLP (TCSMLP trained with CLIP) and the comprehensive tea rating results combining five dimensions. Figure a shows the MLP structure diagram for metabolomics, b shows the metabolomics training loss curve, c shows the metabolomics three-class ROC plot, and d shows the metabolomics confusion matrix; e shows the MLP structure diagram for GC-MS, f shows the GC-MS training loss curve, g shows the GC-MS three-class ROC plot, and h shows the GC-MS confusion matrix; i shows the MLP structure diagram for leaf substrate, j shows the leaf substrate training loss curve, k shows the leaf substrate three-class ROC plot, and l shows the leaf substrate confusion matrix; m shows the image of tea leaves stacked using CLIP. The three-class ROC curves are obtained using image data, metabolomics, GCMS, soup color, and leaf residue as text data; n is the confusion matrix obtained using CLIP to collect images of stacked tea leaves as image data, and metabolomics, GCMS, soup color, and leaf residue as text data; o is the three-class ROC curve obtained by training ResNet on images of stacked tea leaves, training TCSMLP on metabolomics, GCMS, and leaf residue, and training XGBoost on soup color, with weighted classification across five data dimensions; p is the confusion matrix obtained by training ResNet on images of stacked tea leaves, training TCSMLP on metabolomics, GCMS, and leaf residue, and training XGBoost on soup color, with weighted classification across five data dimensions. Detailed Implementation

[0098] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0099] 1. Experimental materials

[0100] We collected 114 samples of 38 varieties of Longjing green tea, with each variety tested in triplicate, from five suppliers in Hangzhou, Zhejiang Province. Details of price and origin are provided in Table 1. The tea samples were categorized into three different quality grades based on human sensory evaluation, with specific information provided in the next paragraph. All samples were purchased directly from the retail market, packaged in sealed aluminum foil bags, and stored at 4°C until analysis.

[0101] 2. Sensory evaluation

[0102] Tea sensory evaluation grading

[0103] Tea experts conducted sensory evaluations based on the GB / T 23666-2018 green tea quality standard. The evaluation included appearance, aroma, taste, liquor color, and infused leaf appearance, with a weighted total score. Each sample (3 grams) was steeped in 150 ml of freshly boiled water for 4 minutes, and the following attributes were evaluated: appearance, liquor color, aroma, taste, and infused leaf appearance. The weighted scores for appearance, liquor color, aroma, taste, and infused leaf appearance were 25%, 10%, 25%, 30%, and 10%, respectively.

[0104] Appearance: The shape, tenderness, color, breakage, and cleanliness of the dried tea leaves are evaluated.

[0105] Tea liquor color: Evaluate the type, intensity, brightness, depth, and turbidity of the tea liquor color.

[0106] Aroma: Evaluate the type, concentration, purity, and persistence of the aroma of the tea liquor.

[0107] Taste: Evaluate the concentration, body, mellowness, astringency, purity, and freshness of the tea soup.

[0108] Leaf base: Evaluate the tenderness, color, brightness, depth, and uniformity (including the uniformity of tenderness and color) of the leaf base.

[0109] 3. Appearance dimension data collection and preprocessing

[0110] Tea photography

[0111] We used a custom computer vision system to capture images of tea samples, such as... Figure 1 As shown in Figure a, the system was surrounded by a black opaque acrylic panel to simulate a darkroom environment. It was equipped with a Sony ILCE-7M2 digital SLR camera with a 50mm fixed-focus lens, positioned 350mm from the sample. Three LED light strips provided stable white light illumination. Each sample contained 5g of tea leaves, placed on a 400×400mm platform. Raw images were saved in JPEG format using RGB color mode. All images were taken using consistent camera settings: manual mode (M), aperture f / 4.5, shutter speed 1 / 160, ISO 100, auto white balance, 0 exposure compensation, wide-area focus, single-point focus mode, and multi-point / center-based metering mode. 200 images were taken for each sample.

[0112] The original image has a resolution of 6000×4000 pixels. Unlike typical fruit and vegetable images (with clear object and background boundaries), tea leaf images consist of stacked leaves, thus requiring no precise segmentation. Image preprocessing was performed by extracting a 224×224 pixel region of interest (ROI) from the original image, centered as shown below. Figure 2As shown in Figure a, this simple extraction technique effectively captures the color, shape, and texture of tea leaves without requiring complex segmentation algorithms.

[0113] from Figure 2 As can be clearly seen in image a, tea images of different qualities exhibit similar visual features, posing a challenge to image-based classification algorithms. High-quality tea typically has a complete, uniform shape and color with fewer impurities, while low-quality tea is often curled, shriveled, faded, and contains visible impurities, including broken leaves.

[0114] Approximately 200 images were collected for each sample, with the dataset allocated 75% for training and validation and 25% for testing. To augment the dataset, data augmentation techniques were employed, including image rotation (90° and 180°) and flipping (vertical and horizontal), such as... Figure 3 As shown in b, this process increased the dataset sixfold, enhancing the model's generalization ability and reducing the risk of overfitting. Table 1 provides detailed information on the tea image dataset and the data augmentation method.

[0115] 4. Data Acquisition and Preprocessing for Leaf Base and Soup Color: Texture and Color Feature Extraction

[0116] Before photographing the tea liquor and infused leaf color, we followed the methods outlined in GB / T23776-2018, the standard for sensory evaluation of tea. Before photographing, the infused leaves were separated from the tea liquor. The evaluation of the tea liquor color was conducted after the first infusion. Thirty-eight tea samples were divided into three batches, each containing 12 to 13 samples. For each sample, 3 grams of tea leaves were placed in a tasting cup, 150 ml of boiling water was added, and the cup was steeped for 4 minutes with the lid on. The tea liquor was then poured into tasting bowls and allowed to cool for 25 minutes until no more steam was visible. The tea liquor was placed in white porcelain bowls to suppress light transmission, and images were captured using a photographic system. Due to the limited number of samples, only one image of the tea liquor color was taken for each sample, resulting in a total of 38 images. For the tea liquor color image dataset, considering that the color of the tea liquor and the amount of tea fragments are the key features of this dataset, it is necessary to exclude factors that affect these features, such as the teacup and the tabletop. For each tea liquor color image, the original image is first cropped to the center using a 1600px × 1600px rectangle. Then, the cropped result is divided into 3×3 equally spaced sections, and five images (top, bottom, left, right, and center) are taken as the tea liquor color image dataset, thus expanding the size of the single-class tea liquor color dataset from 38 images to 190 images. The same approach is used for the tea leaf image dataset.

[0117] Similarly, the tea leaves were placed in a white porcelain bowl, and the shooting parameters were the same as when shooting the stacked tea leaves. One image of the tea leaf was taken for each sample, resulting in a total of 38 images. Color features were extracted from the tea infusion, including R (red value), G (green value), B (blue value), H (hue), S (saturation), V (value), L* (lightness), a* (red-green axis), and b* (yellow-blue axis). Color and texture features were extracted from the tea leaves. In addition to the above nine color features, contrast, dissimilarity, homogeneity, correlation, and second-order angular moments (0°, 45°, 90°, and 135° in four directions) were also extracted. Preprocessing:

[0118] When extracting color features from the soup color and leaf base, the `cvtColor` function from the OpenCV library is used to convert the image to a specific color space, such as HSV or Lab. Then, the `split` function is used to calculate color features based on the pixels of the entire image. When extracting morphological features from the soup color and leaf base, each image is first converted to a grayscale space. Then, the grayscale levels 0-255 are mapped to levels 0-16, thereby compressing the dimensionality of the grayscale co-occurrence matrix (GCMM). Finally, morphological features are calculated based on the GCMM with offsets in four directions.

[0119] 5. Aroma Data Acquisition and Preprocessing

[0120] Volatile compounds were extracted from a tea sample (0.5 g). The sample was sealed in a 20 mL glass bottle, and 10 μL of ethyl decanoate (internal standard, concentration 10 mg / L) and 5 mL of boiling deionized water were added sequentially. After equilibration for 5 minutes, the bottle was placed in a 60 °C water bath, and the gaseous volatiles were adsorbed using a fiber coated with divinylbenzene / chlorinated polystyrene / polydimethylsiloxane (50 / 30 μm, StableFlex 2 cm, Supelco, Belfort, PA, USA) for 60 minutes. The volatiles were then desorbed by heating at 250 °C for 5 minutes in a GC-MS injector. Volatile analysis was performed using an Agilent 7890 gas chromatograph coupled with an Agilent HP 5977MSD ion trap mass spectrometer (Wilmington, Delaware, USA), equipped with an HP-5MS capillary column (30 m × 250 μm × 0.25 μm). Gas chromatography conditions were as follows: injector temperature 250℃, carrier gas was high-purity helium (99.999%), flow rate was 1.0 mL / min (split-mode-free). Temperature program was: 40℃ for 2 minutes, ramped to 85℃ at 2℃ / min, held for 2 minutes; ramped to 180℃ at 2.5℃ / min, held for 2 minutes; ramped to 230℃ at 10℃ / min, held for 2 minutes. Mass spectrometry (MS) was performed in 70 eV electron impact mode (EI), with a mass scan range of 40–400 m / z and an ion source temperature of 230℃.

[0121] The identification of volatiles was performed using the NIST mass spectrometry search program (NIST Library 20) and a GC-MS data analysis program developed based on Microsoft Excel. GC peak identification was achieved by matching mass spectra and retention indices. The semi-quantitative concentrations of volatiles were calculated by comparing the total ion current response of the mass spectrometer with the response of the internal standard.

[0122] 6. Flavor Data Acquisition and Processing

[0123] To extract metabolites, 1 gram of sample was mixed with 50 grams of boiling water (100°C), shaken thoroughly, and heated in a 100°C water bath for 30 minutes. Shaking was repeated twice, once every 10 minutes. The mixture was then centrifuged at 8,000 rpm for 10 minutes to remove the supernatant. Next, the mixture was centrifuged at 12,000 rpm (13,800 × g) at 4°C for 15 minutes, and the supernatant was collected and transferred to a syringe for analysis. For quality control (QC), equal volumes of supernatant from each sample were mixed for machine analysis.

[0124] LC-MS / MS analysis was performed using a UHPLC system (Vanquish, Thermo Fisher Scientific) equipped with a UPLCHSS T3 column (2.1 × 100 mm, 1.8 μm) coupled to an Orbitrap Exploris 120 mass spectrometer (Thermo Fisher Scientific). The mobile phase consisted of 5 mmol / L ammonium acetate and 5 mmol / L acetic acid (A) in water, along with acetonitrile (B). The autosampler was maintained at 4°C, with an injection volume of 2 μL. The Orbitrap Exploris 120 mass spectrometer was operated in information-dependent acquisition mode, with Xcalibur (Thermo Fisher) software as the control software. The electrospray ionization (ESI) source conditions are as follows: protection gas flow rate 50 Arb, auxiliary gas flow rate 15 Arb, capillary temperature 320℃, full scan resolution 60,000, MS / MS resolution 15,000, collision energy 10 / 30 / 60 NCE mode, and spray voltage 3.8 kV (positive ion mode) or 3.4 kV (negative ion mode).

[0125] The raw data was converted to mzXML format using ProteoWizard, and then peak detection, extraction, alignment, and integration were performed using an internal XCMS (R language) based program. Metabolite annotation was performed using the internal MS2 database (BiotreeDB) with an annotation threshold of 0.3.

[0126] 7. Statistical Analysis

[0127] All experiments were performed in triplicate. Multivariate analysis was performed using SIMCA 13.0 (Umetrics AB, Umeå, Sweden). Statistical analysis was performed using IBM statistical software (version 25.0, SPSS Inc., Chicago, USA), employing one-way ANOVA and significance testing (P < 0.05). Graphs were plotted using Origin (version 2018, OriginLabCorp., Northampton, USA).

[0128] Principal Component Analysis (PCA) is a commonly used dimensionality reduction method. It transforms a set of correlated variables into a set of smaller, uncorrelated variables called principal components. The first principal component explains the largest variance, and subsequent principal components are arranged in descending order of variance. PCA reduces data dimensionality while preserving as much information as possible. Before performing PCA, the data is standardized to mitigate the impact of outliers.

[0129] Orthogonal Partial Least Squares Discriminant Analysis (OPLS-DA) is a statistical method used to distinguish and interpret class differences in multivariate datasets. OPLS-DA enhances the interpretability and predictive accuracy of the model by decomposing predictor variables into class-related and class-independent orthogonal components. It is widely used in metabolomics, genomics, and related fields to identify biomarkers and categorical features. Compared to PLS-DA, OPLS-DA more effectively removes class-independent noise, thereby improving the robustness and interpretability of the model.

[0130] 8. Training Model Structure and Design

[0131] A multimodal grading model for Longjing tea

[0132] 8.1. Overall Structure of the Model

[0133] This invention provides a multimodal grading model for Longjing tea that integrates five human evaluation criteria. The model includes:

[0134] Input module: Responsible for receiving and preprocessing multimodal data, including appearance images, soup color images, leaf images, aroma data, and taste data, and performing data augmentation, standardization, and segmentation operations.

[0135] Deep Convolutional Module: Based on an improved ResNet convolutional neural network, it extracts multi-dimensional features from appearance images and converts the features into corresponding scores through linear mapping.

[0136] Integrated learning module: Improved XGBoost and TCSMLP (CLIP-supervised MLP) are used to perform regression prediction on aroma, taste, liquor color and leaf residue data to achieve quantitative evaluation of tea quality.

[0137] Contrastive learning module: The improved CLIP model is used to predict the category of tea images, and the ensemble learning supervision mechanism is combined to improve the generalization ability of the model.

[0138] Output module: Linearly weights the scores of each evaluation dimension to output the final tea quality grade and visualizes the evaluation results.

[0139] 8.2. Workflow

[0140] The workflow of the multimodal Longjing tea grading model of the present invention includes:

[0141] 1. Data Acquisition and Preprocessing: Samples of Longjing tea of ​​different prices and grades were collected, and images of appearance, liquor color, and leaf residue were obtained, along with data on volatile compounds and metabolites. Image data underwent enhancement, cropping, filtering, and normalization. Chemical analysis data were cleaned, screened, filled, and standardized.

[0142] 2. Model Training:

[0143] Deep convolution module training: ResNet network is used to extract multi-dimensional features of appearance images and automatic scoring is achieved through a comment-rating mapping mechanism.

[0144] Ensemble learning module training:

[0145] XGBoost is used for regression prediction of soup color data.

[0146] TCSMLP Training:

[0147] ① A multimodal supervisor is constructed by pre-training TCSMLP using the CLIP supervision mechanism.

[0148] ② Generate image embeddings based on the CLIP visual encoder (VisualEncoder) and store category features in a cache:

[0149] Key = VisualEncoder(I k )∈R NK×C

[0150] ③ Use one-hot encoding to process text category tags:

[0151] Value = OneHot(L k )∈R NK×N

[0152] ④ Calculate the similarity score for CLIP supervision:

[0153]

[0154] ⑤ Combine CLIP classifier and TCSMLP for final prediction:

[0155] Logits=α*Similarity*Value+Query*W T

[0156] ⑥ Finally, TCSMLP was used to perform regression prediction on aroma, taste, and leaf residue data, and the loss function was optimized to improve prediction accuracy.

[0157] 8.3 Reasoning and Grading:

[0158] 1. Process the multimodal data of the input tea samples, and perform feature extraction and scoring calculation through each sub-model;

[0159] 2. A linear weighted average is used to calculate the overall score of the tea, and the tea grade is determined in combination with the preset scoring criteria;

[0160] 3. Generate the final evaluation results, including numerical scores and grading suggestions.

[0161] 8.4. Mathematical Description of Each Module

[0162] ResNet residual block calculation formula:

[0163] h l+1 =h l +σ(W l *h l +b l )

[0164] Among them, h l F(h) represents the input of the l-th layer. l, W l W represents the transformation after convolution and activation function. l Here, σ is the weight of the l-th layer, and b is the activation function. l It is a bias.

[0165] TCSMLP Calculation Formula (CLIP-Supervised MLP):

[0166] h (l) =σ(W (l) h (l-1) +b (l) )

[0167] Among them, h (l) W represents the input of the l-th layer. (l) and b (l) These are the weights and biases of the l-th layer, respectively, and σ is the activation function.

[0168] XGBoost loss function:

[0169]

[0170] Where loss is the loss function, Ω(f k ) is the regularization term, K is the number of decision trees, n is the number of samples, and y i These are the true labels of the samples. These are model predictions;

[0171] Final score calculation:

[0172] Result=0.25*Appearance+0.1*SoupColor+0.25*Aroma+0.3*Taste+0.1*InfusedLeaf

[0173] 8.5. Tea Grading Standards

[0174] The relationship between total score and grade is: C<88≤B<90≤A.

[0175] Scoring criteria for each dimension:

[0176] Appearance: C < 88 ≤ B < 91 ≤ A;

[0177] Aroma: C < 88 ≤ B < 92 ≤ A;

[0178] Taste: C < 88 ≤ B < 91 ≤ A;

[0179] Soup color: C≤87<B≤89<A;

[0180] Leaf base: C≤87<B≤89<A.

[0181] This invention achieves intelligent grading of Longjing tea by using multimodal data fusion, CLIP supervised learning, deep learning, and ensemble learning techniques, thereby improving the accuracy and generalization ability of grading and providing an efficient and accurate method for tea quality assessment.

[0182] All experiments were conducted on a computer running Windows 10, equipped with an 8-core i7-9750H CPU (2.6GHz), 16GB DDR4 memory, and an NVIDIA GeForce GTX 1060 GPU (6GB video memory, CUDA 10.2.89). CNN models were deployed using the PyTorch framework (version 1.10.2) based on Python 3.6.3 to ensure stability. Libraries used for image preprocessing, feature extraction, and machine learning included py-opencv 3.4.2, numpy 1.19.5, and scikit-learn 0.24.2. For the GC-MS and metabolomics datasets, the regression model converged after 200 epochs of training, while the classification model for the image dataset converged after 10 epochs. Parameters from the best-performing model on the training set were saved for testing. Each training experiment was repeated five times with random initialization to minimize error, and the average results were reported.

[0183] 8.6 Detailed introduction of each module

[0184] 8.6.1 ResNet

[0185] ResNet (Residual Network) is a deep neural network architecture proposed by Kaiming He et al. at Microsoft Research Asia in 2015. By introducing residual connections, ResNet addresses the vanishing and exploding gradient problems during deep network training, effectively increasing network depth and improving performance. The core concept of ResNet is adding shortcut connections, allowing input information to bypass intermediate layers and be directly passed to subsequent layers. This mechanism ensures that even if deep layers fail to learn useful features, the model can still rely on input information from earlier layers, avoiding network performance degradation. Residual connections simplify the learning process by allowing the network to learn only the residuals (i.e., the differences between input and output).

[0186] ResNet consists of multiple residual blocks concatenated. Each block has two paths: an identity mapping path, which directly passes the input information to the output; and a residual mapping path, which transforms the input information before merging it with the output of the identity mapping. This structure facilitates gradient flow during backpropagation, thereby mitigating the problems of vanishing or exploding gradients.

[0187] In this invention, ResNet and its extended models are selected to classify tea morphology data to verify the effectiveness of the residual structure. The architecture used is ResNet-18, including an input layer (224×224×3), convolutional layers (7×7, 64 filters, stride 2, output size 112×112), max pooling layers (3×3, stride 2, output size 56×56), and convolutional layers with progressively increasing filter numbers (64, 128, 256, 512 filters), with the output size gradually decreasing. This is followed by average pooling layers and fully connected layers with an output size of 1×3 for classification. The floating computation of this model is approximately 1.8×10⁻⁶. 9 .

[0188] The ResNet-18 model was pre-trained on the ImageNet-1K dataset. After loading the pre-trained parameters, the model was fine-tuned using a stacked tea leaf image dataset containing 5,822 training images and 1,941 test images. The training hyperparameters were set as follows: learning rate = 5e-5, batch size = 128, and number of training epochs = 10. A dynamic learning rate adjustment strategy was employed during training, with a learning rate decay parameter set. The learning rate was decayed when the number of training epochs reached a preset threshold, ensuring the model effectively converged during training. To increase the size of the training set and improve the model's classification accuracy and generalization ability, data augmentation techniques such as 90° rotation, vertical and horizontal flipping, and 180° rotation were used, increasing the training set size fivefold.

[0189] After training each classification model used to evaluate tea appearance, the classification accuracy parameters that performed best on the validation set were saved. These models were then used to infer individual tea appearance images, generating an evaluation descriptor sequence for each image. This process was applied to the entire tea appearance image dataset, resulting in multiple descriptor sequence matrices. To establish the conversion relationship between tea appearance descriptors and appearance scores, a pre-built database containing descriptors for each category and their corresponding scores was used, facilitating the conversion of the descriptor matrix to the score sequence. Specifically, seven evaluation dimensions were used for stacked tea appearance: straightness, smoothness, tenderness, greenness, gloss, color uniformity, and integrity, with each dimension including multiple categories, totaling 46 categories. Therefore, each processed stacked tea image generated a 1×46 descriptor sequence vector. The complete stacked tea appearance dataset contains 7,763 images, resulting in a 7,763×46 matrix. Using the descriptor-score database, matrix operations were performed to convert this matrix into a 7,763×1 score sequence, with each entry representing the appearance score of the corresponding image.

[0190] 8.6.2 Multilayer Perceptron (MLP)

[0191] A Multilayer Perceptron (MLP) is a fundamental artificial neural network (ANN) architecture consisting of multiple layers: an input layer, one or more hidden layers, and an output layer. An MLP is a feedforward network, meaning information flows unidirectionally from input to output, and all layers are fully connected. Through the combination of multiple layers of neurons and non-linear activation functions, MLPs are able to learn the complex relationships between inputs and outputs.

[0192] The learning process involves forward and backpropagation. During forward propagation, the input signal is passed through the hidden layers, generating the model's predictions. Backpropagation calculates the gradient of the loss function relative to the model parameters, updating the weights. Through repeated forward and backpropagation, the model adjusts its weights and biases, gradually improving the prediction results and bringing them closer to the true values.

[0193] In this invention, a custom MLP architecture was selected as the regression model for GC-MS and metabolomics data to evaluate its effectiveness. For metabolomics data, the architecture includes an input layer (18 features), three hidden layers (256, 128, and 64 nodes), and an output layer. The GC-MS model architecture includes an input layer (312 features), three hidden layers (209, 140, and 94 nodes), and an output layer. For the appearance of dispersed tea leaves, the architecture includes an input layer (20 features), three hidden layers (256, 128, and 64 nodes), and an output layer. The training parameters for both models were set as follows: num_epochs = 200, learning rate = 0.01 (metabolics) or 0.001 (GC-MS), weight_decay = 0.003, batch size = 16. To ensure robustness, five-fold cross-validation was applied.

[0194] 8.6.3 XGBoost

[0195] XGBoost (eXtreme Gradient Boosting) is a high-performance gradient boosting framework that improves prediction accuracy by iteratively building decision trees. The goal of each tree is to correct the errors of the previous tree. XGBoost also incorporates optimization techniques such as regularization to prevent overfitting.

[0196] Key parameters of XGBoost include:

[0197] max_depth: The maximum depth of the tree (increasing the depth increases complexity and the risk of overfitting).

[0198] eta (learning rate): controls the step size of weight updates.

[0199] num_round: The number of rounds of promotion.

[0200] In this experiment, the parameters were set as follows: max_depth = 3, eta = 0.1, num_round = 100, and the loss function was regression.

[0201] 8.6.4 Tip Clip and TCSMLP

[0202] CLIP (Contrastive Language-Image Pre-Training) is a multimodal pre-trained neural network released by OpenAI in 2021. It represents an efficient and transferable method for learning through natural language supervision. The basic CLIP model is pre-trained on a massive dataset of 400 million text-image pairs. The core idea of ​​this model is to learn the alignment between images and text using a large amount of text and image pairing data. The model consists of two main components: a text encoder, which converts text into a low-dimensional vector representation; and an image encoder, which converts images into similar vector representations. In the prediction phase, the CLIP model generates predictions by calculating the cosine similarity between text and image vectors, making CLIP particularly suitable for zero-shot and few-shot learning tasks. Figure 9 (a–c)

[0203] For zero-shot classification of tea quality grading datasets (including appearance images, GCMS data, metabolomics data, and leaf bottom images), feature engineering can be used to convert the latter four types of data into feature vectors of specific dimensions. These vectors are then combined with the appearance images output by the image encoder to obtain a new encoding vector I1. The text encoder outputs text encoding vectors T1, T2, and T3 for special grades of tea, grade 1 tea, and grade 2 tea. The cosine similarity between I1 and T1, T2, and T3 is calculated, and the category corresponding to the highest text encoding vector is taken as the category of the current combined input data.

[0204] The key issue is that the basic CLIP model, pre-trained with 400 million text-image pairs, may fail to learn domain-specific data such as tea quality grading, leading to lower classification accuracy. To further improve CLIP's transfer performance in downstream tasks while considering time and space resource costs, this paper introduces a few-shot image classification method—Tip Clip—which improves CLIP accuracy without requiring additional downstream training. This method utilizes CLIP to construct a cache model, storing classification knowledge from downstream training data (in this case, tea quality grading) in a non-training manner. Subsequently, during testing, the prediction results of the cache model and the basic CLIP model are linearly summed to obtain more reliable classification results.

[0205] In this method, a contrastive language-image pre-trained (CLIP) model is used to train on existing text (leaf bottom, metabolomics, GC-MS) and image (tea appearance images) datasets. However, since its pre-training is performed on general data, it may not obtain specialized representations for tea quality grades, leading to reduced classification accuracy. To enhance the transfer performance of CLIP and address time and space constraints, a few-sample image classification method, Tip-CLIP, is proposed. This method improves the accuracy of CLIP without requiring additional downstream training. Tip-CLIP utilizes CLIP to create a cached model ( Figure 9 (d) Classification knowledge acquired from downstream training data (especially knowledge about tea grading) is preserved in a non-training manner. During testing, the classification result is derived from a linear combination of the predictions of the cached model and the CLIP base model, resulting in more reliable classification results. After enhancing the CLIP model with tea grading knowledge, the improved Tip-CLIP model was used for supervised regression tasks on metabolomics, GC-MS, and leaf residue datasets. The Tip-CLIP supervised multilayer perceptron (TCSMLP), developed under CLIP supervision, significantly improved the fitting performance, with R... 2 The value exceeded that of the original MLP model.

[0206] 9. Results and Analysis

[0207] (1) Evaluation results of Longjing green tea

[0208] According to GB / T 23776-201824, the sensory evaluation standard for tea, we conducted standardized sensory evaluations on 38 Longjing green tea samples. Comments and scores for each evaluation indicator were recorded, and a weighted sum was calculated to determine the total score, ranging from 85 to 92 (Table 1). Most samples exhibited the typical flat and straight appearance of Longjing tea; however, six samples showed over-roasting, affecting their fresh aroma and leading to a decrease in aroma score. Market preferences, processing methods, and manufacturer philosophies influenced the degree of roasting. Although high-temperature roasting was more common in the past, most samples showed a medium degree of roasting. In addition to fresh aroma, some samples also had floral notes, enhancing the aroma score. Ten samples exhibited floral aromas, a relatively high proportion. Variety, leaf tenderness, and processing methods significantly affected grading. Longjing tea was evaluated based on appearance, liquor color, aroma, taste, and infused leaf appearance, divided into two main categories: appearance and internal quality. Tea picked in early spring is generally fresher and commands a higher price.

[0209] Ideally, appearance and internal quality should be consistent, provided processing is proper. However, due to processing variations, sometimes teas with excellent appearance may have lower internal quality grades. GBT 23776-2018 stipulates that appearance accounts for 25%, and internal quality accounts for 75% (of which taste accounts for 30%, aroma 25%, liquor color 10%, and infused leaf 10%), with taste having the greatest impact on the overall score. According to the West Lake Longjing group standard TXHLJ001-202125, samples are divided into Grade 2, Grade 1, Premium, and Special Grade. In this evaluation, due to over-roasting, the aroma score of the planned Special Grade standard sample dropped to 88, resulting in a lower overall score than the Premium sample. The Special Grade sample should score slightly higher than the Superior Grade sample in all aspects. Based on expert recommendations, three tea samples (Grade 2 raw tea 4.8, Grade 1 raw tea 3.31, and Special Grade raw tea 3.21) were selected as standards. A total score below 88 is Grade 2, 88 to 90 is Grade 1, and 90 and above is Superior Grade. Of the 38 samples, 16 were rated as excellent (Class A), 16 as first-class (Class B), and 6 as second-class (Class C).

[0210] Table 1 Sensory Evaluation Table for Longjing Green Tea: Appearance

[0211]

[0212]

[0213] Table 2 Sensory Evaluation Table for Longjing Green Tea: Liquor Color

[0214]

[0215]

[0216] Table 3 Sensory Evaluation Table for Longjing Green Tea: Aroma

[0217]

[0218]

[0219] Table 4 Sensory Evaluation Table for Longjing Green Tea: Infused Leaf

[0220]

[0221]

[0222] Table 5 Sensory Evaluation Table for Longjing Green Tea: Taste

[0223]

[0224]

[0225] Table 6 Sensory Evaluation Table for Longjing Green Tea: Total Score and Rating

[0226]

[0227]

[0228]

[0229] 8.2 Construction of a Tea Quality Identification Model Based on Appearance Dimension

[0230] This method combines visual images of tea leaves with human sensory evaluations, calculating appearance scores by associating the generated reviews and ratings with the model. The accuracy of the model's classification is evaluated by establishing a range of sensory evaluation ratings (C < 88 ≤ B < 91 ≤ A). The final results are 12 A grades, 16 B grades, and 10 C grades.

[0231] This method combines visual images with human sensory evaluation comments. The appearance score is derived from model-generated comments and a rating scale. Initial tests showed that the comment rating model, excluding deductions and additions, had a better fit; therefore, only the 7-dimensional sub-model was retained for training. Tea appearance is evaluated from seven dimensions: straightness, smoothness, tenderness, greenness, gloss, color uniformity, and integrity. The camera only captures the front of the leaves, thus omitting the evaluation of flatness. A sub-model for each dimension generates an evaluation score that matches the rating scale. The scores for all dimensions are summed to determine the overall appearance score.

[0232] The total number of sample appearance images is 7,763. After data augmentation, the amount of data increased sixfold, with 75% used for training and augmentation, and 25% used for testing.

[0233] Using a finely tuned ResNet-18 convolutional neural network (CNN) model ( Figure 4 a) Classifying tea samples based on visual images. Accuracy and loss graphs across seven dimensions ( Figure 4 The results (see figure) show that after seven training epochs, the model converged and demonstrated high accuracy and low loss on both the training and test sets. The ResNet-18 model achieved test accuracy greater than 0.85 in almost all dimensions, with the highest accuracy in the green dimension (0.976) and the lowest accuracy in the tenderness dimension (0.863). The final tea appearance score was derived from the seven sub-dimensional model. The overall classification accuracy was 83.77%, as shown in the confusion matrix (…). Figure 4 As shown in i).

[0234] Table 7. Appearance Dimension Evaluation Score Correspondence Table

[0235]

[0236]

[0237] 8.3 Construction of Tea Quality Recognition Model Based on the Dimensions of Infused Leaves and Soup Color

[0238] The infused leaves and soup color contribute 10% to the tea score. The infused leaves are evaluated by tenderness, color, and uniformity, in a similar way to the appearance of dry tea. Since it is difficult to identify the edges of the infused leaves, only color and texture features are extracted. For the soup color, sensory evaluation focuses on color type, chroma, brightness, and clarity, and color feature extraction is adopted.

[0239] For the classification of infused leaves, the scoring range (C ≤ 87 < B ≤ 89 < A) results in 17 A-level samples, 12 B-level samples, and 9 C-level samples. A similar procedure is adopted for the scoring range of the soup color (C ≤ 87 < B ≤ 89 < A), with results of 15 A-level samples, 16 B-level samples, and 7 C-level samples. One image is obtained for each category, and data set augmentation increases the number of images in each dimension to 190. Data preprocessing follows the same steps as for the appearance of tea. The data set is expanded by cropping, and the number of infused leaf and soup color images increases from 38 to 190, including the color and texture features of the infused leaf images and the color features of the soup color images.

[0240] XGBoost is used for the classification of the soup color Figure 6 where a is the schematic diagram of the structure of XGBoost. r, R 2 , logRMSE is shown in Table 1, R 2 is 0.9969, with a high fitting degree. The training loss curve and test loss curve of the soup color ( Figure 6 in b - c) show that after training about 150 rounds, the model converges and shows high accuracy and low loss on both the training set and the test set. The ROC curve ( Figure 6 in d) shows that the classification of soup color category C is better than other categories (AUC 0.81), and the confusion matrix shows ( Figure 6 in e) that the accuracy of the single-dimensional classification of the soup color is 70.05%.

[0241] Therefore, we replace MLP with TCSMLP for classification. TCSCLIP is used for the classification of the infused leaves, R 2 is 0.9996, and the structure of the MLP part in its model is as shown in Figure 10 i in Figure 10 j and k in<000 is the training loss curve and the three-class ROC curve respectively, indicating that the model converges after training about 150 rounds, Figure 10 and l in

[0242] Table 8 Regression Index Table of Training Models for Each Dimension

[0243]

[0244] 8.4 Construction of a tea quality identification model based on aroma dimension:

[0245] This method analyzes the volatile compounds produced after brewing 38 Longjing tea samples (each sample in triplicate) in boiling water, and classifies them using SPME-GC-MS. Approximately 300 aroma compounds were identified using retention time and the NIST 20 database, including aldehydes, ketones, alcohols, terpenes, acids, benzenes, cycloalkanes, aromatic compounds, and amino acid derivatives (Supplementary Data 2). Principal component analysis (PCA), as an unsupervised statistical method, was used for 298 variables to identify differences and associations between variables and samples. The contribution rates of PC1 and PC2 were 6.9% and 6.3%, respectively, indicating that PCA has limited ability to differentiate tea grades based on aroma. Figure 7 (a). However, orthogonal partial least squares discriminant analysis (OPLS-DA) successfully distinguished tea grades ( Figure 7 (b–d)

[0246] The aroma scores of tea leaves, as determined by human reviewers, are used as input labels into the model. A regression strategy is employed to output the model's scores for the aroma dimension of tea leaves. By defining the ABC grade score intervals (C<88≤B<92≤A, A: 14, B: 16, C: 8) for the taste dimension human reviewers' scores, the classification for the single dimension of taste is obtained.

[0247] GC-MS data uses TCSCLIP for classification, R 2 The value is 0.9998, and the structure of the MLP part in its model is as follows: Figure 10 As shown in e. Figure 10 f and g are the training loss curve and the three-class ROC curve, respectively, indicating that the model converged after about 150 more training rounds. At the same time, the AUC for the three classes is close to 1, which shows that the model has high classification accuracy. Figure 10 h is its confusion matrix, and the accuracy for single-dimensional classification of flavor reaches 99.12%.

[0248] 8.5 Construction of a Tea Quality Identification Model Based on Taste Dimension

[0249] A non-targeted metabolomics approach using liquid chromatography-mass spectrometry (LC-MS) was employed for digital sensory evaluation and analysis of taste compounds in tea infusion. Metabolomics is widely used to analyze metabolomics profiles and correlate chemical components with the sensory characteristics of tea. In this invention, metabolomics profiles were generated from 38 Longjing green tea samples, which were categorized into three quality grades. After removing biases and normalizing missing values, 12,287 ions were retained in the positive mode and 9,648 ions in the negative mode (Supplementary Data 3). This method identified 718 metabolites in the positive mode and 265 metabolites in the negative mode as candidate differential metabolites (Supplementary Data 4). These samples were evaluated using PCA and OPLS-DA methods. PCA results showed that the first and second principal components explained 21.5% and 11.9% of the variance, respectively, but failed to effectively distinguish tea grades. However, OPLS-DA showed good clustering results (Supplementary Data 4). Figure 4 The (c–e) method can effectively distinguish between classes C, B, and A.

[0250] Due to the complexity and comprehensiveness of tea evaluation, this method employs a regression strategy to address the classification problem across the overall dimension. When considering label selection for each group of metabolomics data, unlike previous literature which used tea type or specific grading as the label input model, this method uses the human evaluation scores for tea's flavor as the label input model. Naturally, the model's output becomes the model's score for the tea, and not the same tea classification labels as traditional methods. By defining the ABC score intervals (C<88≤B<91≤A, A: 14, B: 18, C: 6) for the human evaluation scores in the flavor dimension, the classification under the flavor dimension is obtained.

[0251] A regression model was established to score the flavor dimension of tea samples of different grades. Metabolomics can detect a wide variety of metabolites; each sample alone yielded 983 substances identified by secondary mass spectrometry. When 100 metabolites were input into the MLP model, overfitting occurred, with an accuracy of only about 30%. Therefore, this method adopted a strategy of reducing the input model dimension. After reducing the dimension to 18, the model's fit significantly improved, reaching 99%. Thus, this method selected the 18 substances with the highest secondary mass spectrometry scores to input into the model.

[0252] Figure 10 Figure 'a' is a schematic diagram of the MLP component used in metabolomics. After training the entire dataset, the training loss curve, the three-class ROC curves, and the confusion matrix were obtained. With R... 2 The increase in value,

[0253] The same data obtained from metabolomics were classified using TCSCLIP. 2 The value is 0.9999, and the structure of the MLP part in its model is as follows: Figure 10 As shown in Figure a. Figure 10 Figures b and c show the training loss curve and the three-class ROC curve, respectively, indicating that the model converged after about 150 more training rounds. At the same time, the AUC for all three classes is close to 1, indicating high classification accuracy. Figure 10 In the diagram, d is the confusion matrix, and the accuracy for single-dimensional taste classification reaches 97.37%.

[0254] Tea quality grading based on artificial intelligence

[0255] After creating evaluation models for tea appearance, aroma, taste, liquor color, and infused leaf appearance, the best-performing method was selected as the sub-model for the overall evaluation. Each sub-model generates scores for its corresponding dimension, which are used to calculate the final overall score. ResNet was used to train the appearance dimension, while TCSMLP was applied to the taste, aroma, and infused leaf appearance datasets. Due to its superior performance, XGBoost was used for the liquor color dataset.

[0256] To address the issue of inconsistent data volume across different dimensions of the dataset and to objectively evaluate the accuracy and effectiveness of these methods in practical applications, the tea liquor color, leaf residue, GC-MS, and metabolomics data were randomly paired with their corresponding appearance image data, with each tea category having an equal pairing probability. This ensures that each image can be paired with its corresponding GC-MS and metabolomics data for accurate evaluation.

[0257] The predicted scores for each evaluation dimension—appearance (25%), liquor color (10%), aroma (25%), taste (30%), and infused leaf (10%)—were weighted and summed to calculate the final tea score. Based on the total score, the tea was graded as C < 88 ≤ B < 90 ≤ A, with 16 grades for A, 16 for B, and 6 for C, and sample sizes of 3303, 3218, and 1242 respectively.

[0258] The ROC curve and confusion matrix (10o, p in the figure) of the multimodal system Long-Tea show that the system has a classification accuracy of 92.00%, which can accurately classify tea grades.

[0259] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A multimodal intelligent grading method for Longjing tea integrating five-dimensional evaluation criteria, characterized in that, include: S1. Collect samples of Longjing tea with different prices and grades; S2. Collect the appearance image dataset, liquor color image dataset, leaf bottom image dataset, aroma dataset, and taste dataset of Longjing tea samples. Preprocess the appearance image dataset, liquor color image dataset, leaf bottom image dataset, aroma dataset, and taste dataset respectively, and divide the dataset into training set, test set, and validation set. S3. Construct a multimodal Longjing tea grading model. Train, test, and validate the multimodal Longjing tea grading model using training, testing, and validation sets to obtain the optimal multimodal Longjing tea grading model. The multimodal Longjing tea grading model includes an input module, a deep convolution module, a contrastive learning module, an ensemble learning module, and an output module. The input module preprocesses the input image, preprocesses the chemical analysis data, and segments and guides the input data downstream. The deep convolutional module first uses the SimCLR self-supervised representation learning framework to pre-train the delabeled data, and then transfers it to the ResNet convolutional model to classify the tea shape image in 7 sub-comment dimensions, and performs comment-rating transformation on the tea shape image based on linear mapping. The contrastive learning module uses CLIP, which combines a cache built based on a few-shot dataset, to perform zero-shot learning prediction of tea image categories, and builds a multimodal ensemble learning supervisor based on this CLIP. The integrated learning module uses a multimodal supervisor to dynamically adjust the training process of XGBoost and MLP, and uses the adjusted and trained models XGBoost and TCSMLP to perform regression prediction on aroma data, taste data, soup color data and leaf residue data. TCSMLP is an MLP trained under CLIP supervision. The output module performs linear weighting on the scores of five human evaluation criteria, visualizes the tea quality evaluation process in text, and outputs the tea quality evaluation results. The SimCLR self-supervised visual representation learning framework of the multimodal Longjing tea grading model learns better visual representations through contrastive learning. The ResNet convolutional model of the multimodal Longjing tea grading model includes a core structure residual block and residual connection operations; The last layer of the ResNet convolutional model outputs the classification probability through a fully connected layer, and the cross-entropy loss drives the parameter update. The MLP portion of the TCSMLP in the multimodal Longjing tea grading model is a feedforward neural network, consisting of an input layer, hidden layers, and an output layer. The calculation formula for each hidden layer is as follows: Where l represents the l-th layer in the network, numbered starting from 1, h (l-1) W represents the output of the (l-1)th layer. (l) and b (l) These are the weights and biases of the l-th layer, respectively, and σ² is the ReLU activation function. For the input layer, h... (l-1) For the model input, and for the output layer, the regression task directly calculates the weights, biases and hidden layer outputs to obtain the final output without going through an activation function; The MLP objective function of the TCSMLP for the multimodal Longjing tea grading model is designed based on mean squared error, and the specific formula is as follows: in, This represents the mean squared error loss function used in the MLP network of TCSMLP, which measures the squared error between the model's predicted values ​​and the true values. n is the total number of samples, and y... i It is the true value of the i-th sample. It is the model's prediction for the i-th sample. This represents the squared prediction error for that sample. It is the normalization coefficient, used to calculate the average squared error value of all samples; The XGBoost method of the multimodal Longjing tea grading model is an ensemble learning method based on gradient boosting. The working process of the multimodal Longjing tea grading model includes: Step 1: Construct a multimodal ensemble learning supervisor, including: building a cache based on a collected small sample image-text dataset; using the VisualEncoder module of the pre-trained CLIP to embed and encode the image data for each category; and constructing the cache index as shown in the formula: Where Key represents the key vector in the image buffer, VisualEncoder is the visual encoder module in the CLIP model, used for feature extraction from image samples, R represents the real number field, indicating that the value of Key is in the real number range, and I k is the image input of the k-th sample, N represents the number of categories, and the number of rows 𝑁𝐾 represents the total number of 𝑁 categories × 𝐾 image samples per category, with a total of 𝑁𝐾 samples; C represents the embedding feature dimension of the visual encoder output, that is, the vector dimension of each image after encoding; OneHot is used to perform one-hot encoding on the text labels corresponding to each category of image, and the cache key-value pair is constructed as shown in the following formula: Where Value represents the value vector in the image buffer, OneHot represents the one-hot encoding module used to encode the category labels one-hot, and L k It is the text label or class label corresponding to the kth image sample; Where Query represents the query vector obtained after the image to be recognized is extracted by the visual encoder, I represents the input image encoding; 1 indicates that there is only one input image sample at present, so the final output is an embedding vector with a dimension of 1×C; For the image to be predicted, input the image into VisualEncoder to obtain the corresponding embedded encoding vector as the query value; calculate the similarity between the cache index and the query value using the following formula, and then retrieve the key value from the cache for decoding based on the similarity: Where Similarity represents the similarity between the query vector and the key vector, β is a scaling factor used to adjust the range of variation of the similarity function, and Key T Indicates the transpose of the key; The similarity is calculated using dot product; therefore, the final output similarity has a dimension of 1×N×K. Based on the decoding results, the weighted combination of the classifier in the pre-trained CLIP model is used to perform zero-shot predictions on the image data, which serve as the labels for the ensemble learning supervisor. Where Logits represents the final predicted output vector, used for label judgment in supervised learning, and α is the regularization coefficient used to balance the contributions of similarity-weighted labels and the original CLIP classifier; W T It is the weight transpose matrix of the CLIP model classifier, used to map the query to the predicted scores of each category; Step 2: The input module preprocesses the tea appearance images, specifically including: cropping, filtering, and channel normalization of the tea appearance images; cropping, normalizing, extracting color features, and extracting texture features of the liquor color and leaf bottom images; preprocessing the aroma and flavor datasets, specifically including cleaning, filtering, filling, and standardization of the datasets; randomly pairing the tea appearance image dataset, liquor color image dataset, leaf bottom image dataset, aroma dataset, and flavor dataset according to their categories, following a rectangular distribution; and importing the data pairs into downstream sub-models for processing based on their different modalities, specifically inputting the appearance image dataset into ResNet, the leaf bottom image dataset, aroma dataset, and flavor dataset into TCSMLP, and the liquor color image dataset into XGBoost; Step 3: SimCLR in the deep convolution module first performs self-supervised pre-training on the dataset of tea appearance evaluation dimensions, and then transfers the learned representations to ResNet-18 to classify the tea appearance images according to seven sub-evaluation dimensions, specifically including straightness, smoothness, tenderness, greenness, gloss, color uniformity, and integrity; and performs linear transformation from evaluation vectors to scores based on the evaluation-score mapping table. Step 4: The ensemble learning module performs regression fitting on the soup color and leaf appearance image data, aroma data, and taste data. Specifically, this includes establishing a dynamic adjustment mechanism for the regularization term of the network training based on the positive and negative feedback of the multimodal supervisor labels during the training process, thereby optimizing the training of the MLP to obtain the TCSMLP. The dynamic adjustment mechanism is defined as follows: Let the overall objective function of the TCSMLP model be... ,in The task loss is the mean squared error used in MLP regression. The variable is the regularization term, and λ(t) is the dynamic adjustment coefficient, controlled by the feedback signal of the supervisor. The dynamic coefficient is constructed as follows: Where λ0 represents the initial regularization coefficient, γ represents the feedback sensitivity coefficient, M represents the number of influence factors of the regularization coefficient, and β m F represents the weight of the m-th influence factor. m (t) represents the feedback function of the m-th influencing factor at time step t; When m=1, the feedback function F m The definition of (t) is as follows: When m=2, the feedback function F m The definition of (t) is as follows: in, This represents the loss function value of TCSMLP at time step t, where sign is the standard signal function and Label is the value of the loss function. supervisor Label represents the label recorded by the supervisor. pred The label represents the current output label of the model; 'sign' is a Boolean-based signal function. If the TCSMLP output is inconsistent with the supervisor's recorded label, enhance regularization; otherwise, reduce regularization. The optimized TCSMLP was used to perform regression prediction on aroma and taste data, and XGBoost was used to perform regression prediction on soup color image data. Step 5: The output module performs linear weighting on the model results of the five human evaluation criteria and sums them to obtain the final result. S4. Grade Longjing tea using the optimal multimodal Longjing tea grading model.

2. The multimodal intelligent grading method for Longjing tea integrating five-dimensional evaluation criteria as described in claim 1, characterized in that, S2 includes: Collect an appearance image dataset: Take images of tea samples before brewing and use data augmentation methods to enhance the data; Collect images of tea liquor color and tea leaf base: Brew tea samples, remove the tea leaves from the tea liquor after brewing, and then take images of tea liquor color and tea leaf base. Aroma dataset collection: Extract volatile compounds from brewed tea samples and identify the types and concentrations of volatile compounds; Collect a flavor dataset: Identify the types and concentrations of metabolites from brewed tea.

Images