Tea withering intelligent classification method based on multi-modal feature fusion and deep interaction

Abstract

The invention provides a tea withering intelligent classification method based on multi-modal feature fusion and deep interaction. The system is composed of a YOLOv8 enhanced feature extraction network, a random forest spectral feature processing module, a Transform deep fusion module, and a feature standardization and projection technology and intelligent classification decision module. And through collaborative learning of C2f module specialized optimization and adaptive attention weight distribution, cross-modal deep fusion of RGB image visual features and hyperspectral chemical component features is realized. Triple feature sequence construction and a multi-level depth feature interaction mechanism are innovatively introduced, feature representation is performed from the view angles of images, spectrums and fusion, and unified representation and intelligent weight distribution of heterogeneous features are realized by combining adaptive attention and long-distance dependence modeling. According to the method, the problem that the classification precision is limited due to complex alignment and feature isomerism of multi-modal data is effectively solved, 93.9% of classification accuracy is obtained on 81,054 verification samples, and the classification accuracy is improved by 6.4% compared with that of an official YOLOv8 model.

Description

Technical Field

[0001] This invention relates to the interdisciplinary field of artificial intelligence, computer vision, deep learning, and agricultural information technology, specifically to an intelligent classification system and method for tea withering based on the YOLOv8 deep neural network architecture, hyperspectral data processing, random forest ensemble learning, and Transformer multimodal fusion technology. This system achieves accurate identification, dynamic monitoring, and intelligent grading of tea withering status by constructing an advanced multimodal feature extraction network, an attention-based deep fusion model, a triple feature sequence construction technology, and an intelligent classification decision engine. This provides technical support for the intelligent upgrading of the tea processing industry and the industrial application of precision tea-making technology. Background Technology

[0002] Withering is a crucial step in the production of white tea, directly impacting its quality, flavor, and market value. Traditional methods of judging withering conditions rely primarily on the experience and observation of tea masters, which suffers from significant drawbacks such as strong subjectivity, inconsistent judgment standards, and low efficiency, making it difficult to adapt to the standardized and automated production requirements of the modern tea industry.

[0003] With the rapid development of deep learning, computer vision, hyperspectral imaging, and multimodal data fusion technologies, intelligent detection and classification systems for tea quality based on artificial intelligence have gradually become a research hotspot. Existing technical solutions can be mainly divided into the following categories: The first category is detection methods based on a single visual sensor, such as pure RGB image analysis, which can only capture changes in the appearance of tea leaves and lacks a deep perception of their internal biochemical components; the second category is single-spectral detection systems, which can only analyze changes in chemical composition and cannot fully reflect the complex characteristics of the withering process; the third category is classification and recognition methods based on traditional machine learning, which have limited feature extraction capabilities and insufficient multimodal fusion depth.

[0004] Existing systems generally suffer from the following technical bottlenecks:

[0005] (1) Insufficient feature extraction, lacking the ability to deeply fuse and collaboratively analyze RGB images and hyperspectral data;

[0006] (2) Multimodal alignment is difficult, and there are technical challenges in aligning and mapping heterogeneous data in the feature dimension;

[0007] (3) The fusion strategy is crude, and traditional simple splicing or weighted averaging methods cannot achieve true cross-modal information interaction;

[0008] (4) The classification accuracy is limited. Existing methods are not accurate enough in identifying fine-grained withering grades, which makes it difficult to meet the requirements of precision tea making. Summary of the Invention

[0009] This invention addresses the core technical bottlenecks in existing tea withering detection technologies, such as insufficient depth of multimodal feature fusion, weak cross-modal information interaction capabilities, low classification accuracy, and poor system generalization ability. It proposes an intelligent tea withering classification system and method based on multimodal feature fusion and deep interaction. This system innovatively constructs four core technical modules:

[0010] (1) Multimodal deep feature extraction module, based on improved YOLOv8 neural network and random forest ensemble learning technology, realizes high-dimensional extraction and representation learning of visual features and hyperspectral chemical composition features of RGB images;

[0011] (2) The multimodal fusion module based on the Transformer architecture realizes deep interaction and intelligent weight allocation of image features and spectral features through self-attention mechanism and cross-modal attention;

[0012] (3) Triple feature sequence construction and five-level interaction mechanism: The feature sequence design adopts image perspective, spectral perspective and fusion perspective, which significantly improves the cross-modal understanding ability of the system.

[0013] (4) Multi-level intelligent classification decision module, based on the dual strategy of feature-level fusion and decision-level fusion, realizes accurate identification of withering level and confidence assessment.

[0014] The core technological innovations of this invention are as follows: Firstly, it combines the YOLOv8 deep detection network with random forest ensemble learning for multimodal data processing of tea withering. It achieves 512-dimensional fusionable image feature extraction through C2f module specialization optimization, and simplifies 224-dimensional hyperspectral data into 121-dimensional key features through professional feature engineering. It innovatively constructs a Transformer-based cross-modal deep fusion network, enabling global information interaction between image and spectral features, improving classification accuracy by more than 15.93% compared to traditional methods. Secondly, it proposes a three-fold feature sequence construction technique in the field of tea withering, and designs an adaptive weight allocation algorithm through a five-level deep feature interaction mechanism.

[0015] A smart classification system for tea withering based on multimodal feature fusion and deep interaction, comprising the following claims:

[0016] 1. A tea withering intelligent classification system based on multimodal feature fusion and deep interaction, characterized by comprising the following steps:

[0017] Step 1: Acquire RGB image data and hyperspectral reflectance data of tea leaves during the withering process;

[0018] Step 2: Construct a comprehensive system that includes a YOLOv8 image feature extraction module, a random forest spectral feature processing module, a Transformer deep fusion module, and an intelligent classification decision module;

[0019] Step 3: In the YOLOv8 image feature extraction module, build a tea withering-specific classifier based on the official yolov8m-cls.pt. A 512-dimensional fusionable feature vector is output through C2f module specialization optimization and GAP pooling technology.

[0020] Step 4: Introduce professional feature engineering techniques into the random forest spectral feature processing module to optimize the 224-dimensional hyperspectral data into 121-dimensional key features. Accurate extraction of spectral features is achieved through the ensemble of 100 decision trees and feature importance weighting.

[0021] Step 5: The Transformer deep fusion architecture is adopted as the basic framework for cross-modal information fusion. A triple feature sequence and a five-level deep interaction mechanism are constructed, in which adaptive attention weight allocation is the core mechanism inside the Transformer architecture to achieve intelligent fusion.

[0022] Step 6: Accurately identify the withering level through multi-level classification control theory and establish an intelligent decision-making and prediction engine.

[0023] 2. The system according to claim 1, wherein the YOLOv8 image feature extraction module in step 3 specifically comprises:

[0024] We constructed a tea withering classification system based on the official yolov8m-cls.pt with significant improvements. The purpose of this system is to comprehensively extract the visual features of the withering state from 640×640 pixel RGB images, providing high-quality image feature representations for subsequent multimodal fusion.

[0025] (1) Specialized optimization design of C2f module: Special improvements were made to address the withering characteristics of tea leaves:

[0026]

[0027]

[0028]

[0029]

[0030] Color sensitivity enhancement is achieved by using an improved convolution kernel in the second layer to extract the color gradient changes of tea leaves (green → yellow-green → brownish-yellow); morphological feature optimization is achieved by adding dilated convolution in the third layer to expand the receptive field and capture leaf curling information (flattened → curled → shrunken); multi-scale fusion enables different bottleneck layers to capture complete withering features from details to the whole.

[0031] (2) Hierarchical feature learning architecture: Progressive feature extraction is achieved through a 4-layer CNN:

[0032] Layer 1 (low-level features): Detects edges and basic shapes, outputs a 320×320×64 feature map, and extracts basic geometric features such as tea leaf outline and veins;

[0033] Layer 2 (Intermediate Features): Recognizes color and texture, outputs a 160×160×128 feature map, and forms color gradient and texture change patterns;

[0034] Layer 3 (Advanced Features): Understanding complex morphology, outputting an 80×80×256 feature map, identifying the degree of leaf curling, edge inward curling, and overlapping relationships;

[0035] Layer 4 (Semantic Features): Forms an abstract representation of wilting, outputs a 20×20×512 feature map, and abstracts high-level semantic features related to wilting level.

[0036] (3) Global average pooling feature compression:

[0037]

[0038] Where f i For the feature value of the i-th channel, the complex multidimensional feature map of 20×20×512 is compressed into a 512-dimensional feature vector, with each channel corresponding to a representative value, retaining the most important withering-related information.

[0039] 3. The system according to claim 1, wherein the random forest spectral feature processing module in step 4 specifically comprises:

[0040] This module uses professional feature engineering technology and ensemble learning algorithms to extract the most discriminative withering-related features from 224-dimensional hyperspectral data in the 397.66-1003.81nm band, achieving efficient processing and accurate classification of spectral information.

[0041] (1) Professional feature engineering optimization: Convert the 224-dimensional original spectral data into 121-dimensional key features:

[0042] The calculation of basic spectral features (29 dimensions) includes raw reflectance and derivative features:

[0043] First derivative characteristics:

[0044]

[0045] Second derivative characteristics:

[0046]

[0047] Where R(λ) i ) represents the wavelength λ i The reflectance at a given point, where Δλ is the wavelength interval. The derivative characteristic can enhance the subtle variations in the spectral curve.

[0048] The Moisture Index (14 dimensions) is accurately calculated and includes multiple professional moisture testing indicators:

[0049] Standardized difference moisture index:

[0050] Corrected Standardized Difference Moisture Index:

[0051] Vegetation index (5-dimensional) calculation reflects leaf physiological activity:

[0052] Normalized Difference Vegetation Index:

[0053] Enhanced vegetation index:

[0054] (2) Random Forest Ensemble Classification: Achieving accurate classification of spectral features using 100 decision trees.

[0055]

[0056] An integrated voting mechanism ensures the stability and accuracy of classification results. Feature importance is reduced through Gini impurity quantification.

[0057]

[0058] 4. The system according to claim 1, wherein the feature standardization and projection technique in step 3 specifically comprises:

[0059] This technology maps heterogeneous features of different modalities to a unified representation space through mathematical transformation, solving the problems of dimensional inconsistency and numerical range differences between image features (512 dimensions) and spectral features (121 dimensions), laying the foundation for subsequent deep fusion.

[0060] Multimodal feature standardization uses Z-score normalization:

[0061]

[0062]

[0063] Where μ and σ are the feature mean and standard deviation on the training set, respectively, to standardize the feature distribution and eliminate the differences in the numerical range of features of different modalities.

[0064] Linear projection onto a 256-dimensional unified space:

[0065]

[0066]

[0067] in This is a learnable projection weight matrix. The choice of 256 dimensions is based on an optimal balance between information retention (94.2%) and computational complexity.

[0068] 5. The system according to claim 1, characterized in that the triple feature sequence construction technique in step 3:

[0069] This technology constructs feature sequences from image perspective, spectral perspective, and fusion perspective to provide multi-dimensional feature representations for Transformer deep fusion, fully exploring the complementarity and synergy of different modal information.

[0070] Image perspective sequence construction emphasizes appearance and morphological features:

[0071]

[0072] It is specifically designed to capture visual changes in tea leaves during the withering process, such as color gradation, shape changes, texture details, and spatial structure.

[0073] Spectral perspective sequence construction emphasizes intrinsic compositional features:

[0074]

[0075] It specifically captures information on intrinsic changes in tea leaves during the withering process, such as moisture content, chlorophyll changes, cell structure, and evolution of biochemical components.

[0076] The fusion perspective sequence construction adopts an adaptive weighting mechanism:

[0077] F enhanced =W2·GELU(W1·[P image ;P spectral ])

[0078] W adaptive =Sigmoid(W att ·[P image ;P spectral ])

[0079] X fusion=[p image ⊙W adaptive ,P spectral ⊙(1-W adaptive ),F enhanced ]

[0080] Where ⊙ represents element-wise multiplication, the adaptive weighting mechanism balances image and spectral information, and realizes intelligent weight allocation.

[0081] 6. The system according to claim 1, characterized in that the five-level deep feature interaction mechanism in step 3 is an internal implementation strategy of the Transformer deep fusion architecture, specifically as follows:

[0082] This mechanism achieves deep fusion of cross-modal information through a multi-level feature interaction strategy within the Transformer framework, breaking through the limitations of traditional simple splicing methods and realizing true inter-modal collaborative enhancement.

[0083] Level 1 - Adaptive Attention Weight Dynamic Interaction:

[0084]

[0085] Where τ = 0.1 is a learnable temperature parameter that controls the sharpness of the attention distribution. Dynamic interaction automatically adjusts according to the withering stage.

[0086] Level 2 - Long-distance dependency global interaction:

[0087] MultiHead(X)=Concat(head1,…,head8)W O

[0088] Cross-modal correlation modeling is achieved through an 8-head attention mechanism, such as the correlation between spectral moisture information and the degree of leaf curling in an image.

[0089] Level 3 - Intelligent Interaction for Feature Quality Assessment:

[0090]

[0091] Dynamically adjust mode weights based on signal-to-noise ratio to improve system robustness.

[0092] Level 4 - Multi-level progressive feature interaction:

[0093] X (l) =LayerNorm(X (l-1) +MultiHead(X (l-1) ))

[0094] Achieving progressive interaction from basic alignment to advanced blending through 4 layers of Transformers.

[0095] Level 5 - Importance-Aware Pooling Interactions:

[0096]

[0097] It enables dynamic weight allocation and noise suppression.

[0098] 7. The system according to claim 1, wherein the Transformer deep fusion network in step 3, as the overall architectural framework, specifically comprises:

[0099] This network provides a basic architecture for cross-modal feature deep fusion, adopts a multi-head self-attention mechanism as the core computing unit, distinguishes different modal perspectives through position encoding, and ensures training stability through feedforward networks and residual connections, providing a computing foundation for the five-level deep feature interaction mechanism described in claim 6.

[0100] Multi-head self-attention calculation:

[0101]

[0102] MultiHead(X)=Concat(head1,…,head8)W O

[0103] Location coding mechanisms distinguish between three perspectives: image, spectrum, and fusion.

[0104]

[0105]

[0106] 8. The system according to claim 1, wherein the multi-level classification control theory in step 5 is specifically implemented as follows:

[0107] The classification system adopts a three-layer fusion strategy: feature-level fusion, decision-level fusion, and adaptive weighted fusion, which achieves high-precision withering level recognition through multi-level information integration.

[0108] Feature-level fusion:

[0109] Decision-level fusion: P final =0.3·P image +0.3·P spectral +0.4·P fusion

[0110] Grade determination: WitheringGrade = argmax i∈{1,2,3,4,5,6} P final [i]

[0111] Confidence assessment: Confidence = max(P) final Samples with a value below 0.8 are marked as uncertain. Attached Figure Description

[0112] Figure 1 This is the overall system flowchart of the present invention.

[0113] Figure 2 This is a diagram of the random forest architecture for extracting spectral features in this invention.

[0114] Figure 3 This is a diagram of the Transformer model architecture of the present invention.

[0115] Figure 4 This is a diagram of the C2f module architecture of the present invention.

[0116] Figure 5 This is a diagram of the YOLOv8 network architecture of the present invention.

[0117] Figure 6 This is a classification result diagram of the present invention. Detailed Implementation

[0118] The following detailed description of a multimodal intelligent classification system for tea withering according to the present invention is provided in conjunction with specific embodiments. This embodiment is based on a large-scale dataset of 81,054 complete withering sample pairs, including RGB image data, 224-dimensional hyperspectral reflectance data, and complete withering process records.

[0119] Example 1: Intelligent Classification System for Tea Withering Based on Multimodal Feature Fusion

[0120] This embodiment constructs a complete system integrating four core modules: multimodal feature extraction, deep fusion, intelligent classification, and quality control. Figure 1 The system flowchart is shown. This system adopts a modular design, where each module can operate independently or work collaboratively, exhibiting good scalability and maintainability.

[0121] Step 1: Multimodal data acquisition and preprocessing

[0122] (1) RGB Image Data Acquisition: A high-resolution RGB camera was used, with a resolution of 640×640 pixels to ensure the clarity and detail of the tea images. Standardized lighting conditions (2000±100Lux) were used for image acquisition to eliminate the impact of lighting variations on image quality. The acquired images underwent color standardization, size normalization, and quality screening preprocessing to ensure the consistency and reliability of the input data.

[0123] (2) Hyperspectral Data Acquisition: Reflectance data of 224 spectral channels in the 397.66-1003.81 nm wavelength range were acquired using a hyperspectral imaging device. The spectral data sampling precision was set to a 2.7 nm interval, covering the visible and near-infrared bands to comprehensively capture the chemical composition information of tea. Dark current correction, whiteboard correction, and spectral smoothing were performed on the hyperspectral data to improve the signal-to-noise ratio.

[0124] (3) Withering grade labeling: Three senior tea masters were invited to independently label the withering samples and establish a 6-level withering grade standard (Level 1 - fresh and fragrant leaves, Level 2 - initial water loss, Level 3 - moderate withering, Level 4 - fully withered, Level 5 - deep withering, Level 6 - excessive withering). The labeling consistency reached more than 95%.

[0125] Step 2: Construction of the multimodal feature extraction module (e.g.) Figure 5 , Figure 2 (As shown)

[0126] Based on the improved YOLOv8 deep neural network architecture and random forest ensemble learning technology, a dual-modal feature extraction system is constructed to achieve comprehensive extraction of visual features and hyperspectral chemical composition features of RGB images.

[0127] (1) Design of YOLOv8 image feature extraction network:

[0128] A tea withering-specific classification network based on the official yolov8m-cls.pt was adopted, such as... Figure 5 As shown. The network input is set to a 640×640×3 RGB image, and the backbone network uses the C2f module instead of the traditional C3 module to enhance the ability to extract tea withering features.

[0129] The forward propagation calculation process of the C2f module is as follows:

[0130] F split =Split(Conv 1×1 (F in ))

[0131] F out =Concat([F split [0],Bottleneck n (…Bottleneck1(F split [1]))])

[0132] Where F in Using input features, feature reuse and gradient flow optimization are achieved through channel segmentation and residual connection design.

[0133] The tea leaf withering specialization optimization design includes: - Enhanced color sensitivity: Optimizing convolution kernel parameters for tea color gradations (green → yellow-green → brownish-yellow) - Morphological feature optimization: Employing dilated convolution to expand the receptive field and capture leaf curling and shrinking information - Multi-scale fusion: Fusion of features from different levels to achieve a complete representation from detailed texture to overall morphology.

[0134] Network parameter settings: initial number of channels 32, gradually increasing to 512; SiLU activation function is used to enhance nonlinear expression; Dropout probability 0.1 to prevent overfitting; initial learning rate 0.001, using cosine annealing scheduling.

[0135] The global average pooling layer compresses the 20×20×512 feature map into a 512-dimensional feature vector:

[0136]

[0137] It outputs a 512-dimensional fusionable feature vector, providing a high-quality image feature representation for subsequent multimodal fusion.

[0138] (2) Random forest spectral feature extraction and processing:

[0139] A professional spectral feature engineering workflow was constructed to convert 224-dimensional hyperspectral data into 121-dimensional key features, such as... Figure 2 As shown. Feature engineering comprises four levels:

[0140] Basic spectral features (29 dimensions): calculation of original reflectance, first derivative, and second derivative features to enhance the information on changes in the spectral curve.

[0141] First derivative:

[0142] Second derivative:

[0143] Key band features (9 dimensions): Select characteristic wavelengths closely related to the biochemical components of tea, including chlorophyll absorption peak (680nm), red edge reflection peak (750nm), and moisture absorption bands (950nm, 970nm).

[0144] Moisture Index (14 dimensions): Calculates professional moisture detection indicators such as NDWI and MNDWI to accurately quantify changes in the moisture content of tea leaves.

[0145] Vegetation Index (5-dimensional): Calculates vegetation vitality indicators such as NDVI, EVI, and SAVI to reflect the physiological state of leaves.

[0146] Feature selection optimization strategies are based on statistical methods and expert knowledge:

[0147] Feature_Scorei =0.4·Correlation i +0.4·MI i +0.2·Expert_Weight i

[0148] The random forest classifier uses an ensemble of 100 decision trees, with each tree employing bootstrap sampling and random feature selection to improve classification stability and generalization ability. Ensemble voting mechanism:

[0149]

[0150] Step 3: Construction of Transformer Deep Fusion Architecture and Multimodal Interaction Mechanism (e.g.) Figure 3 (As shown)

[0151] A Transformer deep fusion network is constructed as the overall architecture framework. Based on this, a five-level deep feature interaction mechanism and adaptive weight allocation are implemented to achieve cross-modal deep fusion of RGB images and hyperspectral features, such as... Figure 3 As shown.

[0152] (1) Feature standardization and projection:

[0153] The numerical range differences between the 512-dimensional image features and the 121-dimensional spectral features were eliminated by Z-score normalization.

[0154]

[0155]

[0156] Heterogeneous features are mapped to a unified 256-dimensional space through learnable linear projection:

[0157]

[0158]

[0159] (2) Construction of triple feature sequences:

[0160] Constructing a triple feature sequence from the image perspective, the spectral perspective, and the fusion perspective:

[0161] Image viewpoint sequence: X image =[P image ,P image ,P image ]

[0162] Spectral perspective sequence: X spectral =[P spectral ,P spectral ,P spectral]

[0163] The fused perspective sequence is generated through an adaptive weighting mechanism:

[0164] W adaptive =Sigmoid(W att ·[P image ;P spectral ])

[0165] X fusion =[P image ⊙W adaptive ,P spectral ⊙(1-W adaptive ),F enhanced ]

[0166] (3) Implementation of multi-head self-attention mechanism:

[0167] An 8-head attention mechanism is employed, with each head having a dimension of 32 (256 / 8). Multi-head self-attention calculation:

[0168] MultiHead(Q,K,V)=Concat(head1,…,head8)W O

[0169] Each attention head:

[0170]

[0171] Position encoding distinguishes different modal perspectives:

[0172]

[0173]

[0174] Step 4: Construction of Multi-level Classification Decision Module

[0175] A dual-strategy classification system with feature-level fusion and decision-level fusion is constructed to achieve accurate identification of withering levels.

[0176] (1) Feature-level fusion:

[0177] The standardized image and spectral features are then stitched and fused together.

[0178]

[0179] Dimensionality reduction to a unified 256-dimensional space via a fully connected layer:

[0180] F unified =W fusion F concat +b fusion

[0181] (2) Decision-level integration:

[0182] Calculate the classification probabilities of the image, spectrum, and fused features separately:

[0183]

[0184]

[0185]

[0186] The final prediction is obtained through weighted fusion:

[0187] P final =0.3·P image +0.3·P spectral +0.4·P fusion

[0188] The weights are determined through a grid search optimization on the validation set.

[0189] (3) Confidence assessment and quality control:

[0190] Calculate prediction confidence: Confidence = max(P) final )

[0191] When the confidence level is below the threshold (0.8), the system automatically marks it as an uncertain sample and recommends manual re-examination to ensure the reliability of the classification results.

[0192] Performance test results: The accuracy of wilting level classification reached 93.9%, precision 93.3%, recall 93.3%, and F1 score 93.4%; compared with the official YOLOv8 classification model (87.5%), the overall improvement was 6.4%; compared with single-modal methods (image 90.1%, spectrum 85.3%), the improvements were 4.2% and 10.1%, respectively.

[0193] like Figure 6 As shown in the figure, the classification results demonstrate the system's ability to classify tea samples at different withering levels. These results validate the effectiveness of the multimodal feature fusion and classification algorithm, proving that the system can accurately identify different degrees of withering.

[0194] The above description is merely a preferred embodiment of a multimodal intelligent classification system for tea withering. The scope of protection for this system is not limited to the above embodiments; all technical solutions falling within this conceptual framework are within the scope of protection of this invention. It should be noted that for those skilled in the art, any improvements and variations made without departing from the principles of this invention should also be considered within the scope of protection of this invention. This invention can be widely applied in the tea processing industry, agricultural informatization, food quality testing, intelligent manufacturing, and other related fields, possessing significant practical value and promising prospects for promotion.

Claims

1. A tea withering intelligent classification method based on multi-modal feature fusion and deep interaction, characterized in that, Comprise the following steps: Step 1: by acquiring the RGB image data, hyperspectral reflectance data, environmental temperature and humidity parameters and wilting timing information in the process of tea wilting; Step 2: build a comprehensive system including YOLOv8 image feature extraction module, random forest spectral feature processing module, Transformer deep fusion module and intelligent classification decision module; Step 3: build a special classifier for tea wilting based on the official yolov8m-cls.pt in the YOLOv8 image feature extraction module, realize 512-dimensional fusion feature vector output through C2f module specialization optimization and GAP pooling technology; Step 4: introduce professional feature engineering technology in the random forest spectral feature processing module, optimize 224-dimensional hyperspectral data to 121-dimensional key features, realize accurate extraction of spectral features through 100 decision tree integration and feature importance weighting; Step 5: realize cross-modal information fusion of RGB image and hyperspectral data by adopting Transformer deep fusion architecture, build triple feature sequence and five-level deep interaction mechanism, realize intelligent fusion through adaptive attention weight distribution; Step 6: realize accurate identification of wilting grade through multi-level classification control theory, establish intelligent decision and prediction engine.

2. The system of claim 1, wherein, The multi-modal data in step 1 is specifically: Use RGB camera to acquire 640x640 pixel tea color image, capture wilting related features such as leaf color change (green→yellow green→brown yellow), morphological characteristics (degree of curling, edge involution), texture information (leaf vein definition, surface gloss) and spatial structure (arrangement density, overlapping relationship); through hyperspectral imaging equipment, 224 spectral channels of reflectance data in the wavelength range of 397.66-1003.81 nm are collected, the internal moisture content, chlorophyll change, cell structure and biochemical composition evolution of tea are detected; record the temperature, humidity change and wilting time parameters of wilting environment. The system continuously monitors the complete wilting process, builds a large-scale multi-modal data set (81,054 sample pairs) covering 6 wilting grades, realizes accurate identification and classification of wilting state.

3. The system of claim 1, wherein, The YOLOv8 image feature extraction module in step 2 is specifically: Build a special tea wilting classification system based on the official yolov8m-cls.pt deep improvement, the purpose of this system design is to extract the visual features of wilting state from RGB image, and provide high-quality image feature representation for subsequent multi-modal fusion. (1) C2f module specialization optimization design: specially improved for tea wilting features: Among them, color sensitivity enhancement: use improved convolution kernel to extract tea color gradient change in the 2nd layer; Morphological feature optimization: increase the receptive field of the 3rd layer by adding dilated convolution to capture leaf curl information; multi-scale fusion: different bottleneck layers capture complete wilting features from details to the whole. (2) Hierarchical feature learning architecture: realize progressive feature extraction through 4-layer CNN: Layer 1 (low-level features): Detect edges and basic shapes, output 320x320x64 feature maps, extract basic geometric features such as tea leaf contours; Layer 2 (medium-level features): Identify colors and textures, output 160x160x128 feature maps, form texture and color patterns; Layer 3 (high-level features): Understand complex morphology, output 80x80x256 feature maps, identify leaf curling degree and overlapping relationship; Layer 4 (semantic features): Form withering abstract representation, output 20x20x512 feature maps, abstract withering level related features. (3) Global average pooling feature compression: where f i is the eigenvalue of the i-th channel, and the complex multi-dimensional feature map is compressed into a 512-dimensional feature vector, with each channel corresponding to a representative numerical value. (4) Feature vector standardization processing: where μ image and σ image are the mean and standard deviation of all training sample image features, respectively, to realize the standardization of feature distribution.

4. The system of claim 1, wherein, The random forest spectral feature processing module in step 2 is specifically: This module uses professional feature engineering technology and ensemble learning algorithm to extract the most discriminative withering related features from high-dimensional hyperspectral data, achieving efficient processing and accurate classification of spectral information. (1) Professional feature engineering optimization: Convert 224-dimensional original spectral data to 121-dimensional key features: Basic spectral features (29 dimensions) calculation: First derivative features: Second derivative features: where R(λ i ) is the reflectivity at wavelength λ i and Δλ is the wavelength interval. Key band features (9 dimensions) extraction: R 680 : chlorophyll a absorption peak (green leaf extent indicator) R 750 : red edge reflection peak (cell structure state) R 950 : moisture absorption band (water content detection) R 970 : Moisture secondary absorption band (moisture precision measurement) Moisture index (14 dimensions) accurate calculation: Standardized Difference Moisture Index: Modified normalized difference water index: Vegetation index (5 dimensions) calculation: Normalized Difference Vegetation Index: Enhanced Vegetation Index: (1) Feature selection optimization strategy: Feature_Score i = 0.4 · Correlation i + 0.4 · MI i + 0.2 · Expert_Weight i where Correlation i is the Pearson correlation coefficient with the wilting scale, MI i is the mutual information, Expert_Weight i is the expert experience weight. (3) Random forest ensemble classification: Achieve accurate classification of spectral features through 100 decision trees: Feature importance calculation: Confidence assessment:

5. The system of claim 1, wherein, The feature standardization and projection technology in step 3 is specifically: This technology maps heterogeneous features of different modalities to a unified representation space through mathematical transformation, laying the foundation for subsequent deep fusion and solving key technical problems such as inconsistent feature dimensions and numerical range differences. Multi-modal feature standardization: Where μ and σ are the mean and standard deviation of the features on the training set, achieving Z-score standardization of feature distribution. Linear projection to unified space: wherein is a learnable projection weight matrix and b is a bias vector. The projection matrix is learned through end-to-end training, ensuring that the projected features are most beneficial for the wilting classification task. Dimension selection optimization: The selection of 256 dimensions is based on the balance between information retention and computational complexity: Information retention: Image features retain 94.2% of the original information, and spectral feature expression ability is expanded by 211%; Computational complexity: Self-attention complexity O(n 2 • d) = O(9 x 256), 75% reduction in computation compared to 512 dimensions; Experimental verification: 256 dimensions achieve the optimal performance balance point in withering classification tasks.

6. The system of claim 1, wherein, The triple feature sequence construction technology in step 4 is: This technology provides multi-dimensional feature representation for Transformer deep fusion by constructing image, spectral and fusion perspective feature sequences, fully exploiting the complementarity and synergy of different modal information. Image perspective sequence construction: Emphasize color gradient, shape change, texture details and other appearance features, specifically capture visual change information during the withering process of tea leaves. Spectral perspective sequence construction: Emphasize water content, chlorophyll change, cell structure and other internal composition features, specifically capture biochemical change information during the withering process of tea leaves. Fusion perspective sequence construction: F enhanced = W2 * GELU(W1 * [P image ; P spectral ]) W adaptive = Sigmoid(W att · [P image ; P spectral ]) X fusion = [P image ⊙W adaptive , P spectral ⊙(1-W adaptive ), F enhanced ] Where ⊙ represents element-wise multiplication, and [·;·] represents concatenation operation. The fusion perspective balances image and spectral information through an adaptive weight mechanism, achieving intelligent weight allocation.

7. The system of claim 1, wherein, The five-level deep feature interaction mechanism in step 5 is specifically implemented as: The mechanism realizes the deep fusion of cross-modal information through multi-level feature interaction strategies, breaking through the limitations of traditional simple splicing methods and achieving true inter-modal collaborative enhancement. Level 1: Dynamic interaction of adaptive attention weight: where τ = 0.1 is a learnable temperature parameter. Dynamic interaction is represented by: Level 2: Global interaction of long-range dependency: MultiHead(X) = Concat(head1,..., head8) W O Implementing cross-modal associations Time accumulation effect modeling and complex non-linear relationship mining. Level 3: Intelligent interaction of feature quality assessment: Smart interaction policy: image quality degradation (SNR image <20 dB), a image x 0.6; poor lighting (variance of brightness < 100), a spectral x 1.

4. Level 4: Multi-level progressive feature interaction: X (l) = LayerNorm(X (l-1) + MultiHead(X (l-1) )) X (l) = LayerNorm(X (l) + FFN(X (l) )) Layer1: Basic alignment; Layer2: Semantic interaction; Layer3: Advanced fusion; Layer4: Feature integration. Level 5: Importance-aware pooling interaction: Achieving dynamic weight allocation, key information preservation, and noise suppression.

8. The system of claim 1, wherein, The Transformer deep fusion network in step 5 is specifically: This network uses a multi-head self-attention mechanism to realize the deep fusion of cross-modal features, ensures training stability and feature expression ability through position encoding, feedforward network, and residual connection, and generates high-quality fusion feature representation. Multi-head self-attention calculation: MultiHead(X) = Concat(head1,..., head h )W O where each attention head: Position encoding mechanism: Inject position information into different modal features to distinguish image, spectrum, and fusion perspectives. Feedforward network and residual connection: X' = LayerNorm(X + MultiHead(X)) X'' = LayerNorm(X' + FFN(X')) FFN(x) = max(0, xW1 + b1)W2 + b2 9. The system of claim 1, wherein, The multi-level classification control theory in step 6 is specifically implemented as: The classification system uses a three-layer fusion strategy: feature-level fusion, decision-level fusion, and adaptive weighted fusion, achieving high-precision wilting grade recognition through multi-level information integration, effectively balancing accuracy and robustness. Feature-level fusion: Decision-level fusion: Final prediction probability: P final = 0.3 · P image + 0.3 · P spectral + 0.4 · P fusion where the weight parameters are optimized through the validation set: α = 0.3, β = 0.3, γ = 0.

4.

10. A tea wilting intelligent classification system based on multi-modal feature fusion and deep interaction, characterized by: The system adopts a multi-level deep fusion architecture, including a YOLOv8 image feature extraction module, a random forest spectral feature processing module, a Transformer deep fusion module, and an intelligent classification decision module. Through innovative triple feature sequence construction and five-level deep feature interaction mechanism, the system realizes the cross-modal deep fusion of RGB images and hyperspectral data. Based on the official yolov8m-cls.pt deep improvement, the system outputs 512-dimensional fusion feature vectors through C2f module specialization optimization and GAP pooling technology; it uses professional feature engineering to optimize 224-dimensional hyperspectral data into 121-dimensional key features, and realizes accurate spectral classification through 100 decision tree integration; combined with Z-score standardization and linear projection technology, it maps heterogeneous features to a 256-dimensional space, and constructs triple feature sequences of image perspective, spectral perspective and fusion perspective; through attention-driven Transformer deep fusion network, it realizes adaptive weight allocation and global information interaction. The system can dynamically adjust the importance of each modality according to the wilting stage: when the light is strong, the spectral weight is increased (35%), when the shape is obvious, the image weight is increased (40%), and in the late wilting stage, the grade weight is increased (45%). On 81,054 validation samples, the system's wilting grade classification accuracy is 93.9%, the precision is 93.3%, the recall is 93.3%, and the F1 score is 93.4%. Compared with the official YOLOv8 classification model (87.5%), the overall improvement is 6.4%, compared with single modal method (image 90.1%, spectrum 85.3%), the improvement is 4.2% and 10.1% respectively, and the average inference time is <100 ms. The system effectively solves the key technical problems of poor recognition accuracy, limited fusion effect, complex spatio-temporal alignment of multi-modal data, and feature heterogeneity of traditional fusion algorithms, and has significant technical innovation and practical value.

Images