Multi-modal fusion evaluation method and system for grading green tea

Abstract

The invention provides a multi-modal fusion evaluation method and system for green tea grading. The method is applied to the technical field of artificial intelligence deep learning, and comprises the steps that to-be-detected hyperspectral data of to-be-detected green tea are collected through a constructed collection system, and the to-be-detected hyperspectral data comprise to-be-detected spatial data and to-be-detected spectral data; performing feature extraction on the to-be-detected spatial data by adopting a residual network model to obtain an image feature vector; inputting the spectral data to be detected into the Transform model to obtain a spectral feature vector; performing interaction association on the image feature vector and the corresponding spectral feature vector by adopting a cross attention mechanism to generate a fusion feature vector; and inputting the fusion feature vector into a trained grading model to obtain a grading result, and training the grading model by taking the green tea grade as a label. The feature representation is automatically learned from the original image of the green tea to classify the grade or evaluate the quality.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence and deep learning technology, and in particular to a multimodal fusion evaluation method and system for grading green tea. Background Technology

[0002] Tea, as a natural beverage, is widely loved by people around the world, and its quality directly determines its market value and consumer experience. Traditional green tea grading relies primarily on sensory evaluation by professional tea tasters using sight, smell, and taste. While this method is authoritative, it has inherent limitations such as strong subjectivity, low efficiency, susceptibility to individual tea taster conditions, and difficulty in large-scale standardization. With the rapid development of computer vision and artificial intelligence technologies, automated and objective quality inspection technologies based on deep learning have become a research hotspot in the agricultural and food industry. Among these, the use of image feature extraction networks to analyze green tea appearance images has shown potential to replace manual visual grading to a certain extent.

[0003] However, the quality of green tea is not determined solely by its appearance. It is a complex system composed of multiple dimensions such as appearance, color, aroma, liquor color, and taste. It is difficult to fully and accurately reflect its overall grade based on a single visual modality.

[0004] Therefore, there is an urgent need to develop a multimodal fusion evaluation method and system for green tea grading to solve the above problems. Summary of the Invention

[0005] This disclosure provides a multimodal fusion evaluation method and system for green tea grading, which solves the problems of existing green tea grading, such as strong subjectivity, low efficiency, susceptibility to the individual state of tea tasters, difficulty in achieving large-scale standardization, and the fact that the quality of green tea is determined by multiple dimensions, and it is difficult to comprehensively and accurately reflect its overall grade based on a single visual modality.

[0006] According to a first aspect of this disclosure, a multimodal fusion evaluation method for grading green tea is provided. The method includes: acquiring hyperspectral data of the green tea to be tested using a constructed acquisition system, wherein the hyperspectral data includes spatial data and spectral data.

[0007] The residual network model is used to extract features from the spatial data to be detected to obtain image feature vectors. The training process of the residual network model adopts a transfer learning strategy. The spectral data to be detected is input into the Transformer model to obtain spectral feature vectors.

[0008] A cross-attention mechanism is used to interactively associate the image feature vector and the corresponding spectral feature vector to generate a fused feature vector;

[0009] The fused feature vector is input into the trained grading model to obtain the grading result. The grading model is trained with green tea grade as the label.

[0010] Furthermore, the acquisition system includes a hyperspectral imager, a standard whiteboard, a monitor, a darkroom, a halogen lamp light source, and a glass dish;

[0011] The glass dish was placed in a dark box and used to hold green tea;

[0012] The halogen lamp light source is fixed on the side wall inside the dark box to provide an imaging light source;

[0013] The hyperspectral imager is fixed to the top inside the dark box and is used to collect the hyperspectral data;

[0014] The display is connected to the hyperspectral imager and is used to display the acquisition process and visualize the acquired hyperspectral data;

[0015] The standard whiteboard is located directly below the hyperspectral imager to ensure that the lighting conditions for data acquisition remain constant.

[0016] Furthermore, the residual network model includes any one of ResNet-18, ResNet-34, ResNet-50, ResNet-101, and ResNet-152; wherein ResNet-18 and ResNet-34 use BasicBlock residual blocks, and ResNet-50, ResNet-101, and ResNet-152 use BottleNeck residual blocks;

[0017] The BasicBlock residual block includes a first convolutional layer and a second convolutional layer in sequence. Each of the first and second convolutional layers is followed by a batch normalization layer and a ReLU activation function.

[0018] The BottleNeck residual block includes a third convolutional layer, a fourth convolutional layer, and a fifth convolutional layer in sequence. Each of the third, fourth, and fifth convolutional layers is followed by a batch normalization layer and a ReLU activation function.

[0019] Furthermore, the residual network model sequentially includes an input layer, a 7×7 convolutional layer, a batch normalization layer, a ReLU activation function, a max pooling layer, a residual block layer, an averaging layer, a fully connected layer, and an output layer. The residual block layer includes several BasicBlock residual blocks or several BottleNeck residual blocks.

[0020] Furthermore, the construction of the residual network model includes the following steps:

[0021] The construction of the residual network model includes the following steps:

[0022] The hyperspectral data of different green teas were collected by the constructed acquisition system, the hyperspectral data were labeled, and the labeled hyperspectral data were divided into training set and test set.

[0023] Based on transfer learning, the residual network model is trained using the training set, and the trained residual network model is validated using the test set.

[0024] Furthermore, the Transformer model includes an encoder and a decoder;

[0025] The encoder is composed of N identical encoder layers stacked together, where N is a positive integer. Each encoder layer includes two sub-layers, and each sub-layer includes a multi-head attention layer and a front feedback layer. Each sub-layer is equipped with residual connections and layer normalization.

[0026] The decoder consists of N identical decoder layers stacked together, each decoder layer including two sub-layers and a masked multi-head attention layer; wherein...

[0027] The multi-head attention layer This can be expressed by the following formula: ,

[0028] in, The formulas representing the effects of different self-attention mechanisms are as follows: This represents the concatenation function, where Q represents the query vector, K represents the key vector, V represents the value vector, and h represents the concatenation function. , , The number of groups formed by different linear projections. This indicates the output linear projection matrix; ,in, The formula representing the role of self-attention mechanism is as follows: , and These represent the i-th self-attention mechanism, respectively. The weight matrices corresponding to K and V.

[0029] Furthermore, the cross-attention mechanism specifically includes the following steps:

[0030] The image feature vector is processed by a residual network model to generate Q, K, and V corresponding to each element, and the spectral feature vector is processed by a Transformer model to generate Q, K, and V corresponding to each element.

[0031] The Q, K, and V corresponding to the image feature vector are all linearly processed to obtain the first linear Q, the first linear K, and the first linear V. The Q, K, and V corresponding to the spectral feature vector are all linearly processed to obtain the second linear Q, the second linear K, and the second linear V.

[0032] The first attention weight is obtained by performing a dot product between the second linear Q and the corresponding first linear K. The first attention weight is then performed by performing a dot product between the first attention weight and the corresponding first linear V, and finally added to the image feature vector to obtain the image enhancement feature.

[0033] The second linear K and the corresponding first linear Q are multiplied by a dot product to obtain the second attention weight. The second attention weight and the corresponding second linear V are multiplied by a dot product and then added to the spectral feature vector to obtain the spectral enhancement feature.

[0034] The image enhancement features and spectral enhancement features are input into the stitching layer and stitched together to generate a fused feature vector.

[0035] According to a second aspect of this disclosure, a multimodal fusion evaluation system for green tea grading is provided, comprising the following modules:

[0036] The acquisition module is used to acquire the hyperspectral data of the green tea to be tested through the constructed acquisition system. The hyperspectral data to be tested includes spatial data and spectral data.

[0037] An extraction module, connected to the acquisition module, is used to extract features from the spatial data to be detected using a residual network model to obtain an image feature vector. The training process of the residual network model adopts a transfer learning strategy. The spectral data to be detected is input into the Transformer model to obtain a spectral feature vector.

[0038] The fusion module, connected to the extraction module, is used to interactively associate the image feature vector and the corresponding spectral feature vector using a cross-attention mechanism to generate a fused feature vector;

[0039] The discrimination module, connected to the fusion module, is used to input the fused feature vector into the trained grading model to obtain the grading result. The grading model is trained with green tea grade as the label.

[0040] According to a third aspect of this disclosure, an electronic device is provided, including a memory, a processor, and a computer program; wherein the computer program is stored in the memory and configured to be executed by the processor to implement the method as described in the first aspect.

[0041] According to a fourth aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon; the computer program is executed by a processor to implement the method as described in the first aspect.

[0042] The beneficial effects of this disclosure are:

[0043] This disclosure constructs an acquisition system to collect hyperspectral data to be detected, introduces a residual network model to extract image features, and adopts different deep models such as ResNet-18, ResNet-34, ResNet-50, ResNet-101, and ResNet-152. Based on transfer learning, comparative experiments are conducted to explore the influence of network depth on the feature extraction effect of green tea images.

[0044] This disclosure employs the Transformer model, in which the encoder then performs deep encoding on the sequence. Its output can effectively characterize key information such as the overall morphology of the spectrum, the position of characteristic peaks, and their intensity, providing more robust features for subsequent feature fusion.

[0045] This disclosure employs a multimodal fusion model based on a cross-attention mechanism, enabling the features of green tea images and spectra to interact and complement each other, thereby generating a unified fusion representation.

[0046] It should be understood that the description in the Summary of the Invention is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0047] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. The drawings are provided for a better understanding of the invention and are not intended to limit the scope of this disclosure. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0048] Figure 1 A flowchart of the multimodal fusion evaluation method for green tea grading provided in this disclosure embodiment is shown;

[0049] Figure 2 A schematic diagram of the structure of the BasicBlock residual block provided in an embodiment of this disclosure is shown;

[0050] Figure 3 A schematic diagram of the BottleNeck residual block provided in an embodiment of this disclosure is shown;

[0051] Figure 4 A schematic diagram of the structure of the residual network model provided in this embodiment of the present disclosure is shown;

[0052] Figure 5 A schematic diagram of the structure of transfer learning provided in an embodiment of this disclosure is shown;

[0053] Figure 6 A schematic diagram of the structure of the Transformer model provided in this embodiment of the present disclosure is shown;

[0054] Figure 7 A schematic diagram of the structure of the multimodal fusion model provided in the embodiments of this disclosure is shown;

[0055] Figure 8 The following is a line graph showing the accuracy of different residual network models provided in the embodiments of this disclosure after using the cross-attention mechanism;

[0056] Figure 9 The following are bar charts showing the corresponding parameter values ​​of different residual network models provided in the embodiments of this disclosure after using the cross-attention mechanism;

[0057] Figure 10 A framework diagram of a multimodal fusion evaluation system for green tea grading provided in an embodiment of this disclosure is shown.

[0058] Figure 11 A block diagram of an electronic device provided according to an embodiment of the present disclosure is shown. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0060] Furthermore, the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0061] First, let me explain the following:

[0062] Hyperspectral imaging is a technology developed based on multispectral imaging. It uses an imaging spectrometer to continuously image a target object within the ultraviolet to near-infrared (200–2500 nm) spectral range, covering tens to hundreds of narrow but continuous spectral bands. This technology can acquire a corresponding two-dimensional spatial image at each wavelength point, thus combining all images in spectral order to form a three-dimensional data "cube." Therefore, hyperspectral imaging can not only capture the morphological features of a target but also simultaneously acquire its detailed spectral information.

[0063] Residual Neural Networks (ResNet) are a deep convolutional neural network (CNN) architecture that effectively mitigates the vanishing gradient problem. It addresses this issue by introducing residual blocks, which bypass certain intermediate layers and directly link activation values ​​from one layer to subsequent layers, creating a residual block. These residual blocks are stacked to create ResNets, enabling network structures exceeding 100 layers. This significantly increases the depth of networks that can be effectively trained, resulting in superior performance in tasks such as image classification and resolving the degradation problem that occurs with increasing network depth.

[0064] Transfer learning is a machine learning strategy that leverages existing knowledge to solve new tasks. Its main principle is to transfer knowledge learned from one task (source domain) to another related but different task (target domain). This reduces reliance on labeled samples in the target domain, improving the model's learning efficiency and generalization ability. Thus, even when new task data is scarce or training costs are high, it can still efficiently build high-performance models, increasing the learning speed of the network model and accelerating model convergence. Unlike traditional machine learning, which assumes that training and test data follow the same distribution, transfer learning allows for distributional differences between the source and target domains and bridges these gaps through knowledge sharing.

[0065] Cross-attention is an important variant of the attention mechanism. It is a mechanism that establishes a connection between two different input sequences and calculates attention weights. Its core function is to allow one sequence (target sequence) to selectively focus on the information of another sequence (source sequence), thereby establishing a connection between the two different sequences.

[0066] Multimodal data fusion aims to integrate complementary information from different sensors or data sources into a global space, thereby building a more powerful recognition model. The core of multimodal fusion lies in the fact that data from different modalities can describe the characteristics of the same object from different perspectives.

[0067] This disclosure provides a multimodal fusion evaluation method for green tea grading. See [link to relevant documentation]. Figure 1This includes the following steps:

[0068] S1. Collect the hyperspectral data of the green tea to be tested using the constructed acquisition system. The hyperspectral data to be tested includes spatial data and spectral data.

[0069] Existing computer vision methods can only capture the external morphology, color, and texture features of tea leaves, but cannot obtain the internal chemical components. Spectroscopic analysis techniques can only reflect chemical properties, but lose the visual-physical characteristics of the tea leaves. Based on the above, this disclosure presents a data acquisition system for simultaneously acquiring image and physical data of green tea.

[0070] The acquisition system includes a hyperspectral imager, a standard whiteboard, a monitor, a dark box, a halogen lamp light source, and a glass dish;

[0071] The glass dish was placed in a dark box and used to hold green tea;

[0072] The halogen lamp light source is fixed on the side wall inside the dark box to provide an imaging light source;

[0073] The hyperspectral imager is fixed to the top inside the dark box and is used to collect the hyperspectral data;

[0074] The display is connected to the hyperspectral imager and is used to display the acquisition process and visualize the acquired hyperspectral data;

[0075] The standard whiteboard is located directly below the hyperspectral imager to ensure that the lighting conditions for data acquisition remain constant.

[0076] (1) The hyperspectral imager continuously images green tea within a specific spectral range using an imaging spectrometer. At each wavelength point, a corresponding two-dimensional spatial image of the green tea is generated. Then, all the two-dimensional spatial images of different wavelengths are stacked and combined in the order of spectral bands to finally form three-dimensional data. Therefore, the hyperspectral data to be detected includes spatial data and spectral data to be detected. Among them, the spatial data to be detected includes visual features such as the shape, color, texture, and uniformity of the tea leaves, which are consistent with the appearance information of tea leaves observed by the human eye. For example, a certain spatial information is 512×512 pixels, and each pixel represents the visual information of a physical point on the surface of the tea leaves. The spectral data to be detected includes the relevant optical properties of the chemical components inside the tea leaves, such as tea polyphenols and amino acids. Different components will produce different reflection and absorption effects on light of specific wavelengths.

[0077] This disclosure utilizes the GaiaSky-MINI2-VN hyperspectral imager, which has a wavelength range of 400 nm to 1000 nm and 176 channels, providing rich spectral information for the research.

[0078] As a core component, the hyperspectral imager can acquire hyperspectral data of green tea in the visible to near-infrared band. With its 176 spectral channels, it can obtain detailed spectral information, providing data support for in-depth research on the optical properties of tea.

[0079] (2) Place the standard whiteboard directly below the hyperspectral imager, ensuring that it fills or exceeds the entire field of view. Under the same lighting conditions and hyperspectral imager settings as when the sample was collected, acquire a hyperspectral image as the standard whiteboard to record the maximum reflected signal that the hyperspectral imager in the current system can receive, including complete information such as light source intensity, spatial non-uniformity, and camera response characteristics of the hyperspectral imager.

[0080] Acquiring black frames with the lens cap of the hyperspectral imager closed can effectively record and deduct background noise and stray light interference, thereby ensuring the consistency and reliability of hyperspectral data under different experimental conditions.

[0081] By collecting reflection information and black frames from a standard white board, it is possible to correct for lighting fluctuations and sensor response differences during the experiment.

[0082] Hyperspectral imagers acquire data using slit scanning, and the distance between the slit and the imaging lens can lead to inconsistent brightness during imaging. Therefore, lens calibration can be performed using the aforementioned standard white board and black frame to eliminate potential distortions or artifacts and ensure consistent brightness at all locations during imaging.

[0083] In addition, the data acquired by the system is also affected by various other factors, such as the light source spectrum, light source intensity, lens transmittance, spectrometer diffraction efficiency, camera spectral response efficiency, and the reflectance of the object being measured. The final acquired data is the image's pixel brightness value (Digital Number, DN value), which records the grayscale value of the photographed green tea. To obtain the reflectance of the hyperspectral data to be detected, the acquired DN value needs to be corrected for reflectance. Specifically, reflectance correction can be performed using the acquired black and white frames, using the following formula: ,

[0084] in, The reflectance of the hyperspectral image after black-and-white correction is, i.e. This is the hyperspectral image data after lens correction. Black frame data, This is white frame data.

[0085] (3) The display can observe and control the image acquisition process in real time, and can intuitively view the acquired hyperspectral data to be detected through the display, thereby ensuring clear image quality and accurate content, and avoiding repeated acquisition.

[0086] (4) All tea data were collected in a darkroom environment to shield external light sources to the greatest extent, effectively isolate ambient light, thereby improving measurement accuracy and reducing image noise introduced by external light.

[0087] (5) Halogen lamp light source: Used to provide a light source to illuminate the tea sample and ensure the illumination required for image acquisition. It helps to improve the quality and accuracy of the obtained spectral data to be detected.

[0088] (6) The glass dish holds and positions the tea sample. Its flat bottom and high light transmittance ensure that light passes through the sample evenly during the spectral scanning process, which helps to obtain stable and reliable hyperspectral data to be detected.

[0089] During the acquisition of the hyperspectral data to be detected, the exposure time and gain parameters are kept constant to facilitate subsequent image correction processing.

[0090] Preprocessing of the hyperspectral data to be detected can be performed, such as image cropping, outlier removal, normalization, and data format conversion, which will not be elaborated on here.

[0091] S2. The residual network model is used to extract features from the spatial data to be detected to obtain image feature vectors. The training process of the residual network model adopts a transfer learning strategy. The spectral data to be detected is input into the Transformer model to obtain spectral feature vectors.

[0092] The residual network model includes any one of ResNet-18, ResNet-34, ResNet-50, ResNet-101, and ResNet-152, each containing two or three convolutional layers; wherein ResNet-18 and ResNet-34 use BasicBlock residual blocks, and ResNet-50, ResNet-101, and ResNet-152 use BottleNeck residual blocks.

[0093] See Figure 2 The BasicBlock residual block includes a first convolutional layer and a second convolutional layer, both of which are 3×3 convolutional layers. Each convolutional layer is followed by a batch normalization (BN) layer and a ReLU activation function.

[0094] It's important to note that when the input and output dimensions of the BasicBlock residual block are the same (number of channels, height, and width are all identical), the path is an identity mapping. When downsampling or increasing the number of channels is required, the input of the BasicBlock residual block can be directly passed through the first branch to the adder added between the batch normalization layer and the ReLU activation function after the second convolutional layer. The first branch includes a 1×1 convolutional layer and a batch normalization layer, which are used to transform the input of the BasicBlock residual block to a dimension that matches the output dimension.

[0095] See Figure 3 The BottleNeck residual block includes a third convolutional layer, a fourth convolutional layer, and a fifth convolutional layer. The third convolutional layer is a 1×1 convolutional layer, the fourth convolutional layer is a 3×3 convolutional layer, and the fifth convolutional layer is a 1×1 convolutional layer. Each convolutional layer is followed by a BN layer and a ReLU activation function.

[0096] Similarly, the input of the BottleNeck residual block can be passed through the second branch to the adder added between the batch normalization layer and the ReLU activation function after the fifth convolutional layer. The second branch includes a 1×1 convolutional layer and a batch normalization layer, forming a downsampling module.

[0097] Networks such as ResNet-18, ResNet-34, and ResNet-50 all include several BasicBlock residual blocks or BottleNeck residual blocks.

[0098] In one specific embodiment, see Figure 4 The residual network model consists of an input layer, a 7×7 convolutional layer, a batch normalization layer, a ReLU activation function, a max pooling layer, a residual block layer, an averaging layer, a fully connected layer, and an output layer. The residual block layer consists of several BasicBlock residual blocks or several BottleNeck residual blocks.

[0099] Specifically, the construction process of the residual network model is as follows:

[0100] Hyperspectral data of different green teas were collected using a constructed acquisition system. This hyperspectral data was then labeled and divided into training and testing sets. For example, the green tea samples were provided by a company, and the collected varieties were pan-fired green teas from different regions and seasons. The hyperspectral data could be randomly divided into training and testing sets in a 7:3 ratio to ensure a balanced sample proportion for each grade of green tea and avoid category bias. Experts then graded the teas according to industry tea grading standards. Multiple experts could score the same hyperspectral data, and the average score would be calculated to determine the corresponding grade.

[0101] Based on transfer learning, the residual network model is trained using the training set, and the trained residual network model is validated using the test set.

[0102] Due to the high cost of sample labeling, this disclosure employs transfer learning to address the overfitting problem with small samples. Specifically, transfer learning is used to train the residual network model to obtain the corresponding transfer parameters and weights. See [link to relevant documentation]. Figure 5 Given a source domain DS and its corresponding source task TS, and a target domain DT and its corresponding target task TT, where DS ≠ DT or TS ≠ TT, the goal of transfer learning is to learn the target conditional probability distribution P(YT|XT) in DT based on the information obtained from DS and TS, where XT represents the sample data in the target domain, and YT represents the label space of the target domain task. In most cases, it is assumed that the number of labeled sample data in the target domain is finite, exponentially less than the number of labeled sample data in the given source domain. Transfer learning can be used to transfer the weight parameters of a model trained on an image dataset in the source domain to a residual network model, enabling effective identification of tea features from hyperspectral data. Transfer learning can accelerate the learning progress of the residual network model and reduce its convergence time. Furthermore, this disclosure employs a full parameter transfer method for cross-domain adaptation.

[0103] The Transformer model is based entirely on the self-attention mechanism, which eliminates the limitation of recurrent neural networks on sequence dependencies and greatly improves parallel computing capabilities and long-distance dependency capture capabilities.

[0104] In one embodiment, see Figure 6 The Transformer model consists of an encoder and a decoder for sequence-to-sequence tasks. The output of the encoder is shifted to the right and then input into the decoder for processing, yielding a spectral feature vector. The encoder converts the input sequence into a continuous vector representation containing contextual information. The encoder can be composed of N identical encoder layers stacked together, where N is a positive integer. Each encoder layer contains two sub-layers, each including a multi-head attention layer and a pre-feedback layer. Residual connections and layer normalization are provided between each sub-layer.

[0105] The decoder is used to convert the continuous vector representation output by the encoder into a target sequence (such as target language text). The decoder is also composed of N identical decoder layers stacked together. In addition to the two sub-layers that are the same as those in the encoder, each decoder layer also includes a masked multi-head attention layer, which is located before the sub-layers. The masked multi-head attention layer is used to focus on information in the input sequence that is related to the current output. A linear layer can also be set in the decoder to map the output of the sub-layers.

[0106] The following is a related explanation of the multi-head attention layer:

[0107] The self-attention mechanism works by ensuring that each element in the input sequence pays attention to information from other elements, thereby generating a continuous vector representation that includes global context. This is used to extract important information from the input sequence. The computational process of the self-attention mechanism mainly relies on… (Query, query vector) (Key, key vector) and Transformation of the (Value, value vector) matrix. Here, Q represents the information needed, K represents the information identifier that the source sequence can provide, and V represents the specific information content of the source sequence. The following are the detailed formulas and steps of the self-attention mechanism:

[0108] Calculate the current element and The similarity (i.e., calculating the dot product between Q and K) is then processed. Normalization yields attention weights, which are then used to... Weighted summation yields the context representation of the current element. The working process of self-attention mechanism can be represented by the following formula. :

[0109] ,

[0110] in, , , They are respectively through the input matrix Weight matrix in Transformer model , , Obtained by linear transformation for Dimensions Let T denote the activation function, and T denote the transpose. Used for scaling to prevent the dot product result from becoming too large. The gradient of the activation function vanishes. The activation function is used to normalize the weights so that they represent the attention distribution. The weight matrix consists of trainable parameters learned by the Transformer model during training.

[0111] Multi-head attention layer will , , The Transformer model is divided into h groups using different linear projections. Each group's self-attention is calculated separately. Finally, the self-attentions from each group are concatenated and linearly projected to obtain the final output, the spectral feature vector. This allows the Transformer model to simultaneously capture attention patterns from different subspaces (such as syntactic dependencies and semantic associations), improving its expressive power. (Multi-head attention layer) This can be expressed by the following formula:

[0112] ,

[0113] in, Formulas representing the effects of different attention mechanisms , and These represent the i-th self-attention mechanism, respectively. , The corresponding weight matrix, This indicates the output linear projection matrix. This represents the concatenation function.

[0114] Furthermore, since the self-attention mechanism itself does not contain positional information and is insensitive to the order of the input sequence, see [link to relevant documentation]. Figure 6 The Transformer model actively injects positional features into the sequence through positional encoding. These features are pre-calculated according to a preset, fixed mathematical formula (i.e., the sine and cosine functions), then directly added to the word vectors before being input into the Transformer model. The sine and cosine functions are expressed by the following formula:

[0115] ,

[0116] ,

[0117] in, Indicates the position of the element in the sequence. Indexed by word vector dimension, This represents the dimension of the input sequence. The positional encoding is added to the input word vector and then fed into the encoder, enabling the model to perceive the relative or absolute position of elements. The word vector is a vector of numbers representing a word, automatically learned from the spectral data to be detected during training.

[0118] S3. A cross-attention mechanism is used to interactively associate the image feature vector and the corresponding spectral feature vector to generate a fused feature vector;

[0119] Unlike the multi-head attention layer that focuses on the relationships between elements within a single sequence, cross attention deals with two different sequences and focuses on the interaction between them.

[0120] This disclosure establishes the correlation between image features and spectral features through a cross-attention mechanism, generating a fused feature vector to solve the problem of the one-sidedness of single-modal features.

[0121] See Figure 7 Given two input sequences: an image feature vector and a spectral feature vector. To achieve information exchange between the two modalities, the image feature vector and the spectral feature vector are used as the source and target sequences, respectively, for cross-modal cross-attention computation.

[0122] The specific steps of the cross-attention mechanism include:

[0123] S31. The image feature vector is processed by a residual network model to generate Q, K, and V corresponding to each element, and the spectral feature vector is processed by a Transformer model to generate Q, K, and V corresponding to each element.

[0124] S32. The Q, K, and V corresponding to the image feature vectors are all linearly processed (corresponding to...). Figure 7 The letter R in the first linear vector is used to obtain the first linear Q, the first linear K, and the first linear V. The Q, K, and V corresponding to the spectral feature vectors are all linearly processed to obtain the second linear Q, the second linear K, and the second linear V.

[0125] S33. Perform a dot product calculation on the second linear Q and the corresponding first linear K to obtain the first attention weight. Perform a dot product calculation on the first attention weight and the corresponding first linear V, and then add it to the image feature vector to obtain the image enhancement feature.

[0126] S34. Perform a dot product calculation on the second linear K and the corresponding first linear Q to obtain the second attention weight. Perform a dot product calculation on the second attention weight and the corresponding second linear V, and then add it to the spectral feature vector to obtain the spectral enhancement feature.

[0127] S35. Input the image enhancement features and spectral enhancement features into the splicing layer and splice them to generate a fused feature vector.

[0128] S4. Input the fused feature vector into the trained grading model to obtain the grading result. The grading model is trained with green tea grade as the label.

[0129] The fused feature vector is input into the grading model, and the grading model is trained using green tea grade as the label to obtain the trained grading model. The hyperspectral data to be detected is then input into the trained grading model to obtain the grading result.

[0130] See Figure 7The hierarchical model is the corresponding classifier, which consists of a first fully connected layer, a ReLU activation function, a regularized layer, a second fully connected layer, and a classifier connected in sequence. Finally, it outputs the probabilities corresponding to different levels, and the level with the highest probability is the final hierarchical result.

[0131] It should be noted that for cases with a maximum probability of less than 0.5, the sample is identified as a retest sample and requires manual verification to rule out issues such as sample contamination and collection errors.

[0132] In one specific embodiment, the method steps provided in this disclosure are applied, see [link to relevant documentation]. Figure 8 All residual network models showed significant performance improvements after generating fused feature vectors; the lightweight ResNet18 performed excellently after fusion, with performance comparable to or even better than the most complex ResNet152, while the number of parameters was greatly reduced. This fully demonstrates that in multimodal tasks, data fusion strategies are more effective than simply increasing model depth.

[0133] See Figure 9 As can be seen from the bar chart analysis, the performance of all models was greatly improved after fusion, and the lightweight ResNet18 / 34 achieved the best performance while having the fewest parameters. This proves that the quality of multimodal data is more important than the complexity of the model. In practical applications, data fusion should be given priority and lightweight models should be selected.

[0134] Table 1 presents the comparative experimental results of multiple models, including the accuracy and parameter count of the green tea grading model. Images represent classification using only the residual network model on hyperspectral data; image + spectral fusion represents using the residual network model to extract image features and the Transformer model to extract spectral feature vectors, finally fused using a cross-attention mechanism. It can be seen that using the cross-attention mechanism significantly improves the classification accuracy of all residual network models, and the performance improvement far exceeds the improvement brought by simply increasing model depth. Notably, the lightweight ResNet18 achieves an accuracy of 98.33% after fusing multimodal data, only 0.42 percentage points lower than the largest ResNet152, but its parameter count (approximately 11.8 million) is only one-fifth of the latter (approximately 59 million). This fully demonstrates that in multimodal tasks, data fusion strategies are more efficient and practical than blindly increasing model size.

[0135] Table 1 Comparison of Multimodal Fusion Models

[0136]

[0137] Based on the above technical solution, this disclosure constructs an acquisition system to collect hyperspectral data to be detected, introduces a residual network model to extract image features, and employs different deep models such as ResNet-18, ResNet-34, ResNet-50, ResNet-101, and ResNet-152. Comparative experiments based on transfer learning are conducted to explore the impact of network depth on the feature extraction effect of green tea images. These models will automatically learn discriminative feature representations from the original green tea images for subsequent classification or quality assessment tasks.

[0138] This disclosure employs the Transformer model, in which the encoder then performs deep encoding on the sequence. Its output can effectively characterize key information such as the overall morphology of the spectrum, the position of characteristic peaks, and their intensity, providing more robust features for subsequent feature fusion.

[0139] This disclosure employs a multimodal fusion model based on a cross-attention mechanism, enabling the features of green tea images and spectra to interact and complement each other, thereby generating a unified fusion representation.

[0140] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this disclosure is not limited to the described order of actions, because according to this disclosure, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this disclosure.

[0141] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0142] The above is a description of the method embodiments. This disclosure also provides a multimodal fusion evaluation system 1000 for green tea grading. See [link to relevant documentation]. Figure 10 ,include:

[0143] The acquisition module 1010 is used to acquire the hyperspectral data of the green tea to be tested through the constructed acquisition system. The hyperspectral data to be tested includes spatial data and spectral data to be tested.

[0144] The extraction module 1020, connected to the acquisition module 1010, is used to extract features from the spatial data to be detected using a residual network model to obtain an image feature vector. The training process of the residual network model adopts a transfer learning strategy. The spectral data to be detected is input into the Transformer model to obtain a spectral feature vector.

[0145] The fusion module 1030, connected to the extraction module 1020, is used to interactively associate the image feature vector and the corresponding spectral feature vector using a cross-attention mechanism to generate a fused feature vector;

[0146] The discrimination module 1040 is connected to the fusion module 1030 and is used to input the fused feature vector into the trained grading model to obtain the grading result. The grading model is trained with green tea grade as the label.

[0147] Other details can be found in the previous methods section and will not be repeated here.

[0148] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the described module can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0149] Figure 11 A schematic block diagram of an electronic device 1100 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0150] Electronic device 1100 includes a computing unit 1101, which can perform various appropriate actions and processes according to a computer program stored in ROM 1102 or a computer program loaded into RAM 1103 from storage unit 1108. RAM 1103 may also store various programs and data required for the operation of electronic device 1100. The computing unit 1101, ROM 1102, and RAM 1103 are interconnected via bus 1104. I / O interface 1105 is also connected to bus 1104.

[0151] Multiple components in electronic device 1100 are connected to I / O interface 1105, including: input unit 1106, such as keyboard, mouse, etc.; output unit 1107, such as various types of displays, speakers, etc.; storage unit 1108, such as disk, optical disk, etc.; and communication unit 1109, such as network card, modem, wireless transceiver, etc. Communication unit 1109 allows electronic device 1100 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0152] The computing unit 1101 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1101 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1101 performs the various methods and processes described above, such as the multimodal fusion evaluation method for green tea grading. For example, in some embodiments, the multimodal fusion evaluation method for green tea grading can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 1108. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 1100 via ROM 1102 and / or communication unit 1109. When the computer program is loaded into RAM 1103 and executed by the computing unit 1101, one or more steps of the multimodal fusion evaluation method for green tea grading described above can be performed. Alternatively, in other embodiments, the computing unit 1101 may be configured by any other suitable means (e.g., by means of firmware) to perform the multimodal fusion evaluation method for green tea grading.

[0153] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0154] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0155] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0156] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including voice input, speech input, or tactile input).

[0157] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0158] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0159] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0160] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A multi-modal fusion evaluation method for green tea grading, characterized in that, The application relates to a green tea grade detection method and system. The application comprises the following steps: Collecting the to-be-detected hyperspectral data of the to-be-detected green tea through a constructed collection system, wherein the to-be-detected hyperspectral data comprises to-be-detected spatial data and to-be-detected spectral data; Extracting features of the to-be-detected spatial data by using a residual network model to obtain an image feature vector, wherein a migration learning strategy is adopted in the training process of the residual network model; and inputting the to-be-detected spectral data into a Transformer model to obtain a spectral feature vector; Interacting and correlating the image feature vector and the corresponding spectral feature vector by using a cross-attention mechanism to generate a fusion feature vector; 2. The multi-modal fusion evaluation method for green tea grading as claimed in claim 1 wherein, Inputting the fusion feature vector into a trained hierarchical model to obtain a hierarchical result, wherein the hierarchical model is trained by taking the grade of green tea as a label. The collection system comprises a hyperspectral imager, a standard white board, a display, a dark box, a halogen lamp light source and a glass dish; The glass dish is arranged in the dark box and used for containing green tea; The halogen lamp light source is fixed on the side wall in the dark box and used for providing an imaging light source; The hyperspectral imager is fixed on the top of the dark box and used for collecting the hyperspectral data; The display is connected with the hyperspectral imager and used for displaying the collection process and visually displaying the collected hyperspectral data; 3. The multi-modal fusion evaluation method for green tea grading as claimed in claim 1 wherein, The standard white board is arranged directly below the hyperspectral imager and used for ensuring that the light collection condition is unchanged. The residual network model comprises any one of ResNet-18, ResNet-34, ResNet-50, ResNet-101 and ResNet-152; wherein ResNet-18 and ResNet-34 adopt a BasicBlock residual block, and ResNet-50, ResNet-101 and ResNet-152 adopt a BottleNeck residual block; The BasicBlock residual block comprises a first convolutional layer and a second convolutional layer in sequence, and a batch normalization layer and a ReLU activation function are arranged after the first convolutional layer and the second convolutional layer; 4. The multi-modal fusion evaluation method for green tea grading as claimed in claim 3 wherein, The BottleNeck residual block comprises a third convolutional layer, a fourth convolutional layer and a fifth convolutional layer in sequence, and a batch normalization layer and a ReLU activation function are arranged after the third convolutional layer, the fourth convolutional layer and the fifth convolutional layer.

5. The multi-modal fusion evaluation method for green tea grading as claimed in claim 1 wherein, The residual network model comprises an input layer, a 7*7 convolutional layer, a batch normalization layer, a ReLU activation function, a maximum pooling layer, a residual block layer, an average layer, a full connection layer and an output layer in sequence, and the residual block layer comprises a plurality of BasicBlock residual blocks or a plurality of BottleNeck residual blocks. The construction of the residual network model comprises the following steps: Collecting the hyperspectral data of different green teas through the constructed collection system, labeling the hyperspectral data and dividing the labeled hyperspectral data into a training set and a test set; 6. The multi-modal fusion evaluation method for green tea grading as claimed in claim 1 wherein, Based on migration learning, training the residual network model by using the training set and verifying the trained residual network model by using the test set. The Transformer model comprises an encoder and a decoder; The encoder is stacked by N identical encoder layers, N is a positive integer, each encoder layer includes two sub-layers, each sub-layer includes a multi-head attention layer and a front feedback layer, and each sub-layer is provided with a residual connection and layer normalization; The decoder is stacked by N identical decoder layers, each decoder layer includes two sub-layers and a mask multi-head attention layer; wherein, The multi-head attention layer is expressed by the following equation: ， wherein, The formula representing the role of different self-attention mechanisms, denotes a concatenation function, Q denotes a query vector, K denotes a key vector, V denotes a value vector, and h denotes a vector obtained by , , The number of groups into which the different linear projections are divided, respectively, denotes an output linear projection matrix; ， in, The formula representing the role of self-attention mechanism is as follows: , and These represent the i-th self-attention mechanism, respectively. The weight matrices corresponding to K and V.

7. The multi-modal fusion evaluation method for green tea grading as claimed in claim 1 wherein, The cross-attention mechanism specifically includes the following steps: The image feature vector is processed by a residual network model to generate Q, K and V corresponding to each element respectively, and the spectral feature vector is processed by a Transformer model to generate Q, K and V corresponding to each element respectively; The Q, K and V corresponding to the image feature vector are all subjected to linear processing to obtain first linear Q, first linear K and first linear V, and the Q, K and V corresponding to the spectral feature vector are all subjected to linear processing to obtain second linear Q, second linear K and second linear V; The second linear Q and the corresponding first linear K are dot product calculated to obtain a first attention weight, and the first attention weight and the corresponding first linear V are dot product calculated and then added to the image feature vector to obtain an image enhanced feature; The second linear K and the corresponding first linear Q are dot product calculated to obtain a second attention weight, and the second attention weight and the corresponding second linear V are dot product calculated and then added to the spectral feature vector to obtain a spectral enhanced feature; The image enhanced feature and the spectral enhanced feature are input into a splicing layer for splicing to generate a fusion feature vector.

8. A multi-modal fusion evaluation system for green tea grading for implementing the method of any one of claims 1 to 7, characterized in that, It comprises the following modules: The acquisition module is used for acquiring the to-be-detected hyperspectral data of the green tea through the constructed acquisition system, and the to-be-detected hyperspectral data includes to-be-detected spatial data and to-be-detected spectral data; The extraction module is connected with the acquisition module and is used for extracting features of the to-be-detected spatial data by using a residual network model to obtain an image feature vector, and a transfer learning strategy is used in the training process of the residual network model; and the to-be-detected spectral data is input into a Transformer model to obtain a spectral feature vector; The fusion module is connected with the extraction module and is used for interacting and correlating the image feature vector and the corresponding spectral feature vector by using a cross-attention mechanism to generate a fusion feature vector; The discrimination module is connected with the fusion module and is used for inputting the fusion feature vector into a trained grading model to obtain a grading result, and the grading model is trained with green tea grades as labels.

9. An electronic device, comprising: It comprises a memory, a processor and a computer program; wherein the computer program is stored in the memory and is configured to be executed by the processor to realize the multi-modal fusion evaluation method for green tea grading according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, A computer program is stored thereon; the computer program is executed by a processor to realize the multi-modal fusion evaluation method for green tea grading according to any one of claims 1 to 7.

Images