Tobacco appearance quality prediction method and system based on multi-modal spectral information fusion

Abstract

This invention discloses a method and system for predicting the appearance quality of tobacco leaves based on multimodal spatial-spectral information fusion. The method includes: acquiring RGB images, hyperspectral images, and near-infrared images of tobacco leaves; performing dimensionality reduction processing on the hyperspectral images; performing data augmentation processing on the RGB images and the dimensionality-reduced hyperspectral images; fusing the visible light images, hyperspectral, and near-infrared data of tobacco leaves using a multimodal feature extraction network architecture to obtain a fusion result; and predicting the classification result of tobacco leaf appearance quality based on the fusion result. The method and system for predicting the appearance quality of tobacco leaves based on multimodal spatial-spectral information fusion of this invention establishes a three-channel discriminant model based on multimodal digital images + hyperspectral + near-infrared data. It extracts RGB image features by building a visual encoding module and extracts tobacco leaf appearance features in hyperspectral and near-infrared data by building a convolution module. The correlation of appearance quality is achieved through feature-level feature fusion.

Description

Technical Field

[0001] This invention relates to the field of tobacco leaf appearance quality inspection technology, and more specifically, to a method and system for predicting tobacco leaf appearance quality based on multimodal spatial-spectral information fusion. Background Technology

[0002] As the core raw material for cigarette production, the quality of tobacco leaves directly determines the style and quality of the final product. Tobacco leaf grading is an essential link in the tobacco industry chain, aiming to classify tobacco leaves into different grades based on their appearance and internal quality, in order to optimize resource allocation and stabilize product quality.

[0003] However, the current mainstream tobacco grading methods still heavily rely on manual assessment. This method is not only labor-intensive and inefficient, but more importantly, its assessment results are easily influenced by the grader's personal experience and environmental factors, making it difficult to standardize grading criteria and resulting in a lack of objectivity and consistency. Therefore, a new grading system needs to be developed.

[0004] Therefore, there is an urgent need for a method and system for predicting the appearance quality of tobacco leaves based on the fusion of multimodal spatial-spectral information. Summary of the Invention

[0005] The purpose of this invention is to provide a method and system for predicting the appearance quality of tobacco leaves based on multimodal spatial-spectral information fusion, so as to solve the problems in the prior art mentioned above, and to achieve automated grading of tobacco leaf appearance quality objectively, accurately and efficiently, thereby promoting the modernization of the tobacco industry.

[0006] This invention provides a method for predicting the appearance quality of tobacco leaves based on multimodal spatial-spectral information fusion, comprising:

[0007] Acquire RGB, hyperspectral, and near-infrared images of tobacco leaves;

[0008] Dimensionality reduction processing of hyperspectral images of tobacco leaves;

[0009] Data augmentation processing was performed on the RGB images and dimensionality-reduced hyperspectral images of tobacco leaves;

[0010] Visible light images, hyperspectral and near-infrared data of tobacco leaves are fused using a multimodal feature extraction network architecture to obtain a fusion result. The multimodal feature extraction network architecture includes: a hyperspectral feature extraction module based on a two-dimensional convolutional neural network, a visual Transformer encoded image feature extraction module, and a spectral feature extraction module based on a one-dimensional convolutional neural network.

[0011] The classification results of tobacco leaf appearance quality are predicted based on the fusion results.

[0012] The method for predicting the appearance quality of tobacco leaves based on multimodal spatial-spectral information fusion, as described above, preferably includes the following: Dimensionality reduction processing of the hyperspectral image of the tobacco leaves:

[0013] Principal component analysis was used to reduce the dimensionality of the hyperspectral images of tobacco leaves: the principal components with a cumulative variance explanation rate of 99% were retained as the optimal principal components.

[0014] The method for predicting the appearance quality of tobacco leaves based on multimodal spatial-spectral information fusion, as described above, preferably includes the following step: Data augmentation processing of the RGB image and the dimensionality-reduced hyperspectral image of the tobacco leaves, comprising:

[0015] By using at least one of random horizontal flipping, random vertical flipping, and random rotations of 90 degrees, 180 degrees, and 270 degrees, the sample size of each subcategory in the RGB image and the dimensionality-reduced hyperspectral image of tobacco leaves is expanded to 500, and a common random seed is set during the data augmentation process.

[0016] The method for predicting the appearance quality of tobacco leaves based on multimodal spatial-spectral information fusion, as described above, preferably involves fusing visible light images, hyperspectral data, and near-infrared data of tobacco leaves using a multimodal feature extraction network architecture to obtain a fusion result, including:

[0017] In the hyperspectral feature extraction module based on a two-dimensional convolutional neural network, the input is a hyperspectral data cube H×W×B, and feature extraction is performed through two-dimensional convolution.

[0018] In the visual Transformer encoded image feature extraction module, the input is a spatially registered RGB image, and the visual Transformer is used as the backbone network for feature extraction to capture the global spatial dependencies in the RGB image of the tobacco leaf.

[0019] In the spectral feature extraction module based on a one-dimensional convolutional neural network, the input is N×P of parallel sampled near-infrared spectral data, and feature extraction is performed through one-dimensional convolution.

[0020] The feature vectors extracted from the hyperspectral feature extraction module based on the two-dimensional convolutional neural network, the visual Transformer encoded image feature extraction module, and the spectral feature extraction module based on the one-dimensional convolutional neural network are concatenated, and the vector concatenation result is used as the fusion result.

[0021] The method for predicting the appearance quality of tobacco leaves based on multimodal spatial-spectral information fusion, as described above, preferably includes the following: In the hyperspectral feature extraction module based on a two-dimensional convolutional neural network, the input is a hyperspectral data cube H×W×B, and feature extraction is performed through two-dimensional convolution, including:

[0022] The input hyperspectral data is processed through a series of two-dimensional convolutional layers. Each convolutional layer is followed by batch normalization and ReLU activation functions. Max pooling is then used to reduce the dimensionality of the data. After three convolutional layers, activation, and max pooling operations, the extracted features are then combined using average pooling.

[0023] In the spectral feature extraction module based on a one-dimensional convolutional neural network, the input is N×P of parallel-sampled near-infrared spectral data, and feature extraction is performed through one-dimensional convolution, including:

[0024] The input spectral data is processed through a series of one-dimensional convolutional layers. Each one-dimensional convolutional layer is followed by batch normalization and ReLU activation functions. The data is then reduced in dimensionality through max pooling. After three one-dimensional convolutional layers, activation, and max pooling operations, the extracted features are then integrated using average pooling.

[0025] The method for predicting the appearance quality of tobacco leaves based on multimodal spatial-spectral information fusion, as described above, preferably includes the following: In the visual Transformer encoded image feature extraction module, the input is a spatially registered RGB image, and a visual Transformer is used as the backbone network for feature extraction to capture the global spatial dependencies in the RGB image of the tobacco leaves, including:

[0026] The image is segmented into a series of non-overlapping image blocks, and each image block is converted into a one-dimensional sequence. The one-dimensional sequence is then globally modeled using the self-attention mechanism of the Transformer encoder.

[0027] The method for predicting the appearance quality of tobacco leaves based on multimodal spatial-spectral information fusion, as described above, preferably includes the following steps: segmenting the image into a series of non-overlapping image patches, converting each image patch into a one-dimensional sequence, and globally modeling the one-dimensional sequence using the self-attention mechanism of a Transformer encoder;

[0028] Batch partitioning and encoding operations are performed using a block grid counting module, a block vector embedding mapping module, and a category-guided location injection module.

[0029] Encoding is performed using a Transformer encoder that includes a residual self-attention update module and a residual feedforward reshaping module;

[0030] Multi-head self-attention calculation is performed using the QKV projection generation module, the scaling correlation aggregation module, and the multi-head stitching shaping module.

[0031] The tobacco leaf appearance quality prediction method based on multimodal spatial-spectral information fusion described above preferably includes the following step: performing batch partitioning and encoding operations using a block grid counting module, a block vector embedding mapping module, and a category-guided position injection module.

[0032] The size parameters of the input RGB image are obtained through a block grid counting module, and the image is divided into grids according to a preset block side length to determine the number of image blocks required for subsequent sequence modeling. The number of image patches can be calculated using the following formula:

[0033] (1)

[0034] in: Indicates the number of image patches. Indicates the height of the input image. This represents the height and width of the input image. Indicates the side length of the block;

[0035] The block vector embedding mapping module is used to process the first... Flattening the image blocks yields And through a linear mapping matrix Projected to dimension The feature space is obtained The mapping relationship can be represented by the following formula:

[0036]

[0037] in, For the first The initial embedding vector of each image patch, For flattened block vectors, the length is... , Represents the projection matrix. Indicates the embedding dimension. Indicates block index and The total number of blocks, To represent the number of channels;

[0038] Learnable category tags are introduced through the category-guided location injection module. Mark the learnable categories The concatenation is appended to the front of the block embedding sequence, and the overlay position encoding matrix of the concatenated sequence is used. The initial sequence entering the encoder is obtained. The initial sequence is represented by the following formula. Construction method:

[0039]

[0040] in, This represents an input sequence containing category labels and position codes. Represents the category label vector, Represents a block embedding sequence. Represents the position encoding matrix,

[0041] The encoding using a Transformer encoder, which includes a residual self-attention update module and a residual feedforward reshaping module, includes:

[0042] The visual encoder consists of stacked Transformer modules, each layer containing a residual self-attention update module and a residual feedforward renormalization module.

[0043] The residual self-attention update module is used to update the first... -1 layer output First perform layer normalization Then input the multi-head self-attention operator. The intermediate output is obtained by adding the residual from the original input. The intermediate output is obtained through the following formula. Update relationship:

[0044]

[0045] in, Indicates the first The attention sublayer output of the layer, Indicates the sequence features of the previous layer. Representation layer normalization, Indicates a multi-head self-attention operator;

[0046] The output of the residual self-attention update module is processed by the residual feedforward renormalization module. Stratigraphic normalization Post-input multilayer perceptron and with The residuals are summed to obtain the layer output. It can be expressed by the following formula:

[0047]

[0048] in, Indicates the first The final output of the layer, This represents the output of the attention sublayer. Representation layer normalization, To represent a multilayer perceptron, Indicates the number of encoder layers and ,

[0049] The multi-head self-attention calculation using the QKV projection generation module, the scaling correlation convergence module, and the multi-head stitching and shaping module includes:

[0050] The input sequence features are generated using the QKV projection generation module. Perform three sets of linear projections to obtain the query branches. Key branches The linear projection relationship of the sum branch is expressed by the following formula:

[0051]

[0052] in, Represents the features of the input sequence. , and Represents three sets of projection matrices. Generate a query matrix. Generate the key matrix. Used to characterize value branches The linear projection relationship;

[0053] The scaling relevance aggregation module utilizes the inner product relevance between the query and the key to calculate attention weights, and then performs weighted aggregation on the value branches to obtain the single-head attention output, expressed by the following formula:

[0054] (7)

[0055] in, This indicates single-head attention output. Represents the query matrix. Represents the key matrix. This represents the transpose of the key matrix. The scaling factor represents the dimension of the key vector. Generate normalized weights and apply them to The weighted aggregation yields the results;

[0056] The multi-head concatenation and shaping module concatenates the outputs of each attention head in the feature dimension and maps them back to a unified dimension through the output projection matrix to obtain the multi-head self-attention output, which is expressed by the following formula:

[0057]

[0058] in, This indicates the output of multi-head self-attention. This indicates the output of each attention head. Indicates the number of heads. Indicates feature dimension splicing, This indicates the output projection matrix.

[0059] The method for predicting the appearance quality of tobacco leaves based on multimodal spatial-spectral information fusion as described above, preferably, includes the following: Predicting the classification result of the tobacco leaf appearance quality based on the fusion result includes:

[0060] The fusion result is input into a multilayer perceptron classifier for processing, and the classification result of tobacco leaf appearance quality is output. Specifically, this includes: using a global label discriminant output module for classification output: the global label discriminant output module will process the fusion result... Class label vector after layer Transformer encoder As a global feature representation of the image, it is input into the multilayer perceptron classification head for prediction, expressed by the following formula:

[0061]

[0062] in, Indicates the predicted output. Indicates the first The category label vector after the layer, This represents a category header, used to classify... Mapped to the task output space, Indicates the number of encoder layers.

[0063] The present invention also provides a tobacco leaf appearance quality prediction system based on multimodal spatial-spectral information fusion using the above method, comprising:

[0064] The data acquisition module is used to acquire RGB images, hyperspectral images, and near-infrared images of tobacco leaves;

[0065] The dimension reduction module is used to perform dimension reduction processing on the hyperspectral images of tobacco leaves;

[0066] The data augmentation module is used to perform data augmentation processing on the RGB image and the dimension-reduced hyperspectral image of tobacco leaves;

[0067] The multimodal feature fusion module is used to fuse visible light images, hyperspectral and near-infrared data of tobacco leaves using a multimodal feature extraction network architecture to obtain a fusion result. The multimodal feature extraction network architecture includes: a hyperspectral feature extraction module based on a two-dimensional convolutional neural network, a visual Transformer encoded image feature extraction module, and a spectral feature extraction module based on a one-dimensional convolutional neural network.

[0068] The classification module is used to predict the classification result of the tobacco leaf appearance quality based on the fusion result.

[0069] This invention provides a method and system for predicting the appearance quality of tobacco leaves based on multimodal spatial-spectral information fusion. It introduces hyperspectral imaging technology, which simultaneously incorporates spatial and narrow-band spectra, into the analysis of tobacco leaf appearance quality. By acquiring spectral information in the 400-1000 nm visible and near-infrared range, it constructs a correlation between the hyperspectral band and various tobacco leaf appearance quality attributes. Near-infrared spectral information is used as a supplement to provide more accurate molecular descriptive information, compensating for the loss of spectral information in hyperspectral data due to balancing acquisition efficiency and spatial dimension. This provides data support for accurate discriminant analysis. This paper proposes a complete multi-attribute evaluation framework for tobacco leaves based on the fusion of hyperspectral imaging and high-resolution near-infrared spectroscopy. This framework systematically integrates hyperspectral data dimensionality reduction, multimodal feature extraction, and feature fusion techniques to construct independent models for the six core quality attributes of tobacco leaves, thereby achieving accurate evaluation of each attribute. A three-channel discrimination model based on multimodal digital images + hyperspectral + near-infrared data is built. By constructing a visual coding module to extract RGB image features and a convolution module to extract tobacco leaf appearance features in hyperspectral and near-infrared spectroscopy, the correlation of appearance quality is achieved through feature-level feature fusion. Attached Figure Description

[0070] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described below with reference to the accompanying drawings, wherein:

[0071] Figure 1 A flowchart illustrating an embodiment of the tobacco leaf appearance quality prediction method based on multimodal spatial-spectral information fusion provided by the present invention;

[0072] Figure 2 A schematic diagram of example data for tobacco leaves;

[0073] Figure 3 A schematic diagram illustrating the dimensionality reduction process for HIS data;

[0074] Figure 4 This is a schematic diagram of the data amplification process;

[0075] Figure 5 For multimodal feature extraction network architecture;

[0076] Figure 6 Architecture diagram of the image feature extraction module for visual Transformer encoding;

[0077] Figure 7 This is a structural diagram of a Transformer encoder;

[0078] Figure 8 This is a structural block diagram of an embodiment of the tobacco leaf appearance quality prediction system based on multimodal spatial spectrum information fusion provided by the present invention. Detailed Implementation

[0079] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. The descriptions of the exemplary embodiments are merely illustrative and are in no way intended to limit the present disclosure or its application or use. The present disclosure may be implemented in many different forms and is not limited to the embodiments described herein. These embodiments are provided so that the present disclosure will be thorough and complete, and will fully express the scope of the disclosure to those skilled in the art. It should be noted that, unless specifically stated otherwise, the relative arrangement of components and steps, the composition of materials, numerical expressions, and values ​​set forth in these embodiments should be interpreted as exemplary only and not as limiting.

[0080] The terms “first,” “second,” and similar terms used in this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different parts. Terms such as “including” or “contains” mean that the element preceding the term encompasses the element listed after it, and do not exclude the possibility of encompassing other elements as well. Terms such as “above” and “below” are used only to indicate relative positional relationships; when the absolute position of the described object changes, this relative positional relationship may also change accordingly.

[0081] In this disclosure, when a specific component is described as being located between a first component and a second component, an intermediary component may or may not be present between the specific component and the first or second component. When a specific component is described as connecting to other components, the specific component may be directly connected to the other components without having an intermediary component, or it may not be directly connected to the other components but may have an intermediary component.

[0082] All terms used in this disclosure (including technical or scientific terms) have the same meaning as understood by one of ordinary skill in the art to which this disclosure pertains, unless otherwise specifically defined. It should also be understood that terms defined in a general dictionary, such as a dictionary, should be interpreted as having a meaning consistent with their meaning in the context of the relevant art, and not as having an idealized or highly formalized meaning, unless expressly defined herein.

[0083] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, they should be considered part of the specification.

[0084] Hyperspectral technology can simultaneously acquire spatial and spectral information of a target object, forming a "data cube" that integrates image and spectrum, thereby accurately capturing the macroscopic characteristics of the sample. Near-infrared technology enables in-depth characterization of the sample's composition. Furthermore, because both technologies possess non-destructive testing characteristics, they show great promise in the non-destructive evaluation of agricultural product quality (such as freshness, component content, and defects). The combined use of near-infrared and hyperspectral technologies essentially represents a synergistic enhancement of spectral depth and spatial breadth, offering significant advantages in the analysis of complex chemical systems. It is expected to provide a stable, efficient, and automated solution for evaluating the appearance quality of tobacco leaves, including color, maturity, identity, oil content, structure, and chroma.

[0085] With the rapid development of deep learning technology, especially the widespread application of models represented by Convolutional Neural Networks (CNNs), some related methods have been used in tobacco leaf data mining. Leveraging the powerful feature learning capabilities of CNNs in image recognition, research has been conducted on areas such as tobacco leaf grading and damage identification based on digital images. However, RGB images are broadband images, providing relatively low data granularity, only reflecting information related to appearance color and condition. Richer physical and chemical features, such as oil content, identity, and structure, urgently need to be incorporated into the tobacco leaf appearance quality evaluation system. Furthermore, CNN models, designed based on local convolutional kernels, have inherent limitations in capturing global long-range dependencies in images. Currently, hyperspectral imaging has been studied in areas such as tobacco leaf mold and damage, but systematic exploration in the evaluation of tobacco leaf appearance quality (the intrinsic quality of the tobacco leaf itself) remains lacking.

[0086] The processing of the tobacco leaf appearance quality prediction method based on multimodal spatial-spectral information fusion of the present invention is developed and run under the PyTorch framework. The computer operating system used is Windows 11, the processor model is i7-14675HX, the graphics card is NVIDIA GeForce RTX 4060, the memory is 16GB, the programming environment is Python 3.8, and the CUDA version is 12.6.

[0087] like Figure 1 As shown, the tobacco leaf appearance quality prediction method based on multimodal spatial-spectral information fusion provided in this embodiment includes the following steps in actual implementation:

[0088] Step S1: Acquire RGB image, hyperspectral image and near-infrared image of tobacco leaves.

[0089] In one embodiment of the invention, data was collected from 48 tobacco stations in Henan Province, representing a broad geographical area. Data collection was conducted under standardized experimental conditions, including color temperature (5300–5800 K), working surface illuminance (1800–2200 Lx), temperature (20–24℃), and relative humidity (65–75%), to eliminate interference from ambient light, temperature, and humidity. The resulting hyperspectral files numbered 378, with each hyperspectral image measuring 856,1092,480 pixels. The number of RGB images of tobacco leaves also totaled 378, with a size of 856,10921,3 pixels. Examples of the data are shown below. Figure 2 As shown.

[0090] Step S2: Perform dimensionality reduction processing on the hyperspectral image of the tobacco leaves.

[0091] The "curse of dimensionality" caused by the hundreds of bands contained in hyperspectral data not only increases the computational burden but also challenges the performance of classification models due to information redundancy. To alleviate this problem and extract the most relevant features, this invention performs dimensionality reduction processing on hyperspectral images of tobacco leaves. Specifically, as... Figure 3 As shown, Principal Component Analysis (PCA) was used to reduce the dimensionality of the hyperspectral images of tobacco leaves: the principal components with a cumulative variance explanation rate of 99% were retained as the optimal principal components to construct an information-dense feature space with significantly reduced dimensionality. The data dimensionality reduction process is as follows: Figure 3 As shown.

[0092] Step S3: Perform data augmentation processing on the RGB image and the dimensionality-reduced hyperspectral image of the tobacco leaves.

[0093] To address the issues of small original dataset size (378 samples) and class imbalance, this invention employs data augmentation techniques to prevent model overfitting. Specifically, by using at least one of random horizontal flipping, random vertical flipping, and random rotations of 90 degrees, 180 degrees, and 270 degrees, the sample size of each subclass in the RGB images and dimensionality-reduced hyperspectral images of tobacco leaves is expanded to 500. Furthermore, during the data augmentation process, a common random seed is set to ensure consistency in spatial transformation between paired RGB images and hyperspectral data. Both RGB and hyperspectral images are uniformly sized to 300×300 pixels after transformation. The data augmentation process is as follows: Figure 4 As shown.

[0094] Step S4: Use a multimodal feature extraction network architecture to fuse the visible light image, hyperspectral and near-infrared data of tobacco leaves to obtain the fusion result.

[0095] Among them, such as Figure 5 As shown, the multimodal feature extraction network architecture includes: a 2D CNN Spectral Feature Extraction Module (2DSFEM) based on a two-dimensional convolutional neural network, a ViT Image Feature Extraction Module (ViTIFEM) based on a visual Transformer-coded image feature extraction module, and a 1D CNN Spectral Feature Extraction Module (1DSFEM) based on a one-dimensional convolutional neural network. This multimodal feature extraction network architecture enables the effective fusion of visible light images, hyperspectral data, and near-infrared data of tobacco leaves to achieve simultaneous classification of six core quality attributes. In the end-to-end framework, the 1DSFEM, 2DSFEM, and ViTIFEM modules are trained simultaneously to perform cross-modal feature fusion and optimization.

[0096] In one embodiment of the tobacco leaf appearance quality prediction method based on multimodal spatial-spectral information fusion of the present invention, step S4 may specifically include:

[0097] Step S41: In the hyperspectral feature extraction module based on the two-dimensional convolutional neural network, the input is a hyperspectral data cube H×W×B, and feature extraction is performed through two-dimensional convolution.

[0098] The hyperspectral feature extraction module based on a two-dimensional convolutional neural network can capture deep features of hyperspectral data. Specifically, the input hyperspectral data is processed through a series of two-dimensional convolutional layers, each followed by batch normalization and ReLU activation functions to improve the network's convergence speed and nonlinear expressive power. Max pooling is used to reduce the dimensionality of the data, further reducing the data dimension and enhancing the model's robustness. After three layers of two-dimensional convolutional layers, activation, and max pooling, the extracted features are integrated using average pooling to preserve important spatial and spectral information. This module can capture spatial-spectral features (referred to as spatial-spectral features) in hyperspectral images, providing accurate feature representations for subsequent classification tasks.

[0099] Step S42: In the visual Transformer encoded image feature extraction module, the input is a spatially registered RGB image. The visual Transformer is used as the backbone network for feature extraction to capture the global spatial dependencies in the RGB image of the tobacco leaf.

[0100] In this invention, the visual Transformer encoded image feature extraction module serves as the core mechanism of the model, such as... Figure 6 As shown, the image is segmented into a series of fixed-size, non-overlapping image patches. Each patch is then linearly mapped and transformed into a one-dimensional sequence of feature vectors, while spatial location information of the image patches is preserved through positional encoding. Subsequently, as... Figure 7 As shown, the one-dimensional sequence is globally modeled using the self-attention mechanism of the Transformer encoder. This mechanism enables the model to establish long-distance dependencies between any two patches in the image, thereby gaining a deeper understanding of the global relationships of the macroscopic morphology, texture distribution, color, etc., of tobacco leaves. In a specific implementation, these feature sequences are fed into the Transformer encoder, where the global dependencies between image patches are modeled using a multi-head self-attention mechanism. In one embodiment of the tobacco leaf appearance quality prediction method based on multimodal spatial-spectral information fusion of the present invention, step S42 may specifically include:

[0101] Step S421: Perform batch partitioning and encoding operations using the block grid counting module, block vector embedding mapping module, and category-guided position injection module.

[0102] In one embodiment of the tobacco leaf appearance quality prediction method based on multimodal spatial-spectral information fusion of the present invention, step S421 may specifically include:

[0103] Step S4211: Obtain the size parameters of the input RGB image through the block grid counting module, and divide the image into grids according to the preset block side length to determine the number of image blocks required for subsequent sequence modeling. The number of image patches can be calculated using the following formula:

[0104] (1)

[0105] in: Indicates the number of image patches. Indicates the height of the input image. This represents the height and width of the input image. This indicates the side length of the block.

[0106] Step S4212: The block vector embedding mapping module is used to process the first... Flattening the image blocks yields And through a linear mapping matrix Projected to dimension The feature space is obtained The mapping relationship can be represented by the following formula:

[0107]

[0108] in, For the first The initial embedding vector of each image patch, For flattened block vectors, the length is... , Represents the projection matrix. Indicates the embedding dimension. Indicates block index and The total number of blocks, To represent the number of channels;

[0109] Step S4213: Introduce learnable category tags through the category-guided location injection module. Mark the learnable categories The concatenation is appended to the front of the block embedding sequence, and the overlay position encoding matrix of the concatenated sequence is used. The initial sequence entering the encoder is obtained. The initial sequence is represented by the following formula. Construction method:

[0110]

[0111] in, This represents an input sequence containing category labels and position codes. Represents the category label vector, Represents a block embedding sequence. This represents the position encoding matrix. Position encoding matrix The concatenated sequence of the category label vector and the block embedding sequence is added item by item to obtain the input sequence containing the category label and position code. .

[0112] The category-guided location injection module can extract category labels for classification tasks, thereby obtaining a global visual feature representation.

[0113] Step S422: Encode using a Transformer encoder that includes a residual self-attention update module and a residual feedforward reshaping module.

[0114] In one embodiment of the tobacco leaf appearance quality prediction method based on multimodal spatial-spectral information fusion of the present invention, step S422 may specifically include:

[0115] Step S4221, as follows Figure 7 As shown, the visual encoder consists of stacked Transformer modules, with each layer containing a residual self-attention update module and a residual feedforward renormalization module.

[0116] Step S4222: Update the residual self-attention module for the first... -1 layer output First perform layer normalization Then input the multi-head self-attention operator. The intermediate output is obtained by adding the residual from the original input. The intermediate output is obtained through the following formula. Update relationship:

[0117]

[0118] in, Indicates the first The attention sublayer output of the layer, Indicates the sequence features of the previous layer. Representation layer normalization, This represents a multi-head self-attention operator.

[0119] Step S4223: The output of the residual self-attention update module is updated through the residual feedforward renormalization module. Stratigraphic normalization Post-input multilayer perceptron and with The residuals are summed to obtain the layer output. It can be expressed by the following formula:

[0120]

[0121] in, Indicates the first The final output of the layer, This represents the output of the attention sublayer. Representation layer normalization, To represent a multilayer perceptron, Indicates the number of encoder layers and .

[0122] Step S423: Perform multi-head self-attention calculation using the QKV projection generation module, the scaling correlation convergence module, and the multi-head stitching shaping module.

[0123] In one embodiment of the tobacco leaf appearance quality prediction method based on multimodal spatial-spectral information fusion of the present invention, step S423 may specifically include:

[0124] Step S4231: Use the QKV projection generation module to process the features of the input sequence. Perform three sets of linear projections to obtain the query branches. Key branches The linear projection relationship of the sum branch is expressed by the following formula:

[0125]

[0126] in, Represents the features of the input sequence. , and Represents three sets of projection matrices. Generate a query matrix. Generate the key matrix. Used to characterize value branches The linear projection relationship.

[0127] Step S4232: The scaling relevance aggregation module utilizes the inner product relevance between the query and the key to calculate the attention weights, and then performs weighted aggregation on the value branches to obtain the single-head attention output, expressed by the following formula:

[0128] (7)

[0129] in, This indicates single-head attention output. Represents the query matrix. Represents the key matrix. This represents the transpose of the key matrix. The scaling factor represents the dimension of the key vector. Generate normalized weights and apply them to The weighted aggregation yields the result.

[0130] Step S4233: The outputs of each attention head are concatenated in the feature dimension using the multi-head splicing and shaping module, and then mapped back to a unified dimension using the output projection matrix to obtain the multi-head self-attention output, which is expressed by the following formula:

[0131]

[0132] in, This indicates the output of multi-head self-attention. This indicates the output of each attention head. Indicates the number of heads. Indicates feature dimension splicing, This indicates the output projection matrix, used to complete dimensional shaping and fusion.

[0133] Therefore, the visual Transformer-encoded image feature extraction module uses the visual Transformer as its backbone and efficiently extracts global spatial features from visible light images through a self-attention mechanism.

[0134] Step S43: In the spectral feature extraction module based on a one-dimensional convolutional neural network, the input is N×P of parallel sampled near-infrared spectral data, and feature extraction is performed through one-dimensional convolution.

[0135] Near-infrared spectroscopy uses data collected multiple times from the same sample as input. A spectral feature extraction module based on a one-dimensional convolutional neural network can capture deep features of the near-infrared spectral data. Specifically, the input spectral data passes through a series of one-dimensional convolutional layers, each followed by batch normalization and ReLU activation functions to improve the network's convergence speed and nonlinear expressive power. Max pooling is then used to reduce the data dimensionality, further enhancing the model's robustness. After three layers of one-dimensional convolutional layers, activation, and max pooling, the extracted features are integrated using average pooling to preserve important spatial and spectral information. This module can capture vibrational features in the near-infrared spectrum, providing accurate feature representations for subsequent classification tasks.

[0136] Step S44: The feature vectors extracted from the hyperspectral feature extraction module based on the two-dimensional convolutional neural network, the visual Transformer encoded image feature extraction module, and the spectral feature extraction module based on the one-dimensional convolutional neural network are concatenated, and the vector concatenation result is used as the fusion result.

[0137] The feature vectors extracted by the three modules—the hyperspectral feature extraction module based on a two-dimensional convolutional neural network, the visual Transformer encoded image feature extraction module, and the spectral feature extraction module based on a one-dimensional convolutional neural network—are concatenated together to form a comprehensive joint feature vector.

[0138] Step S5: Predict the classification result of tobacco leaf appearance quality based on the fusion result.

[0139] In this invention, such as Figure 5 As shown, the fusion result is input into a multilayer perceptron classifier for processing, outputting classification results for six quality attributes of tobacco leaf appearance (color, maturity, structure, identification, oil content, and chroma). Specifically, this includes classification output using a global label discriminant output module. Specifically, the global label discriminant output module will process the classification results of the six quality attributes of tobacco leaf appearance (color, maturity, structure, identification, oil content, and chroma). Class label vector after layer Transformer encoder As a global feature representation of the image, it is input into the multilayer perceptron classification head for prediction, expressed by the following formula:

[0140]

[0141] in, Indicates the predicted output. Indicates the first The category label vector after the layer, This represents a category header, used to classify... Mapped to the task output space, Indicates the number of encoder layers.

[0142] Correspondingly, such as Figure 8 As shown, the present invention also provides a tobacco leaf appearance quality prediction system based on multimodal spatial-spectral information fusion using the above method, comprising:

[0143] The data acquisition module 11 is used to acquire RGB images, hyperspectral images, and near-infrared images of tobacco leaves;

[0144] Dimensionality reduction module 12 is used to perform dimension reduction processing on the hyperspectral image of tobacco leaves;

[0145] The data augmentation module 13 is used to perform data augmentation processing on the RGB image and the dimension-reduced hyperspectral image of the tobacco leaves;

[0146] The multimodal feature fusion module 14 is used to fuse visible light images, hyperspectral and near-infrared data of tobacco leaves using a multimodal feature extraction network architecture to obtain a fusion result. The multimodal feature extraction network architecture includes: a hyperspectral feature extraction module based on a two-dimensional convolutional neural network, a visual Transformer encoded image feature extraction module, and a spectral feature extraction module based on a one-dimensional convolutional neural network.

[0147] The classification module 15 is used to predict the classification result of the tobacco leaf appearance quality based on the fusion result.

[0148] The tobacco leaf appearance quality prediction method and system based on multimodal spatial-spectral information fusion provided in this invention introduces hyperspectral imaging technology, which simultaneously includes spatial and narrow-band spectra, into the analysis of tobacco leaf appearance quality. By acquiring spectral information in the 400-1000nm visible and near-infrared ranges, a correlation is constructed between the hyperspectral bands and various tobacco leaf appearance quality attributes. Near-infrared spectral information is used as a supplement to provide more accurate molecular description information, compensating for the loss of spectral dimensional information in hyperspectral data due to balancing acquisition efficiency and spatial dimension, thus providing data for accurate discriminant analysis. Support; a complete multi-attribute evaluation framework for tobacco leaves based on the fusion of hyperspectral imaging and high-resolution near-infrared spectroscopy is proposed. This framework systematically integrates hyperspectral data dimensionality reduction, multimodal feature extraction, and feature fusion techniques, constructing independent models for the six core quality attributes of tobacco leaves to achieve accurate evaluation of each attribute; a three-channel discrimination model based on multimodal digital image + hyperspectral + near-infrared data is built. By building a visual coding module to extract RGB image features and a convolution module to extract tobacco leaf appearance features in hyperspectral and near-infrared spectroscopy, the correlation of appearance quality is achieved through feature-level feature fusion.

[0149] The embodiments of this disclosure have now been described in detail. To avoid obscuring the concept of this disclosure, some details known in the art have not been described. Those skilled in the art can fully understand how to implement the technical solutions disclosed herein based on the above description.

[0150] While specific embodiments of this disclosure have been described in detail by way of examples, those skilled in the art should understand that the examples are for illustrative purposes only and not intended to limit the scope of this disclosure. Those skilled in the art should understand that modifications can be made to the above embodiments or equivalent substitutions can be made to some technical features without departing from the scope and spirit of this disclosure. The scope of this disclosure is defined by the appended claims.

Claims

1. A method for predicting the appearance quality of tobacco leaves based on multimodal spatial-spectral information fusion, characterized in that, include: Acquire RGB, hyperspectral, and near-infrared images of tobacco leaves; Dimensionality reduction processing of hyperspectral images of tobacco leaves; Data augmentation processing was performed on the RGB images and dimensionality-reduced hyperspectral images of tobacco leaves; Visible light images, hyperspectral and near-infrared data of tobacco leaves are fused using a multimodal feature extraction network architecture to obtain a fusion result. The multimodal feature extraction network architecture includes: a hyperspectral feature extraction module based on a two-dimensional convolutional neural network, a visual Transformer encoded image feature extraction module, and a spectral feature extraction module based on a one-dimensional convolutional neural network. The classification results of tobacco leaf appearance quality are predicted based on the fusion results.

2. The method for predicting the appearance quality of tobacco leaves based on multimodal spatial-spectral information fusion according to claim 1, characterized in that, The dimensionality reduction processing of the hyperspectral image of tobacco leaves includes: Principal component analysis was used to reduce the dimensionality of the hyperspectral images of tobacco leaves: the principal components with a cumulative variance explanation rate of 99% were retained as the optimal principal components.

3. The method for predicting the appearance quality of tobacco leaves based on multimodal spatial-spectral information fusion according to claim 1, characterized in that, The data augmentation processing of the RGB image and the dimensionality-reduced hyperspectral image of the tobacco leaves includes: By using at least one of random horizontal flipping, random vertical flipping, and random rotations of 90 degrees, 180 degrees, and 270 degrees, the sample size of each subcategory in the RGB image and the dimensionality-reduced hyperspectral image of tobacco leaves is expanded to 500, and a common random seed is set during the data augmentation process.

4. The method for predicting the appearance quality of tobacco leaves based on multimodal spatial-spectral information fusion according to claim 1, characterized in that, The method utilizes a multimodal feature extraction network architecture to fuse visible light images, hyperspectral, and near-infrared data of tobacco leaves to obtain a fusion result, including: In the hyperspectral feature extraction module based on a two-dimensional convolutional neural network, the input is a hyperspectral data cube H×W×B, and feature extraction is performed through two-dimensional convolution. In the visual Transformer encoded image feature extraction module, the input is a spatially registered RGB image, and the visual Transformer is used as the backbone network for feature extraction to capture the global spatial dependencies in the RGB image of the tobacco leaf. In the spectral feature extraction module based on a one-dimensional convolutional neural network, the input is N×P of parallel sampled near-infrared spectral data, and feature extraction is performed through one-dimensional convolution. The feature vectors extracted from the hyperspectral feature extraction module based on the two-dimensional convolutional neural network, the visual Transformer encoded image feature extraction module, and the spectral feature extraction module based on the one-dimensional convolutional neural network are concatenated, and the vector concatenation result is used as the fusion result.

5. The method for predicting the appearance quality of tobacco leaves based on multimodal spatial-spectral information fusion according to claim 4, characterized in that, In the hyperspectral feature extraction module based on a two-dimensional convolutional neural network, the input is a hyperspectral data cube H×W×B, and feature extraction is performed through two-dimensional convolution, including: The input hyperspectral data is processed through a series of two-dimensional convolutional layers. Each convolutional layer is followed by batch normalization and ReLU activation functions. Max pooling is then used to reduce the dimensionality of the data. After three convolutional layers, activation, and max pooling operations, the extracted features are then combined using average pooling. In the spectral feature extraction module based on a one-dimensional convolutional neural network, the input is N×P of parallel-sampled near-infrared spectral data, and feature extraction is performed through one-dimensional convolution, including: The input spectral data is processed through a series of one-dimensional convolutional layers. Each one-dimensional convolutional layer is followed by batch normalization and ReLU activation functions. The data is then reduced in dimensionality through max pooling. After three one-dimensional convolutional layers, activation, and max pooling operations, the extracted features are then integrated using average pooling.

6. The method for predicting the appearance quality of tobacco leaves based on multimodal spatial-spectral information fusion according to claim 4, characterized in that, In the visual Transformer encoded image feature extraction module, the input is a spatially registered RGB image. A visual Transformer is used as the backbone network for feature extraction to capture the global spatial dependencies in the RGB image of the tobacco leaf, including: The image is segmented into a series of non-overlapping image blocks, and each image block is converted into a one-dimensional sequence. The one-dimensional sequence is then globally modeled using the self-attention mechanism of the Transformer encoder.

7. The method for predicting the appearance quality of tobacco leaves based on multimodal spatial-spectral information fusion according to claim 6, characterized in that, The process of segmenting an image into a series of non-overlapping image patches, converting each patch into a one-dimensional sequence, and globally modeling the one-dimensional sequence using the self-attention mechanism of a Transformer encoder includes: Batch partitioning and encoding operations are performed using a block grid counting module, a block vector embedding mapping module, and a category-guided location injection module. Encoding is performed using a Transformer encoder that includes a residual self-attention update module and a residual feedforward reshaping module; Multi-head self-attention calculation is performed using the QKV projection generation module, the scaling correlation aggregation module, and the multi-head stitching shaping module.

8. The method for predicting the appearance quality of tobacco leaves based on multimodal spatial-spectral information fusion according to claim 7, characterized in that, The batch partitioning and encoding operations performed using the block grid counting module, the block vector embedding mapping module, and the category-guided position injection module include: The size parameters of the input RGB image are obtained through a block grid counting module, and the image is divided into grids according to a preset block side length to determine the number of image blocks required for subsequent sequence modeling. The number of image patches can be calculated using the following formula: （1） in: Indicates the number of image patches. Indicates the height of the input image. This represents the height and width of the input image. Indicates the side length of the block; The block vector embedding mapping module is used to process the first... Flattening the image blocks yields And through a linear mapping matrix Projected to dimension The feature space is obtained The mapping relationship can be represented by the following formula: ， in, For the first The initial embedding vector of each image patch, For flattened block vectors, the length is... , Represents the projection matrix. Indicates the embedding dimension. Indicates block index and The total number of blocks, To represent the number of channels; Learnable category tags are introduced through the category-guided location injection module. Mark the learnable categories The concatenation is appended to the front of the block embedding sequence, and the overlay position encoding matrix of the concatenated sequence is used. The initial sequence entering the encoder is obtained. The initial sequence is represented by the following formula. Construction method: ， in, This represents an input sequence containing category labels and position codes. Represents the category label vector, Represents a block embedding sequence. Represents the position encoding matrix, The encoding using a Transformer encoder, which includes a residual self-attention update module and a residual feedforward reshaping module, includes: The visual encoder consists of stacked Transformer modules, each layer containing a residual self-attention update module and a residual feedforward renormalization module. The residual self-attention update module is used to update the first... -1 layer output First perform layer normalization Then input the multi-head self-attention operator. The intermediate output is obtained by adding the residual from the original input. The intermediate output is obtained through the following formula. Update relationship: ， in, Indicates the first The attention sublayer output of the layer, Indicates the sequence features of the previous layer. Representation layer normalization, Indicates a multi-head self-attention operator; The output of the residual self-attention update module is processed by the residual feedforward renormalization module. Stratigraphic normalization Post-input multilayer perceptron and with The residuals are summed to obtain the layer output. It can be expressed by the following formula: ， in, Indicates the first The final output of the layer, This represents the output of the attention sublayer. Representation layer normalization, To represent a multilayer perceptron, Indicates the number of encoder layers and , The multi-head self-attention calculation using the QKV projection generation module, the scaling correlation convergence module, and the multi-head stitching and shaping module includes: The input sequence features are generated using the QKV projection generation module. Perform three sets of linear projections to obtain the query branches. Key branches The linear projection relationship of the sum branch is expressed by the following formula: ， in, Represents the features of the input sequence. , and Represents three sets of projection matrices. Generate a query matrix. Generate the key matrix. Used to characterize value branches The linear projection relationship; The scaling relevance aggregation module utilizes the inner product relevance between the query and the key to calculate attention weights, and then performs weighted aggregation on the value branches to obtain the single-head attention output, expressed by the following formula: （7）， in, This indicates single-head attention output. Represents the query matrix. Represents the key matrix. This represents the transpose of the key matrix. The scaling factor represents the dimension of the key vector. Generate normalized weights and apply them to The weighted aggregation yields the results; The multi-head concatenation and shaping module concatenates the outputs of each attention head in the feature dimension and maps them back to a unified dimension through the output projection matrix to obtain the multi-head self-attention output, which is expressed by the following formula: ， in, This indicates the output of multi-head self-attention. This indicates the output of each attention head. Indicates the number of heads. Indicates feature dimension splicing, This indicates the output projection matrix.

9. The method for predicting the appearance quality of tobacco leaves based on multimodal spatial-spectral information fusion according to claim 1, characterized in that, The classification result for predicting the appearance quality of tobacco leaves based on the fusion result includes: The fusion result is input into a multilayer perceptron classifier for processing, and the classification result of tobacco leaf appearance quality is output. Specifically, this includes: using a global label discriminant output module for classification output: the global label discriminant output module will process the fusion result... Class label vector after layer Transformer encoder As a global feature representation of the image, it is input into the multilayer perceptron classification head for prediction, expressed by the following formula: ， in, Indicates the predicted output. Indicates the first The category label vector after the layer, This represents a category header, used to classify... Mapped to the task output space, Indicates the number of encoder layers.

10. A tobacco leaf appearance quality prediction system based on multimodal spatial-spectral information fusion using the method described in any one of claims 1-9, characterized in that, include: The data acquisition module is used to acquire RGB images, hyperspectral images, and near-infrared images of tobacco leaves; The dimension reduction module is used to perform dimension reduction processing on the hyperspectral images of tobacco leaves; The data augmentation module is used to perform data augmentation processing on the RGB image and the dimension-reduced hyperspectral image of tobacco leaves; The multimodal feature fusion module is used to fuse visible light images, hyperspectral and near-infrared data of tobacco leaves using a multimodal feature extraction network architecture to obtain a fusion result. The multimodal feature extraction network architecture includes: a hyperspectral feature extraction module based on a two-dimensional convolutional neural network, a visual Transformer encoded image feature extraction module, and a spectral feature extraction module based on a one-dimensional convolutional neural network. The classification module is used to predict the classification result of the tobacco leaf appearance quality based on the fusion result.

Images