Tobacco leaf sorting and grading method and system based on multi-source feature fusion

Abstract

The invention provides a tobacco leaf sorting and grading method and system based on multi-source feature fusion, and the method comprises the steps: preprocessing the physicochemical data and original images of tobacco leaf samples, and respectively constructing a feature data sequence and an image set; extracting an image feature value sequence through a pre-trained convolutional neural network; a similarity matrix is constructed based on the feature data sequence, the most similar samples are iteratively merged, and the matrix is updated until a preset number of classification samples are obtained; fusing the feature data sequence of the classification sample and the image feature value sequence as input, and training a multi-layer perceptron network to obtain a grading model; and finally, fusing the multi-source feature input model of the to-be-graded sample to output a grading result. According to the method, the classification precision can be optimized through multi-source feature fusion and dynamic sample combination.

Description

Technical Field

[0001] The present invention relates to the technical field of tobacco leaf quality detection, and more specifically, to a tobacco leaf sorting and grading method and system based on multi-source feature fusion. Background Art

[0002] Tobacco leaf grading is a critical step in tobacco production and processing, and its accuracy directly impacts the quality and market value of tobacco products. Traditional tobacco leaf grading relies primarily on manual experience, based on observation of external features such as leaf color, leaf size, and maturity. However, manual grading has numerous limitations, including low efficiency, significant subjective influence, and difficulty ensuring consistent grading results. With the advancement of computer technology, tobacco leaf grading methods based on image processing and machine learning have gained increasing attention. These methods analyze features such as texture and color in tobacco leaf images to achieve automated grading. However, image features alone cannot fully reflect the quality characteristics of tobacco leaves, as their intrinsic physical and chemical properties are equally important. In recent years, some studies have attempted to combine physical and chemical test data of tobacco leaves with image features to improve grading accuracy. However, most of these methods simply perform weighting or concatenation of the two data sources, failing to fully exploit the deep correlations between multi-source features. Consequently, the generalization and accuracy of grading models remain to be improved.

[0003] In the process of implementing the embodiments of the present invention, the inventors found that there are at least the following problems or defects in the prior art: the existing tobacco leaf grading methods either rely on manual experience, are inefficient and greatly affected by subjective factors; or are graded based only on image features or a single dimension of physical and chemical data, which cannot fully reflect the quality of tobacco leaves; even if attempts are made to integrate multi-source data, the deep-level correlations between features cannot be effectively explored, and the grading accuracy and model generalization ability are insufficient. Summary of the Invention

[0004] The present invention provides a tobacco leaf sorting and grading method and system based on multi-source feature fusion.

[0005] In a first aspect of the present invention, a tobacco leaf sorting and grading method based on multi-source feature fusion is provided, comprising: Preprocessing the physical and chemical test data of tobacco leaf samples to construct a feature data sequence of each tobacco leaf sample; preprocessing the original tobacco leaf images to obtain an image set of each tobacco leaf sample; Inputting the image set into a pre-trained convolutional neural network to obtain a sequence of image feature values for each tobacco leaf sample; Based on the feature data sequence, a similarity matrix is constructed between samples, the two samples corresponding to the maximum element value in the similarity matrix are merged into a new sample, the feature data sequence and the image feature value sequence of the new sample are calculated, a new similarity matrix is constructed, and the two samples corresponding to the maximum element value in the new similarity matrix are merged into a new sample, and the above process is repeated until a preset number of tobacco leaf samples are obtained as classification samples; The characteristic data sequence of the classification sample and the image characteristic value sequence are fused to obtain input data, which is input into the multi-layer perceptron network for training to obtain a trained tobacco leaf grading model; The characteristic data sequence and image feature value sequence of the tobacco leaf sample to be graded are fused and input into the tobacco leaf grading model to output the grading result.

[0006] Furthermore, constructing the characteristic data sequence of each tobacco leaf sample includes: According to the preset detection indicators, multiple feature dimensions are obtained. Based on the preprocessed physical and chemical test data, the values of each tobacco leaf sample under each indicator are extracted and placed into the feature data sequence of each tobacco leaf sample in the order of the feature dimensions.

[0007] Furthermore, the preprocessing of the original tobacco leaf image includes: acquiring the corresponding original tobacco leaf image according to the multiple feature dimensions; The contrast of the three RGB channels of each original tobacco leaf image is enhanced to obtain the enhanced original tobacco leaf image; The enhanced original tobacco leaf image is normalized, and the pixel values of the three RGB channels are linearly mapped from the original dynamic range to the predefined normalized interval.

[0008] Furthermore, obtaining an image set of each tobacco leaf sample includes: In each pre-processed original tobacco leaf image, the tobacco leaf region image is cut out according to a preset size based on the center position of each tobacco leaf sample and is placed in the image set of each tobacco leaf sample.

[0009] Furthermore, the image set is input into a pre-trained convolutional neural network to obtain a sequence of image feature values of each tobacco leaf sample, including: Based on the image set, the image set of each tobacco leaf sample is used as input data, and the physical and chemical data of each tobacco leaf sample in each feature dimension is used as the feature label of the corresponding tobacco leaf image. Through the pre-trained convolutional neural network, the k-dimensional image feature value of each tobacco leaf sample in each feature dimension is output, 1≤k≤8, and the image feature value sequence of each tobacco leaf sample is obtained.

[0010] Furthermore, constructing a similarity matrix between samples based on the feature data sequence includes: Place each tobacco leaf sample into a sample collection; Based on the cosine similarity algorithm, the similarity value of the feature data sequences of any two samples in the sample set is calculated; according to the increasing function, the similarity value is added to the Euclidean distance between the feature data sequences of the two corresponding samples to obtain the similarity measure of the any two samples; All the similarity measures obtained are constructed into a symmetrical similarity matrix, whose diagonal element value is 1 and the off-diagonal element value is the similarity measure of the tobacco leaf sample corresponding to the row and column.

[0011] Furthermore, the step of merging the two samples corresponding to the maximum element value in the similarity matrix into a new sample, calculating the feature data sequence and the image feature value sequence of the new sample, and constructing a new similarity matrix includes: Select the maximum non-diagonal element value in the similarity matrix, and according to the feature data sequences of the two samples corresponding to it, perform weighted averaging on the physical and chemical data and image feature values of the same feature dimension to obtain the feature data sequence and image feature value sequence of the merged new sample; In the sample set, the two samples corresponding to the maximum element value are deleted, and the merged new sample is added to update the sample set; A new similarity matrix is constructed based on the feature data sequence of each sample in the updated sample set.

[0012] Furthermore, the feature data sequence of the classification sample and the image feature value sequence are fused to obtain input data, including: splicing the m-dimensional feature data sequence and the k-dimensional image feature value sequence of each classification sample into an m+k-dimensional fused data sequence; For the fused data sequence of each classified sample, the preset number of feature dimensions is used as the sliding window length, and the sliding window is gradually shifted from the maximum feature dimension with a step size of 1 to obtain multiple groups of fused data for each classified sample as input data.

[0013] Furthermore, the multilayer perceptron network includes: at least 2 fully connected layers, 1 batch normalization layer and 1 Softmax classification layer, wherein each fully connected layer is followed by an activation function, and the input data serves as the input of the first fully connected layer.

[0014] In a second aspect of the present invention, a tobacco leaf sorting and grading system based on multi-source feature fusion is provided, comprising: The data preprocessing module is used to preprocess the physical and chemical data of tobacco leaf samples and construct the characteristic data sequence of each sample; preprocess the original tobacco leaf images and obtain the image set of each sample; An image feature extraction module is used to input the image set into a pre-trained convolutional neural network to obtain a sequence of image feature values for each sample; The classification sample generation module is used to construct a sample similarity matrix based on the feature data sequence, merge the samples corresponding to the maximum element value in the similarity matrix into a new sample, calculate the feature data sequence and image feature value sequence of the new sample, construct a new similarity matrix, and repeat the merging until a preset number of classification samples are obtained; The grading model training module fuses the feature data sequence of the classification sample and the image feature value sequence and inputs them into the multi-layer perceptron network for training to obtain a trained tobacco grading model; The tobacco leaf grading module fuses the feature data sequence and image feature value sequence of the tobacco leaf sample to be graded, inputs them into the grading model, and outputs the grading results.

[0015] According to the above-mentioned embodiment of the present invention, there are at least the following beneficial effects: the present invention first constructs a similarity matrix between samples and gradually merges samples to generate classification samples, which can effectively reduce the number of samples, reduce the complexity of data processing, and retain key information, thereby improving the representativeness of the data. This method can better capture the intrinsic relationship between samples, provide a higher-quality data basis for subsequent grading model training, and thus improve the accuracy and generalization ability of the grading model. Secondly, the characteristic data sequence of the tobacco leaf sample and the image feature value sequence are fused and input into the multi-layer perceptron network for training, which can make full use of multi-source feature information. This fusion method can fully explore the deep correlation between physical and chemical data and image features, so that the grading model can more comprehensively evaluate the quality characteristics of tobacco leaves, and further improve the accuracy and reliability of the grading results. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The above and other objects, features and advantages of the exemplary embodiments of the present invention will become readily apparent by reading the following detailed description with reference to the accompanying drawings, in which several embodiments of the present invention are shown by way of example and not limitation, in which: Figure 1 A schematic diagram of a process for tobacco leaf sorting and grading based on multi-source feature fusion provided in one embodiment of the present invention; Figure 2 A schematic structural diagram of a tobacco sorting and grading system based on multi-source feature fusion provided by one embodiment of the present invention; Figure 3 The figure schematically shows the structure of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0017] The principles and spirit of the present invention will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided solely to enable those skilled in the art to better understand and implement the present invention, and are not intended to limit the scope of the present invention in any way. Rather, these embodiments are provided to make the present invention more thorough and complete, and to fully convey the scope of the present invention to those skilled in the art.

[0018] Those skilled in the art will appreciate that the embodiments of the present invention may be implemented as a system, apparatus, device, method, or computer program product. Therefore, the present invention may be implemented in the following forms: entirely in hardware, entirely in software (including firmware, resident software, microcode, etc.), or in a combination of hardware and software.

[0019] It should be noted that any number of elements in the drawings is for illustration only and not for limitation, and any naming is only for distinction and does not have any limiting meaning.

[0020] Reference below Figure 1 , Figure 1 This is a flow chart of a tobacco leaf sorting and grading method based on multi-source feature fusion provided by one embodiment of the present invention. Figure 1 As shown, a tobacco leaf sorting and grading method based on multi-source feature fusion includes: S1 preprocesses the physical and chemical test data of tobacco leaf samples to construct a feature data sequence of each tobacco leaf sample; preprocesses the original tobacco leaf image to obtain an image set of each tobacco leaf sample; S2 inputs the image set into a pre-trained convolutional neural network to obtain a sequence of image feature values for each tobacco leaf sample; S3 constructs a similarity matrix between samples based on the feature data sequence, merges the two samples corresponding to the maximum element value in the similarity matrix into a new sample, calculates the feature data sequence and image feature value sequence of the new sample, constructs a new similarity matrix, and merges the two samples corresponding to the maximum element value in the new similarity matrix into a new sample, repeating the above process until a preset number of tobacco leaf samples are obtained as classification samples; S4 fuses the feature data sequence of the classification sample and the image feature value sequence to obtain input data, which is input into the multi-layer perceptron network for training to obtain a trained tobacco leaf grading model; S5 fuses the feature data sequence and image feature value sequence of the tobacco leaf sample to be graded and inputs them into the tobacco leaf grading model to output the grading result.

[0021] It should be noted that the present invention relates to a tobacco leaf sorting and grading method based on multi-source feature fusion, which aims to achieve accurate grading by integrating the physical and chemical test data and image features of tobacco leaf samples. Physical and chemical test data refers to tobacco leaf indicators measured by chemical and physical methods, such as moisture content, nicotine content, etc. These data are constructed into feature data sequences after preprocessing. Image features are extracted from tobacco leaf images through convolutional neural networks. These images are preprocessed, including contrast enhancement and normalization, to highlight the appearance features of the tobacco leaves, such as color and texture. By constructing a similarity matrix between samples and gradually merging samples, classified samples are finally generated. After the feature data sequences and image feature value sequences of these classified samples are fused, they are input into a multi-layer perceptron network for training to obtain a tobacco leaf grading model. The tobacco leaf samples to be graded are input into the model after the same processing, and the grading results are output.

[0022] Specifically, when preprocessing the physical and chemical test data of tobacco leaf samples, a normalization method is used to map the data to a range of 0 to 1, eliminating dimensional differences between different indicators. When constructing a feature data sequence, the numerical values of each tobacco leaf sample are extracted based on preset test indicators, such as moisture content and nicotine content, and are sequentially placed into the feature data sequence. When preprocessing the raw tobacco leaf images, contrast enhancement is performed on each of the three RGB channels of each image, followed by normalization, mapping the pixel values to a predefined range. When acquiring an image set, the center position of the tobacco leaf sample is used as a reference, and the tobacco leaf region image is cropped to a preset size and placed into the image set. After input into a convolutional neural network, the k-dimensional image feature values of each tobacco leaf sample in each feature dimension are output, resulting in an image feature value sequence. When constructing a similarity matrix between samples, each tobacco leaf sample is placed into the sample set, and the similarity value between any two samples is calculated using the cosine similarity algorithm. The similarity measure is then combined with the Euclidean distance to construct a symmetric similarity matrix. When merging samples, select the two samples corresponding to the maximum element value in the similarity matrix. The physical and chemical data and image feature values of the same feature dimensions are weighted averaged to obtain the feature data sequence and image feature value sequence of the new sample. The sample set is updated and a new similarity matrix is constructed. When fusing the feature data sequence and image feature value sequence of the classified samples, the m-dimensional feature data sequence and the k-dimensional image feature value sequence are concatenated into an m+k-dimensional fused data sequence. Using the preset number of feature dimensions as the sliding window length, multiple sets of fused data are gradually shifted to serve as the input data. The multilayer perceptron network includes at least two fully connected layers, one batch normalization layer, and one softmax classification layer. The input data is used as the input to the first fully connected layer.

[0023] Preferably, the Z-score normalization method can be used when preprocessing physical and chemical test data, and the difference between each data point and the mean is calculated and divided by the standard deviation to obtain a normalized data value. Contrast enhancement can use a histogram equalization method, and the normalization process can map the pixel value to a range of 0 to 255. When inputting a convolutional neural network, a pre-trained model such as AlexNet or ResNet can be used, and the specific dimension of the k-dimensional image eigenvalue can be set according to the model output layer, for example, 8 dimensions. When constructing a similarity matrix, the increasing function can use a linear function, and the calculation of the similarity measure can consider the importance of different indicators and assign different weights. When merging samples, the weight of the weighted average can be adjusted according to the reliability of the sample or the amount of data. When fusing the feature data sequence and the image feature value sequence, the sliding window length can be set according to the length and complexity of the feature data sequence, for example, 5. The number of neurons in the fully connected layer of the multilayer perceptron network can be adjusted according to the dimension of the input data and the complexity of the grading task, for example, 128 neurons. The activation function can use the ReLU function, and the number of output categories of the Softmax classification layer is determined according to the number of tobacco leaf grading levels, for example, 5 levels.

[0024] In some embodiments, constructing a characteristic data sequence for each tobacco leaf sample includes: According to the preset detection indicators, multiple feature dimensions are obtained. Based on the preprocessed physical and chemical test data, the values of each tobacco leaf sample under each indicator are extracted and placed into the feature data sequence of each tobacco leaf sample in the order of the feature dimensions.

[0025] It should be noted that constructing a feature data sequence for each tobacco leaf sample involves converting the physical and chemical test data of the tobacco leaf samples into a structured data format suitable for subsequent processing. Physical and chemical test data refers to various indicators of tobacco leaf samples obtained through chemical analysis and physical measurements, such as moisture content, nicotine content, and sugar content. This data reflects the inherent quality characteristics of the tobacco leaves. The feature data sequence arranges this physical and chemical test data in a specific order to form a vector for subsequent processing. In this way, complex tobacco leaf sample characteristics can be quantified in numerical form, providing basic data for subsequent similarity calculations and model training.

[0026] Specifically, when constructing a feature data sequence, the system first divides the data into multiple feature dimensions based on the preset detection indicators. These detection indicators are determined in advance based on the actual needs of tobacco leaf grading, such as moisture content, nicotine content, sugar content, etc. Based on the preprocessed physical and chemical test data, the numerical values of each tobacco leaf sample under each indicator are extracted. The preprocessing process includes operations such as data cleaning and normalization to ensure the quality and consistency of the data. Then, these values are placed in the feature data sequence of each tobacco leaf sample in the order of the feature dimensions. For example, if there are three preset detection indicators, namely moisture content, nicotine content, and sugar content, then the feature data sequence of each tobacco leaf sample will be a vector containing three numerical values, corresponding to the numerical values of these three indicators. Such a feature data sequence can clearly represent the characteristics of each tobacco leaf sample under different indicators, providing convenience for subsequent processing.

[0027] Preferably, when constructing the feature data sequence, the values of each indicator can be normalized, mapping the values to the range of 0 to 1 to eliminate dimensional differences between different indicators. A typical normalization method is maximum-minimum normalization, which subtracts the minimum value of each indicator from the value of that indicator, and then divides the result by the difference between the maximum and minimum values of that indicator. This ensures that the values of all indicators are within the same range, facilitating subsequent similarity calculations.

[0028] Furthermore, different indicators can be assigned different weights based on actual needs to reflect their importance in tobacco leaf grading. For example, if moisture content has a significant impact on tobacco leaf quality, a higher weight could be assigned to moisture content. Weightings can be determined based on expert experience or through data-driven approaches, such as analyzing historical data to determine the degree of impact of each indicator on tobacco leaf quality. This approach can more accurately reflect the characteristics of tobacco leaf samples, improving the accuracy and reliability of subsequent processing.

[0029] In some embodiments, the preprocessing of the original tobacco leaf image includes: acquiring a corresponding original tobacco leaf image according to the multiple feature dimensions; The contrast of the three RGB channels of each original tobacco leaf image is enhanced to obtain the enhanced original tobacco leaf image; The enhanced original tobacco leaf image is normalized, and the pixel values of the three RGB channels are linearly mapped from the original dynamic range to the predefined normalized interval.

[0030] It should be noted that the purpose of preprocessing the original tobacco leaf image is to enhance the image's feature information, making it more suitable for subsequent feature extraction and analysis. Specifically, this process involves contrast enhancement of the three RGB channels of the original tobacco leaf image and normalization of the enhanced image. Contrast enhancement is intended to highlight details and features in the image, making the color and texture of the tobacco leaves in the image clearer. Normalization linearly maps the image's pixel values from the original dynamic range to a predefined normalization interval. This is typically done to eliminate the effects of differences in lighting conditions on the image while providing a unified data format for subsequent image processing and analysis.

[0031] Specifically, the original tobacco leaf image refers to the image obtained directly from the tobacco leaf sample. These images contain the appearance information of the tobacco leaf, such as color, texture, and shape. The three RGB channels refer to the red, green, and blue channels of the image, respectively. Each channel contains the pixel value information of the image under the corresponding color spectrum. Contrast enhancement can be achieved through a variety of methods, such as histogram equalization, which is a commonly used image processing technique that enhances the contrast of the image by adjusting the pixel value distribution of the image. Normalization processing usually maps pixel values from the original range, such as 0 to 255, to a new interval, such as 0 to 1. This ensures that the pixel values between different images are comparable, and also facilitates subsequent calculations and processing.

[0032] Preferably, during the contrast enhancement process, an adaptive histogram equalization (CLAHE) method can be used. This method can better process local details in the image and avoid image distortion caused by over-enhancement. In the normalization process, the pixel value can be mapped to the range of 0 to 1 by subtracting the minimum value of the image from each pixel value and then dividing it by the difference between the maximum and minimum values of the image. In addition, to further optimize the image processing effect, the image can be denoised before normalization, for example, using a Gaussian filter to remove random noise in the image, thereby improving the image quality and the accuracy of subsequent feature extraction.

[0033] In some embodiments, obtaining an image set of each tobacco leaf sample includes: In each pre-processed original tobacco leaf image, the tobacco leaf region image is cut out according to a preset size based on the center position of each tobacco leaf sample and is placed in the image set of each tobacco leaf sample.

[0034] It should be noted that the purpose of acquiring image collections of each tobacco leaf sample is to extract regional images relevant to the tobacco leaf sample from the preprocessed raw tobacco leaf images for subsequent feature extraction and analysis. This process is achieved by cropping the central region of the tobacco leaf sample in the image, ensuring that the extracted image region accurately reflects the appearance characteristics of the tobacco leaf sample while minimizing background interference. Image collections are the collection of images of multiple tobacco leaf samples into a single dataset for subsequent processing and analysis.

[0035] Specifically, the pre-processed original tobacco leaf image refers to the image after contrast enhancement and normalization processing, which has enhanced the color and texture characteristics of the tobacco leaves and eliminated the influence of lighting conditions. Taking the center position of each tobacco leaf sample as the reference refers to locating the center point of the tobacco leaf sample in the image, which can usually be determined by image segmentation or manual annotation. Cropping the tobacco leaf area image according to the preset size means cropping the area containing the tobacco leaf sample from the image according to the preset size. The preset size can be adjusted according to the actual size and shape of the tobacco leaf sample, for example, it can be set to 100×100 pixels or 200×200 pixels, etc., to ensure that the cropped image area can completely contain the tobacco leaf sample. The purpose of placing the cropped image into the image set of each tobacco leaf sample is to form a data set containing multiple tobacco leaf sample images, which is convenient for subsequent batch processing and analysis.

[0036] Preferably, in order to more accurately locate the center position of the tobacco leaf sample, an image segmentation algorithm, such as a threshold-based segmentation or edge detection algorithm, can be used to automatically identify the outline of the tobacco leaf sample and determine its center point. The preset size can be dynamically adjusted according to the actual size and shape of the tobacco leaf sample, for example, by analyzing the aspect ratio of the tobacco leaf sample in the image to determine the size of the intercepted area. In addition, to further improve the quality of the image collection, the image can be cropped or padded after interception to ensure that all images are of the same size to facilitate subsequent processing. For example, if the intercepted image area is smaller than the preset size, the image size can be adjusted by padding with background pixels; if it is larger than the preset size, it can be cropped.

[0037] In some embodiments, inputting the image set into a pre-trained convolutional neural network to obtain a sequence of image feature values for each tobacco leaf sample includes: Based on the image set, the image set of each tobacco leaf sample is used as input data, and the physical and chemical data of each tobacco leaf sample in each feature dimension is used as the feature label of the corresponding tobacco leaf image. Through the pre-trained convolutional neural network, the k-dimensional image feature value of each tobacco leaf sample in each feature dimension is output, 1≤k≤8, and the image feature value sequence of each tobacco leaf sample is obtained.

[0038] It should be noted that the purpose of inputting the image set into a pre-trained convolutional neural network is to extract a sequence of feature values from the images that is useful for tobacco leaf grading. A convolutional neural network (CNN) is a deep learning model that can automatically learn features in an image, such as edges, texture, and shape. The image feature value sequence refers to the feature vectors output by the convolutional neural network. These feature vectors can reflect the intrinsic characteristics of the image and provide a basis for subsequent grading. By using the image set of each tobacco leaf sample as input data and the physical and chemical data of each tobacco leaf sample in each feature dimension as the feature label of the corresponding tobacco leaf image, the convolutional neural network can output k-dimensional image feature values for each tobacco leaf sample in each feature dimension, thereby obtaining a sequence of image feature values for each tobacco leaf sample.

[0039] Specifically, an image set refers to a collection of preprocessed tobacco leaf sample images, which contain appearance information such as the color and texture of the tobacco leaves. A pretrained convolutional neural network refers to a pretrained model, typically trained on a large-scale image dataset to learn common image features. Using a set of images of each tobacco leaf sample as input data means inputting these images into a convolutional neural network for processing. Using the physical and chemical data of each tobacco leaf sample in each feature dimension as the feature label of the corresponding tobacco leaf image means associating the physical and chemical test data of the tobacco leaf sample with the image features so that the convolutional neural network can learn the relationship between the image features and the physical and chemical data. The k-dimensional image feature value output by the convolutional neural network refers to the dimension of the feature vector output by the network. The value of k can be set according to actual needs, such as 8 or 16 dimensions. The image feature value sequence arranges these feature vectors in sample order to form a sequence for subsequent processing and analysis.

[0040] Preferably, when constructing a convolutional neural network, a commonly used architecture such as AlexNet or ResNet can be selected. Taking AlexNet as an example, it contains multiple convolutional layers and pooling layers for extracting local features of the image, and fully connected layers for learning the global relationship between features. The input parameters include an image set and corresponding physical and chemical data labels. During the training process, the backpropagation algorithm can be used to adjust the network weights to minimize the difference between the predicted feature values and the actual physical and chemical data labels. In order to improve the generalization ability of the model, data enhancement techniques such as random cropping, rotation, and flipping of images can be used during training. In addition, hyperparameters such as learning rate and batch size can be set, for example, the learning rate is set to 0.001 and the batch size is set to 32. When outputting the image feature values, the feature vector can be normalized to have a uniform scale to facilitate subsequent similarity calculation and model training.

[0041] In some embodiments, constructing a similarity matrix between samples based on the feature data sequence includes: Place each tobacco leaf sample into a sample collection; Based on the cosine similarity algorithm, the similarity value of the feature data sequences of any two samples in the sample set is calculated; according to the increasing function, the similarity value is added to the Euclidean distance between the feature data sequences of the two corresponding samples to obtain the similarity measure of the any two samples; All the similarity measures obtained are constructed into a symmetrical similarity matrix, whose diagonal element value is 1 and the off-diagonal element value is the similarity measure of the tobacco leaf sample corresponding to the row and column.

[0042] It's important to note that the purpose of constructing a similarity matrix between samples is to quantify their similarity so that similar samples can be merged through clustering, reducing the number of samples while preserving key information. The similarity matrix is a symmetric matrix whose diagonal elements are 1, indicating the similarity between a sample and itself, and whose off-diagonal elements represent the similarity measure between different samples. By calculating the similarity between sample feature data sequences and combining it with the Euclidean distance, a comprehensive similarity measure can be derived for subsequent sample merging operations.

[0043] Specifically, the sample set refers to the set containing the characteristic data sequences of all tobacco leaf samples. The cosine similarity algorithm is a commonly used similarity calculation method, which measures the similarity between two vectors by calculating the cosine value of the angle between them. Specifically, the closer the cosine similarity value between the characteristic data sequences of two samples is to 1, the more similar they are. The increasing function is used to add the similarity value to the Euclidean distance between the corresponding characteristic data sequences of the two samples to obtain a comprehensive similarity measure. The Euclidean distance is the straight-line distance between two vectors in space and is used to measure the difference between samples. In this way, a symmetric similarity matrix can be obtained, in which each element represents the similarity measure between two samples.

[0044] Preferably, when constructing the similarity matrix, normalized cosine similarity can be used, that is, the cosine similarity value is adjusted to the range of 0 to 1 so that it can be uniformly compared with other similarity measures. For the increasing function, a linear function can be selected, such as adding the cosine similarity value to the inverse of the Euclidean distance, so that the similarities and differences between samples can be considered at the same time. When calculating the Euclidean distance, the feature data sequence can be standardized to eliminate the dimensional differences between different feature dimensions. In addition, in order to improve computational efficiency, matrix operations can be used to simultaneously calculate the similarity values between all sample pairs instead of calculating them pair by pair. This can significantly reduce computing time and complexity, especially when processing large-scale sample sets.

[0045] In some embodiments, merging the two samples corresponding to the maximum element value in the similarity matrix into a new sample, calculating the feature data sequence and image feature value sequence of the new sample, and constructing a new similarity matrix includes: Select the maximum non-diagonal element value in the similarity matrix, and according to the feature data sequences of the two samples corresponding to it, perform weighted averaging on the physical and chemical data and image feature values of the same feature dimension to obtain the feature data sequence and image feature value sequence of the merged new sample; In the sample set, the two samples corresponding to the maximum element value are deleted, and the merged new sample is added to update the sample set; A new similarity matrix is constructed based on the feature data sequence of each sample in the updated sample set.

[0046] It's important to note that the process of merging the two samples corresponding to the maximum element value in the similarity matrix into a new sample is intended to gradually reduce the number of samples through clustering while preserving key information. This process involves taking a weighted average of the sample's feature data sequence and the image feature value sequence to generate new sample features. The sample set and similarity matrix are updated to reflect the changes after the sample merge, ensuring that subsequent operations are based on the most recent sample set. By repeating this process, a preset number of classification samples are ultimately obtained, which will be used for subsequent classification model training.

[0047] Specifically, the maximum element value in the similarity matrix represents the two samples with the highest similarity in the sample set. The feature data sequence and image feature value sequence of these two samples are the basis for merging. Weighted averaging means that when calculating the feature data and image feature values of the new sample, different weights are assigned to the corresponding data of the original sample. The size of the weight can be determined based on the reliability of the sample or other factors. The merged new sample replaces the original two samples and is added to the updated sample set. The new similarity matrix is reconstructed based on the updated sample set to reflect the similarity relationship after the sample is merged. This process is repeated until a preset number of classified samples are obtained. These classified samples will be used for subsequent hierarchical model training.

[0048] Preferably, the setting of weights can be determined based on the variance of the feature data sequence of the sample. Samples with smaller variances have more stable features and can therefore be given higher weights. When calculating the feature data sequence and image feature value sequence of a new sample, a weighted average can be performed on each feature dimension to ensure the representativeness of the new sample in each feature dimension. When updating the sample set, it is necessary to remove the two merged samples from the set and add the newly generated sample. When constructing a new similarity matrix, the similarity between all sample pairs can be recalculated, or only the similarity involving new samples can be calculated to improve computational efficiency. In addition, in order to further optimize the merging process, the feature data sequence of the new sample can be normalized after each merge to ensure that the feature data of all samples are on the same scale, which is convenient for subsequent similarity calculations and model training.

[0049] In some embodiments, the feature data sequence of the classified samples and the image feature value sequence are fused to obtain input data, including: splicing the m-dimensional feature data sequence and the k-dimensional image feature value sequence of each classified sample into an m+k-dimensional fused data sequence; For the fused data sequence of each classified sample, the preset number of feature dimensions is used as the sliding window length, and the sliding window is gradually shifted from the maximum feature dimension with a step size of 1 to obtain multiple groups of fused data for each classified sample as input data.

[0050] It should be noted that the purpose of fusing the feature data sequence of the classified samples with the image feature value sequence is to generate a comprehensive feature representation for subsequent grading model training. This fusion approach fully leverages the advantages of multi-source data, enabling the grading model to more comprehensively assess tobacco leaf quality characteristics. The fused data serves as input to a multi-layer perceptron network for training, resulting in a model capable of accurate grading. This process is a key step in achieving precise grading. By integrating feature information from different sources, the accuracy and generalization of the grading model can be improved.

[0051] Specifically, the feature data sequence refers to the vector representation of the physical and chemical test data of tobacco leaf samples obtained through preprocessing and clustering, while the image feature value sequence refers to the vector representation of the image features extracted by the convolutional neural network. The two sequences are fused by concatenating them into a higher-dimensional vector. For example, if the feature data sequence is m-dimensional and the image feature value sequence is k-dimensional, the fused data sequence will be m+k-dimensional. The sliding window length is a parameter operating on the fused data sequence, which determines the length of each data segment processed. A step size of 1 means that the sliding window moves one unit each time, starting from the maximum feature dimension. This step shift means starting from the highest dimension of the fused data sequence and gradually moving towards lower dimensions, sequentially obtaining multiple sets of fused data for each classification sample. This fused data is used as input data for training the multilayer perceptron network.

[0052] Preferably, the sliding window length can be adjusted according to the length and complexity of the fused data sequence. For example, if the fused data sequence is long, a larger sliding window length can be set to reduce the dimension of the data and improve computational efficiency. Before inputting into the multi-layer perceptron network, the fused data can be standardized to ensure that data of different feature dimensions have the same scale. The construction of the multi-layer perceptron network can include multiple fully connected layers, each followed by an activation function, such as ReLU, to introduce nonlinearity. The batch normalization layer can be used to accelerate the training process and improve the stability of the model. The Softmax classification layer is used to output the grading results, and the number of its output categories is determined according to the number of grades of tobacco leaf grading. During the training process, the cross entropy loss function can be used to optimize the model parameters, and the performance of the model can be improved by adjusting hyperparameters such as the learning rate and batch size.

[0053] In some embodiments, the multilayer perceptron network includes: at least 2 fully connected layers, 1 batch normalization layer and 1 Softmax classification layer, wherein each fully connected layer is followed by an activation function, and the input data serves as the input of the first fully connected layer.

[0054] It should be noted that the Multilayer Perceptron (MLP) network is a feedforward neural network used to classify input data. It consists of multiple fully connected layers, a batch normalization layer, and a softmax classification layer. The fully connected layers learn the mapping between input features and output labels, the batch normalization layer accelerates training and improves model stability, and the softmax classification layer converts the output into a probability distribution for classification. This network structure effectively processes fused feature data and learns complex feature relationships, thereby achieving accurate grading of tobacco leaf samples.

[0055] Specifically, a fully connected layer in a multilayer perceptron network means that each neuron is connected to all neurons in the previous layer, which is used to extract high-level features of the input data. At least two fully connected layers means that the network contains at least two layers of neurons for feature extraction and transformation. The batch normalization layer is a normalization technique used to adjust the distribution of each small batch of data, thereby reducing internal covariate shift, accelerating the training process, and improving the generalization ability of the model. The softmax classification layer is a commonly used classification layer that converts the input feature vector into a probability distribution, and each output value represents the probability that the sample belongs to a certain category. The input data refers to the feature data sequence after fusion processing. This data serves as the input of the first fully connected layer and is processed layer by layer by the network to ultimately output the classification result.

[0056] Preferably, when constructing a multi-layer perceptron network, the number of neurons in the first fully connected layer can be set to 128, and the number of neurons in the second fully connected layer can be set to 64. These parameters can be adjusted according to the dimension of the input data and the complexity of the grading task. Each fully connected layer is followed by an activation function ReLU, which is used to introduce nonlinearity so that the network can learn complex feature relationships. The batch normalization layer can be placed after each fully connected layer to normalize each small batch of data. The number of output categories of the Softmax classification layer is determined according to the number of levels of tobacco leaf grading, for example, 5 levels. During the training process, the cross-entropy loss function can be used to optimize the model parameters, and the performance of the model can be improved by adjusting hyperparameters such as the learning rate and batch size. For example, the learning rate can be set to 0.001 and the batch size can be set to 32. In addition, to prevent overfitting, a Dropout layer can be added to the network to randomly discard the output of a part of the neurons, thereby improving the generalization ability of the model.

[0057] The above-described embodiments of the present invention have the following beneficial effects: This method can improve the accuracy and efficiency of tobacco leaf grading. By integrating physical and chemical test data with image features, a multi-dimensional feature representation is constructed, fully leveraging the physical and chemical properties and visual features of tobacco leaves for comprehensive judgment. An iterative sample merging strategy based on a similarity matrix can optimize sample distribution, reduce the interference of redundant data on model training, and retain the most representative sample features, providing a higher-quality data foundation for subsequent model training.

[0058] Image features extracted through convolutional neural networks can capture deep visual information about tobacco leaves. Combined with sliding window feature fusion methods, this method enhances data representation and enables the model to learn richer feature combinations. Batch normalization and softmax classification layers within a multi-layer perceptron network improve model stability and classification performance, ultimately producing more reliable grading results. This method can adapt to the grading needs of tobacco leaves from different production areas and varieties, demonstrating strong generalization and practical application value.

[0059] like Figure 2 As shown, some embodiments provide a tobacco leaf sorting and grading system based on multi-source feature fusion, the system comprising: The data preprocessing module 201 is used to preprocess the physical and chemical data of tobacco leaf samples to construct a characteristic data sequence of each sample; preprocess the original tobacco leaf images to obtain an image set of each sample; An image feature extraction module 202 is configured to input the image set into a pre-trained convolutional neural network to obtain a sequence of image feature values for each sample; The classification sample generation module 203 is used to construct a sample similarity matrix based on the feature data sequence, merge the samples corresponding to the maximum element value in the similarity matrix into a new sample, calculate the feature data sequence and image feature value sequence of the new sample, construct a new similarity matrix, and repeat the merging until a preset number of classification samples are obtained; The classification model training module 204 fuses the feature data sequence of the classification sample and the image feature value sequence and inputs them into the multi-layer perceptron network for training to obtain a trained tobacco grading model; The tobacco leaf grading module 205 fuses the feature data sequence and the image feature value sequence of the tobacco leaf sample to be graded, inputs the result into the grading model, and outputs the grading result.

[0060] It is understandable that the modules recorded in the tobacco sorting and grading system based on multi-source feature fusion are similar to those in the reference Figure 1 The steps in the tobacco sorting and grading method based on multi-source feature fusion described above correspond to each other. Therefore, the operations, features, and beneficial effects described above for the tobacco sorting and grading method based on multi-source feature fusion are also applicable to the tobacco sorting and grading system based on multi-source feature fusion and the modules contained therein, and will not be repeated here.

[0061] Reference below Figure 3 , which shows a schematic structural diagram of an electronic device 300 suitable for implementing some embodiments of the present invention. The electronic devices in some embodiments of the present invention may include, but are not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 3 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0062] like Figure 3As shown, electronic device 300 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes based on programs stored in a read-only memory (ROM) 302 or programs loaded from a storage device 308 into a random access memory (RAM) 303. RAM 303 also stores various programs and data required for the operation of electronic device 300. Processing device 301, ROM 302, and RAM 303 are interconnected via a bus 304. An input / output (I / O) interface 305 is also connected to bus 304.

[0063] Typically, the following devices may be connected to the I / O interface 305: an input device 306 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 308 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 309. The communication device 309 may allow the electronic device 300 to communicate with other devices wirelessly or by wire to exchange data. Figure 3 The electronic device 300 is shown with various devices, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead. Figure 3 Each block shown in the figure may represent one device, or may represent multiple devices as needed.

[0064] Furthermore, the storage medium of the embodiment of the present application stores program instructions that can implement all the above methods, wherein the program instructions can be stored in the above storage medium in the form of a software product, including a number of instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, or a terminal device such as a computer, server, mobile phone, or tablet.

[0065] The above descriptions merely illustrate some preferred embodiments of the present invention and the underlying technical principles. Those skilled in the art should understand that the scope of the invention encompassed by the embodiments of the present invention is not limited to technical solutions formed by specific combinations of the aforementioned technical features. It also encompasses other technical solutions formed by any combination of the aforementioned technical features or their equivalents, without departing from the aforementioned inventive concept. For example, a technical solution formed by replacing the aforementioned features with (but not limited to) technical features with similar functions disclosed in the embodiments of the present invention.

Claims

1. A tobacco leaf sorting and grading method based on multi-source feature fusion, characterized in that: The following steps are involved: Preprocessing the physical and chemical test data of tobacco leaf samples to construct the characteristic data sequence of each tobacco leaf sample; Preprocessing the original tobacco leaf image to obtain an image set of each tobacco leaf sample; Inputting the image set into a pre-trained convolutional neural network to obtain a sequence of image feature values for each tobacco leaf sample; Based on the feature data sequence, a similarity matrix is constructed between samples, the two samples corresponding to the maximum element value in the similarity matrix are merged into a new sample, the feature data sequence and the image feature value sequence of the new sample are calculated, a new similarity matrix is constructed, and the two samples corresponding to the maximum element value in the new similarity matrix are merged into a new sample, and the above process is repeated until a preset number of tobacco leaf samples are obtained as classification samples; The characteristic data sequence of the classification sample and the image characteristic value sequence are fused to obtain input data, which is input into the multi-layer perceptron network for training to obtain a trained tobacco leaf grading model; The characteristic data sequence and image feature value sequence of the tobacco leaf sample to be graded are fused and input into the tobacco leaf grading model to output the grading result.

2. The tobacco leaf sorting and grading method based on multi-source feature fusion according to claim 1, characterized in that: The step of constructing a characteristic data sequence for each tobacco leaf sample includes: According to the preset detection indicators, multiple feature dimensions are obtained. Based on the preprocessed physical and chemical test data, the values of each tobacco leaf sample under each indicator are extracted and placed into the feature data sequence of each tobacco leaf sample in the order of the feature dimensions.

3. The tobacco leaf sorting and grading method based on multi-source feature fusion according to claim 2, characterized in that: The preprocessing of the original tobacco leaf image comprises: acquiring the corresponding original tobacco leaf image according to the multiple feature dimensions; The contrast of the three RGB channels of each original tobacco leaf image is enhanced to obtain the enhanced original tobacco leaf image; The enhanced original tobacco leaf image is normalized, and the pixel values of the three RGB channels are linearly mapped from the original dynamic range to the predefined normalized interval.

4. The tobacco leaf sorting and grading method based on multi-source feature fusion according to claim 3 is characterized in that: The step of obtaining an image set of each tobacco leaf sample comprises: In each pre-processed original tobacco leaf image, the tobacco leaf region image is cut out according to a preset size based on the center position of each tobacco leaf sample and is placed in the image set of each tobacco leaf sample.

5. The tobacco leaf sorting and grading method based on multi-source feature fusion according to claim 4 is characterized in that: The image set is input into a pre-trained convolutional neural network to obtain a sequence of image feature values of each tobacco leaf sample, including: Based on the image set, the image set of each tobacco leaf sample is used as input data, and the physical and chemical data of each tobacco leaf sample in each feature dimension is used as the feature label of the corresponding tobacco leaf image. Through the pre-trained convolutional neural network, the k-dimensional image feature value of each tobacco leaf sample in each feature dimension is output, 1≤k≤8, and the image feature value sequence of each tobacco leaf sample is obtained.

6. The tobacco leaf sorting and grading method based on multi-source feature fusion according to claim 2, characterized in that: The process of constructing a similarity matrix between samples based on the feature data sequence includes: Place each tobacco leaf sample into a sample collection; Based on the cosine similarity algorithm, the similarity value of the feature data sequences of any two samples in the sample set is calculated; according to the increasing function, the similarity value is added to the Euclidean distance between the feature data sequences of the two corresponding samples to obtain the similarity measure of the any two samples; All the similarity measures obtained are constructed into a symmetrical similarity matrix, whose diagonal element value is 1 and the off-diagonal element value is the similarity measure of the tobacco leaf sample corresponding to the row and column.

7. The tobacco leaf sorting and grading method based on multi-source feature fusion according to claim 6, characterized in that: The method of merging the two samples corresponding to the maximum element value in the similarity matrix into a new sample, calculating the feature data sequence and the image feature value sequence of the new sample, and constructing a new similarity matrix includes: Select the maximum non-diagonal element value in the similarity matrix, and according to the feature data sequences of the two samples corresponding to it, perform weighted averaging on the physical and chemical data and image feature values of the same feature dimension to obtain the feature data sequence and image feature value sequence of the merged new sample; In the sample set, the two samples corresponding to the maximum element value are deleted, and the merged new sample is added to update the sample set; A new similarity matrix is constructed based on the feature data sequence of each sample in the updated sample set.

8. The tobacco leaf sorting and grading method based on multi-source feature fusion according to claim 5 or 7, characterized in that: Fusing the feature data sequence of the classification sample and the image feature value sequence to obtain input data, including: splicing the m-dimensional feature data sequence and the k-dimensional image feature value sequence of each classification sample into an m+k-dimensional fused data sequence; For the fused data sequence of each classified sample, the preset number of feature dimensions is used as the sliding window length, and the sliding window is gradually shifted from the maximum feature dimension with a step size of 1 to obtain multiple groups of fused data for each classified sample as input data.

9. The tobacco leaf sorting and grading method based on multi-source feature fusion according to claim 8, characterized in that: The multi-layer perceptron network includes: at least 2 fully connected layers, 1 batch normalization layer and 1 Softmax classification layer, wherein each fully connected layer is followed by an activation function, and the input data is used as the input of the first fully connected layer.

10. A tobacco leaf sorting and grading system based on multi-source feature fusion, characterized in that: include: Data preprocessing module, used to preprocess the physical and chemical data of tobacco leaf samples and construct characteristic data sequences of each sample; Preprocess the original tobacco leaf images to obtain image sets of each sample; An image feature extraction module is used to input the image set into a pre-trained convolutional neural network to obtain a sequence of image feature values for each sample; The classification sample generation module is used to construct a sample similarity matrix based on the feature data sequence, merge the samples corresponding to the maximum element value in the similarity matrix into a new sample, calculate the feature data sequence and image feature value sequence of the new sample, construct a new similarity matrix, and repeat the merging until a preset number of classification samples are obtained; The grading model training module fuses the feature data sequence of the classification sample and the image feature value sequence and inputs them into the multi-layer perceptron network for training to obtain a trained tobacco grading model; The tobacco leaf grading module fuses the feature data sequence and image feature value sequence of the tobacco leaf sample to be graded, inputs them into the grading model, and outputs the grading results.

Images