Fresh tobacco leaf maturity image evaluation method and system

Abstract

The invention belongs to the technical field of fresh tobacco leaf evaluation, and particularly relates to a fresh tobacco leaf maturity image evaluation method and system, and the method comprises the following steps: S1, obtaining fresh tobacco leaf image data which comprises multispectral image data and three-dimensional structure information data; s2, preprocessing the obtained fresh tobacco leaf image data, and constructing a fresh tobacco leaf image sample library; s3, constructing a maturity evaluation model, wherein the maturity evaluation model comprises a multi-modal feature extraction layer, a space-time attention fusion layer and a multi-task prediction layer; s4, adopting the fresh tobacco leaf image sample library to train the maturity evaluation model; s5, performing maturity evaluation on the to-be-evaluated fresh tobacco leaf image by adopting the trained maturity evaluation model, and generating an evaluation result; according to the method, the accuracy, reliability and practicability of tobacco maturity evaluation are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of fresh tobacco leaf evaluation, and in particular relates to a fresh tobacco leaf maturity image evaluation method and system. Background Art

[0002] In modern agricultural production, tobacco leaves are an important cash crop, and quality control is crucial to the development of the tobacco industry. The maturity of fresh tobacco leaves, in particular, directly impacts subsequent processing quality, economic benefits, and the quality stability of the final product. Therefore, accurate and real-time monitoring of fresh tobacco leaf maturity has become a critical issue that needs to be addressed within the industry.

[0003] Currently, tobacco leaf maturity assessment primarily relies on traditional manual judgment. This method relies on professionals visually observing the leaves' color, shape, and other visual characteristics for qualitative evaluation. Its advantages lie in the lack of complex equipment, relatively low cost, and simple operation, making it valuable for small-scale growers or in preliminary screening scenarios. However, this manual approach to maturity assessment has significant drawbacks. Firstly, due to differences in evaluators' experience, visual perception, and subjective judgment criteria, the assessment results are highly subjective, making it difficult to ensure consistency and accuracy. Secondly, manual assessment is inefficient, especially in large-scale cultivation environments, where massive amounts of tobacco leaf samples are required. It is difficult to complete efficient and accurate assessments quickly, falling far short of the demands of efficient operations in the modern tobacco industry.

[0004] In addition to manual judgment, traditional technical solutions also include chemical analysis. This method assesses the maturity of tobacco leaves by subjecting tobacco leaf samples to complex chemical treatments and determination of specific component content in the laboratory. Although chemical analysis can provide relatively objective data, such as determining the maturity of tobacco leaves by measuring changes in the content of certain maturity-related chemical components, its operation is cumbersome and requires specialized laboratory equipment, technicians, and strict experimental procedures. It is not only time-consuming and labor-intensive, but also damages the integrity of the tobacco leaf samples and cannot obtain results immediately on site. This greatly limits its practical application and makes it difficult to promote to large-scale real-time field monitoring scenarios.

[0005] With the advancement of technology, computer vision and machine learning algorithms are increasingly being applied to tobacco leaf maturity assessment, aiming to achieve a degree of automated evaluation. Some existing technologies attempt to use computer vision to capture visible light image information of tobacco leaves and combine it with simple machine learning models for analysis and prediction. These methods have improved the efficiency and objectivity of assessments compared to traditional methods. However, they often rely solely on single visible light image information, overlooking the importance of multi-source, heterogeneous data on fresh tobacco leaves as a crucial information source. Tobacco leaves undergo changes in internal composition and external morphology at different growth stages and maturity levels. Relying solely on visible light images makes it difficult to fully and accurately capture these key features, resulting in limited prediction accuracy and an inability to meet the requirements for high-precision and reliable tobacco leaf maturity assessment.

[0006] Furthermore, existing computer vision and machine learning-based technical solutions mostly employ single-modal image processing or simple machine learning model architectures, failing to fully exploit the potential value of multi-source, heterogeneous data. Tobacco leaf growth is a dynamic process, and its morphological characteristics change over time and space. Current evaluation methods may not effectively capture the changing patterns of these key characteristics, making them particularly inadequate when faced with the variability of complex natural environments and difficult to adapt to diverse practical application scenarios.

[0007] In summary, the existing fresh tobacco leaf maturity assessment technology has certain deficiencies in accuracy and real-time performance. Summary of the Invention

[0008] In order to solve the above technical problems, the present invention provides a fresh tobacco leaf maturity image assessment method to solve the problem that the existing fresh tobacco leaf maturity assessment technology has certain deficiencies in accuracy and real-time performance.

[0009] The technical solution adopted in the present invention is as follows:

[0010] The present invention discloses a method for evaluating the maturity of fresh tobacco leaves using images, comprising the following steps:

[0011] S1, acquiring fresh tobacco leaf image data, wherein the fresh tobacco leaf image data includes multispectral image data and three-dimensional structure information data;

[0012] S2, preprocessing the acquired fresh tobacco leaf image data and building a fresh tobacco leaf image sample library;

[0013] S3, constructing a maturity assessment model, wherein the maturity assessment model includes a multimodal feature extraction layer, a spatiotemporal attention fusion layer, and a multi-task prediction layer;

[0014] S4, uses the fresh tobacco leaf image sample library to train the maturity assessment model;

[0015] S5, using the trained maturity assessment model, performs maturity assessment on the fresh tobacco leaf image to be assessed and generates an assessment result.

[0016] Furthermore, S1 includes the following steps:

[0017] S11, collecting fresh tobacco leaf images using multispectral imaging technology to obtain multispectral image data of the fresh tobacco leaves;

[0018] S12, scanning the fresh tobacco leaves using a three-dimensional laser scanning technology to obtain three-dimensional structural information data of the fresh tobacco leaves.

[0019] Furthermore, the multimodal feature extraction layer includes a convolutional neural network and a graph neural network;

[0020] The convolutional neural network is used to extract multispectral features from multispectral image data;

[0021] The graph neural network is used to extract spatial structural features from three-dimensional structural information data.

[0022] Furthermore, in the input layer of the graph neural network, coding items based on the dynamic changes of the fresh tobacco leaf growth cycle are used to capture the dynamic change trends of the same spatial position features of fresh tobacco leaves at different growth stages.

[0023] Furthermore, in the graph convolution layer of the graph neural network, a distance weighting factor is used to adjust the feature update of the graph convolution operation.

[0024] Furthermore, the spatiotemporal attention fusion layer uses a bidirectional long short-term memory network combined with an attention mechanism to fuse the features extracted from the multimodal feature extraction layer. The fusion process includes:

[0025] Arrange the features extracted from the multimodal feature extraction layer into a feature vector sequence according to the time sequence of the growth cycle;

[0026] A bidirectional long short-term memory network is used to capture the dependencies in the feature vector sequence and generate a hidden state vector;

[0027] A multiplicative attention mechanism is used to calculate the correlation score between the feature vector of each time step in the feature vector sequence and the corresponding hidden state vector;

[0028] Normalize the relevance scores of all time steps to generate attention weights;

[0029] The attention weights are used to perform weighted summation on the feature vectors to generate the context vector.

[0030] Furthermore, the multi-task prediction layer predicts the maturity category and maturity quantification value of the fresh tobacco leaf image through a fully connected network and generates a prediction result.

[0031] Furthermore, in S4, the process of training the maturity assessment model includes:

[0032] S41, dividing the fresh tobacco leaf image sample library into a training set, a validation set, and a test set;

[0033] S42, construct comprehensive loss function;

[0034] S43, based on the Bayesian optimization algorithm, constructs a Gaussian process model. The Gaussian process model is used to model the relationship between the hyperparameters of the maturity assessment model and the comprehensive loss function, and the optimal hyperparameter combination is selected.

[0035] S44, using the mini-batch gradient descent method based on gradient accumulation, adjusts the maturity assessment model parameters according to the number of mini-batch samples and the number of gradient accumulation steps, and calculates the average gradient of the accumulated gradient;

[0036] S45, based on the average gradient of the accumulated gradient, uses the adversarial network to update the maturity evaluation model parameters;

[0037] S46, minimize the comprehensive loss function and screen the final optimal hyperparameter combination.

[0038] Furthermore, in S5, a Monte Carlo method is used to perform confidence assessment on the evaluation results and generate a comprehensive confidence score.

[0039] The present invention also discloses a fresh tobacco leaf maturity image assessment system, which adopts the above-mentioned fresh tobacco leaf maturity image assessment method and comprises:

[0040] A data acquisition module, configured to acquire fresh tobacco leaf image data, wherein the fresh tobacco leaf image data includes multispectral image data and three-dimensional structure information data;

[0041] A data processing module is used to pre-process the acquired fresh tobacco leaf image data and construct a fresh tobacco leaf image sample library;

[0042] A model construction module is used to construct a maturity assessment model, wherein the maturity assessment model includes a multimodal feature extraction layer, a spatiotemporal attention fusion layer, and a multi-task prediction layer;

[0043] A model training module is used to train a maturity assessment model using a fresh tobacco leaf image sample library;

[0044] The maturity assessment module is used to use the trained maturity assessment model to perform maturity assessment on the fresh tobacco leaf images to be assessed and generate assessment results.

[0045] Beneficial effects of the present invention:

[0046] 1. By acquiring multispectral image data and three-dimensional structural information data as fresh tobacco leaf image data and utilizing multimodal data sources, the maturity assessment model can learn and understand the complex relationship between tobacco leaf maturity and various characteristics from multiple perspectives, thereby more accurately distinguishing tobacco leaves of different maturity levels and significantly improving the accuracy of maturity assessment. In the maturity assessment model, the multimodal feature extraction layer can perform targeted feature extraction on the data of each modality, better mining the maturity-related features in the data of each modality. The spatiotemporal attention fusion layer can focus on the key spatiotemporal feature information in the image data, and the multi-task prediction layer can consider multiple maturity-related prediction tasks, making the model's judgment on tobacco leaf maturity more accurate and precise. The maturity assessment model after training is used to perform maturity assessment on the fresh tobacco leaf images to be evaluated, meeting the real-time requirements in actual production, overcoming the problems of low efficiency of manual evaluation or time-consuming complex chemical analysis methods in existing technologies, and realizing rapid and real-time monitoring of the maturity of fresh tobacco leaves.

[0047] 2. Multispectral imaging technology is used to collect image data of fresh tobacco leaves in the visible and near-infrared bands, and combined with 3D laser scanning technology to obtain the 3D structural information of the tobacco leaves. The fusion of these two data sources provides rich input data for the maturity assessment model, which can more comprehensively describe the characteristics of fresh tobacco leaves, enabling the assessment model to more comprehensively reflect the actual growth status of the tobacco leaves and improve the accuracy of maturity assessment.

[0048] 3. The maturity assessment model utilizes a layered deep learning architecture. The underlying multimodal feature extraction layer consists of a convolutional neural network (CNN) branch and a graph neural network (GNN) branch, processing multispectral image features and three-dimensional structural features, respectively. The middle layer utilizes a spatiotemporal attention fusion layer to capture spatiotemporal dynamic features. The top layer implements multi-task prediction, simultaneously outputting a maturity category and a quantitative value. This approach allows the model to capture a richer set of local and global features, including the texture and color of fresh tobacco leaves, as well as spatial structural characteristics such as leaf curl and thickness variations, thereby providing a more refined maturity classification.

[0049] 4. In the graph neural network (GNN), coding items based on the dynamic changes of the fresh tobacco leaf growth cycle are used to capture the dynamic change trend of the same spatial position features of fresh tobacco leaves in different growth stages, and a distance weighting factor is used to adjust the feature update of the graph convolution operation, so that the graph neural network (GNN) can more accurately extract maturity-related features from the three-dimensional structural information of fresh tobacco leaves. Compared with traditional graph neural networks, it greatly improves the understanding and expression ability of the three-dimensional structural characteristics of fresh tobacco leaves, provides richer and more valuable feature information for subsequent maturity assessment, and significantly improves the accuracy of the model in tobacco leaf maturity assessment in complex natural environments.

[0050] 5. Based on the bidirectional long short-term memory (LSTM) network, the attention mechanism dynamically adjusts the importance of features at different time steps, allowing the model to focus more on those closely related to the spatiotemporal dynamics of tobacco leaf growth. The bidirectional LSTM combined with the attention mechanism can capture the complex correlation weights between different time steps and between multispectral features and spatial structural features. This design not only considers the changing trends in the temporal dimension, but also the feature distribution in the spatial dimension, thereby achieving effective fusion of multimodal features.

[0051] 6. A fully connected network is used to simultaneously predict the maturity category and quantified value of fresh tobacco leaf images, and the Monte Carlo method is used to evaluate the confidence of the assessment results. This provides a reference for the confidence of the assessment results. This feature helps assist decision-making, especially in agricultural practice where key choices need to be made based on the prediction results. It can intuitively reflect the reliability of the maturity assessment model for the fresh tobacco leaf maturity assessment results.

[0052] 7. During the training process of the maturity assessment model, the fresh tobacco leaf image sample library is reasonably divided into training set, validation set and test set, and a comprehensive loss function is constructed to ensure that the maturity assessment model achieves a balance in multi-task learning; the hyperparameters are automatically adjusted through the Bayesian optimization strategy to screen the hyperparameter combination that can enable the maturity assessment model to achieve the current optimal performance after multiple iterations. Under the basic framework of the current optimal hyperparameter combination, the maturity assessment model parameters are adjusted and updated through the combination of mini-batch gradient descent based on gradient accumulation and adversarial network, so that the maturity assessment model is continuously optimized during the training process and the prediction ability of fresh tobacco leaf maturity is gradually improved. Finally, under the optimization of mini-batch gradient descent based on gradient accumulation, according to the results of the maturity assessment model parameter adjustment, the weights in the comprehensive loss function are dynamically adjusted to make the comprehensive loss function continuously approach the minimum value, thereby guiding the maturity assessment model to train in a more optimal direction, screening the final optimal network hyperparameter combination, and improving the effect of fresh tobacco leaf maturity image assessment. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 This is a flow chart of a method for evaluating the maturity of fresh tobacco leaves using images according to the present invention;

[0054] Figure 2 This is a logic block diagram of a fresh tobacco leaf maturity image assessment system according to the present invention. DETAILED DESCRIPTION

[0055] The following embodiments of the present invention are described in further detail with reference to the accompanying drawings and examples. The following examples are used to illustrate the present invention but are not intended to limit the scope of the present invention.

[0056] Example 1:

[0057] like Figure 1 As shown, a method for evaluating the maturity of fresh tobacco leaves by image, comprising the following steps:

[0058] S1, acquiring fresh tobacco leaf image data, wherein the fresh tobacco leaf image data includes multispectral image data and three-dimensional structure information data;

[0059] S2, preprocessing the acquired fresh tobacco leaf image data and building a fresh tobacco leaf image sample library;

[0060] S3, constructing a maturity assessment model, wherein the maturity assessment model includes a multimodal feature extraction layer, a spatiotemporal attention fusion layer, and a multi-task prediction layer;

[0061] S4, uses the fresh tobacco leaf image sample library to train the maturity assessment model;

[0062] S5, using the trained maturity assessment model, performs maturity assessment on the fresh tobacco leaf image to be assessed and generates an assessment result.

[0063] The image source data of fresh tobacco leaves in S1 includes multispectral image data and three-dimensional structure information data. S1 includes the following steps:

[0064] S11, collecting fresh tobacco leaf images using multispectral imaging technology to obtain multispectral image data of the fresh tobacco leaves;

[0065] S12, scanning the fresh tobacco leaves using a three-dimensional laser scanning technology to obtain three-dimensional structural information data of the fresh tobacco leaves;

[0066] In this embodiment, in S11, fresh tobacco leaf images are collected by multispectral imaging technology to obtain multispectral fresh tobacco leaf image data including visible light and near-infrared bands to form multispectral image data. In the fresh tobacco leaf image collection, multispectral fresh tobacco leaf image data including visible light and near-infrared bands are obtained by a multispectral imaging device. The device is placed at a suitable distance from the fresh tobacco leaf plants to ensure that the target tobacco leaf area is covered and the light is sufficient and uniform. By controlling the spectral filter switching mechanism of the multispectral imaging device, it is allowed to pass through the visible light band (400-700nm) and the near-infrared band (700-1100nm) in turn and irradiate the fresh tobacco leaves; the fresh tobacco leaves produce different degrees of reflection and absorption of light in different bands. After these light rays are reflected back to the imaging device, they are received by the optical sensor and converted into electrical signals. After the signal processing and analog-to-digital conversion modules inside the device, the electrical signals are converted into digital signals, thereby generating multispectral fresh tobacco leaf image data including visible light and near-infrared bands. These data are the multispectral image data.

[0067] In S12, the three-dimensional laser scanning technology is implemented by a three-dimensional laser scanning device. Specifically, the fresh tobacco leaves are scanned by the three-dimensional laser scanning device to obtain the three-dimensional structural information of the fresh tobacco leaves. First, the position and angle of the three-dimensional laser scanning device are calibrated to ensure that the scanning range can fully cover the fresh tobacco leaves to be inspected. After starting the device, the laser emitter of the three-dimensional laser scanning device will emit dense laser beams to the surface of the fresh tobacco leaves. These laser beams are reflected on the surface of the fresh tobacco leaves, and the reflected light is captured by the receiver of the three-dimensional laser scanning device. The device calculates and processes the laser beams based on the parameters such as the laser emission angle, reflection time and optical path difference through the built-in algorithm of the three-dimensional laser scanning device. After collecting and integrating a large number of data points, a three-dimensional point cloud model of the fresh tobacco leaves is constructed to obtain high-density three-dimensional point cloud data, and then generate three-dimensional structural information that can accurately reflect the characteristics of the fresh tobacco leaves, such as shape, surface undulation and spatial position relationship, thereby constituting three-dimensional structural information data.

[0068] In S2, the acquired fresh tobacco leaf image data is preprocessed and a fresh tobacco leaf image sample library is constructed. In this embodiment, the fresh tobacco leaf image sample library is constructed to standardize the multispectral image data (multispectral image data) and the three-dimensional structure information data (three-dimensional structure information data).

[0069] Specifically, for the processing of multispectral image data, the original grayscale value of the image pixel in the i-th band is G i , calculate the grayscale mean μ of the band in the entire sample library i and standard deviation σ i , and then by the formula The grayscale value of each pixel is standardized to a distribution with a mean of 0 and a standard deviation of 1 to eliminate the influence of factors such as lighting under different acquisition conditions. The standardized data has zero mean and unit variance, which helps to reduce the influence of factors such as lighting changes under different acquisition conditions and improve the stability and accuracy of model training.

[0070] For the processing of three-dimensional structural information data, first determine the maximum value X of the x, y, and z coordinates in the fresh tobacco leaf image sample library. max \Y max \Z max and minimum value X min \Y min \Z min , and then use the formula Normalize the coordinate values to the range [0,1]. This normalization process ensures that differences in spatial dimensions between samples do not affect subsequent analysis. This is particularly important when using deep learning models, as it prevents certain features from dominating the model's learning process due to their large numerical range. Specifically, after normalization, principal component analysis (PCA) can be performed on the normalized three-dimensional structural data using feature dimensionality reduction techniques. This identifies the main components in the data, reduces data dimensionality while retaining most of the key information, reduces data redundancy, improves model training efficiency, and uncovers key feature combinations hidden in the data.

[0071] In S3, the multimodal feature extraction layer includes a convolutional neural network and a graph neural network;

[0072] The convolutional neural network is used to extract multispectral features from multispectral image data;

[0073] The graph neural network is used to extract spatial structural features from three-dimensional structural information data.

[0074] In this embodiment, the process of extracting multispectral features from multispectral image data by the convolutional neural network is as follows: when processing the multispectral image data, first, the pre-processed and standardized multispectral image data is input into the input layer of the convolutional neural network. Suppose the input multispectral fresh tobacco leaf image is I, and its size is H×W×C, where: H is the height, W is the width, and C is the number of bands.

[0075] Then, a series of convolutional layers of the convolutional neural network are used to perform convolution operations on the input multispectral fresh tobacco leaf image to extract local features and output feature maps. After the convolution operation, the activation function is used to perform nonlinear transformation on the output feature maps. Specifically, the convolution kernel of the convolution layer is used to extract local features and output feature maps. Let the convolution kernel of the lth convolution layer be K l , size k l ×kl ×C l , where k l is the spatial size of the convolution kernel, C l Is the number of input channels, then the output feature map F of this layer l The calculation formula is:

[0076]

[0077] In the formula, σ is the activation function (ReLU function can be used), s l is the step size, b l is the bias term, p and q refer to the position index of the convolution kernel in the horizontal and vertical directions respectively, and m is the channel index of the input image.

[0078] Through the alternating operation of multiple convolutional and pooling layers, local and global features in the multispectral image are gradually extracted, including features related to texture, color, and the maturity of the fresh tobacco leaves. Specifically, through the alternating operation of pooling and convolutional layers, the feature map is downsampled. The features extracted by the convolutional and pooling layers are integrated in the output layer to output the multispectral features of the fresh tobacco leaf image.

[0079] Finally, the features extracted by the convolutional layer are passed to the output layer of the convolutional neural network, and the multispectral features of the fresh tobacco leaf image are output through the output layer.

[0080] In this embodiment, the graph neural network includes an input layer, a graph convolution layer, and an output layer;

[0081] The input layer of the graph neural network is used to build the graph structure and initialize node features;

[0082] The graph convolution layer of the graph neural network is used to update the feature vector of the node through graph convolution operations and extract the three-dimensional spatial structural features of the fresh tobacco leaves through multi-layer graph convolution operations;

[0083] The output layer of the graph neural network is used to map the extracted three-dimensional spatial structure features to the target output space.

[0084] The output layer of the graph neural network maps the three-dimensional spatial structural features to the target output space. The multi-task prediction layer is responsible for predicting the maturity category and maturity quantification value of the fresh tobacco leaf image. The output layer of the graph neural network focuses on providing processed three-dimensional spatial structural features. These features serve as the input of the multi-task prediction layer, which uses a fully connected network to complete the maturity category prediction and maturity quantification value prediction.

[0085] When processing three-dimensional structure information data through a graph neural network, the three-dimensional structure information data in the input layer of the graph neural network is constructed into a graph structure, which can be expressed as:

[0086] G=(V,E)

[0087] Where V is a node set, including several nodes, representing key sampling points on the surface or inside of fresh tobacco leaves; E is an edge set, representing the spatial connection relationship between nodes; for node v i ∈V, each node v i The eigenvector x i Initialize the three-dimensional coordinates (x, y, z) and local geometric attributes of the point. The three-dimensional coordinates describe the spatial position of the node, and the local geometric attributes include curvature, normal vector, local thickness, etc., which describe the geometric characteristics of the node. Pass the constructed graph structure to the graph convolution layer.

[0088] The node features are updated through the graph convolution layer of the graph neural network, and the three-dimensional spatial structural features of the fresh tobacco leaves are extracted through multi-layer graph convolution operations. The specific process is as follows:

[0089] The feature is updated through the graph convolution layer. Let the weight matrix of the graph convolution layer be W, then the node v i After a layer of graph convolution operation, the feature vector x i The update formula is:

[0090]

[0091] Where, For node v i The updated feature vector, σ is the activation function, N(v i ) is the node v i The neighbor node set, W is the weight matrix of the graph convolution layer, x j is the neighbor node v j The original eigenvector of , b is the bias term.

[0092] Among them, the core of the graph convolution operation is to aggregate the information of neighboring nodes and update the node features through nonlinear transformation. Each layer of graph convolution will aggregate the information of neighboring nodes and update the node features through the weight matrix and activation function. Through multi-layer graph convolution operations, the three-dimensional spatial structural characteristics of fresh tobacco leaves are gradually extracted, including the curling degree, thickness change characteristics and overall shape of the fresh tobacco leaves.

[0093] The extracted three-dimensional spatial structural features are mapped to the target output space through the output layer of the graph neural network for subsequent tasks.

[0094] As a preferred solution, in the input layer of the graph neural network, a coding term based on the dynamic changes in the growth cycle of fresh tobacco leaves is used to capture the dynamic change trend of the same spatial position features of fresh tobacco leaves at different growth stages. The specific process is as follows:

[0095] The growth cycle of fresh tobacco leaves is divided into T growth stages;

[0096] For each node v i In the tth growth stage, in addition to the original spatial information, the feature vector also adds a coding item related to the growth stage, expressed as g(t). The coding value calculated according to the growth stage is added to the node v as an additional feature. i In the eigenvector of

[0097] The calculation formula of the coding term g(t) is as follows:

[0098]

[0099] In the formula, α is an adjustable hyperparameter used to control the rate at which the coding term changes with growth stage; T is the total number of stages in the fresh tobacco leaf growth cycle, which determines the granularity of the growth cycle division and affects the range and frequency of the coding term changes; t is the tth growth stage of the fresh tobacco leaf, indicating the growth stage the node is in at the current moment. The value of t ranges from 1 to T, identifying the growth stage of the node at the current moment, ensuring that the coding term reflects the dynamic changes in the growth process. The coding term enables the graph neural network (GNN) to capture the dynamic changes in the characteristics of the tobacco leaf at the same spatial location at different growth stages.

[0100] As a preferred solution, in the graph convolution layer of the graph neural network, a distance weighting factor is used to adjust the feature update of the graph convolution operation. The specific process is as follows:

[0101] Calculate the distance weighting factor d ij , the distance weighting factor d ij The calculation formula is as follows:

[0102]

[0103] Where, d ij For node v i With neighbor node v j The distance weighting factor between ij For node v i With neighbor node v j The Euclidean distance between them, β is the distance weight adjustment parameter;

[0104] The distance weighting factor is used to adjust the feature update of the graph convolution operation, then the node v i After a layer of graph convolution operation, the feature vector x i The updated formula is adjusted to:

[0105]

[0106] Where σ is the activation function, N(v i ) is the node v i The neighbor node set, W is the weight matrix of the graph convolution layer, x j is the neighbor node v j The original eigenvector of , b is the bias term.

[0107] The distance weighting factor enables the model to pay more attention to the information of nodes that are closer when transmitting information. This is because in the three-dimensional structure of fresh tobacco leaves, areas with similar distances are more likely to have close structural and physiological connections physically. This allows for more effective extraction of local subtle features such as leaf curl and thickness changes, which is of great significance for accurately assessing the maturity of tobacco leaves.

[0108] In S3, the spatiotemporal attention fusion layer uses a bidirectional long short-term memory network (LSTM) combined with an attention mechanism to fuse the features extracted from the multimodal feature extraction layer.

[0109] In this embodiment, the features extracted in the multimodal feature extraction layer include multispectral features (including features such as the color and texture of fresh tobacco leaves) and spatial structural features (including features such as the curl degree and thickness variation of fresh tobacco leaves). The spatiotemporal attention fusion layer uses a bidirectional long short-term memory network (LSTM) combined with an attention mechanism to fuse the multispectral features and spatial structural features. The process is as follows:

[0110] The features extracted from the multimodal feature extraction layer (including the multispectral features and spatial structure features of fresh tobacco leaves) are arranged into a feature vector sequence according to the time sequence of the growth cycle;

[0111] The growth cycle of fresh tobacco leaves is divided into T stages. The multispectral features and spatial structure feature vectors of fresh tobacco leaves are serialized. The extracted multispectral features and spatial structure features are arranged into a feature vector sequence according to the time sequence of the growth cycle. Each time step corresponds to the feature representation of a specific growth stage or time period.

[0112] A bidirectional long short-term memory network is used to capture the dependencies in the feature vector sequence and generate a hidden state vector containing temporal context information.

[0113] A bidirectional long short-term memory (LSTM) network is used to process feature vector sequences. The bidirectional LSTM can capture dependencies in the time series from both the forward and backward directions. The forward LSTM captures temporal dependencies from the past to the future, generating a series of hidden states. The backward LSTM captures temporal dependencies from the future to the past, generating another series of hidden states. The bidirectional LSTM concatenates these forward and backward hidden states as the final hidden state vector, thereby providing a more comprehensive understanding of the dynamic changes in fresh tobacco leaves throughout their growth cycle. Specifically, the feature vector sequence is input into the bidirectional LSTM to generate hidden state vectors containing contextual information in the temporal dimension. These hidden state vectors not only contain information about the current time step but also incorporate information from previous and subsequent time steps. In a bidirectional LSTM network, the hidden state is a state representation maintained internally by the network that captures all input information to date. For each time step, the LSTM unit generates a hidden state vector.

[0114] A multiplicative attention mechanism is used to calculate the correlation score between the feature vector of each time step in the feature vector sequence and the corresponding hidden state vector;

[0115] The multiplicative attention mechanism is used to calculate the correlation score (also called attention score) between the feature vector of each time step and the corresponding current hidden state through the dot product operation. The calculation formula is as follows:

[0116] e t =q ′ ×h t

[0117] Where q ′ The query vector is the hidden state of another time step, the context vector, or other specific vector used to compare with the hidden state of each time step; h t is the hidden state vector of the bidirectional long short-term memory network at time step t, which contains all relevant information before and after the time step.

[0118] Normalize the relevance scores of all time steps to generate attention weights;

[0119] Apply the softmax function to normalize the relevance scores of all time steps to obtain the attention weight, which is calculated as follows:

[0120]

[0121] Where, e t is the correlation score, and T is the total length of the feature vector sequence.

[0122] The feature vectors are weighted and summed using the attention weights to generate the context vector. The calculation formula is as follows:

[0123]

[0124] Where, α t is the attention weight at time step t, is the hidden state vector of the bidirectional long short-term memory network at time step t, and T is the total length of the feature vector sequence.

[0125] The context vector is a weighted summation result obtained by applying the multiplicative attention mechanism to the feature vectors of each time step in the feature vector sequence. It represents the most relevant part of the entire feature vector sequence and best reflects the information concentration of the spatiotemporal dynamic characteristics of the fresh tobacco leaf growth process. In the maturity assessment of fresh tobacco leaves, the context vector captures the key information of the color, morphological changes and spatial structural characteristics of the leaves at different growth stages. The output context vector of the spatiotemporal attention fusion layer is received by the multi-task prediction layer and further processed to complete the fresh tobacco leaf maturity prediction task.

[0126] The final context vector is obtained by weighted summation of feature vectors using attention weights. This allows the model to focus on the parts of the multimodal features (multispectral features and spatial structure features) that are closely related to the spatiotemporal dynamics of tobacco leaf growth. This allows the model to focus more on the parts closely related to the spatiotemporal dynamics of tobacco leaf growth, such as the key time points of leaf color and morphological changes at different growth stages, and the changing trends of spatial features in different parts. At a certain key growth stage, changes in leaf color or morphology may be given higher weights. By adding the attention mechanism, it can focus more on extracting more important features.

[0127] In S3, the multi-task prediction layer predicts the maturity category and maturity quantification value of the fresh tobacco leaf image through a fully connected network and generates a prediction result. The specific process is as follows:

[0128] The feature vector fused by the spatiotemporal attention fusion layer is input into the multi-task prediction layer to predict the maturity of the fresh tobacco leaf image and generate prediction results, including maturity category prediction results and maturity quantitative numerical prediction results.

[0129] Maturity category prediction: A fully connected network is used to perform weighted summation on the fused feature vectors. The activation function is used to convert the weighted summation result into probability values for each maturity category. The category with the highest probability is selected as the final output fresh tobacco leaf maturity category.

[0130] In this embodiment, the neurons in the fully connected network perform weighted summation on the input feature vectors and convert the results into probability values for each maturity category through an activation function. The activation function adopts the softmax function, wherein the softmax function is used to convert the score of each category into a probability value to ensure that the sum of the probabilities of all categories is 1; the maturity categories include early immaturity, middle immaturity, early moderate maturity, middle moderate maturity, late moderate maturity, early overmaturity, and late overmaturity, which realizes the accurate assessment of the maturity of fresh tobacco leaves, which can not only help farmers determine the optimal harvest time, but also provide a more accurate quality control basis for tobacco processing.

[0131] Maturity quantification prediction: In the fully connected network of the multi-task prediction layer, a numerical result is directly calculated from the input feature vector through weight and bias operations. This numerical result represents the actual maturity quantification value of the fresh tobacco leaf.

[0132] In this embodiment, this numerical result is obtained based on the mapping relationship between the fresh tobacco leaf features and the maturity values learned by the model during the training process; during the training process, the mean square error loss function is used to minimize the mean square error (MSE) loss between the predicted values and the actual annotations, and the weights and biases of the network are continuously adjusted so that in the evaluation stage, a quantitative value corresponding to the actual maturity of the fresh tobacco leaves can be accurately output, thereby achieving accurate quantitative evaluation of the maturity of the fresh tobacco leaves.

[0133] In S4, the process of training the maturity assessment model includes:

[0134] S41, dividing the fresh tobacco leaf image sample library into a training set, a validation set, and a test set;

[0135] S42, construct comprehensive loss function;

[0136] S43, based on the Bayesian optimization algorithm, constructs a Gaussian process model. The Gaussian process model is used to model the relationship between the hyperparameters of the maturity assessment model and the comprehensive loss function, and the optimal hyperparameter combination is selected.

[0137] S44, using the mini-batch gradient descent method based on gradient accumulation, adjusts the maturity assessment model parameters according to the number of mini-batch samples and the number of gradient accumulation steps, and calculates the average gradient of the accumulated gradient;

[0138] S45, based on the average gradient of the accumulated gradient, uses the adversarial network to update the maturity evaluation model parameters;

[0139] S46, minimize the comprehensive loss function and screen the final optimal hyperparameter combination.

[0140] In S41, the preprocessed fresh tobacco leaf image sample library is divided into a training set, a validation set, and a test set in a ratio of 7:1.5:1.5 to ensure that the data distribution of each subset is reasonable and representative.

[0141] In S42, a comprehensive loss function is constructed through classification cross entropy loss, mean square error loss and regularization term. The specific process is as follows:

[0142] Construct a classification cross entropy loss for classification tasks (maturity category prediction) to measure the accuracy of maturity category prediction, that is, to measure the difference between the predicted category and the true category. The specific formula is:

[0143]

[0144] Where N is the number of samples, C is the number of categories, and y ij is the true label (0 or 1) of sample i belonging to category j, p ij is the predicted probability;

[0145] Construct the mean square error loss for the regression task (maturity quantification numerical prediction) to measure the difference between the predicted value and the true value of the maturity value. The specific formula is:

[0146]

[0147] Where, and are the actual maturity value and the predicted maturity value respectively;

[0148] A regularization term is constructed based on the uncertainty assessment index to constrain the prediction uncertainty of the maturity assessment model. The specific formula is:

[0149] L reg =λU

[0150] Where λ is the regularization coefficient and U is the uncertainty index, which prevents overfitting and improves the generalization ability of the model;

[0151] The comprehensive loss function is calculated based on the classification cross entropy loss, mean square error loss and regularization term. The specific formula is:

[0152] L 综合 =δL ce +μL m +τL reg

[0153] Where δ, μ, and τ are the weights of the classification cross entropy loss, mean square error loss, and regularization term, respectively.

[0154] The comprehensive loss function combines classification, regression, and regularization tasks to ensure that the model achieves a balance in multi-task learning.

[0155] In S43, based on the Bayesian optimization strategy, a Gaussian process model is used to model the relationship between the hyperparameters of the maturity assessment model and the comprehensive loss function. The process of screening the current optimal hyperparameter combination is as follows:

[0156] An optimized hyperparameter space for the maturity assessment model is defined. In this embodiment, the optimal hyperparameter combination selected optimizes the overall architecture and training method of the maturity assessment model from a macro perspective. This includes the value ranges of key hyperparameters, such as the number and size of convolutional neural networks (CNNs), the weight adjustment parameter β of the distance weighting factor in graph neural networks (GNNs), the hidden layer size of LSTM units, the number of neurons in the fully connected layer, and the learning rate. These hyperparameters are configuration parameters that need to be determined before the maturity assessment model is constructed. They determine the basic structure of the maturity assessment model and the general direction of the training method, and have a significant impact on the model performance in the early stages of model training.

[0157] The Bayesian optimization algorithm is used to construct a Gaussian process model. The Gaussian process model is a probabilistic model that models the relationship between hyperparameters and the comprehensive loss function to approximate the relationship between hyperparameters and the comprehensive loss function.

[0158] The optimal hyperparameter combination is selected. During each round of training, based on the training results of the current maturity assessment model (such as the loss value on the validation set), the probability of performance improvement of the maturity assessment model under different hyperparameter combinations is calculated using a Gaussian process model. The hyperparameter combination that is most likely to reduce the comprehensive loss function is selected for trial. After multiple iterations, the hyperparameter combination that enables the maturity assessment model to achieve the current optimal performance during the entire training process is selected.

[0159] The current optimal hyperparameter combination screened out through the above process provides a basic framework for the training of the maturity assessment model. Under this basic framework, the training of steps S44, S45 and S46 is performed.

[0160] In S44, the mini-batch gradient descent method based on gradient accumulation is used to adjust the maturity assessment model parameters according to the number of mini-batch samples and the number of gradient accumulation steps, and the average gradient of the accumulated gradient is calculated as follows:

[0161] Preset small batch sample size S Z And the number of steps S of gradient accumulation V , each time S is selected from the training set ZSamples are input into the maturity assessment model. During the forward propagation process, the data passes through the bottom multimodal feature extraction layer, the middle spatiotemporal attention fusion layer, and finally reaches the top multi-task prediction layer. The loss value between the prediction result and the true label is calculated. For example, a combination of the cross entropy loss function and the mean square error loss function is used to measure the loss of multiple tasks. During this training process, the number of small batch samples is dynamically adjusted according to the number of iterations of maturity assessment model training and the change in loss value. The parameters of the maturity assessment model are adjusted using the small batch gradient descent method with gradient accumulation. When the loss value decreases rapidly in the early stage of maturity assessment model training, the number of small batch samples is appropriately increased to utilize the gradient information of more samples to accelerate the convergence of the maturity assessment model. When the loss value decreases gradually, the number of small batch samples is reduced and the parameters of the maturity assessment model are adjusted to avoid overfitting.

[0162] In the back propagation phase, the gradient of the network parameters corresponding to the current small batch of samples is calculated, but the parameters are not updated immediately. Instead, these gradients are accumulated. When the number of accumulated steps reaches the preset number of gradient accumulation steps S, V When , the average value of the accumulated gradient is calculated to generate the average gradient of the accumulated gradient. During the entire training process, the above steps are repeated continuously, the number of small batch samples is dynamically adjusted, and the model is gradually optimized through gradient accumulation to generate the parameter adjustment results of the maturity assessment model.

[0163] Through this small-batch gradient descent method based on gradient accumulation, it is possible to use larger-scale sample gradient information to adjust model parameters under limited hardware resources, so that the maturity assessment model is continuously optimized during the training process, and the ability to predict the maturity of fresh tobacco leaves is gradually improved.

[0164] In S45, the process of using the adversarial network to update the maturity evaluation model parameters based on the average gradient of the accumulated gradient is as follows:

[0165] Through the adversarial training mechanism, a Generative Adversarial Network (GAN) architecture is constructed and combined with a maturity assessment model. The GAN architecture includes a generator and a discriminator. The generator uses the Adam optimizer, and the discriminator uses the Adagad optimizer. The generator's task is to set a small learning rate and generate a data distribution similar to the features of real fresh tobacco leaf images based on a random noise vector, attempting to deceive the discriminator. The discriminator adaptively adjusts the learning rate to distinguish between real fresh tobacco leaf image features and fake features generated by the generator.

[0166] During training, real multispectral image data and 3D structural information data, along with data generated by the generator, are fed into the discriminator. The discriminator outputs a probability of whether the input data is real or fake. By minimizing the discriminator's loss function, it can accurately distinguish between real and fake data, while maximizing the generator's loss function, prompting the generator to produce more realistic data.

[0167] Finally, the generator (Adam optimizer) updates the parameters of the maturity assessment model based on the average gradient of the cumulative gradients calculated in step S44. During this adversarial process, the parameters of each layer of the maturity assessment model are updated and optimized together with the parameters of the adversarial network (GAN). This allows the maturity assessment model to learn more robust feature representations during continuous adversarial training, and is able to better cope with changes in fresh tobacco leaf image data under different environments, thereby improving the accuracy and stability of fresh tobacco leaf maturity assessment.

[0168] In this embodiment, the maturity assessment model parameters adjusted and updated in S44 and S45 include parameters such as weights and bias items of each layer in the maturity assessment model, with the purpose of better fitting the training data under the current optimal hyperparameter combination screened out in S43.

[0169] In S46, the process of minimizing the comprehensive loss function and screening the final optimal hyperparameter combination is as follows:

[0170] Initialize the weights of the classification cross entropy loss, mean square error loss, and regularization term according to the current optimal hyperparameter combination screened in S43;

[0171] Under the optimization of the mini-batch gradient descent method based on gradient accumulation, the parameters of the model are adjusted according to the maturity assessment results. By dynamically adjusting the weights in the comprehensive loss function so that the comprehensive loss function continuously approaches the minimum value, the model is guided towards a more optimal training direction.

[0172] Specifically, the process of dynamically adjusting the weights in the comprehensive loss function is as follows:

[0173] During model training, the losses of the classification task (maturity category prediction) and the regression task (maturity quantification numerical prediction) are continuously monitored. Based on the changing trends of the losses of the classification and regression tasks, the weights of the classification cross entropy loss and the mean squared error loss are dynamically adjusted. If the loss of the classification task decreases slowly, the weight of the classification cross entropy loss in the comprehensive loss function is appropriately increased. If the loss of the regression task fluctuates greatly, the weight of the mean squared error loss is increased. The weight of the regularization term is dynamically adjusted according to the changing trend of uncertainty during training. The Monte Carlo Dropout method is used to calculate the variance of the prediction results, and the weight of the regularization term is adjusted accordingly.

[0174] During the model training process, the hyperparameter combination is dynamically adjusted for the classification cross entropy loss, mean square error loss, and the weight of the regularization term to screen out the final optimal network hyperparameter combination.

[0175] Specifically, this is a more refined adjustment. Based on the hyperparameter framework determined by S43, the maturity assessment model dynamically optimizes the weights of the comprehensive loss function according to the actual performance of the classification task (maturity category prediction), regression task (maturity quantification numerical prediction) and regularization term during training, and dynamically adjusts the hyperparameter combination to screen out the final optimal network hyperparameter combination.

[0176] By minimizing the comprehensive loss function through the above process, the performance of the maturity assessment model on different tasks is balanced, and the optimal network hyperparameter combination is finally selected to improve the effect of fresh tobacco leaf maturity image assessment.

[0177] In S5, the maturity assessment model after training is used to perform maturity assessment on the fresh tobacco leaf image to be assessed, and an assessment result is generated. In this embodiment, the process of performing maturity assessment on the fresh tobacco leaf image to be assessed is as follows:

[0178] S51, extracting and fusing features of the fresh tobacco leaf image to be evaluated;

[0179] In S51, the image of the fresh tobacco leaf to be evaluated is preprocessed so that it is consistent with the image specifications of the training set and input into the trained maturity assessment model. The image of the fresh tobacco leaf to be evaluated is first subjected to feature extraction in the multimodal feature extraction layer. Specifically, multispectral features are extracted by a convolutional neural network (CNN), and three-dimensional structural information is extracted by a graph neural network (GNN); the extracted features are passed to the spatiotemporal attention fusion layer, and the extracted features are fused through a bidirectional long short-term memory network (LSTM) combined with an attention mechanism to capture the temporal and spatial dynamic characteristics of fresh tobacco leaves at different growth stages.

[0180] S52, performing maturity evaluation on the fresh tobacco leaf image to be evaluated and generating an evaluation result;

[0181] In S52, the features of the fresh tobacco leaf image to be evaluated after fusion processing are passed to the multi-task prediction layer to perform maturity evaluation on the fresh tobacco leaf image to be evaluated, and an evaluation result is generated. The evaluation result includes a maturity category prediction result and a maturity quantification numerical prediction result;

[0182] For maturity category prediction, in the fully connected network of the multi-task prediction layer, neurons perform a weighted summation of input features and convert the results into probability values for each maturity category using the softmax function. The fully connected network finally outputs the category with the highest probability as the final output fresh tobacco leaf maturity category.

[0183] For the quantitative prediction of maturity, in the fully connected network of the multi-task prediction layer, a numerical result is calculated directly from the input features through weight and bias operations. This numerical value represents the actual quantitative value of maturity of fresh tobacco leaves.

[0184] S53, using the Monte Carlo (dropout) method to perform confidence assessment on the evaluation results and generate a comprehensive confidence score.

[0185] In S53, the Monte Carlo method is used to calculate the uncertainty measure of the maturity category prediction result of the fresh tobacco leaf image to be evaluated and the stability measure of the maturity quantification numerical prediction result, and generate a comprehensive confidence score. The specific process includes:

[0186] S531, performing multiple forward propagations on the same input fresh tobacco leaf image to be evaluated, applying the Monte Carlo Dropout method to randomly discard a certain proportion of neurons during each forward propagation to generate multiple different prediction results;

[0187] In S531, when the model has been trained and the weights are fixed, multiple forward propagations are performed on the same input image. During each forward propagation, a certain proportion of neurons are randomly discarded (i.e., Dropout is applied). Each forward propagation will obtain a prediction result of a maturity category and a quantitative value, which can simulate different states of the maturity assessment model and generate multiple different prediction results.

[0188] S532, calculates uncertainty measures for maturity category prediction results;

[0189] In S532, for each maturity category prediction result obtained through forward propagation, the frequency of each category is counted. The frequency of the most frequently occurring maturity category is subtracted from 1, and the difference is used as a measure of uncertainty. The smaller the difference, the higher the confidence level. For example, if a category consistently dominates across multiple predictions, the confidence level for that category is high.

[0190] S533, computing maturity quantification to measure the stability of numerical prediction results;

[0191] In S533, for each maturity quantization numerical prediction result obtained by forward propagation, the standard deviation of these prediction results is calculated. The smaller the standard deviation, the more stable the prediction result and the higher the confidence level.

[0192] S534 , generating a comprehensive confidence score by combining the uncertainty measure of the maturity category prediction result of the fresh tobacco leaf image to be evaluated and the stability measure of the maturity quantification numerical prediction result in a weighted or normalized manner.

[0193] In S534, the uncertainty measure of the maturity category prediction result of the fresh tobacco leaf image to be evaluated and the stability measure of the maturity quantification numerical prediction result are combined by weighting or normalization to obtain the final evaluation result and a comprehensive confidence score. This score can intuitively reflect the reliability of the maturity assessment model for the fresh tobacco leaf maturity assessment result.

[0194] Test process:

[0195] In this embodiment, the maturity of fresh tobacco leaves was evaluated in the tobacco production field to verify the superiority of the fresh tobacco leaf maturity image assessment technology proposed in this solution. Comparative experiments with various existing technology models were conducted, as follows:

[0196] A total of 2,800 tobacco plants were randomly selected from five tobacco planting bases in different geographical locations, and 5,200 fresh tobacco leaf samples were collected according to different growth stages and parts, covering all maturity states from young to over-mature.

[0197] This project explored eight representative tobacco-growing regions with diverse ecological environments. Using a random sampling method, 1,347 tobacco plants were randomly selected from each region. Careful and comprehensive sample collection was conducted on each plant, based on the different growth stages (including tender leaves, rapid growth, maturity, and overmaturity) and plant location (top, middle, and bottom). Ultimately, a total of 5,129 fresh tobacco leaf samples were obtained, covering a wide range of maturity states, from extremely young to severely overmature, providing a rich and diverse data base for the experiment.

[0198] In the image acquisition process, a professional-grade camera equipped with a high-sensitivity sensor and a variety of high-precision spectral filters is used. Its spectral response range accurately covers the 400-900nm band, which can accurately capture the subtle characteristic differences of fresh tobacco leaves under different spectra. At the same time, a 3D laser scanner with ultra-high resolution and fast scanning capabilities is used to perform a full-scale, high-precision acquisition of the three-dimensional structure of fresh tobacco leaves. The entire experimental process is carried out in a strictly controlled environment of light, temperature (maintained at 22.5℃±1℃) and humidity (relative humidity maintained at 60%±5%) to ensure the stability of the data acquisition process and the consistency of the data results to the greatest extent possible.

[0199] After extensive research and screening, we selected several widely used and representative models for fresh tobacco leaf maturity assessment for comparison. These included the Support Vector Machine (SVM) and Random Forest (RF) models based on traditional machine learning algorithms, as well as the YOLOv5 and Faster R-CNN models based on deep learning architectures. These models have a solid foundation in application and technical characteristics in this field, providing comprehensive and valuable references for this comparative experiment.

[0200] Each fresh tobacco leaf sample was photographed from multiple angles to acquire multispectral images, resulting in a total of 15,387 images. During the acquisition process, detailed spectral reflectance data for each band was recorded. For example, the average reflectance of sample A at 605nm was 0.4567, and the average reflectance at 720nm was 0.2345. Simultaneously, a 3D laser scanner was used to perform a comprehensive and detailed scan of each sample, acquiring high-density 3D point cloud data. For example, the point cloud density of sample B was precisely calculated to be 120.34 points per square centimeter.

[0201] The collected multispectral images were preprocessed, and their pixel value range was precisely adjusted to the [0,1] interval through complex algorithms to eliminate interference caused by different equipment and environmental factors. The three-dimensional point cloud data was carefully streamlined and aligned to remove redundant points and ensure data accuracy, significantly improving the efficiency of subsequent data processing. After a series of rigorous processing, the data was constructed into a complete sample library of fresh tobacco leaf images. Then, the sample library was carefully divided into a training set (3590 samples), a validation set (769 samples), and a test set (770 samples) in a ratio of 7:1.5:1.5 to ensure that the data distribution of each subset is reasonable and representative.

[0202] A hierarchical deep learning model is constructed to output maturity categories, quantitative values, and confidence scores. The hierarchical deep learning model includes a multimodal feature fusion module, a spatiotemporal attention module, and a multi-task prediction module. In the multimodal feature fusion module, convolutional neural networks (CNN) and graph convolutional networks (GCN) are used to extract deep features from multispectral images and three-dimensional structural data, respectively. A complex adaptive weight fusion strategy is used to achieve efficient fusion of different modal features. The spatiotemporal attention module is based on a long short-term memory network (LSTM) and an advanced attention mechanism, which can keenly capture subtle changes in fresh tobacco leaf characteristics at different temporal and spatial scales. The multi-task prediction module accurately outputs maturity categories, quantitative values, and confidence scores through a fully connected layer.

[0203] During the training process, the small batch gradient descent method based on gradient accumulation is used, and the number of small batch samples S is set Zis 33, the number of steps of gradient accumulation is S V The convergence value momentum factor of the mini-batch gradient descent method is 5, and the momentum factor is 0.92. The momentum factor is used to balance training efficiency and model convergence. To further enhance the robustness of the model, an adversarial training mechanism was introduced. High-quality adversarial examples were generated through a generative adversarial network (GAN), enabling the model to learn more generalizable features during training. A Bayesian optimization algorithm was used to conduct a detailed hyperparameter search. The hyperparameter space covered several key parameters, including the number of convolutional layers of the convolutional neural network (CNN) (range [3, 5, 7]), the adjacency matrix weights of the graph neural network (GNN) (range [0.2, 0.4, 0.6]), and the number of hidden units in the LSTM (range [50, 70, 90]). After 37 rounds of rigorous iteration, the optimal hyperparameter combination was finally determined. At the same time, a comprehensive loss function was constructed, which includes cross entropy loss, mean square error loss, and a regularization term based on uncertainty evaluation indicators. By continuously minimizing the loss function, the hyperparameter combination of the model is continuously optimized to ensure that the model performance reaches the best state.

[0204] The test set samples are input into the fully trained model for evaluation. The input data is first normalized to meet the model's input requirements. The data then passes through the multimodal feature extraction layer, the spatiotemporal attention fusion layer, and the multi-task prediction module, ultimately outputting accurate evaluation results. For example, the maturity category of sample C is predicted to be moderately mature, with a prediction probability of 0.8923, a maturity quantization value of 0.7654, and a confidence score of 0.9321. The maturity category of sample D is predicted to be immature, with a prediction probability of 0.9125, a maturity quantization value of 0.3246, and a confidence score of 0.8873.

[0205] The samples are tested using the existing model and the model of this solution to generate a test result evaluation table, as shown in the following table:

[0206]

[0207]

[0208] The table above clearly shows that our proposed model significantly outperforms existing models in terms of accuracy for multispectral feature extraction, 3D structural feature extraction, and fusion features. In terms of processing time, our proposed model also demonstrates significant advantages in maturity category prediction and quantitative numerical prediction. It also outputs reliable and accurate confidence scores, providing strong assurance for the reliability of the assessment results.

[0209] By calculating the maturity category prediction accuracy and maturity quantification numerical prediction mean square error of this solution and the existing technology, the superiority of this solution can be better reflected, as shown in the following table:

[0210] Model Maturity category prediction accuracy Maturity quantification of numerical prediction mean square error Maturity Assessment Model of This Solution 93.45％ 0.0234 SVM model 65.32％ 0.0876 RF Model 68.45％ 0.0765 YOLOv5 model 72.56％ 0.0654 Faster R-CNN model 74.67％ 0.0543

[0211] The model accuracy evaluation table shows that the maturity category prediction accuracy of this proposed model is 18.78 percentage points higher than that of the relatively well-performing Faster R-CNN model, and the mean square error of the maturity quantification prediction is 0.0309 lower than that of the Faster R-CNN model. This fully demonstrates that this proposed technology has significant and undeniable advantages over existing technologies in terms of the accuracy and reliability of fresh tobacco leaf maturity assessment.

[0212] This experiment validated the advanced and practical nature of this proposed fresh tobacco leaf maturity image assessment technology, significantly improving the maturity assessment model's predictive capabilities and accuracy. Compared to existing technologies, this technology demonstrates significant advantages in key areas such as feature extraction, prediction accuracy, and processing efficiency, providing a more efficient, accurate, and reliable innovative solution for fresh tobacco leaf maturity assessment during tobacco production.

[0213] like Figure 2 As shown, this embodiment further provides a fresh tobacco leaf maturity image assessment system, which adopts the above-mentioned fresh tobacco leaf maturity image assessment method, including:

[0214] A data acquisition module, configured to acquire fresh tobacco leaf image data, wherein the fresh tobacco leaf image data includes multispectral image data and three-dimensional structure information data;

[0215] A data processing module is used to pre-process the acquired fresh tobacco leaf image data and construct a fresh tobacco leaf image sample library;

[0216] A model construction module is used to construct a maturity assessment model, wherein the maturity assessment model includes a multimodal feature extraction layer, a spatiotemporal attention fusion layer, and a multi-task prediction layer;

[0217] A model training module is used to train a maturity assessment model using a fresh tobacco leaf image sample library;

[0218] The maturity assessment module is used to use the trained maturity assessment model to perform maturity assessment on the fresh tobacco leaf images to be assessed and generate assessment results.

[0219] The fresh tobacco leaf maturity image assessment system in this solution and the fresh tobacco leaf maturity image assessment method provided above have the same concept, and the contents of the operations performed by each module have been described in detail in the method embodiment and will not be repeated here.

[0220] Example 2:

[0221] The remaining features of Example 2 are the same as those of Example 1. The difference is that Example 2 constructs an adaptive feature fusion mechanism (AFFM) within the graph neural network structure to dynamically fuse features of different levels and types to fully tap the potential value of three-dimensional structural information. During the multi-layer graph convolution operation of the graph neural network, feature maps of different scales and semantic levels are generated.

[0222] The adaptive feature fusion mechanism includes the following steps:

[0223] The feature maps of different scales and semantic levels generated by the multi-layer graph convolution operation are analyzed, and the importance score of each feature map is calculated. The calculation formula of the importance score is as follows:

[0224]

[0225] Where L is the total number of graph convolution layers, l is the lth graph convolution layer of the graph neural network, and F l is the feature map of the lth convolutional layer, ω i The weights are pre-set based on the degree of correlation between each element in the feature map and the indicators related to the maturity of fresh tobacco leaves, and ReLU is the activation function;

[0226] According to the calculated importance score, the feature maps of different levels are fused by weighted fusion. The fused feature map F f for:

[0227]

[0228] Where S l is the importance score of the l-th graph convolution layer, F l is the feature map of the l-th graph convolution layer, and L is the total number of graph convolution layers.

[0229] This adaptive fusion strategy can dynamically adjust the contribution of features at different levels according to the characteristics of the three-dimensional structural features of tobacco leaves, enabling the model to better capture multi-scale features related to maturity.

[0230] In addition, the adaptive feature fusion mechanism also considers different types of feature fusion. The 3D structural information of fresh tobacco leaves not only contains spatial geometric features, but also some implicit features related to the growth environment, such as the impact of light distribution on the surface morphology of tobacco leaves.

[0231] In this embodiment, the adaptive feature fusion mechanism further includes an environmental feature embedding submodule (EFES), which uses environmental sensors to collect environmental features, including light intensity data and temperature data;

[0232] The environmental features are encoded into low-dimensional vectors and fused with the spatial structure features extracted by the graph neural network to generate a new feature map that combines the spatial structure features and the environmental features. The calculation formula for the new fused feature map that combines the spatial structure features and the environmental features is as follows:

[0233] F new =F f +γ·E

[0234] Where, F f The fused feature map is formed by fusing the feature maps of all levels of graph convolution layers in the graph neural network in a weighted fusion manner; E is a low-dimensional vector encoded by environmental features; γ is a learnable fusion coefficient used to control the influence of environmental features on the final feature representation.

[0235] Through the design of this adaptive feature fusion module, the graph neural network (GNN) in this solution can efficiently fuse various features from the three-dimensional structural information of fresh tobacco leaves, comprehensively improving its ability to process complex three-dimensional structural features. This innovative fusion strategy enables the model to more accurately extract maturity-related features when dealing with tobacco leaves grown in different environments, providing a more comprehensive and reliable basis for subsequent maturity assessment. Compared to existing technologies, it represents a significant improvement in the comprehensiveness and effectiveness of feature fusion.

[0236] The above are only embodiments of the present invention. Common knowledge such as the known specific structures and characteristics in the scheme are not described in detail here. Ordinary technicians in the field are aware of all common technical knowledge in the technical field of the invention before the application date or priority date, can obtain all existing technologies in the field, and have the ability to apply conventional experimental means before that date. Ordinary technicians in the field can improve and implement this scheme in combination with their own abilities under the inspiration given by this application. Some typical known structures or known methods should not become obstacles for ordinary technicians in the field to implement this application. It should be pointed out that for those skilled in the art, without departing from the structure of the present invention, several variations and improvements can be made, which should also be regarded as the scope of protection of the present invention. These will not affect the effect of the implementation of the present invention and the practicality of the patent. The scope of protection required by this application shall be based on the content of its claims, and the specific implementation methods and other records in the specification can be used to interpret the content of the claims.

Claims

1. A method for assessing the maturity of fresh tobacco leaves using images, characterized by: The following steps are involved: S1, acquiring fresh tobacco leaf image data, wherein the fresh tobacco leaf image data includes multispectral image data and three-dimensional structure information data; S2, preprocessing the acquired fresh tobacco leaf image data and building a fresh tobacco leaf image sample library; S3, constructing a maturity assessment model, wherein the maturity assessment model includes a multimodal feature extraction layer, a spatiotemporal attention fusion layer, and a multi-task prediction layer; S4, uses the fresh tobacco leaf image sample library to train the maturity assessment model; S5, using the trained maturity assessment model, performs maturity assessment on the fresh tobacco leaf image to be assessed and generates an assessment result.

2. The method for evaluating the maturity of fresh tobacco leaves according to claim 1, wherein: S1 includes the following steps: S11, collecting fresh tobacco leaf images using multispectral imaging technology to obtain multispectral image data of the fresh tobacco leaves; S12, scanning the fresh tobacco leaves using a three-dimensional laser scanning technology to obtain three-dimensional structural information data of the fresh tobacco leaves.

3. The method for evaluating the maturity of fresh tobacco leaves according to claim 2, wherein: The multimodal feature extraction layer includes a convolutional neural network and a graph neural network; The convolutional neural network is used to extract multispectral features from multispectral image data; The graph neural network is used to extract spatial structural features from three-dimensional structural information data.

4. The method for evaluating the maturity of fresh tobacco leaves according to claim 3, wherein: In the input layer of the graph neural network, coding items based on the dynamic changes of the fresh tobacco leaves' growth cycle are used to capture the dynamic change trends of the same spatial position features of fresh tobacco leaves at different growth stages.

5. The method for evaluating the maturity of fresh tobacco leaves according to claim 3, wherein: In the graph convolution layer of the graph neural network, a distance weighting factor is used to adjust the feature update of the graph convolution operation.

6. The method for evaluating the maturity of fresh tobacco leaves according to claim 1, wherein: The spatiotemporal attention fusion layer uses a bidirectional long short-term memory network combined with an attention mechanism to fuse the features extracted from the multimodal feature extraction layer. The fusion process includes: Arrange the features extracted from the multimodal feature extraction layer into a feature vector sequence according to the time sequence of the growth cycle; A bidirectional long short-term memory network is used to capture the dependencies in the feature vector sequence and generate a hidden state vector; A multiplicative attention mechanism is used to calculate the correlation score between the feature vector of each time step in the feature vector sequence and the corresponding hidden state vector; Normalize the relevance scores of all time steps to generate attention weights; The feature vectors are weighted summed using attention weights to generate a context vector.

7. The method for evaluating the maturity of fresh tobacco leaves according to claim 1, wherein: The multi-task prediction layer predicts the maturity category and maturity quantification value of the fresh tobacco leaf image through a fully connected network and generates a prediction result.

8. The method for evaluating the maturity of fresh tobacco leaves according to claim 1, wherein: In S4, the maturity assessment model training process includes: S41, dividing the fresh tobacco leaf image sample library into a training set, a validation set, and a test set; S42, construct comprehensive loss function; S43, based on the Bayesian optimization algorithm, constructs a Gaussian process model. The Gaussian process model is used to model the relationship between the hyperparameters of the maturity assessment model and the comprehensive loss function, and the optimal hyperparameter combination is selected. S44, using the mini-batch gradient descent method based on gradient accumulation, adjusts the maturity assessment model parameters according to the number of mini-batch samples and the number of gradient accumulation steps, and calculates the average gradient of the accumulated gradient; S45, based on the average gradient of the accumulated gradient, uses the adversarial network to update the maturity evaluation model parameters; S46, minimize the comprehensive loss function and screen the final optimal hyperparameter combination.

9. The method for evaluating the maturity of fresh tobacco leaves according to claim 1, wherein: In S5, the Monte Carlo method is used to conduct confidence assessment on the evaluation results and generate a comprehensive confidence score.

10. A fresh tobacco leaf maturity image assessment system, using the fresh tobacco leaf maturity image assessment method according to any one of claims 1 to 9, characterized in that: include: A data acquisition module, configured to acquire fresh tobacco leaf image data, wherein the fresh tobacco leaf image data includes multispectral image data and three-dimensional structure information data; A data processing module is used to pre-process the acquired fresh tobacco leaf image data and construct a fresh tobacco leaf image sample library; A model construction module is used to construct a maturity assessment model, wherein the maturity assessment model includes a multimodal feature extraction layer, a spatiotemporal attention fusion layer, and a multi-task prediction layer; A model training module is used to train a maturity assessment model using a fresh tobacco leaf image sample library; The maturity assessment module is used to use the trained maturity assessment model to perform maturity assessment on the fresh tobacco leaf images to be assessed and generate assessment results.

Images