Method for predicting ripening degree of dry-cured ham based on feature voting gated multimodal fusion

Abstract

The application discloses a dry-cured ham maturity prediction method based on feature voting gated multimodal fusion, and belongs to the technical field of machine vision and food detection. The method comprises the following steps: collecting machine vision images and near-infrared spectrum data of the ham, and respectively performing pretreatment; extracting visual depth features through a deep residual network embedded with an attention mechanism, and extracting spectrum depth features through a multi-scale one-dimensional convolutional neural network; mapping the two types of features to a common subspace and splicing into a joint feature vector; adopting a TopK voting gating mechanism, dynamically screening key features through context enhancement and Bayesian uncertainty scoring, and forming a sparse fusion feature vector; and finally, based on a base classifier group composed of a support vector machine, a random forest and a multilayer perception mechanism, combining a logistic regression meta-learner to perform integrated decision-making, and outputting a maturity prediction result. The application realizes balanced complementation and adaptive fusion of multimodal features, and significantly improves the prediction accuracy and model generalization ability.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of machine vision and food processing inspection, specifically involving a method for predicting the maturity of dry-cured ham based on feature voting gating multimodal fusion. Background Technology

[0002] Determining the maturity of dried ham is a core quality control step in its food processing, directly impacting product quality and commercial value. Traditional methods rely primarily on manual sensory evaluation or single physicochemical tests, such as moisture content, fat oxidation, and protein hydrolysis. These methods often suffer from subjectivity, low efficiency, and sample destruction, thus having limitations. Traditional manual assessment relies on experienced technicians who determine the maturity stage based on subjective indicators like color, firmness, and aroma. This method is highly susceptible to individual differences and cannot be quantified. Traditional dried ham testing techniques face several limitations in terms of technology and the application of modern analytical techniques, hindering the modernization and large-scale production of dried ham processing technology.

[0003] In recent years, machine vision and near-infrared spectroscopy have made progress in food quality inspection. In tasks such as food quality inspection and ripeness classification, multi-source information, such as visible light images, near-infrared spectroscopy, and sampling from different angles and locations, is often used to improve robustness. Machine vision excels at characterizing color and texture differences caused by anisotropy in appearance; near-infrared spectroscopy can reflect changes in internal components such as moisture, fat, and protein. However, relying solely on machine vision can only extract surface color and texture features and cannot perceive changes in internal chemical composition; while relying solely on near-infrared spectroscopy can detect internal components, it is easily affected by surface roughness and light scattering, and lacks spatial distribution information. In actual training, a single feature source often dominates, resulting in low participation of other feature sources during multimodal fusion. This leads to unbalanced feature utilization and fails to truly achieve complementary advantages. Simple full feature fusion introduces irrelevant background noise or redundant bands, reducing the model's signal-to-noise ratio and recognition accuracy. In addition, existing methods mostly rely on a single classifier. In complex environments, the generalization ability of a single model is limited, and it is prone to overfitting or sensitivity to specific noise, resulting in large fluctuations in prediction results between adjacent maturity months. Summary of the Invention

[0004] The technical problem to be solved by this invention is that existing non-destructive testing methods based on machine vision and near-infrared spectroscopy have problems such as insufficient multimodal feature fusion, unbalanced feature utilization, and weak generalization ability due to reliance on a single model in the prediction of the maturity of dry-cured ham. As a result, the prediction accuracy is not high and confusion is easily generated between adjacent maturity levels.

[0005] To address the aforementioned problems, this invention provides a method for predicting the maturity of dried ham based on feature voting gating multimodal fusion. This method utilizes a Top-K voting gating mechanism, constructing a voting gating network to adaptively select key features, thus solving the problem of insufficient utilization of near-infrared or visible light features of ham and shallow fusion. Furthermore, it employs a Stacking hybrid ensemble strategy to integrate the decision advantages of heterogeneous classifiers, addressing the weak generalization ability of a single model. The specific steps are as follows: Step S1: Collect machine vision image data and near-infrared spectral data of complete dry-cured ham samples with different fermentation and maturation times; Step S2: The machine vision image data is enhanced using the Retinex algorithm to extract the region of interest and uniformly adjust the image size; the near-infrared spectral data is preprocessed by multivariate scattering correction and Savitzky-Golay smoothing filtering in sequence. Step S3: Input the preprocessed image data into a deep residual network with an embedded attention mechanism to extract visual depth feature vectors; Step S4: Input the preprocessed spectral data into a one-dimensional convolutional neural network to extract the spectral depth feature vector; Step S5: Using two independent projection heads, the visual depth feature vector and the spectral depth feature vector are mapped to a common subspace of the same dimension, and the mapped features are concatenated to obtain a joint feature vector; Step S6: For the joint feature vector, local context features are extracted sequentially through one-dimensional convolution, and global context features are extracted through multi-head self-attention mechanism. The two are then fused to enhance information. The importance score of the enhanced features is calibrated through Bayesian uncertainty estimation. The Top-K key features are dynamically selected based on the calibrated scores to generate a sparse fusion feature vector. Step S7: Simultaneously input the sparse fused feature vector into a base classifier group composed of support vector machine, random forest and multilayer perceptron to obtain the primary probability distribution output by each base classifier; after concatenating the primary probability distributions, input them into the logistic regression meta-learner to obtain the final dry-cured ham maturity prediction result.

[0006] Optionally, step S3 specifically includes: Convert the preprocessed RGB format image to HSV color space and CIE L*a*b* color space; Calculate the first, second, and third color moments for each channel of the HSV color space and the CIE L*a*b* color space respectively; Extract texture features based on the gray-level co-occurrence matrix from the RGB format image; The calculated color moment features are concatenated with the extracted texture features to form an explicit feature vector; The original RGB image is input into the deep residual network with an embedded attention mechanism, which outputs a first visual feature vector. The explicit feature vector is concatenated with the first visual feature vector to output the final visual depth feature vector.

[0007] Optionally, in step S4, the one-dimensional convolutional neural network uses convolutional kernels of various sizes to capture spectral peak features of different widths, and the lengths of the convolutional kernels include 3, 5, and 7. Optionally, in step S4, the spectral feature bands of interest when extracting the spectral depth feature vector include: water feature region 960 nm to 980 nm, protein feature region 1020 nm and 1510 nm, and lipid oxidation region 1210 nm and 1720 nm to 1760 nm.

[0008] Optionally, in step S5, the projection head consists of a first fully connected layer, a batch normalization layer, an activation function layer, and a second fully connected layer connected in sequence.

[0009] Optionally, in step S6, the local context features extracted by one-dimensional convolution and the global context features extracted by multi-head self-attention mechanism are fused. Specifically, the local context features and the global context features are connected by residuals and then subjected to layer normalization to obtain a context-aware feature vector.

[0010] Optionally, in step S6, the importance score is calibrated using Bayesian uncertainty estimation, specifically as follows: The importance score of each feature dimension in the context-aware feature vector is predicted by using a first evaluation head network and a second evaluation head network in parallel. and uncertainty variance ; The calibrated score is calculated for each feature dimension using the exponential decay function. The calculation formula is: ,in This is a preset adjustment factor.

[0011] Optionally, in step S6, the Top-K key features are dynamically selected based on the calibrated scores to generate a sparse fusion feature vector, specifically as follows: The score vector containing scores calibrated across all feature dimensions Sort the scores in descending order and take the Kth largest score as the dynamic threshold. ; Generate a binary mask vector m, wherein the scoring vector The score is greater than or equal to the dynamic threshold. The mask value corresponding to the feature dimension is 1, otherwise it is 0; The joint feature vector The scoring vector The binary mask vector m is weighted and fused to obtain a sparse fused feature vector. The calculation formula is: ,in This indicates element-wise multiplication.

[0012] Optionally, in step S2, the image size is uniformly adjusted to 224 pixels × 224 pixels; in the training process of step S7, K-fold cross-validation is used to generate training data for training the logistic regression meta-learner.

[0013] Optionally, it also includes: calculating modal characteristics using the balance degree MCB; The modal characteristics are calculated using the equilibrium degree MCB formula as follows: ,in, The number of the Top-K key features. A set of indices for visual feature dimensions. A set of indices for spectral feature dimensions. Let be the binary mask vector.

[0014] Compared with the prior art, the present invention has the following beneficial effects: 1. Effectively fusing multimodal complementary information solves the problem of insufficient feature fusion and improves feature discrimination: This invention targets the appearance and internal chemical changes of ham, respectively extracting visual depth features including color and texture through a deep residual network with an embedded attention mechanism, and extracting spectral depth features reflecting key components such as water, protein, and fat oxidation through a multi-scale one-dimensional convolutional neural network. Using independent projectors, the two types of heterogeneous features are mapped to a common subspace and concatenated, achieving effective alignment and complementarity of appearance and internal component information at the semantic level. This method overcomes the limitations of single-modal information, providing a more complete and more correlated fusion feature foundation for accurate discrimination from the source.

[0015] 2. Adaptive feature selection via Top-K voting gating solves the problem of unbalanced feature utilization and suppresses noise interference: The core of this invention lies in the Top-K dynamic feature selection mechanism. This mechanism first performs local convolution and global self-attention context enhancement on the joint feature vector to uncover deep correlations between features. Subsequently, it innovatively introduces Bayesian uncertainty estimation, predicts the importance score and uncertainty variance of features through a parallel evaluation head network, and uses an exponential decay function to penalize high-uncertainty features. Finally, based on the calibrated scores, the Top-K key features are dynamically selected to generate a sparse fusion feature vector. This process can adaptively focus on the most relevant and reliable combination of visual and spectral features for each sample, automatically filtering out redundant and high-noise features, thereby significantly improving the balanced utilization of multimodal features and effectively avoiding model overfitting caused by irrelevant features or noise.

[0016] 3. By leveraging the advantages of heterogeneous classifiers through a Stacking ensemble learning strategy, this invention addresses the weak generalization ability of single models and improves prediction accuracy and stability: To address the ambiguity of decision boundaries caused by gradual maturity changes, this invention employs a Stacking ensemble strategy. Sparse fused feature vectors are simultaneously input into a heterogeneous base classifier group composed of support vector machines, random forests, and multilayer perceptrons, fully utilizing their decision diversity to obtain a preliminary probability distribution. Then, a logistic regression meta-learner learns and fuses these preliminary distributions to form the final decision. This two-level decision framework effectively compensates for the limitations of single models in handling sample heterogeneity and subtle differences between adjacent maturity levels, significantly improving the overall generalization ability, classification stability, and prediction accuracy in complex scenarios, and significantly reducing misclassifications between adjacent maturity levels.

[0017] In summary, this invention systematically solves the core problems of traditional methods in predicting the maturity of dry-cured ham, such as unbalanced feature utilization, poor fusion effect, and weak model generalization ability, through a complete technical solution from multimodal feature deep extraction, dynamic adaptive fusion to heterogeneous model intelligent decision-making. It provides an efficient, accurate, and stable solution for intelligent quality inspection in the food processing field. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1This is a schematic diagram of the overall process of the method for predicting the maturity of dry-cured ham based on feature voting gating multimodal fusion provided by the present invention.

[0020] Figure 2 This is a schematic diagram of the network structure provided by the present invention.

[0021] Figure 3 These are partial original images of dry-cured ham samples at different stages of maturity, acquired by a machine vision system in an embodiment of the present invention.

[0022] Figure 4 This is the original near-infrared spectrum curve of the dry-cured ham sample in the embodiment of the present invention.

[0023] Figure 5 This is a comparison chart of the confusion matrix results of various comparison models and the final model on the test set in the embodiments of the present invention. Attached image description: Figure 5 The correct answer is: (A) ResNet18 model built based on image data; (B) 1DCNN model built based on near-infrared data; (C) model built based on feature splicing and fusion; (D) model built based on decision weighting and fusion; (E) model built based on Top-K voting + Stacking decision. Detailed Implementation

[0025] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] This embodiment provides a method for predicting the maturity of dry-cured ham based on feature voting gating multimodal fusion. The overall process of this method is as follows: Figure 1 As shown, its core network architecture is as follows: Figure 2 As shown.

[0027] Step S1: In this embodiment, dry-cured hams from a standard fermentation workshop of a ham factory in Jinhua, Zhejiang Province, were selected as the experimental subjects. The samples covered five fermentation maturity levels: 3 months, 4 months, 5 months, 6 months, and 7 months. 120 hams with intact appearance and no damage were randomly selected from each level, totaling 600 samples. Subsequently, the data from all samples were randomly divided into a training set and a test set at an 8:2 ratio. The training set was used for model building and parameter learning, while the test set was used for final performance evaluation.

[0028] Data acquisition consists of two parallel parts: machine vision image acquisition and near-infrared spectral acquisition.

[0029] Machine vision image acquisition was conducted in a darkroom environment with a standard D65 light source to eliminate ambient light interference. A high-resolution industrial camera (2048×2048 resolution) was used to capture RGB images of each ham from multiple angles, ensuring complete capture of its surface color and texture information. Examples of some of the acquired raw images representing different stages of ham ripeness are shown below. Figure 3 As shown, it can be observed that as the maturation time increases, the surface color of the ham gradually darkens and the texture changes.

[0030] Near-infrared spectroscopy was performed using a portable spectrometer covering the wavelength range of 900 nm to 1700 nm. A fiber optic diffuse reflection probe was used, closely fitted to the ham surface, and 18 representative points were selected for spectral scanning. Spectra at each point were collected three times and averaged to reduce random error; the average spectrum of the 18 points was then used as the raw spectral data for the ham sample. A typical raw near-infrared spectral curve for a dry-cured ham is shown below. Figure 4 As shown, the horizontal axis represents wavelength and the vertical axis represents absorbance. The shape of the curve reflects the absorption characteristics of near-infrared light by the chemical components inside the sample.

[0031] Step S2: Raw data usually contains various interferences and needs to be preprocessed to improve data quality.

[0032] For machine vision images, the Retinex algorithm is first used for image enhancement. This algorithm effectively compensates for uneven lighting and highlights the true colors and details of the image. Next, a thresholding segmentation algorithm is used to accurately extract the muscle region of the ham from the background, i.e., the region of interest. Finally, all extracted regions of interest images are uniformly scaled to 224 pixels × 224 pixels to meet the input size requirements of subsequent deep network processing. This series of processes provides clear and standardized input for visual feature extraction.

[0033] For near-infrared spectral data, preprocessing primarily targets physical scattering and noise. First, multivariate scattering correction is applied to eliminate light scattering effects caused by variations in sample particle size and surface roughness, allowing the spectrum to more accurately reflect chemical composition information. Then, the Savitzky-Golay smoothing filter method is used to remove high-frequency random noise from the spectrum, resulting in a smoother spectral curve that facilitates subsequent feature analysis.

[0034] Step S3: This step aims to extract high-level visual features strongly correlated with maturity from the preprocessed image. The specific implementation process is as follows: First, explicit feature calculation and fusion are performed. The size-normalized RGB image is converted to the HSV color space and the CIE L*a*b* color space, respectively. For the H, S, V channels of HSV and the L*, a*, b* channels of CIE L*a*b*, the first-order color moments (mean values) are calculated respectively. Second-order color moment (standard deviation) ) and third-order color moments (skewness) ),total 3D color statistical characteristics. Let... For the first Channel in pixels The value at the point, the mean with standard deviation The calculation formula is:

[0035]

[0036] Here, H and W represent the height and width of the region of interest (ROI) in the image, respectively. Simultaneously, a Gray-Level Co-occurrence Matrix (GLCM) is constructed from the original RGB image, and four key texture statistics—energy, contrast, correlation, and entropy—are extracted. Subsequently, the calculated 27-dimensional color moment features are concatenated with the 4-dimensional texture features to obtain a 31-dimensional explicit feature vector. This step integrates multi-dimensional color statistics and micro-texture information, providing semantically rich and structured initialization input for the deep network.

[0037] Next, to extract higher-order semantic information from the image, the original RGB image is input into a deep residual network (ResNet-CBAM) with an embedded attention mechanism. This embodiment preferably uses ResNet-18 as the backbone network, with a convolutional block attention module (CBAM) embedded after its residual blocks. This CBAM module sequentially includes a channel attention submodule and a spatial attention submodule. Channel attention adaptively calibrates the weights of each feature channel, emphasizing information-rich channels; spatial attention focuses on key spatial locations in the image. In this way, the network can automatically learn and focus on the areas on the ham surface most relevant to maturity (such as areas of muscle texture variation), while suppressing responses from interfering areas such as fat reflection. The network ultimately outputs a 512-dimensional visual feature vector. The aforementioned 31-dimensional explicit feature vector is concatenated with the 512-dimensional visual depth feature vector along the feature channel dimension to form a 543-dimensional comprehensive visual feature vector, denoted as . This method combines the explicitness of handcrafted features with the adaptive focusing capability of deep attention networks, significantly improving the representation quality of visual features.

[0038] Step S4: This step aims to extract depth features reflecting changes in internal chemical composition from the preprocessed spectrum. The specific implementation process is as follows: First, we focused on mechanism-driven characteristic wavelengths. Based on the core biochemical changes in cured ham during the ripening process, such as water loss, protein hydrolysis, and lipid oxidation, we focused on the following specific spectral ranges during feature extraction: the 960 nm to 980 nm water characteristic region, corresponding to the combination and overtone absorption of OH bonds, reflecting changes in water content; the 1020 nm and 1510 nm protein characteristic regions, corresponding to the overtone absorption of NH bonds, reflecting protein degradation and the generation of free amino acids; and the 1210 nm and 1720 nm to 1760 nm lipid oxidation region, corresponding to the stretching vibration absorption of CH bonds, reflecting intramuscular fat content and its oxidation state. This targeted focus ensured the physicochemical significance of the feature extraction.

[0039] Then, the full-band spectral data is input into a specially designed one-dimensional convolutional neural network (1D-CNN) for deep feature extraction. This network contains three parallel branches, using convolutional kernels of sizes 3, 5, and 7 to extract peak shape features at different scales. Smaller kernels are better at capturing sharp, narrow chemical bond peaks on the spectral curve; larger kernels help perceive gentle baseline drift trends. Through this multi-scale convolutional operation, the network can adaptively learn complex patterns related to maturity in both local spectral shape and global trends. Finally, the 1D-CNN outputs a 128-dimensional spectral depth feature vector, denoted as... .

[0040] Step S5: This step aims to resolve visual features. With spectral depth features The problem of direct fusion of heterogeneous and heterogeneous dimensions is addressed. Two independent projection heads with identical structures are designed. Each projection head consists of two fully connected layers, with the following operation sequence: first fully connected layer (FC), batch normalization layer (BN), ReLU activation function layer, and second fully connected layer (FC). The first projection head will project a 543-dimensional image... Mapped to a common subspace, the second projection head will project 128 dimensions. Mapped to the same common subspace. In this embodiment, the dimension of this common subspace is set to 256. The features obtained after mapping are denoted as follows: and Finally, and The features are concatenated along the feature dimensions to form a 512-dimensional (i.e., 256-dimensional + 256-dimensional) joint feature vector. .

[0041] This step aligns features from different sources and distributions into the same semantic space through nonlinear transformation, unifies the dimensions, and lays the foundation for subsequent deep fusion.

[0042] Step S6: This step is the core of the present invention for achieving efficient and balanced multimodal fusion. Its goal is to extract features from the joint feature vector. The process dynamically selects the Top-K features most critical to the current sample, as follows: Figure 2 The corresponding module is shown in the figure.

[0043] 1. Enhanced Contextual Information First, consider the joint feature vector. Enhancements are made to uncover its rich internal contextual relationships. A two-stream strategy is employed: the first stream utilizes one-dimensional convolution (Conv1D) in... The sequence is slid upwards, with the convolution kernel size set to 3 to capture local correlations and continuity between adjacent feature dimensions. The second flow introduces a multi-head self-attention mechanism... Linear mapping is used to represent query, key, and value vectors. Attention weights are calculated to model long-distance dependencies between any two dimensions of the feature vector, obtaining global contextual information. Subsequently, the local features extracted by one-dimensional convolution and the global features extracted by self-attention are added through a residual connection, followed by layer normalization, and finally fused to generate a context-rich feature vector, called a context-aware feature vector. This step significantly improves the expressive power and discriminative power of the features. This process can be represented by the following formula: Joint eigenvectors (in, (Total dimension), local features Global features will Linear mapping to query vector Key vector Value vector :

[0044]

[0045]

[0046] Calculate and fuse self-attention weights:

[0047] The two-stream features are fused using residual connectivity and layer normalization (LayerNorm):

[0048] 2. Uncertainty perception scoring calibration To address the issue that certain features may exhibit high response values ​​but low reliability due to sample heterogeneity, this invention introduces Bayesian uncertainty estimation for score calibration. Specifically, two parallel fully connected layers are constructed as evaluation heads. The first evaluation head takes a context-aware feature vector as input and outputs an importance score for each feature dimension in relation to maturity classification, denoted as . ( ,in This is the weight matrix; For bias terms; (This refers to the sigmoid function). The second evaluation head outputs the uncertainty variance for each feature dimension, denoted as... ( ,in This is the weight matrix; For bias terms; (This is a softplus function). The variance reflects the stability of the feature during training. The larger the variance, the more uncertain it is and the more likely it is to be noise.

[0049] Subsequently, the importance scores were calibrated using an exponential decay function to obtain the final score for each feature dimension. The calculation formula is:

[0050] in, It is a preset positive adjustment factor used to control the strength of the uncertainty penalty. According to this formula, even if the importance score of a feature... High, but if its uncertainty It's also very large, and its final score is... The noise level will still be significantly reduced. This mechanism effectively avoids misclassifying high-noise, high-fluctuation features as key features, thus improving the robustness of feature selection.

[0051] The adjustment factor in this embodiment The settings are optimized through experiments, and their physical significance lies in controlling the degree to which the model suppresses high-noise features. During the model training phase, [the following steps are taken]. The search range was set to [0.1, 1.0], the step size was set to 0.1, and the classification accuracy on the validation set and modal feature utilization balance (MCB) were used as joint optimization objectives. Experiments showed that when When the value is too small (e.g., less than 0.3), the penalty is too lenient, and the model cannot effectively suppress reflected noise and scattering noise in the near-infrared spectrum of machine vision images. This results in redundant information being mixed into the features selected by Top-K, and the accuracy improvement is not significant. When the value is too large (e.g., greater than 0.7), the penalty is too heavy, and the model is too harsh on the features. This leads to the loss of some effective features with slight fluctuations but containing important semantic information, causing a decrease in the Modal Feature Utilization Balance (MCB) metric and a drop in accuracy. This was determined through parameter sensitivity experiments. When the value of is between [0.3, 0.7], the model can achieve a good balance between feature purity and information richness. In this embodiment, it is preferable to set . = 0.5 to obtain the best overall performance.

[0052] 3. Top-K Dynamic Sparse Routing Based on the calibrated scoring vector Hard screening is performed to achieve feature sparsity.

[0053] First, the rating vector All scores are sorted in descending order. If the number of key features to be retained is set to K, then the Kth largest score after sorting is taken as the dynamic threshold for the current sample. .

[0054] Next, a binary mask vector m is generated based on this threshold. The mask generation rule is: for the score vector... The i-th rating ,like The corresponding mask value ;otherwise .

[0055] Finally, feature weighting and filtering are performed. The original joint feature vector is then processed. Rating vector With binary mask vector Element-wise multiplication yields the final sparse fused feature vector. The calculation formula is as follows:

[0056] Among them, symbols This represents element-wise multiplication. The result of this operation is that only the top K features by score are retained, and these feature values ​​are weighted by their importance scores; the remaining features are completely filtered out and set to zero. This process achieves adaptive feature selection based on the characteristics of each sample, significantly reducing feature dimensionality, eliminating redundancy, and ensuring that the selected features are highly discriminative and reliable.

[0057] To determine the optimal number of key features K, this invention employs a grid search strategy based on a validation set. (Joint feature vector) The total dimension is N=512. To retain discriminative information as much as possible while filtering out noise, the search range of K is set to 20% to 50% of the total dimension, i.e. The search step size was set to 2% of the total dimension, approximately 10 features. Classification accuracy was selected as the primary evaluation metric, while modal feature utilization balance (MCB) was monitored to ensure the model did not overly rely on a single modality due to inappropriate K value selection. Experimental results show that model performance initially increases and then decreases with increasing K value. When K < 130, the feature vector is excessively sparse, causing some weak signals representing changes in trace biochemical components within the cured ham to be mistakenly filtered out as redundant information, resulting in the loss of key information and underfitting. When K > 180, the gating threshold is too low, failing to effectively filter out random environmental noise introduced during machine vision and spectral acquisition, leading to a decrease in the feature signal-to-noise ratio and a drop in accuracy. Grid search revealed that when K = 154 (approximately 30% of the total dimension), the model achieved peak classification accuracy on the validation set, with the optimal MCB metric. Therefore, in this embodiment, K=154 was ultimately selected as the optimal number of features to retain. This setting filters out about 70% of redundancy and noise while retaining the key features for maturity determination to the maximum extent.

[0058] Step S7: This step further enhances the model's generalization ability and final decision accuracy through an ensemble learning strategy. See the architecture below. Figure 2 The right side.

[0059] 1. Level-0 base learner layer Construct an ensemble of three heterogeneous base classifiers: Support Vector Machine (SVM), Random Forest, and Multilayer Perceptron (MLP). These three classifiers differ significantly in their mathematical principles and decision boundary construction.

[0060] To fully leverage the decision diversity of the Stacking ensemble strategy and enable each base classifier to make judgments from different perspectives when faced with indistinguishable samples, this invention, based on validation set data and considering the distribution characteristics of the 154-dimensional sparse features obtained after Top-K filtering, performs targeted parameter tuning on the three base classifiers. The specific configuration is as follows: Support Vector Machine (SVM): To accommodate the highly nonlinear characteristics of spectral data, a radial basis function (RBF) is selected as the kernel. The penalty coefficient (C) is set to C=10 to control the penalty for misclassified samples; the kernel coefficient (Gamma) is set to gamma=0.05 to control the influence range of a single training sample.

[0061] Random Forest: The number of base estimators (n_estimators) is set to 200 decision trees to ensure model stability. The maximum tree depth (max_depth) and the maximum number of features (max_features) are both limited to 12. By limiting the tree depth and the number of features considered during splits, the correlation between decision trees is effectively reduced, thereby significantly improving the overall noise resistance and generalization performance of the random forest.

[0062] Multilayer Perceptron (MLP): The network structure employs a dual-hidden-layer design with 128 and 64 neurons respectively. The input layer receives 154-dimensional sparsely fused features, and dimensionality reduction is performed layer by layer to extract a more compact high-level representation. Activation function: The ReLU function is used across all layers to address the vanishing gradient problem and accelerate convergence. Optimizer: The Adam optimizer is selected, with an initial learning rate set to 0.001. To prevent overfitting on small sample data, a Dropout layer is introduced after the fully connected layers, with a dropout rate (p) set to 0.4; L2 regularization is added to the loss function, and the weight decay coefficient (weight_decay) is set to 1e-4.

[0063] The sparse fusion feature vector obtained in step S6 Simultaneously input these three base classifiers. Each base classifier makes an independent decision and outputs a primary probability distribution vector indicating whether a sample belongs to each maturity category. Let there be C maturity categories in total, then the probability vector output by the k-th base classifier is denoted as... , where k=1,2,3.

[0064] 2. Generation of Meta-learner Training Data To train the upper-level meta-learner and prevent overfitting, K-fold cross-validation is used to generate meta-features. The specific steps are as follows: The original training set is randomly divided into K parts (K=5 in this example). Each part is used as a validation set, and the remaining K-1 parts are used as the training set to train the three base classifiers. Then, the trained classifiers are used to predict the "out-of-bag" prediction probability of the current validation set sample. After traversing all K parts of data, the prediction probabilities of the three base classifiers for each sample in the original training set are obtained. These probability vectors are concatenated in sample order to form a new feature matrix, which serves as the training data for the meta-learner. This method ensures the purity and unbiasedness of the meta-learner training data.

[0065] 3. Level-1 Meta-learner Layer and Final Prediction A logistic regression model was selected as the meta-learner. During the training phase, the feature matrix generated by the K-fold cross-validation, which is composed of the probabilities of the three base classifiers, was input into the logistic regression model, and training was performed using the true maturity label of the sample as the supervision signal. In the prediction phase, for a new ham sample to be tested, its sparse fusion features were first obtained through steps S2 to S6. Then, it is input into the three pre-trained base classifiers to obtain three primary probability vectors. , , Concatenate these three probability vectors into a single long vector. Input the pre-trained logistic regression meta-learner. The meta-learner outputs the final probability distribution of the sample belonging to each maturity category. ( ,in This is the weight matrix; For bias terms; (This is a softmax function). Finally, the category with the highest probability value is selected as the maturity prediction result for the dried ham. This stacking strategy cleverly integrates the advantages of different models, and learns the optimal decision weights through a meta-learner, effectively improving the model's discrimination accuracy and stability when facing complex samples and adjacent maturity boundaries.

[0066] For an unknown dry-cured ham whose maturity is to be predicted, the process of applying the method of this invention is as follows: The ham is placed in a data acquisition device to obtain its machine vision image and near-infrared spectrum. A pre-trained model is then invoked to sequentially perform preprocessing, visual and spectral depth feature extraction, feature projection and joint analysis, and Top-K feature dynamic selection, resulting in a sparse fused feature vector for the sample. Subsequently, The input is fed into three saved base classifier models to obtain a preliminary probability distribution. This distribution is then concatenated and fed into a saved logistic regression meta-learner to output the final maturity prediction category. The entire process enables fast, automated, and lossless maturity determination.

[0067] To objectively evaluate the performance of the method of the present invention, the following evaluation indicators are defined and calculated in this embodiment: (1) Accuracy: The proportion of samples that are correctly classified as a whole.

[0068] (2) F1 score: the harmonic mean of precision and recall, used to comprehensively evaluate the performance of the classification model.

[0069] (3) Kappa coefficient: an index that measures the consistency between the model’s prediction results and the actual results, taking into account the impact of random consistency.

[0070] Furthermore, to quantitatively evaluate the balance of modal feature utilization by the Top-K mechanism of this invention, a modal feature utilization balance index is specifically defined. Let K be the total number of features retained after Top-K filtering, and the index set of the visual feature dimensions be... The index set of spectral feature dimensions is The binary mask vector is The formula for calculating MCB is:

[0071] The MCB value range is [0, 1]. The closer the MCB value is to 1, the closer the number of features from the visual modality and the spectral modality are among the selected Top-K key features, that is, the more balanced the feature utilization; conversely, it indicates that the model may be overly dependent on a single modality.

[0072] The model constructed in this embodiment, namely the fusion model based on Top-K voting and Stacking decision-making, was compared with the performance of several baseline models on the same test set. The comparison models included: a ResNet18 model using only machine vision features, a 1D-CNN model using only near-infrared spectral features, a simple early feature stitching fusion model, and a decision-weighted late-stage fusion model. The confusion matrix results of each model on the test set are compared as follows: Figure 5 As shown, subgraphs A to E correspond to the five models mentioned above. The quantitative comparison results are shown in the table below: Table 1. Comparison of prediction performance and feature utilization of different methods

[0073] The following conclusions can be drawn from Table 1: (1) The method proposed in this invention achieves the best performance in three core performance indicators: accuracy, F1 score and Kappa coefficient. The accuracy is as high as 98.60%, which is 3.53% higher than the second-best performance decision weighted fusion method. This directly proves the effectiveness of the overall technical solution of this invention.

[0074] (2) The method of the present invention achieved a high score of 0.92 on the modal feature utilization balance index, which is much higher than the 0.35 of the traditional feature splicing and fusion. This shows that the Top-K dynamic voting gating mechanism of the present invention is not a simple feature mixing, but can intelligently and evenly select the most discriminative and complementary feature subsets from the two modalities according to the specific situation of each sample, fundamentally solving the common problems of unbalanced feature utilization and single modality dominance in multimodal fusion.

[0075] Combination Figure 5 Visual analysis of the confusion matrix: Figure 5Models A and B, as unimodal models, suffer from severe confusion and numerous prediction errors. Models C and D, as traditional fusion models, still exhibit significant misclassification regions between adjacent maturity levels. In contrast, Model E, corresponding to the method of this invention, shows highly concentrated and bright values ​​on the diagonal of its confusion matrix, while the off-diagonal elements (especially those representing confusion between adjacent months) are very dim, almost zero. This intuitively and powerfully demonstrates that the method of this invention can significantly reduce misclassifications between adjacent maturity levels, solving the technical challenge of blurred decision boundaries caused by gradual maturity changes and similar physicochemical indicators, exhibiting superior classification accuracy and robustness.

[0076] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method for predicting the maturity of dry-cured ham based on feature voting gating multimodal fusion, characterized in that, Includes the following steps: Step S1: Collect machine vision image data and near-infrared spectral data of complete dry-cured ham samples with different fermentation and maturation times; Step S2: The machine vision image data is enhanced using the Retinex algorithm to extract the region of interest and uniformly adjust the image size; the near-infrared spectral data is preprocessed by multivariate scattering correction and Savitzky-Golay smoothing filtering in sequence. Step S3: Input the preprocessed image data into a deep residual network with an embedded attention mechanism to extract visual depth feature vectors; Step S4: Input the preprocessed spectral data into a one-dimensional convolutional neural network to extract the spectral depth feature vector; Step S5: Using two independent projection heads, the visual depth feature vector and the spectral depth feature vector are mapped to a common subspace of the same dimension, and the mapped features are concatenated to obtain a joint feature vector; Step S6: For the joint feature vector, local context features are extracted sequentially through one-dimensional convolution, and global context features are extracted through multi-head self-attention mechanism. The two are then fused to enhance information. The importance score of the enhanced features is calibrated through Bayesian uncertainty estimation. The Top-K key features are dynamically selected based on the calibrated scores to generate a sparse fusion feature vector. Step S7: Simultaneously input the sparse fused feature vector into a base classifier group composed of support vector machine, random forest and multilayer perceptron to obtain the primary probability distribution output by each base classifier; after concatenating the primary probability distributions, input them into the logistic regression meta-learner to obtain the final dry-cured ham maturity prediction result.

2. The method according to claim 1, characterized in that, Step S3 specifically includes: Convert the preprocessed RGB format image to HSV color space and CIE L*a*b* color space; Calculate the first, second, and third color moments for each channel of the HSV color space and the CIE L*a*b* color space respectively; Extract texture features based on the gray-level co-occurrence matrix from the RGB format image; The calculated color moment features are concatenated with the extracted texture features to form an explicit feature vector; The original RGB image is input into the deep residual network with an embedded attention mechanism, which outputs a first visual feature vector. The explicit feature vector is concatenated with the first visual feature vector to output the final visual depth feature vector.

3. The method according to claim 1, characterized in that, In step S4, the one-dimensional convolutional neural network uses convolutional kernels of various sizes to capture spectral peak features of different widths, and the lengths of the convolutional kernels include 3, 5 and 7.

4. The method according to claim 3, characterized in that, In step S4, the spectral feature bands of interest when extracting the spectral depth feature vector include: water feature region 960 nm to 980 nm, protein feature region 1020 nm and 1510 nm, and lipid oxidation region 1210 nm and 1720 nm to 1760 nm.

5. The method according to claim 1, characterized in that, In step S5, the projection head consists of a first fully connected layer, a batch normalization layer, an activation function layer, and a second fully connected layer connected in sequence.

6. The method according to claim 1, characterized in that, In step S6, the local context features extracted by one-dimensional convolution and the global context features extracted by multi-head self-attention mechanism are fused. Specifically, the local context features and the global context features are connected by residuals and then subjected to layer normalization to obtain a context-aware feature vector.

7. The method according to claim 6, characterized in that, In step S6, the importance score is calibrated using Bayesian uncertainty estimation, specifically as follows: The importance score of each feature dimension in the context-aware feature vector is predicted by using a first evaluation head network and a second evaluation head network in parallel. and uncertainty variance ; The calibrated score is calculated for each feature dimension using the exponential decay function. The calculation formula is: ,in This is a preset adjustment factor.

8. The method according to claim 7, characterized in that, In step S6, the Top-K key features are dynamically selected based on the calibrated scores to generate a sparse fusion feature vector, specifically as follows: The score vector containing scores calibrated across all feature dimensions Sort the scores in descending order and take the Kth largest score as the dynamic threshold. ; Generate a binary mask vector m, wherein the scoring vector The score is greater than or equal to the dynamic threshold. The mask value corresponding to the feature dimension is 1, otherwise it is 0; The joint feature vector The scoring vector The binary mask vector m is weighted and fused to obtain a sparse fused feature vector. The calculation formula is: ,in This indicates element-wise multiplication.

9. The method according to claim 1, characterized in that, In step S2, the image size is uniformly adjusted to 224 pixels × 224 pixels; in the training process of step S7, K-fold cross-validation is used to generate training data for training the logistic regression meta-learner.

10. The method according to claim 8, characterized in that, Also includes: Modal characteristics are calculated using the equilibrium degree MCB; The modal characteristics are calculated using the equilibrium degree MCB formula as follows: ,in, The number of the Top-K key features. A set of indices for visual feature dimensions. A set of indices for spectral feature dimensions. Let be the binary mask vector.

Images