Pork quality index accurate determination method and system based on multi-modal spectrum fusion

Abstract

The invention discloses a pork quality index accurate determination method and system based on multi-modal spectrum fusion. Based on visible near-infrared imaging hyperspectrum, near-infrared imaging hyperspectrum and Raman spectrum data of a longissimus dorsi and waist muscle sample of a pig, accurate determination of key pork quality indexes of the pig is realized through a multi-modal fusion network SF-Net, a set of pork quality index accurate determination system is constructed, the system can automatically collect three modal spectrums of a pork sample, and the accuracy of the pork quality indexes is improved. Spectral features and frequency features of meat samples are analyzed and extracted, and feature fusion is performed by using a multi-modal fusion network, so that accurate measurement of the key meat quality indexes of the pigs is realized. Compared with the prior art, on the basis of realizing lossless and efficient determination of the pig key meat quality indexes, the accuracy of meat quality index determination is further improved. According to the method, the meat quality indexes can be accurately measured, meanwhile, the multiple meat quality indexes can be synchronously measured, and a new method is provided for online real-time intelligent evaluation of the meat quality.

Description

Technical Field

[0001] The present invention belongs to the fields of animal husbandry automation, artificial intelligence in production, and agricultural technology promotion, and specifically relates to a method and system for accurately measuring pork quality indicators based on multimodal spectral fusion. Background Art

[0002] Pork, favored by consumers for its high-quality protein, iron, zinc, and other trace elements, is the largest meat commodity produced and consumed in my country. Pork quality not only impacts consumer safety, health, and consumption, but is also directly linked to the economic benefits, market competitiveness, and sustainable development of the livestock industry. Pig breeding is an important means of improving pork quality, and the precise measurement of meat quality indicators is a core component of pig breeding. Accurately measuring meat quality indicators will facilitate the construction of a large-scale meat quality phenotypic database, helping to analyze the genetic regulatory mechanisms underlying complex meat quality traits and accelerating the development and selection of high-quality pig breeds.

[0003] Patent publication number CN118501089 proposes a nondestructive meat quality testing method and system based on near-infrared spectroscopy. This method obtains the type of fresh meat to be tested and near-infrared spectral data, then extracts characteristic wavelengths from the near-infrared spectral data and combines them with a quality scoring model to generate a quality score. This method eliminates the need for analyzing corresponding chemical indicators, thereby improving detection efficiency and reducing the need for specialized personnel, making fresh meat testing more universal. Patent publication number CN119688666A proposes a Raman spectroscopy method for testing the quality of ground pork gel after starch treatment. This method first prepares a ground pork gel containing starch, then measures the strength and water holding capacity of the ground pork gel. Raman spectral data is simultaneously collected, and a quantitative prediction model using uninformative variable elimination and support vector machine (SVM) is constructed. This model effectively captures the complex correlation between spectra and ground pork gel quality, enabling faster prediction of ground pork gel quality compared to traditional physical and chemical testing. However, the two aforementioned methods for predicting pork quality through spectroscopy use only a single spectrum and predict only a few indicators, resulting in insufficient model prediction accuracy and a limited range of practical applications.

[0004] A comprehensive analysis of existing technologies shows that traditional methods for determining meat quality indicators have complex sample pre-treatment processes, limited detection throughput, and irreversible damage to samples. Although this method is highly accurate, it is time-consuming, labor-intensive, costly, and subjective, and cannot meet the needs of rapid determination of various meat quality indicators in a short period of time. With the development of new non-destructive testing technologies such as VNIR, NIR, and Raman, combined with chemometrics, deep learning and other methods, new means have been provided for the non-destructive determination of meat quality. These methods can objectively and efficiently determine the quality of meat. However, existing methods for determining meat quality indicators based on spectral technology still have problems such as limited information captured by a single spectrum and few prediction indicators, which greatly affect their prediction accuracy and practical use value. Summary of the Invention

[0005] (1) Technical issues to be resolved The technical problems to be solved by the present invention include the following aspects. Traditional meat quality index determination methods have problems such as limited detection flux and irreversible sample destruction. These methods are time-consuming, labor-intensive, costly, and somewhat subjective. Although spectral-based meat quality index determination can achieve non-destructive and efficient determination of meat quality indicators, the accuracy is still not high, and most of them predict a single indicator, resulting in a small scope of application of the model and difficulty in accurately measuring various meat quality indicators from multiple angles.

[0006] (2) Technical solution In order to solve the above problems, the present invention provides the following technical solutions, and proposes a method and system for accurately measuring pork quality indicators based on multimodal spectral fusion. The specific solution is as follows.

[0007] A method for accurately measuring pork quality indicators based on multimodal spectral fusion, comprising the following steps: Step S1, collecting visible near-infrared imaging hyperspectral VNIR-HSI, near-infrared imaging hyperspectral NIR-HSI, and Raman spectroscopy data of a pig longissimus dorsi muscle sample, and manually measuring multiple meat quality indicators including intramuscular fat and pH value of the meat sample; Step S2, performing modal normalization processing on the collected VNIR-HSI, NIR-HSI, and Raman spectral data, uniformly reducing the dimensionality to one dimension, and obtaining three one-dimensional spectral data of the meat sample; Step S3, performing frequency domain conversion on the three one-dimensional spectral data obtained in step S2 to obtain frequency data of the three spectra of the meat sample; Step S4, using the three one-dimensional spectral data obtained in step S2 as input, constructing a spectral feature extractor, performing regression prediction on the multiple meat quality indicators described in step S1, and obtaining spectral features of the meat sample; Step S5, using the frequency data of the three spectra obtained in step S3 as input, constructing a frequency feature extractor, performing regression prediction on the multiple meat quality indicators described in step S1, and obtaining the frequency features of the meat sample; Step S6: Using the spectral and frequency data of the three modalities obtained in steps S2 and S3 as input data, constructing a pork quality index prediction network SF-Net based on multimodal spectral fusion, obtaining the spectral and frequency features of the meat sample based on the spectral and frequency feature extractors constructed in steps S4 and S5, and fusing them, performing regression prediction on the multiple meat quality indicators described in step S1, and obtaining a multimodal spectral fusion prediction model; In step S7, the three spectra and frequency data of the pork to be tested are used as input, the multimodal spectrum fusion prediction model established in step S6 is applied, and the meat quality index of the pork to be tested is output as the measurement result.

[0008] Preferably, in step S1, the wavelength range of VNIR-HSI spectrum collection is 400-1000 nm, the wavelength range of NIR-HSI spectrum collection is 900-1700 nm, the Raman excitation wavelength is 532 nm, and 6 points are randomly selected for each sample to be tested to collect Raman spectrum data; the multiple meat quality indicators include color value 24 hours after slaughter, pH value 24 hours after slaughter, drip loss, intramuscular fat, and protein content.

[0009] Preferably, the VNIR-HSI, NIR-HSI, and Raman modal normalization in step S2 specifically includes the following steps: Step S21: Select two characteristic bands with similar backgrounds but significantly different foregrounds in the VNIR-HSI and NIR-HSI images, respectively. Calculate the ratio of the images under the selected bands, perform maximum-minimum normalization, scale the normalized values to a range of 0-255, and finally generate a binary mask containing the meat sample region of interest (ROI) using the OTSU algorithm. Step S22: Using the binary mask obtained in step S21, extract the region of interest (ROI) in the VNIR-HSI and NIR-HSI data of the meat sample in each band, and calculate the average value of all pixel values in the ROI in each band, thereby obtaining one-dimensional VNIR-HSI and NIR-HSI data; In step S23, the Raman spectrum data of the six points randomly collected in step S1 are arithmetic averaged with respect to the original Raman spectrum intensity values of each measurement site at the same wave number to obtain the one-dimensional average Raman spectrum data of the meat sample.

[0010] Preferably, the one-dimensional data of the three spectra in step S3 are converted into the frequency domain, and the fast Fourier transform (FFT) method is used to read the one-dimensional VNIR-HSI, NIR-HSI, and Raman spectral data respectively. After the FFT transformation, a complex number containing amplitude and phase is obtained, and the amplitude of the positive frequency part corresponding to each type of spectral data is retained to represent the frequency data.

[0011] Preferably, the spectral feature extractor constructed in step S4 includes a spectral unimodal feature extractor, a spectral bimodal data layer fusion feature extractor, and a spectral bimodal feature layer fusion feature extractor, and is implemented by the following steps: Step S41, spectral unimodal feature extractor: Based on a multi-layer perceptron (MLP) framework, the network framework includes three fully connected layers. The first fully connected layer is preceded by a conditional judgment. The size of the input layer is dynamically adjusted according to the type of input spectrum. The meat quality index is then regressed through the three fully connected layers in sequence. A five-fold cross-validation is performed, and the optimal model parameters for each fold of the three spectral model training process are retained. Step S42, spectral bimodal data layer fusion feature extractor: Based on the one-dimensional VNIR-HSI and NIR-HSI spectral data obtained in step S2, the data are directly spliced and input into a neural network consisting of three fully connected layers. A five-fold cross-validation regression is performed on each meat quality indicator, and the optimal model parameters in each fold cross-validation are retained to obtain the pre-trained optimal model parameters of the VNIR-HSI and NIR-HSI spectral bimodal data layer fusion model; Step S43, spectral dual-modal feature layer fusion feature extractor: directly splice the two spectral features output by the VNIR-HSI and NIR-HSI single-modal feature extractors in step S41, input them into a neural network composed of 3 fully connected layers, perform five-fold cross-validation regression on each meat quality index, retain the optimal model parameters in each fold cross-validation, and obtain the pre-trained optimal model parameters of the VNIR-HSI and NIR-HSI spectral dual-modal feature layer fusion model.

[0012] Preferably, the frequency modal feature extractor constructed in step S5 is implemented by the following steps: The frequency data obtained in step S3 are respectively used as input. The feature extractor is based on a multi-layer perceptron (MLP) framework. Its network framework mainly includes three fully connected layers. There is a conditional judgment before the first fully connected layer. The size of the input layer is dynamically adjusted according to the type of input spectrum. Then, the meat quality index regression is realized through the three fully connected layers in sequence. A five-fold cross-validation is performed, and the optimal model parameters of each fold in the training process of the three frequency models are retained.

[0013] Preferably, in step S6, the pork quality index prediction network SF-Net based on multimodal spectral fusion is constructed by the following steps: Based on the spectral and frequency feature extractors constructed in steps S4 and S5, the optimal model parameters of each modality obtained during the pre-training process of the three feature extractors are imported in sequence, and the parameters in the training of each feature extractor are frozen to obtain the three unimodal spectral features extracted by the spectral feature extractor, the two spectral cross-modal features extracted by the spectral bimodal feature extractor, and the frequency features extracted by the frequency feature extractor; these features are then directly spliced, and the spliced modal features are input into three fully connected layer networks for feature fusion. The last fully connected layer performs meat quality index regression prediction, and a five-fold cross-validation is performed to retain the optimal model parameters in each fold training process.

[0014] Preferably, the spectrum and frequency modal data of the pork to be tested are input, the optimal model parameters of each fold training process of SF-Net saved in step S6 are imported, model inference is performed, and finally the average value of the prediction results of each fold is output as the final measurement result.

[0015] The present invention also discloses a pork quality index precision measurement system based on multimodal spectral fusion, which uses the aforementioned pig muscle quality index precision measurement method based on multimodal spectral fusion to evaluate pork quality. The system includes the following modules: The data acquisition module is used to obtain visible near-infrared imaging hyperspectral VNIR-HSI, near-infrared imaging hyperspectral NIR-HSI, and Raman spectroscopy data of the longissimus dorsi muscle of the pig, and to measure multiple meat quality indicators including intramuscular fat and pH value according to national standard methods; A spectral data preprocessing module is used to perform modal normalization on the three spectral data of VNIR-HSI, NIR-HSI, and Raman, and uniformly process them into one-dimensional spectral data; A frequency data extraction module is used to convert the one-dimensional data of the three spectra into the frequency domain to obtain the frequency data of the three spectra of the meat sample; The multimodal feature extraction module is used to extract the three modal spectral and frequency features of the meat sample, build regression prediction models based on the spectral and frequency data respectively, perform five-fold cross-validation regression, and retain the optimal model parameters as the feature extractor; The multimodal spectral fusion module is used to fuse the three extracted spectral and frequency features to build a pork quality index prediction model based on multimodal spectral fusion; The precise measurement module is used to receive the spectrum and frequency characteristics of the pork to be tested processed by the spectrum data preprocessing module and the frequency data extraction module, input them into the prediction model constructed by the multimodal feature fusion module, and output the meat quality index of the pork to be tested as the measurement result.

[0016] (3) Beneficial effects Compared with the prior art, the present invention has at least the following positive technical effects.

[0017] Most of the existing non-destructive measurement methods for various meat quality indicators through spectroscopy are based on a single spectrum, however, each spectrum has a preference for the prediction of various meat quality indicators. Meat components are complex, and the molecular signals of different components are in different spectral ranges, and are closely related to the penetration depth of photons. Different types of spectra cover a limited spectral range, so the prediction accuracy of each spectrum for various meat quality indicators is different. The present invention proposes an innovative method, which collects three spectra of VNIR, NIR, and Raman, constructs a spectral single-modal feature extractor, a spectral bimodal data layer fusion feature extractor, and a spectral bimodal feature layer fusion feature extractor, and respectively extracts the shallow features of the three spectral single modes and the deep features of VNIR-HSI and NIR-HSI, and then directly splices the shallow spectral features, deep spectral features, and the three spectral prediction results into a multimodal feature fusion module to effectively fuse the features of different spectral modalities.

[0018] Most of the existing meat quality index prediction methods based on hyperspectral data are to average the reflectivity within the ROI along the wavelength direction of the three-dimensional hyperspectral data to obtain the average reflectivity in each band, which is insufficient for the development and utilization of the information contained in the hyperspectral data. The present invention extracts the frequency data contained in each spectrum through fast Fourier transform, and uses a frequency feature extractor to extract the frequency features contained in the three spectra. The multimodal spectral fusion network SF-Net proposed in the present invention realizes the deep fusion and synergistic enhancement of multi-source information by efficiently integrating the spectral domain and frequency domain features of meat samples. The network utilizes the complementary specific information between different modalities and effectively suppresses redundant interference, thereby significantly improving the accuracy and reliability of meat quality index prediction. At the same time, its multimodal architecture gives the system an inherent robustness advantage, which can effectively resist the negative impact of the quality degradation or interference of a single modality information. Therefore, SF-Net has demonstrated significant technical advantages in terms of information representation dimension, feature collaboration efficiency and system robustness, and has achieved accurate prediction of multiple meat quality indicators.

[0019] In existing spectral meat quality measurement, the difficulty of obtaining true meat quality indicators, the high cost, and low throughput of these factors limit the variety and quantity of meat quality indicators obtained. Therefore, most methods rely on regression prediction of a single or a few meat quality indicators. This paper proposes a SF-Net network to perform regression prediction of multiple meat quality indicators, including key meat quality indicators such as color value 24 hours after slaughter, pH value 24 hours after slaughter, drip loss, intramuscular fat, and protein content, further expanding the model's application scope. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a flow chart of the method for accurately measuring pork quality indicators based on multimodal spectral fusion of the present invention.

[0021] Figure 2 This is a flowchart of the VNIR-HSI data modality normalization processing of the present invention.

[0022] Figure 3 This is a flowchart of the NIR-HSI data modality normalization process of the present invention.

[0023] Figure 4 A diagram of the network structure of the unimodal feature extractor constructed for the present invention.

[0024] Figure 5 This is the network structure diagram of the VNIR-HSI and NIR-HSI spectral dual-modal data layer fusion feature extractor constructed for the present invention.

[0025] Figure 6 This is the network structure diagram of the VNIR-HSI and NIR-HSI spectral dual-modal feature layer fusion feature extractor constructed for the present invention.

[0026] Figure 7 This is a graph showing the prediction results of various meat quality indicators using the three frequency feature extractors constructed in the present invention.

[0027] Figure 8 The structural diagram of the pork quality index prediction network SF-Net based on multimodal spectral fusion is constructed for the present invention.

[0028] Figure 9 This is a performance comparison chart of the CH_Fusion and VNIR-HSI, NIR-HSI, Raman, LLF, and MLF spectral feature extractor models constructed in the present invention.

[0029] Figure 10 This is a performance comparison chart of the SF-Net and CH_Fusion models constructed by the present invention.

[0030] Figure 11 This is a flow chart of the pork quality index precise measurement system based on multimodal spectral fusion of the present invention. DETAILED DESCRIPTION

[0031] The present invention is further described below with reference to the accompanying drawings and implementation examples.

[0032] The present invention discloses a method and system for accurately measuring pork quality indicators based on multimodal spectral fusion, such as Figure 1The flowchart shown is a method for accurately measuring pork quality indicators based on multimodal spectral fusion provided in this embodiment, which includes the following steps.

[0033] Step S1, collecting visible near-infrared imaging hyperspectral VNIR-HSI, near-infrared imaging hyperspectral NIR-HSI, and Raman spectroscopy data of a pig longissimus dorsi muscle sample, and manually measuring multiple meat quality indicators including intramuscular fat and pH value of the meat sample; Specifically, the steps for collecting visible near-infrared imaging hyperspectral VNIR-HSI, near-infrared imaging hyperspectral NIR-HSI, and Raman spectroscopy data of pig longissimus dorsi muscle samples in step S1 are as follows: (1) 372 pigs were collected, and a 20 cm long longissimus dorsi muscle sample was cut from the position between the third and fourth ribs from the bottom of the left half of the pig carcass toward the pig tail; (2) A 2 cm piece of meat was taken from the side between the third and fourth ribs from the bottom, and excess fat and connective tissue were removed. The meat sample was trimmed into a length × width × height of 5 cm × 5 cm × 1 cm; (3) The meat sample was placed in a clean bacterial culture dish, placed in an ice box at 4 ° C, and sent to a hyperspectral imaging darkroom and a Raman spectroscopy darkroom respectively. Excess moisture on the surface of the meat sample was wiped dry with absorbent paper, and then the dish containing the meat sample was placed on a translation stage for Raman spectrum, visible near-infrared imaging hyperspectral, and near-infrared imaging hyperspectral acquisition, respectively.

[0034] Specifically, the steps for manually measuring multiple meat quality indicators of meat samples, including intramuscular fat, protein content, and drip loss, in step S1 are as follows: (1) The remaining meat sample, except for the captured spectrum, is prepared into a sample according to the standard of NY / T 821-2019 "Technical Specifications for the Determination of Pork Quality" for the determination of various meat quality indicators; (2) The protein content is determined using the first method in GB5009.5-2016 "National Food Safety Standard - Determination of Protein in Food", namely the Kjeldahl method. The intramuscular fat content is determined using the methanol-chloroform method with reference to the procedures specified in "NY / T821-2004 Technical Specifications for the Determination of Pork Muscle Quality". Meat color, pH value, and drip loss are determined according to the determination process and methods specified in NY / T 821-2019 "Technical Specifications for the Determination of Pork Quality".

[0035] Step S2: The three spectral data of VNIR-HSI, NIR-HSI and Raman are subjected to modal normalization respectively, and the dimensions are uniformly reduced to one dimension to obtain three one-dimensional spectral data of the meat sample. The modal normalization processing flow chart of VNIR-HSI and NIR-HSI is shown in FIG. Figure 2 and Figure 3 As shown; Specifically, the steps of VNIR-HSI and NIR-HSI modal normalization processing in step S2 are as follows: (1) read the original data of the spectrum and select two bands with large foreground differences and relatively similar backgrounds; (2) divide the images of the two bands pixel by pixel to generate a ratio image, and normalize the image to the range of 0-255 again. The OTSU algorithm is used to process the ratio image to achieve binary segmentation of the background and target areas; (3) dilate the binary image in sequence to fill the internal holes and erode to restore the original image size. The largest connected domain in the image is extracted, and other discrete noise areas are removed. Finally, the image after the connected domain processing is subjected to a secondary erosion operation to eliminate abnormal pixel values caused by reasons such as sample water loss; (4) The processed binary image is used as a mask to extract the region of interest in the original image; (5) the average value of all pixels in the region of interest under each band is calculated to obtain one-dimensional spectral data.

[0036] Step S3, performing frequency domain conversion on the three one-dimensional spectral data obtained in step S2 to obtain frequency data of the three spectra of the meat sample; Specifically, the frequency conversion steps for step S3 are as follows: (1) read the one-dimensional data of VNIR-HSI, NIR-HSI, and Raman spectra respectively; (2) use the FFT method of the Numpy library to perform fast Fourier transform to obtain a complex number containing amplitude and phase, and calculate the modulus of the complex number to obtain the amplitude of each frequency; (3) retain the positive frequency part for analysis, and finally save the amplitudes of these three spectra respectively.

[0037] Step S4, taking the three one-dimensional spectral data obtained in step S2 as input, constructing a spectral feature extractor, performing regression prediction on the multiple meat quality indicators described in step S1, and obtaining the spectral modal features of the meat sample, wherein the spectral feature extractor includes a spectral single modal feature extractor, a spectral bimodal data layer fusion feature extractor, and a spectral bimodal feature layer fusion feature extractor, and their network structures are as follows: Figure 4 、 Figure 5 、 Figure 6 As shown; Specifically, the steps of constructing the spectral feature extractor in step S4 are as follows: (1) Based on the spectral unimodal feature extractor in step S41, step S42, and step S43, the spectral bimodal data layer fusion feature extractor, and the spectral bimodal feature layer fusion feature extractor, a five-fold cross-validation regression is performed on each meat quality index, and the optimal model parameters in each fold cross-validation are retained; (2) The above regression models all use the following hyperparameter settings: Adam is selected as the optimizer, the cosine learning rate scheduler is used to dynamically adjust the learning rate, the mean square error MSE is used as the loss function for model training, and the dropout ratio is set to 0.5.

[0038] Step S5, using the frequency data of the three spectra obtained in step S3 as input, constructing a frequency feature extractor, performing regression prediction on the multiple meat quality indicators described in step S1, and obtaining frequency modal features; Specifically, the construction steps of the frequency feature extractor in step S5 are as follows: (1) The three spectral meat sample frequency features obtained in step S3 are used as input respectively, and five-fold cross-validation regression is performed on each meat quality index; (2) The optimal model parameters in each fold cross-validation of each meat quality index are saved; (3) The following hyperparameter settings are adopted: Adam is selected as the optimizer, the cosine learning rate scheduler is used to dynamically adjust the learning rate, the mean square error (MSE) is used as the loss function for model training, and the dropout ratio is set to 0.5.

[0039] In order to objectively evaluate the effectiveness of the frequency features obtained by the frequency feature extractor of the present invention, this exemplary embodiment performs a five-fold cross-validation regression on each meat quality index using the frequency feature extractor, and takes the average determination coefficient R of each fold test set. 2 As a comparison of model performance. Figure 7 The results show that the prediction performance of the frequency feature extractor VNIR_FFT based on VNIR-HSI and the frequency feature extractor NIR_FFT based on NIR-HSI is better than that of the frequency feature extractor Raman_FFT based on Raman. 24 * The performance of the indicator prediction is the best, and its R 2 is 0.654. NIR_FFT at pH 24 , drip loss, intramuscular fat, and protein content, with R 2 The values are 0.536, 0.512, 0.697 and 0.558 respectively, indicating that different frequency characteristics have certain predictive effects on various meat quality indicators.

[0040] Step S6, using the spectrum and frequency data of the three modes obtained in steps S2 and S3 as input data, constructing a pork quality index prediction network SF-Net based on multimodal spectrum fusion, obtaining the spectrum and frequency features of the meat sample based on the spectrum and frequency feature extractor constructed in steps S4 and S5 and fusing them, performing regression prediction on the multiple meat quality indicators described in step S1, and obtaining a multimodal spectrum fusion prediction model; wherein the structure of the pork quality index prediction network SF-Net based on multimodal spectrum fusion is as follows: Figure 8 As shown; Specifically, the pork quality index prediction network SF-Net based on multimodal spectral fusion includes a feature extraction part and a feature fusion part. The network structure of the feature extraction part consists of two branches: frequency and spectrum. The construction steps of each part of the network are as follows: (1) The frequency branch consists of three frequency feature extractors. The structure of each frequency feature extractor is the same as the corresponding three frequency feature extractor network structures in step S5. In the output part, the prediction results of the three frequency feature extractors are directly spliced as the extraction features of the frequency branch; (2) The spectral branch network structure consists of five basic feature extractors: VNIR-HSI, NIR-HSI, Raman spectrum single-modal feature extractor, VNIR-HSI, NIR-HSI spectrum dual-modal data layer fusion feature extractor, and VNIR-HSI, NIR-HSI spectrum dual-modal feature layer fusion feature extractor. The network structure of each feature extractor is the same as the network structure of the spectral feature extractor in step S4. Specifically, the first fully connected layer output vectors of the three single-modal feature extractors are directly spliced as shallow spectral features, and the first fully connected layer output vectors after VNIR-HSI and NIR-HSI fusion in the VNIR-HSI and NIR-HSI spectral bimodal data layer fusion feature extractor and the VNIR-HSI and NIR-HSI spectral bimodal feature layer fusion feature extractor are spliced as deep spectral features. Then, the shallow spectral features, deep spectral features and the prediction results of the three single-modal feature extractors are spliced to form spectral fusion features, and the spectral fusion vector obtained above is used as the spectral feature extracted by the final spectral branch (3). The feature fusion part consists of 3 fully connected layers, which map the fusion features of each modality to the predicted values of various meat quality indicators of pork.

[0041] Specifically, the steps of the multimodal spectral fusion prediction model in step S6 are as follows: (1) The optimal model parameters of the VNIR-HSI, NIR-HSI, and Raman spectral single-modal feature extractors, the VNIR-HSI and NIR-HSI spectral dual-modal data layer fusion feature extractors, and the VNIR-HSI and NIR-HSI spectral dual-modal feature layer fusion feature extractors in step S4 are loaded and frozen into the SF-Net spectral branch through transfer learning to obtain the spectral modality features; (2) The optimal model parameters of the frequency feature extractor constructed in step S5 are loaded and frozen into the SF-Net frequency branch through transfer learning to obtain the frequency modality features; (3) The spectral and frequency features are fused and spliced, and a five-fold cross-validation regression is performed on each meat quality index, and the optimal model parameters in each fold cross-validation are retained. (4) The model uses the following hyperparameter settings: Adam is selected as the optimizer, the cosine learning rate scheduler is used to dynamically adjust the learning rate, the mean square error (MSE) is used as the loss function for model training, and the dropout ratio is set to 0.5.

[0042] In order to objectively evaluate the effectiveness of the spectral fusion features extracted by the spectral branch of the SF-Net network of the present invention, this exemplary embodiment compares its performance with the five feature extractors that form its basis. These five basic feature extractors are VNIR-HSI single-modal feature extractor, NIR-HSI single-modal feature extractor, Raman single-modal feature extractor, spectral bimodal data layer fusion feature extractor LLF, and spectral bimodal feature layer fusion feature extractor MLF. The features extracted from the spectral modality are connected to three fully connected layers to form a spectral feature regression network CH_Fusion. CH_Fusion and the five models that form its basis are used to regress and predict various meat quality indicators. The regression predictions of the above six models are evaluated using five-fold cross-validation, and the performance comparison is based on the average determination coefficient R calculated on the test set. 2 The prediction results of each method are as follows Figure 9 The results show that the prediction performance of the CH_Fusion model on all indicators exceeds the prediction performance of the other five models on each indicator. 24 * , pH 24 , drip loss, intramuscular fat, and protein prediction coefficient R 2 The results are: 0.723, 0.716, 0.581, 0.745, and 0.616, respectively. This shows that the spectral fusion features extracted from the spectral branches, including deep spectral features and shallow spectral features as well as the prediction results of each single modal spectrum, have positive significance for improving the prediction accuracy of various meat quality indicators.

[0043] In order to objectively evaluate the effectiveness of the multimodal strategy fusion adopted by the pork quality index prediction network SF-Net based on multimodal spectral fusion in the present invention, this exemplary embodiment compares the model performance of the spectral feature CH_Fusion and the spectrum + frequency feature SF-Net, and performs a five-fold cross-validation regression on each meat quality index to compare the average determination coefficient R calculated on the test set. 2 The prediction results of each model are as follows Figure 10 The results show that SF-Net has achieved the best results in the prediction of all indicators. 24 * , pH 24 , drip loss, intramuscular fat, and protein prediction coefficient R 2The results are: 0.734, 0.721, 0.600, 0.762, and 0.625, respectively. This shows that SF-Net has the best overall performance among all models. It can better extract and fuse the spectral and frequency characteristics of each modality, accurately capture the complex relationship between pork quality indicators and the characteristics of each modality, thereby improving the prediction accuracy of each indicator and verifying the effectiveness of the multimodal spectral fusion strategy.

[0044] In step S7, the spectrum and frequency modal data of the pork to be tested are used as input, the multimodal spectrum fusion prediction model established in step S6 is applied, and the meat quality index of the pork to be tested is output as the measurement result.

[0045] Specifically, the steps of using the multimodal spectral fusion prediction model established in step S7 to predict the meat quality index of the pork to be tested are as follows: (1) collect a piece of longissimus dorsi muscle of the pork to be tested, remove the excess connective tissue on the surface, trim the meat sample into a 5 cm × 5 cm × 1 cm piece, wipe off the excess moisture on the surface of the meat sample, and then collect the VNIR-HSI, NIR-HSI, and Raman spectra of the meat sample respectively; (2) use the spectral data preprocessing module and the frequency data extraction module to extract the spectrum and frequency multimodal data of the pork to be tested; (3) use the spectrum and frequency multimodal data of the pork to be tested as input, use the multimodal spectral fusion prediction model established in step S6 to predict various meat quality indexes, and output various meat quality indexes of the pork to be tested as the measurement results.

[0046] The present invention also discloses a pork quality index accurate measurement system based on multimodal spectral fusion, such as Figure 11 The figure shows a flow chart of a pork quality index accurate measurement system based on multimodal spectral fusion provided by this embodiment. The system includes the following modules: The data acquisition module is used to obtain visible near-infrared imaging hyperspectral (VNIR-HSI), near-infrared imaging hyperspectral (NIR-HSI), and Raman spectroscopy data of the longissimus dorsi muscle of the pig, and to measure multiple meat quality indicators including intramuscular fat, protein content, and drip loss according to national standards. The data acquisition module is used to obtain visible near-infrared imaging hyperspectral VNIR-HSI, near-infrared imaging hyperspectral NIR-HSI, and Raman spectroscopy data of the longissimus dorsi muscle of the pig, and to measure multiple meat quality indicators including intramuscular fat and pH value according to national standard methods; A spectral data preprocessing module is used to perform modal normalization on the three spectral data of VNIR-HSI, NIR-HSI, and Raman, and uniformly process them into one-dimensional spectral data; A frequency data extraction module is used to convert the one-dimensional data of the three spectra into the frequency domain to obtain the frequency data of the three spectra of the meat sample; The multimodal feature extraction module is used to extract three spectral and frequency features of meat samples, build regression prediction models based on the spectral and frequency data, perform five-fold cross-validation regression, and retain the optimal model parameters as the feature extractor; The multimodal spectral fusion module is used to fuse the three extracted spectral and frequency features to build a pork quality index prediction model based on multimodal spectral fusion; The precise measurement module is used to receive the spectrum and frequency characteristics of the pork to be tested processed by the spectrum data preprocessing module and the frequency data extraction module, input them into the prediction model constructed by the multimodal feature fusion module, and output the meat quality index of the pork to be tested as the measurement result.

[0047] The specific examples described in this application are merely illustrative of the spirit of the present invention. Persons skilled in the art may make various modifications, additions, or substitutions to the specific examples described herein without departing from the spirit of the present invention or exceeding the scope of the appended claims.

Claims

1. A method for accurately measuring pork quality indicators based on multimodal spectral fusion, characterized in that: The steps include: Step S1, collecting visible near-infrared imaging hyperspectral VNIR-HSI, near-infrared imaging hyperspectral NIR-HSI, and Raman spectroscopy data of a pig longissimus dorsi muscle sample, and manually measuring multiple meat quality indicators including intramuscular fat and pH value of the meat sample; Step S2, performing modal normalization processing on the collected VNIR-HSI, NIR-HSI, and Raman spectral data, uniformly reducing the dimensionality to one dimension, and obtaining three one-dimensional spectral data of the meat sample; Step S3, performing frequency domain conversion on the three one-dimensional spectral data obtained in step S2 to obtain frequency data of the three spectra of the meat sample; Step S4, using the three one-dimensional spectral data obtained in step S2 as input, constructing a spectral feature extractor, performing regression prediction on the multiple meat quality indicators described in step S1, and obtaining spectral features of the meat sample; Step S5, using the frequency data of the three spectra obtained in step S3 as input, constructing a frequency feature extractor, performing regression prediction on the multiple meat quality indicators described in step S1, and obtaining the frequency features of the meat sample; Step S6: Using the spectral and frequency data of the three modalities obtained in steps S2 and S3 as input data, constructing a pork quality index prediction network SF-Net based on multimodal spectral fusion, obtaining the spectral and frequency features of the meat sample based on the spectral and frequency feature extractors constructed in steps S4 and S5, and fusing them, performing regression prediction on the multiple meat quality indicators described in step S1, and obtaining a multimodal spectral fusion prediction model; In step S7, the three spectra and frequency data of the pork to be tested are used as input, the multimodal spectrum fusion prediction model established in step S6 is applied, and the meat quality index of the pork to be tested is output as the measurement result.

2. The method for accurately measuring pork quality indicators based on multimodal spectral fusion according to claim 1 is characterized in that: In step S1, the wavelength range of VNIR-HSI spectrum collection is 400-1000 nm, the wavelength range of NIR-HSI spectrum collection is 900-1700 nm, the Raman excitation wavelength is 532 nm, and 6 points are randomly selected for each sample to be tested to collect Raman spectrum data; the multiple meat quality indicators include color value 24 hours after slaughter, pH value 24 hours after slaughter, drip loss, intramuscular fat, and protein content.

3. The method for accurately measuring pork quality indicators based on multimodal spectral fusion according to claim 1 is characterized in that: The VNIR-HSI, NIR-HSI, and Raman modal normalization in step S2 specifically includes the following steps: Step S21: Select two characteristic bands with similar backgrounds but significantly different foregrounds in the VNIR-HSI and NIR-HSI images, respectively. Calculate the ratio of the images under the selected bands, perform maximum-minimum normalization, scale the normalized values to a range of 0-255, and finally generate a binary mask containing the meat sample region of interest (ROI) using the OTSU algorithm. Step S22: Using the binary mask obtained in step S21, extract the region of interest (ROI) in the VNIR-HSI and NIR-HSI data of the meat sample in each band, and calculate the average value of all pixel values in the ROI in each band, thereby obtaining one-dimensional VNIR-HSI and NIR-HSI data; In step S23, the Raman spectrum data of the six points randomly collected in step S1 are arithmetic averaged with respect to the original Raman spectrum intensity values of each measurement site at the same wave number to obtain the one-dimensional average Raman spectrum data of the meat sample.

4. The method for accurately measuring pork quality indicators based on multimodal spectral fusion according to claim 1 is characterized in that: The one-dimensional data of the three spectra in step S3 are converted into the frequency domain, and the fast Fourier transform (FFT) method is used to read the one-dimensional VNIR-HSI, NIR-HSI, and Raman spectral data respectively. After the FFT transformation, a complex number containing amplitude and phase is obtained, and the amplitude of the positive frequency part corresponding to each spectral data is retained to represent the frequency data.

5. The method for accurately measuring pork quality indicators based on multimodal spectral fusion according to claim 1 is characterized in that: The spectral feature extractor constructed in step S4 includes a spectral unimodal feature extractor, a spectral bimodal data layer fusion feature extractor, and a spectral bimodal feature layer fusion feature extractor, and is implemented by the following steps: Step S41, spectral unimodal feature extractor: Based on a multi-layer perceptron (MLP) framework, the network framework includes three fully connected layers. The first fully connected layer is preceded by a conditional judgment. The size of the input layer is dynamically adjusted according to the type of input spectrum. The meat quality index is then regressed through the three fully connected layers in sequence. A five-fold cross-validation is performed, and the optimal model parameters for each fold of the three spectral model training process are retained. Step S42, spectral bimodal data layer fusion feature extractor: Based on the one-dimensional VNIR-HSI and NIR-HSI spectral data obtained in step S2, the data are directly spliced and input into a neural network composed of three fully connected layers. A five-fold cross-validation regression is performed on each meat quality indicator, and the optimal model parameters in each fold cross-validation are retained to obtain the pre-trained optimal model parameters of the VNIR-HSI and NIR-HSI spectral bimodal data layer fusion model; Step S43, spectral dual-modal feature layer fusion feature extractor: directly splice the two spectral features output by the second fully connected layer of the VNIR-HSI and NIR-HSI single-modal feature extractors in step S41, input them into a neural network composed of 3 fully connected layers, perform five-fold cross-validation regression on each meat quality indicator, retain the optimal model parameters in each fold cross-validation, and obtain the pre-trained optimal model parameters of the VNIR-HSI and NIR-HSI spectral dual-modal feature layer fusion model.

6. The method for accurately measuring pork quality indicators based on multimodal spectral fusion according to claim 1 is characterized in that: The frequency modal feature extractor constructed in step S5 is implemented by the following steps: The frequency data obtained in step S3 are respectively used as input. The feature extractor is based on a multi-layer perceptron (MLP) framework. Its network framework mainly includes three fully connected layers. There is a conditional judgment before the first fully connected layer. The size of the input layer is dynamically adjusted according to the type of input spectrum. Then, the meat quality index regression is realized through the three fully connected layers in sequence. A five-fold cross-validation is performed, and the optimal model parameters of each fold in the training process of the three frequency models are retained.

7. The method for accurately measuring pork quality indicators based on multimodal spectral fusion according to claim 1 is characterized in that: In step S6, a pork quality index prediction network SF-Net based on multimodal spectral fusion is constructed, which is implemented by the following steps: Based on the spectral and frequency feature extractors constructed in steps S4 and S5, the optimal model parameters of each modality obtained during the pre-training process of the spectral and frequency feature extractors are imported in sequence, and the parameters in the training of each feature extractor are frozen, and the three single-modal spectral features extracted by the spectral feature extractor, the two spectral cross-modal features extracted by the spectral bimodal feature extractor, and the frequency features extracted by the frequency feature extractor are obtained in sequence; then these features are directly spliced, and the spliced modal features are input into three fully connected layers for feature fusion. The last fully connected layer performs meat quality index regression prediction, and a five-fold cross-validation is performed to retain the optimal model parameters in each fold training process.

8. The method for accurately measuring pork quality indicators based on multimodal spectral fusion according to claim 1 is characterized in that: Input the spectral and frequency modal data of the pork to be tested, import the optimal model parameters of each fold of SF-Net training process saved in step S6, perform model inference, and finally output the average value of each fold prediction result as the final measurement result.

9. A system for accurately measuring pork quality indicators based on multimodal spectral fusion, which uses the method for accurately measuring pork muscle quality indicators based on multimodal spectral fusion as described in any one of claims 1 to 8 to evaluate pork quality, characterized in that: The system includes the following modules: The data acquisition module is used to obtain visible near-infrared imaging hyperspectral VNIR-HSI, near-infrared imaging hyperspectral NIR-HSI, and Raman spectroscopy data of the longissimus dorsi muscle of the pig, and to measure multiple meat quality indicators including intramuscular fat and pH value according to national standard methods; A spectral data preprocessing module is used to perform modal normalization on the three spectral data of VNIR-HSI, NIR-HSI, and Raman, and uniformly process them into one-dimensional spectral data; A frequency data extraction module is used to convert the one-dimensional data of the three spectra into the frequency domain to obtain the frequency data of the three spectra of the meat sample; The multimodal feature extraction module is used to extract the three modal spectral and frequency features of the meat sample, build regression prediction models based on the spectral and frequency data respectively, perform five-fold cross-validation regression, and retain the optimal model parameters as the feature extractor; The multimodal spectral fusion module is used to fuse the three extracted spectral and frequency features to build a pork quality index prediction model based on multimodal spectral fusion; The precise measurement module is used to receive the spectrum and frequency characteristics of the pork to be tested processed by the spectrum data preprocessing module and the frequency data extraction module, input them into the prediction model constructed by the multimodal feature fusion module, and output the meat quality index of the pork to be tested as the measurement result.

Images