Multi-modal fusion detection method for freshness of pork tenderloin based on deep learning

Abstract

The invention belongs to the field of food detection, and discloses a pork tenderloin freshness multi-modal fusion detection method based on deep learning, which comprises the following steps: making a sample of a pork tenderloin freshness classification model, collecting HSI data, ultraviolet fluorescence image data, electronic nose data and LF-NMR data in the pork tenderloin storage process, and preprocessing the data; respectively designing feature extraction modules for the ultraviolet fluorescence image data and the HSI data; designing a data enhancement module based on a variational auto-encoder to enhance the sample freshness boundary recognition capability of each category; designing an improved dung beetle optimization algorithm to enhance the hyper-parameter optimization ability of the model; a UV-CNN module, an HSI-CNN module, an AAVAE module and an EDBO optimization algorithm are introduced into a BiLSTM network structure, and a CNN-AAVAE-EDBO-BiLSTM classification model of the freshness of the pork tenderloin is constructed; and testing and evaluating the accuracy of the pig tenderloin classification model.

Description

Technical Field

[0001] The present invention belongs to the field of food detection and relates to a multimodal fusion detection method for freshness of pork tenderloin based on deep learning. Background Art

[0002] Pork is rich in protein, carbohydrates, fat, and essential minerals and trace elements for the human body. Its high nutritional value makes it popular among consumers. Among them, pork tenderloin can be used as a raw material for deep-processing products or sold as a finished product, and it faces a wide market demand. Whether fresh pork tenderloin is transported from the slaughterhouse to the processing plant or sold as a finished product to consumers, it needs to be transported through the cold chain and stored at low temperatures to ensure its freshness. However, in this process, irregular behavior or external factors (such as refrigeration temperature and storage time, etc.) will cause microbial growth in the product and a series of biochemical reactions, leading to deterioration and corruption. The determination of volatile basic nitrogen (TVB-N) and thiobarbituric acid (TBARS) content is an important indicator for evaluating the freshness of pork tenderloin. However, the traditional detection methods of these indicators are not suitable for non-destructive testing due to their destructiveness and time-consuming disadvantages.

[0003] Currently, there are methods for non-destructive detection of meat freshness, such as hyperspectral imaging technology (HSI), ultraviolet fluorescence imaging technology, electronic nose technology and low-field nuclear magnetic resonance technology (LF-NMR). Patent CN 117929294A discloses an HSI detection method for mutton quality indicators. The HSI is used to detect the color and texture changes on the surface of mutton to evaluate its freshness. However, this detection method is easily interfered by environmental factors (ambient light, humidity, etc.) and it is difficult to detect the spectral reflectivity of the surface of spoiled meat where microorganisms breed and produce mucus, which affects the evaluation of meat freshness. Ultraviolet fluorescence imaging technology can detect fluorescent substances in mucus produced on the surface of meat due to microbial metabolism during meat corruption, and then evaluate the freshness of meat. For example, patent CN116067931A discloses a method for detecting the freshness of frozen tilapia based on fluorescence response images. The accumulation of fluorescent substances is detected to determine the microbial breeding situation and then evaluate the freshness of frozen tilapia. However, there are some natural compounds in meat, such as tryptophan (fluorescence under 340nm ultraviolet light) and tyrosine (fluorescence under 305nm ultraviolet light) widely present in proteins, which will affect the evaluation of meat freshness. Electronic nose technology evaluates the freshness of meat by capturing the odor information generated during meat corruption, such as patent CN 117665226A discloses a method for evaluating the freshness of meat by detecting ammonia and volatile sulfides produced during meat spoilage using a gas detection device. However, the amount of ammonia and sulfides produced by biochemical reactions in the early stages of meat spoilage is relatively small, and the volatilization rate is unstable, making it difficult to judge the sub-fresh meat in the early stages of spoilage. In addition, during the spoilage process, microbial reproduction and metabolism on the surface of meat will also produce gases that affect the final freshness evaluation result. The initial process of meat spoilage is accompanied by changes in internal physical and chemical properties (water content, protein degradation, etc.). By detecting the migration changes of the internal moisture content of meat, its freshness can be evaluated. For example, patent CN 109033616A proposes a freshness detection method for detecting the migration changes of the internal moisture content of frozen pork due to spoilage using low-field nuclear magnetic resonance (LF-NMR) technology. However, when the ambient temperature changes, this method will cause microorganisms to multiply and metabolize to produce excess moisture, which will affect the detection of the migration changes of the internal moisture content of meat, thereby affecting the meat freshness evaluation. The above patents provide freshness characteristic information from four dimensions: color, microbial metabolism, odor, and water content migration. However, meat freshness is a comprehensive and complex indicator. A single data indicator cannot take into account multiple information factors and is difficult to adapt to the complex and changeable actual application environment, which ultimately affects the accuracy of freshness assessment. Based on this problem, relevant research has invented a multi-source data fusion method for meat freshness to improve the accuracy and reliability of detection.For example, patent CN 109490306A discloses a pork freshness detection method based on the fusion of color appearance and odor data. The patent achieves a certain effect by combining odor information and appearance information. However, the patent only evaluates from the perspective of sensory characteristics and does not take into account the internal changes in the meat spoilage process (microorganism growth, changes in moisture content, etc.). This results in the model being only applicable to specific detection scenarios (no ambient light interference, sufficient volatilization of spoilage odor, etc.), and having problems such as poor generalization and limited applicability. In addition, only a simple machine learning algorithm model is used for freshness assessment, which makes it difficult to fully explore the deep feature associations of multi-source data, resulting in the loss of more feature information during the fusion process, thereby affecting the detection accuracy.

[0004] Based on the shortcomings of the above research, the present invention proposes a CNN-AAVAE-EBDO-BiLSTM classification model for the freshness of pork tenderloin, which integrates four different dimensional data of HSI, ultraviolet fluorescence image, electronic nose and LF-NMR. It provides freshness feature information through four dimensions of color appearance, odor, microbial growth status and internal moisture content change, improves the generalization and applicability of the model, and realizes non-destructive detection of the freshness of pork tenderloin. Summary of the invention

[0005] The purpose of the present invention is to overcome the deficiencies in the prior art and provide a multimodal fusion detection method for freshness of pork tenderloin based on deep learning, which can accurately classify the freshness of pork tenderloin and help inspectors to non-destructively detect the freshness of pork tenderloin;

[0006] The technical solution of the present invention:

[0007] A multimodal fusion detection method for freshness of pork tenderloin based on deep learning, the steps are as follows:

[0008] Step 1: Prepare samples for training and testing the designed pork tenderloin freshness classification model, obtain HSI data, ultraviolet fluorescence image data, electronic nose data and LF-NMR data of pork tenderloin during storage through four data acquisition devices, and preprocess the data;

[0009] Step 2: Design feature extraction modules for UV fluorescence image data and HSI data respectively, analyze key feature information, reduce dimensions and reduce redundant data, so as to facilitate subsequent data fusion;

[0010] Step 3: Design a data augmentation module (AAVAE) based on variational autoencoder (VAE) to enhance the ability to identify the boundaries of each category of sample freshness;

[0011] Step 4: Introduce the UV-CNN module, HSI-CNN module and AAVAE module into the BiLSTM network structure to build a CNN-AAVAE-BiLSTM classification model for pork tenderloin freshness;

[0012] Step 5: Design an improved dung beetle optimization algorithm (EDBO) based on the DBO module to enhance the model hyperparameter optimization capability;

[0013] Step 6: Test and evaluate the accuracy of the CNN-AAVAE-EDBO-BiLSTM classification model for pork tenderloin freshness.

[0014] Furthermore, in step 1, in order to ensure the accuracy of the designed pork tenderloin freshness classification model, samples of the same batch of fresh pork tenderloin are used for training and testing the designed pork tenderloin freshness classification model; the fresh pork tenderloin is divided into block samples of equal mass of 30×30×10 mm, the samples are divided into two parts, and stored in a 4°C refrigerator for later use; the specific steps of data collection and preprocessing are as follows:

[0015] 1-1. The UV fluorescence imaging is collected using a UV fluorescence imaging system, which is mainly composed of a camera, a lens, two UV-LEDs and a polarizing filter installed in front of the light source. The sample is placed in the center of a black sample in a dark box, and the camera collects its top view image as a UV fluorescence image.

[0016] 1-2. Use a hyperspectral instrument system to collect hyperspectral images of the samples; wipe off excess moisture on the sample surface, place it on a black sample, and use a spectral camera to collect its top view image; perform black and white correction on the hyperspectral image to eliminate the interference of noise and dark current, and crop the hyperspectral image into a 512×512 hyperspectral image with the sample as the center of the image to produce the original data set;

[0017] 1-3. Use a nuclear magnetic resonance imaging analyzer to obtain low-field nuclear magnetic resonance (LF-NMR) data of the sample;

[0018] 1-4. Use a portable electronic nose to collect odor information of the sample at a constant ambient temperature of 21℃-25℃.

[0019] According to the national standards "GB 5009.228-2016 National Food Safety Standard - Determination of Volatile Basic Nitrogen in Foods" and "GB 5009.181-2016 National Food Safety Standard - Determination of Thiobarbituric Acid Value (TBA Value) in Foods", and combined with previous studies, the freshness of pork is divided into three grades: fresh meat (TVB-N ≤ 15mg / 100g, TBARS < 0.5mg MDA / kg), sub-fresh meat (15mg / 100g < TVB-N ≤ 20mg / 100g, 0.5mg MDA / kg ≤ TBARS < 1.0mg MDA / kg), and spoiled meat (TVB-N > 20mg / 100g, TBARS ≥ 1.0mg MDA / kg). Among them, the label corresponding to fresh meat is 0, the label corresponding to sub-fresh meat is 1, and the label corresponding to spoiled meat is 2.

[0020] In step 2, in the task of classifying the freshness of pork tenderloin, not only irrelevant features such as blood stains and attachments will cause interference, but also the sensory characteristics of pork tenderloin in the initial stage of spoilage change and the fluorescent substances accumulated by microbial metabolism are not obvious, and some key feature points are easily confused with interference factors such as the background. To address the above problems, an ultraviolet fluorescence image feature extraction module (UV-CNN) and an HSI feature extraction module (HSI-CNN) are designed for ultraviolet fluorescence images and HSI data respectively to enhance the extraction of key feature information on color appearance, texture, and fluorescent substances accumulated by microbial metabolism.

[0021] The ultraviolet fluorescence image feature extraction module (UV-CNN) captures the distribution characteristics and texture characteristics of fluorescent substances on the surface of the sample in the ultraviolet fluorescence image, and then converts the captured distribution characteristics and texture characteristics into one-dimensional feature vectors for subsequent processing; the ultraviolet fluorescence image feature extraction module uses 2DCNN as the feature extraction backbone network and improves the 2D convolution kernels of the same size in it, that is, four 2D convolution kernels of different sizes are used to replace the original 2D convolution kernels of the same size in the four convolutional layers of 2DCNN. Specifically, the first Conv1 uses 64 5×5 2D convolution kernels to be responsible for extracting basic low-level features, the second Conv2 uses 128 3×3 2D convolution kernels to be responsible for extracting more complex intermediate features, the third Conv3 uses 256 3×3 2D convolution cores to be responsible for extracting higher-level features, and the fourth Conv4 uses 512 3×3 2D convolution cores to be responsible for extracting the highest-level features. Then, the extracted features are flattened into one-dimensional vectors through the flattening layer and the fully connected layer for subsequent processing; the multi-scale convolution adopted can provide a strong ability to represent the distribution and texture characteristics of fluorescent substances on the sample surface.

[0022] The HSI feature extraction module (HSI-CNN) captures the color appearance and texture features of the sample surface in the hyperspectral image, and then converts the captured color appearance and texture features into a one-dimensional feature vector for subsequent processing; the HSI feature extraction module uses 3DCNN as the feature extraction backbone network, reduces the network structure of 3DCNN to a five-layer structure to improve the network convergence speed, and introduces a residual connection block with an attention mechanism to assist deep feature transfer. Specifically, the HSI feature extraction module has a total of seven layers. The first 3D convolution layer uses 32 3D convolution kernels of size 3×3×3 to extract preliminary features; the second pooling layer has a pooling window size of 2×2×2 to reduce the depth of the feature map and prevent overfitting; The third layer of residual connection blocks is used to focus on the deep features in the feature map and assist in passing the deep features of the feature map to the next convolutional layer; the fourth layer of 3D convolutional layer uses 64 3D convolution kernels of size 3×3×3 to further extract more complex features; the fifth layer of pooling layer uses the same pooling window as the second layer to further reduce the size of the feature map; the sixth layer uses the same residual connection block as the third layer to assist in passing deep features; the seventh layer is a global average pooling layer, which expands the extracted features into a fixed-length vector for subsequent processing; using fewer convolutional layers can avoid the problem of excessively complex network structures with excessive computational complexity and time-consuming, and introducing residual connections with an attention mechanism can effectively alleviate the problem of gradient vanishing and enhance the ability to focus on key features.

[0023] In step 3, in the pork tenderloin freshness classification task, with the acceleration of microbial activity and enzymatic reaction, the transition of pork tenderloin from a fresh state to a spoiled state is often relatively rapid, and the duration of the sub-fresh stage is relatively short. This results in a smaller number of samples in the sub-fresh category during the sample data collection process, resulting in an uneven sample distribution, which affects the distinction and definition of freshness categories. To solve the above problems, a data augmentation module (AAVAE) is designed with variational autoencoder (VAE) as the backbone network. The data augmentation module consists of an encoder and a decoder. The feature data fused by the feature fusion layer in step 4 is input into the encoder of the data augmentation module for compression and the mean and variance are obtained. Then, latent variables are collected from the latent space distribution with added noise through reparameterization. Finally, the latent variables are reconstructed into new data through the decoder of the data augmentation module. Specifically, an adaptive adjustment strategy is introduced, that is, by calculating the distribution of the number of samples of each category in the data set in step 1, and training each minority category separately and generating supplementary samples until the number of samples of the maximum category is reached. At the same time, randomness is introduced in the reparameterization process, and the reconstruction error and KL divergence are balanced in the damage function. The introduced adaptive adjustment strategy can enable AAVAE to sample and generate new data from the latent space and balance the gap in the number of samples of each category, thereby improving the ability to identify category boundaries. The introduction of randomness in the reparameterization process ensures the continuity of the latent space and enhances the generalization ability of the model. Balancing the reconstruction error and KL divergence in the loss function can normalize the distribution of the latent space and prevent overfitting.

[0024] In step 4, the BiLSTM network is used as the backbone network, and the ultraviolet fluorescence image feature extraction module (UV-CNN) and HSI feature extraction module (HSI-CNN) in step 2 are introduced at the head. UV-CNN and HSI-CNN are connected in parallel. After the ultraviolet fluorescence image and HSI data pass through their corresponding feature extraction modules, the output one-dimensional feature data are combined with the electronic nose data and LF-NMR data to construct a feature fusion layer by splicing and fusion. The fused data is input into the data enhancement module in step 3 for data enhancement and expansion, so as to solve the problem of too few fresh pork tenderloin samples and unbalanced number of samples in each category, and improve the model's accurate recognition of various categories of pork tenderloin freshness. The data set after data enhancement is input into the BiLSTM network for classification.

[0025] In step 5, in the pork tenderloin freshness classification task, not only the selection of features and the balance of the number of samples in each category will affect the classification results, but also the selection of network hyperparameters such as the number of hidden layer neurons, initial learning rate and regularization coefficient in the network model will affect the classification results. The traditional manual parameter setting using trial and error and empirical methods causes the algorithm to either focus too much on global search or over-focus on local development during the search process. To solve the above problems, an improved dung beetle optimization algorithm (EDBO) is designed based on the dung beetle optimization algorithm (DBO). EDBO is used to first calculate the optimal solutions of network hyperparameters such as the number of hidden layer neurons, initial learning rate and regularization coefficient in the CNN-AAVAE-BiLSTM classification model in step 4, and then the optimal solution is sent to the CNN-AAVAE-BiLSTM classification model in step 4 to detect and classify the data set after data enhancement. The improved EDBO algorithm uses DBO as the backbone network of the optimization algorithm, in which adaptive parameter adjustment is introduced to replace the fixed parameter control in the original algorithm, and the search behavior is dynamically adjusted by real-time calculation and updating of parameter values. In the local search stage, a simulated annealing mechanism is introduced, and the acceptance probability of the solution is controlled by the temperature parameter to avoid falling into the local optimum. The introduced adaptive parameter adjustment and simulated annealing mechanism improve the search ability and convergence performance of the algorithm.

[0026] The constructed pork tenderloin freshness CNN-AAVAE-EDBO-BiLSTM classification model consists of HSI-CNN feature extraction module, UV-CNN feature extraction module, AAVAE data enhancement module and BiLSTM network with EDBO algorithm introduced. The pork tenderloin freshness classification model is trained and tested using the data set produced in step 1, and the model classification performance is evaluated by loss function, accuracy, precision, recall and F1 score.

[0027] Compared with the previous technology, the present invention has the following beneficial effects:

[0028] 1. The present invention adopts a multidimensional feature extraction strategy for four different dimensional data, namely three-dimensional data (spatial spectrum information of HSI), two-dimensional data (ultraviolet fluorescence image) and two one-dimensional data (low-field nuclear magnetic resonance and electronic nose) in the feature extraction stage, and uses the HSI-CNN feature extraction module to fully utilize the spatial spectrum three-dimensional information of pork tenderloin hyperspectral imaging, and uses the UV-CNN feature extraction module to effectively capture the complex texture features of the surface of pork tenderloin in the ultraviolet fluorescence image, and performs feature fusion with low-field nuclear magnetic resonance and electronic nose data through the feature fusion layer, breaking the limitation of relying on only a single indicator, providing more comprehensive information on the internal moisture content and volatile compounds of pork tenderloin, and realizing the comprehensive fusion of multimodal data. The comprehensiveness and accuracy of the freshness evaluation of pork tenderloin are improved by the multimodal detection method that integrates visual, tactile and olfactory information.

[0029] 2. Since pork tenderloin changes from fresh to spoiled quickly due to the influence of microorganisms and enzymatic reactions, the number of sub-fresh samples is small, resulting in an uneven distribution of sample categories. To solve this problem, the present invention embeds an improved AAVAE data enhancement module after the feature fusion layer to specifically expand the number of samples in a small number of categories, improve the model's ability to recognize samples in a minority category, and improve the model's accuracy and stability.

[0030] 3. The present invention introduces an improved dung beetle optimization algorithm (EDBO) into the CNN-AAVAE-EDBO-BiLSTM model to automatically search for the optimal hyperparameters required by the BiLSTM network. This improvement overcomes the time-consuming and labor-intensive problems of traditional empirical methods and trial-and-error methods in hyperparameter selection, effectively avoids the reduction of model accuracy caused by human factors, and provides effective support for the accurate classification of pork tenderloin freshness. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 Schematic diagram of the process of the embodiment of the present invention

[0032] Figure 2 This is the improved UV-CNN network structure diagram in the present invention

[0033] Figure 3 This is the improved HSI-CNN network structure diagram in the present invention

[0034] Figure 4 The overall structure diagram of the improved CNN-AAVAE-EDBO-BiLSTM network in the present invention

[0035] Figure 5 This is a confusion matrix diagram of the classification results of the model without the AAVAE module and the model with the AAVAE module in the embodiment of the present invention. DETAILED DESCRIPTION

[0036] The specific implementation of the present invention is further described below in conjunction with the accompanying drawings and technical solutions.

[0037] like Figure 1 As shown, the overall steps of the present invention are as follows

[0038] Step 1: To ensure the accuracy of the designed pork tenderloin freshness classification model, samples of fresh pork tenderloin from the same batch were used for training and testing the designed pork tenderloin freshness classification model. Fresh pork tenderloin was divided into 30×30×10mm block samples of equal mass. A total of 60 samples were divided into two parts and stored in a 4°C refrigerator for later use. One part was used for HSI data, ultraviolet fluorescence image data, electronic nose data and LF-NMR data acquisition. In order to make the range of physicochemical values ​​wider, the storage time was extended to 14 days. All samples were taken out for data acquisition at 0, 3, 5 and 7-14 days, with a total sample size of 330. The other part was used for physicochemical value determination, and the corresponding TVB-N and TBARS values ​​were collected, with a total sample size of 330.

[0039] Step 1-1: The self-built UV fluorescence imaging system used for collecting UV fluorescence images consists of a camera (EOS 550D, Canon, China), a lens (EFS18-55mm, Canon, China), 2 UV-LEDs (365nm, light efficiency 150W) and a polarizing filter installed in front of the light source; the system was turned on half an hour before collecting images to obtain stable lighting conditions; the pork tenderloin was placed in the center of a black sample in a dark box, and the camera collected its top view image as a UV fluorescence image, a total of 330 images;

[0040] Step 1-2: Use a hyperspectral instrument (GaiaSorter-Dual full-band hyperspectral instrument, Sichuan Shuangli Hepu Technology Co., Ltd.) to collect hyperspectral images; turn on the system half an hour before collecting images to obtain stable lighting conditions; wipe the excess moisture on the surface of the pork tenderloin sample with kitchen paper, place it on a black sample, and use a spectral camera to collect its top view image; perform black and white correction on the hyperspectral image to eliminate the interference of noise and dark current, and use the pork tenderloin sample as the center of the image to crop it into a 512×512 hyperspectral image to create an original data set, with a total of 330 pork tenderloin hyperspectral data;

[0041] Step 1-3: Use Shanghai Newmai Nuclear Magnetic Resonance Imaging Analyzer to obtain low-field nuclear magnetic resonance data of pork tenderloin samples, a total of 330;

[0042] Step 1-4: Use the PEN3 portable electronic nose to collect odor information of 330 pork tenderloin samples at a constant ambient temperature of 23±2℃.

[0043] Step 1-5: Determine the TVB-N and TBARS contents of the sample immediately after the non-destructive testing data is collected. As shown in Formula 1, the determination of TVB-N content refers to the national standard GB 5009.228-2016 and is determined by the semi-micro nitrogen determination method. The determination of TBARS content refers to the national standard GB / T35252-2017 and is determined by the direct determination method.

[0044]

[0045] Step 2: In the task of classifying the freshness of pork tenderloin, not only irrelevant features such as blood stains and attachments will cause interference, but also the changes in the sensory characteristics of pork tenderloin in the early stage of corruption and the fluorescent substances accumulated by microbial metabolism are not obvious, and some key feature points are easily confused with interference factors such as background. In response to the above problems, an ultraviolet fluorescence image feature extraction module (UV-CNN) and an HSI feature extraction module (HSI-CNN) are designed for ultraviolet fluorescence images and HSI data, respectively, to enhance the extraction of key feature information of color appearance, texture, and fluorescent substances accumulated by microbial metabolism.

[0046] The ultraviolet fluorescence image feature extraction module (UV-CNN) is as follows Figure 2 As shown in the figure, by capturing the distribution and texture features of fluorescent substances on the surface of pork tenderloin in the ultraviolet fluorescence image, the captured features are then converted into a one-dimensional feature vector for subsequent processing; the UV-CNN module uses the traditional 2DCNN as the feature extraction backbone network, and makes the following improvements to the 2D convolution kernels of the same size in the traditional network, that is, four 2D convolution kernels of different sizes are used to replace the original 2D convolution kernels of the same size in the four convolution layers of the 2DCNN. Specifically, a 512×512×3 original image is input into the first Conv1, which uses 64 5×5 2D convolution kernels to extract basic low-level features and output a 256×256×64 feature map, which is input into the second Conv2 uses 128 3×3 2D convolution kernels to extract more complex intermediate features and output a feature map of 32×32×128. It then inputs the third Conv3, which uses 256 3×3 2D convolution kernels to extract higher-level features and outputs a feature map of 16×16×256. It finally inputs the fourth Conv4, which uses 512 3×3 2D convolution kernels to extract the highest-level features and finally outputs a feature map of 4×4×512. The extracted features are then flattened and fully connected to form a one-dimensional vector of 1024 for subsequent processing. The multi-scale convolution used can provide a powerful representation of the distribution of fluorescent substances and texture features on the surface of pork tenderloin.

[0047] The HSI feature extraction module (HSI-CNN) is as follows Figure 3As shown in the figure, by capturing the color appearance and surface texture features of pork tenderloin in hyperspectral data, the captured features are then converted into a one-dimensional feature vector for subsequent processing; the HSI-CNN module uses traditional 3DCNN as the feature extraction backbone network, reduces the original complex network structure to a five-layer structure to improve the network convergence speed, and introduces a residual block with an attention mechanism to assist deep feature transfer. Specifically, the network has a seven-layer structure. The original hyperspectral data (1, 512, 512, 360) is input into the first 3D convolution layer. 32 3D convolution kernels of size 3×3×3 are used to extract preliminary features. The output feature map has a shape of (1, 512, 512, 32). The feature map is input into the second pooling layer with a pooling window size of 2×2×2 to reduce the depth of the feature map and prevent overfitting. After this layer, the output feature map has a shape of (0.5, 256, 256, 32), which is then connected through the third residual connection block. It is used to focus on the deep features in the feature map and assist in passing the deep features of the feature map to the next convolution layer. The fourth 3D convolution layer uses 64 3D convolution kernels of size 3×3×3 to further extract more complex features and output a feature map of size (0.5, 256, 256, 64). The fifth pooling layer uses the same pooling window as the second layer to further reduce the size of the feature map, so that the feature map output by the fourth 3D convolution layer is output as a feature map of size (0.25, 128, 128, 64) after passing through this layer. The sixth layer uses the same residual connection block as the third layer to assist in passing deep features. The seventh layer is a global average pooling layer, which expands the extracted features into a one-dimensional feature vector of fixed length 64 for subsequent processing. Using fewer convolution layers can avoid the problem of excessively complex network structures with excessive calculations and long time consumption. The introduced residual connection and attention mechanism can effectively alleviate the gradient vanishing problem and enhance the ability to focus on key features.

[0048] Step 3: Design a data enhancement module (AAVAE) based on the VAE module to balance the number of samples in each category. In the pork tenderloin freshness classification task, the transition of pork tenderloin from fresh state to corrupt state is often relatively rapid, the duration of the sub-fresh stage is relatively short, and the number of samples in the sub-fresh category is small, resulting in an uneven number of samples in each category. To address the above problems, a data enhancement module (AAVAE) was designed with a variational autoencoder (VAE) as the backbone network. The feature data fused by the feature fusion layer described in step 4 was input into the encoder of the AAVAE for compression and the mean and variance were obtained. Then, the latent variables were collected from the latent space distribution with added noise through reparameterization. Finally, the latent variables were reconstructed into new data through the decoder part of the AAVAE. The mathematical expression is as follows:

[0049] The encoder outputs mean μ and logarithmic variance logσ 2 : μ, logσ 2=Encoder(x) (2)

[0050] Reparameterization:

[0051] Collecting noise:

[0052] Latent variable z: z=μ+σ·∈ (5)

[0053] Reconstructing data

[0054] Specifically, an adaptive adjustment strategy is introduced, that is, by calculating the distribution of the number of samples in each category in the data set, and training each minority category separately and generating supplementary samples until the number of samples in the maximum category is reached, randomness is introduced in the reparameterization process, and the reconstruction error and KL divergence are balanced in the damage function. The introduced adaptive adjustment strategy can enable AAVAE to sample and generate new data from the latent space and balance the gap in the number of samples in each category, improving the ability to identify category boundaries. The introduction of randomness in the reparameterization process ensures the continuity of the latent space and enhances the generalization ability of the model. Balancing the reconstruction error and KL divergence in the loss function can normalize the distribution of the latent space and prevent overfitting.

[0055] Step 4: With the BiLSTM network as the backbone network, the ultraviolet fluorescence image feature extraction module (UV-CNN) and HSI feature extraction module (HSI-CNN) in step 2 are introduced into the head. UV-CNN and HSI-CNN are connected in parallel. After the ultraviolet fluorescence image and HSI data pass through their corresponding feature extraction modules, the output one-dimensional feature data are combined with the electronic nose data and LF-NMR data to construct a feature fusion layer by splicing and fusion. The fused data is input into the data enhancement module in step 3 for data enhancement and expansion, which solves the problem of too few sub-fresh pork tenderloin samples and unbalanced number of samples in each category, improves the model's accurate recognition of various categories of pork tenderloin freshness, and inputs the data enhancement data set into the BiLSTM network for classification.

[0056] Step 5: Design an improved dung beetle optimization algorithm (EDBO) based on the dung beetle optimization algorithm (DBO). Aiming at the problem that the network hyperparameters in the pork tenderloin freshness classification task are easily disturbed by artificial trial and error and empirical methods, thus affecting the accuracy of the classification results, an improved dung beetle optimization algorithm (EDBO) is designed based on the dung beetle optimization algorithm (DBO). EDBO is used to first calculate the optimal solution of network hyperparameters such as the number of hidden layer neurons, initial learning rate and regularization coefficient in the model, and then the optimal solution is sent to the BiLSTM model to detect and classify the data set after data enhancement; the improved EDBO algorithm uses DBO as the optimization algorithm backbone network, in which adaptive parameter adjustment is introduced to replace the fixed parameter control in the original algorithm, and the search behavior is dynamically adjusted by real-time calculation and updating of parameter values. Its mathematical expression is shown in Formula 7:

[0057] P(t)=P max -(P max -P min )×(t / T) (7)

[0058] Where P(t) represents the reference value at time t, P max With P min Indicates the maximum and minimum values ​​of the parameter; t indicates the current number of iterations, and T indicates the maximum number of iterations. In the local search stage, a simulated annealing mechanism is introduced to control the acceptance probability of the solution through the temperature parameter to avoid falling into the local optimum. Its mathematical expression is shown in Formula 8:

[0059] P(t)=e -t / T (8)

[0060] Where t represents the current iteration number and T represents the maximum iteration number. The introduced adaptive parameter adjustment and simulated annealing mechanism improve the search ability and convergence performance of the algorithm. The maximum iteration number of the improved dung beetle optimization algorithm (EDBO) is 500, the producer ratio is 0.3, the memory size is 10, and the mutation rate range is 0.001.

[0061] The overall structure of the pork tenderloin freshness CNN-AAVAE-EDBO-BiLSTM classification model designed in this embodiment is as follows: Figure 4As shown in the figure, the constructed pork tenderloin freshness CNN-AAVAE-EDBO-BiLSTM classification model consists of HSI-CNN feature extraction module, UV-CNN feature extraction module, AAVAE data enhancement module and BiLSTM network with EDBO algorithm introduced. The pork tenderloin freshness classification model is trained and tested using the data set prepared in step 1, and the model classification performance is evaluated by loss function, accuracy, precision, recall and F1 score.

[0062] The classification results of the CNN-AAVAE-EDBO-BiLSTM model are shown in Table 3. The accuracy, precision, recall and F1 score are all greater than 95%, and the loss rate is 0.00155, indicating that the classification results of the model are relatively excellent. The above model shows good classification ability and high stability.

[0063] Experimental testing

[0064] 1. In order to explore the influence of different indicators on the freshness classification of pork tenderloin, this experiment uses HSI, ultraviolet fluorescence image, low-field nuclear magnetic resonance and electronic nose data as input to test the classification results of each single indicator. At the same time, in order to give full play to the complementary advantages of multimodal data, the classification results are tested after the four types of data are fused at the feature level to evaluate the improvement effect of multimodal data fusion on classification performance. As shown in Table 1, the accuracy of the CNN-BiLSTM model using multimodal data fusion is improved compared with the model with single data input, from 70.769%-89.231% to 90.769%, and the precision, recall and F1 scores are all above 60%. It shows that using multimodal data fusion as the input of the model can extract more information that is conducive to the freshness classification of pork tenderloin.

[0065] 2. Test the performance of the embedded EDBO module. As shown in Table 2, the model embedded with the EDBO module has improved its accuracy by 4.616%, precision by 25.743%, recall by 24.939%, and F1 score by 25.218% compared with the model without the optimization algorithm module; the accuracy is improved by 1.539%, precision by 0.217%, recall by 6.836%, and F1 score by 3.768% compared with the embedded unimproved DBO module, which illustrates the effectiveness and accuracy of the embedded improved EDBO module.

[0066] 3. Test the impact of embedding the AAVAE module on the model. As shown in Table 3, after embedding the AAVAE module, the accuracy of the model after data expansion increased by 1.716%, the precision increased by 12.657%, the recall rate increased by 8.382%, and the F1 score increased by 10.793%. Figure 5 As shown in , the test set confusion matrix of the two models is plotted. Figure 5 As can be seen from (a), in the model test set without data enhancement, the classification accuracy of fresh and corrupted products among the three categories reached more than 95%, while the classification accuracy of sub-fresh products only reached 50%. There were misclassifications in both fresh and corrupted products. Figure 5 As can be seen in (b), after the AAVAE module is used, the number of fresh and semi-fresh data sets is greatly expanded, and the classification accuracy of the three categories exceeds 95%. The improvement in model performance demonstrates the effectiveness of the AAVAE module.

[0067] Table 1

[0068]

[0069] Table 2

[0070]

[0071] Table 3

[0072]

Claims

1. A multimodal fusion detection method for pork tenderloin freshness based on deep learning, characterized in that: The steps are as follows: Step 1: Prepare samples for training and testing the designed freshness classification model of pork tenderloin. Collect hyperspectral imaging (HSI) data, ultraviolet fluorescence image data, electronic nose data, and low-field nuclear magnetic data during the storage of pork tenderloin, and preprocess the data to form a dataset; Step 2: Design feature extraction modules for ultraviolet fluorescence image data and HSI data respectively, analyze key feature information, reduce dimensions and redundant data to facilitate subsequent data fusion; Step 3: Design a data augmentation module based on variational autoencoder to enhance the recognition ability of the boundaries of various freshness categories of samples; Step 4: Introduce the ultraviolet fluorescence image feature extraction module, HSI feature extraction module, and data augmentation module AAVAE into the BiLSTM network structure to construct a freshness CNN-AAVAE-BiLSTM classification model for pork tenderloin; Step 5: Design an improved dung beetle optimization algorithm EDBO based on the dung beetle optimization algorithm to enhance the hyperparameter optimization ability of the model; Step 6: Test and evaluate the accuracy of the freshness CNN-AAVAE-EDBO-BiLSTM classification model for pork tenderloin.

2. The multimodal fusion detection method for freshness of pork tenderloin based on deep learning according to claim 1 is characterized in that: In Step 1, fresh pork tenderloin of the same batch is used to prepare samples for training and testing the designed freshness classification model of pork tenderloin. The fresh pork tenderloin is cut into equal-mass block samples of 30×30×10 mm. The samples are divided into two parts and stored in a 4°C refrigerator for later use. The specific steps of data collection and preprocessing are as follows: 1-1. The ultraviolet fluorescence imaging data is collected using an ultraviolet fluorescence imaging system, which mainly consists of a camera, a lens, 2 UV-LEDs, and a polarizing filter installed in front of the light source. The sample is placed in the center of a black template in a dark box, and its top-view image is collected by the camera as the ultraviolet fluorescence image data; 1-2. The hyperspectral image data of the sample is collected using a hyperspectral instrument system. After wiping off the excess moisture on the sample surface, it is placed on a black template, and its top-view image is collected by a spectral camera. The hyperspectral image data is corrected for black and white to eliminate the interference of noise dark current, and the HSI image data of 512×512 is cropped with the sample as the image center; 1-3. The low-field nuclear magnetic data of the sample is obtained using a nuclear magnetic resonance imaging analyzer; 1-4. The electronic nose data of the sample is collected using a portable electronic nose at a constant ambient temperature of 21°C - 25°C; The freshness of fresh pork tenderloin is divided into three grades: for fresh meat, TVB-N ≤ 15 mg / 100 g, TBARS < 0.5 mg MDA / kg; for sub-fresh meat, 15 mg / 100 g < TVB-N ≤ 20 mg / 100 g, 0.5 mg MDA / kg ≤ TBARS < 1.0 mg MDA / kg; for spoiled meat, TVB-N > 20 mg / 100 g, TBARS ≥ 1.0 mg MDA / kg. Among them, fresh meat corresponds to the label 0, sub-fresh meat corresponds to the label 1, and spoiled meat corresponds to the label 2.

3. The multimodal fusion detection method for freshness of pork tenderloin based on deep learning according to claim 1 is characterized in that: In Step 2, an ultraviolet fluorescence image feature extraction module and an HSI feature extraction module are designed for ultraviolet fluorescence image data and HSI data respectively; The ultraviolet fluorescence image feature extraction module first captures the distribution characteristics and texture characteristics of the fluorescent substances on the sample surface in the ultraviolet fluorescence image data, and then converts the captured distribution characteristics and texture characteristics into a one-dimensional feature vector for subsequent processing; the ultraviolet fluorescence image feature extraction module uses 2DCNN as the feature extraction backbone network, and improves the 2D convolution kernels of the same size, that is, four 2D convolution kernels of different sizes are used in the four convolution layers of 2DCNN to replace the original 2D convolution kernels of the same size, as follows: the first Conv1 uses 64 5×5 2D convolution kernels to extract basic low-level features, the second Conv2 uses 128 3×3 2D convolution kernels to extract more complex intermediate features, the third Conv3 uses 256 3×3 2D convolution kernels to extract higher-level features, and the fourth Conv4 uses 512 3×3 2D convolution kernels to extract the highest-level features; then the extracted multi-level features are expanded into a one-dimensional vector through the flattening layer and the fully connected layer for subsequent processing; The HSI feature extraction module first captures the color appearance and texture features of the sample surface in the HSI image data, and then converts the captured color appearance and texture features into a one-dimensional feature vector for subsequent processing; the HSI feature extraction module uses 3DCNN as the feature extraction backbone network, reduces the network structure of 3DCNN to a five-layer structure to improve the network convergence speed, and introduces a residual connection block with an attention mechanism to assist deep feature transfer, as follows: The HSI feature extraction module has a total of seven layers. The first 3D convolution layer uses 32 3D convolution kernels of size 3×3×3 to extract preliminary features; the second layer The pooling window size of the pooling layer is 2×2×2, which is used to reduce the depth of the feature map and prevent overfitting; the third layer of residual connection blocks is used to focus on the deep features in the feature map and assist in passing the deep features of the feature map to the next convolutional layer; the fourth layer of 3D convolutional layer uses 64 3D convolution kernels of size 3×3×3 to further extract more complex features; the fifth layer of pooling layer uses the same pooling window as the second layer to further reduce the size of the feature map; the sixth layer uses the same residual connection block as the third layer to assist in passing deep features; the seventh layer is a global average pooling layer, which expands the extracted features into a fixed-length vector for subsequent processing.

4. The multimodal fusion detection method for freshness of pork tenderloin based on deep learning according to claim 1 is characterized in that: In step 3, a data enhancement module is designed with a variational autoencoder as the backbone network. The data enhancement module consists of an encoder and a decoder. The feature data fused by the feature fusion layer in step 4 is input into the encoder of the data enhancement module for compression and the mean and variance are obtained. Then, latent variables are collected from the latent space distribution with added noise through reparameterization. Finally, the latent variables are reconstructed into new data through the decoder of the data enhancement module. The details are as follows: an adaptive adjustment strategy is introduced, that is, by calculating the distribution of the number of samples of each category in the data set of step 1, and training each minority category separately and generating supplementary samples until the number of samples of the maximum category is reached. At the same time, randomness is introduced in the reparameterization process, and the reconstruction error and KL divergence are balanced in the damage function. The introduced adaptive adjustment strategy enables the data enhancement module to sample and generate new data from the latent space and balance the gap in the number of samples of each category, thereby improving the ability to identify category boundaries. The introduction of randomness in the reparameterization process ensures the continuity of the latent space and enhances the generalization ability of the model. Balancing the reconstruction error and KL divergence in the loss function can normalize the distribution of the latent space and prevent overfitting.

5. The multimodal fusion detection method for freshness of pork tenderloin based on deep learning according to claim 1, characterized in that: In step 4, the BiLSTM network is used as the backbone network, and the ultraviolet fluorescence image feature extraction module and the HSI feature extraction module of step 2 are introduced in the head. The ultraviolet fluorescence image feature extraction module and the HSI feature extraction module are connected in parallel. After the ultraviolet fluorescence image data and the HSI data pass through the ultraviolet fluorescence image feature extraction module and the HSI feature extraction module respectively, the output one-dimensional feature data and the electronic nose data and the low-field nuclear magnetic resonance data are spliced ​​and fused to construct a feature fusion layer, and the fused data is input into the data enhancement module of step 3 for data enhancement and expansion. The data set after data enhancement is input into the BiLSTM network for classification.

6. The multimodal fusion detection method for freshness of pork tenderloin based on deep learning according to claim 1 is characterized in that: In step 5, an improved dung beetle optimization algorithm is designed based on the dung beetle optimization algorithm. The improved dung beetle optimization algorithm is used to first calculate the optimal solutions of network hyperparameters such as the number of hidden layer neurons, initial learning rate and regularization coefficient in the pork tenderloin freshness CNN-AAVAE-BiLSTM classification model in step 4, and then the optimal solution is sent to the pork tenderloin freshness CNN-AAVAE-BiLSTM classification model in step 4 to detect and classify the data set after data enhancement; the improved dung beetle optimization algorithm uses the dung beetle optimization algorithm as the backbone network of the optimization algorithm, in which adaptive parameter adjustment is introduced to replace the fixed parameter control in the original algorithm, and the search behavior is dynamically adjusted by real-time calculation and updating of parameter values. A simulated annealing mechanism is introduced in the local search stage, and the acceptance probability of the solution is controlled by the temperature parameter to avoid falling into the local optimum.

7. The multimodal fusion detection method for freshness of pork tenderloin based on deep learning according to claim 1, characterized in that: The constructed pork tenderloin freshness CNN-AAVAE-EDBO-BiLSTM classification model consists of HSI-CNN feature extraction module, UV-CNN feature extraction module, AAVAE data enhancement module and BiLSTM network with EDBO algorithm introduced. The pork tenderloin freshness classification model is trained and tested using the dataset produced in step 1, and the model classification performance is evaluated by loss rate, accuracy, precision, recall rate and F1 score.

Images