Method for predicting dynamic flavor of traditional fermented trachinotus ovatus based on multi-modal fusion deep learning

Abstract

The invention relates to a method for predicting the dynamic flavor of traditional fermented trachinotus ovatus based on multi-modal fusion deep learning, and the method comprises the steps: obtaining multi-modal data of to-be-detected traditional fermented trachinotus ovatus, the multi-modal data comprising chemical component analysis data, microbial community data and physical property data; inputting the multi-modal data into a deep learning model to obtain sensory evaluation data, the deep learning model being constructed according to a multi-modal fusion module and a hybrid deep neural network and being obtained based on training of a training set, the training set is a multi-dimensional data matrix comprising historical chemical component data, historical microbial community data, historical physical property data and corresponding sensory evaluation data. According to the method, real-time, dynamic and accurate prediction of the flavor in the fermentation process can be realized.

Description

Technical Field

[0001] The present invention relates to the technical field of traditional fermented food detection, and particularly to a method for predicting the dynamic flavor of traditional fermented Trachinotus ovatus based on multi-modal fusion deep learning. Background Art

[0002] In the field of traditional fermented foods, especially in the production of traditional fermented Trachinotus ovatus, the formation and change of flavor are crucial to product quality. At present, there are certain limitations in the research methods for the flavor of fermented foods. Traditional analysis methods are often static and cannot well capture the dynamic changes of flavor during the fermentation process. Moreover, in flavor analysis, it is mostly based on single-dimensional or a few-dimensional analysis, making it difficult to comprehensively and accurately reflect the complex flavor system. In addition, existing prediction methods have poor accuracy and adaptability when dealing with complex non-linear data such as flavor, with unreasonable dimension selection and integration, making it difficult to explore the deep relationships between dimensions, and it is difficult to accurately predict flavor changes and the model has poor adaptability when facing complex changes during the fermentation process. Summary of the Invention

[0003] The object of the present invention is to provide a method for predicting the dynamic flavor of traditional fermented Trachinotus ovatus based on multi-modal fusion deep learning, so as to realize real-time, dynamic, and accurate prediction of the flavor during the fermentation process, and provide strong support for the quality control and flavor optimization of traditional fermented Trachinotus ovatus.

[0004] To achieve the above object, the present invention provides the following solution:

[0005] A method for predicting the dynamic flavor of traditional fermented Trachinotus ovatus based on multi-modal fusion deep learning includes:

[0006] Obtaining multi-modal data of the traditional fermented Trachinotus ovatus to be detected, where the multi-modal data includes chemical composition analysis data, microbial community data, and physical property data;

[0007] Inputting the multi-modal data into a deep learning model to obtain sensory evaluation data, where the deep learning model is constructed according to a multi-modal fusion module and a hybrid deep neural network and is obtained by training based on a training set, and the training set is a multi-dimensional data matrix including historical chemical composition data, historical microbial community data, historical physical property data, and corresponding sensory evaluation data.

[0008] Optionally, obtaining the training set includes:

[0009] Sampling in a fermentation container to obtain a collected sample;

[0010] Analyzing the collected sample to obtain the historical chemical composition data;

[0011] Analyze the collected sample using high-throughput sequencing technology to obtain the historical microbial community data;

[0012] Preprocess the historical chemical composition data and the historical microbial community data to obtain preprocessed historical chemical composition data and preprocessed historical microbial community data;

[0013] Conduct a sensory evaluation of the collected sample to obtain sensory evaluation data;

[0014] Integrate the preprocessed historical chemical composition data, the preprocessed historical microbial community data, the historical physical property data, and the sensory evaluation data to obtain a multi-dimensional data matrix;

[0015] Obtain the training set based on the multi-dimensional data matrix.

[0016] Optionally, preprocessing the historical chemical composition data and the historical microbial community data includes:

[0017] Remove or correct obvious outliers from the historical chemical composition data, remove low-quality sequences and chimeric sequences from the historical microbial community data, and merge or ignore relative abundances below a preset value to obtain cleaned historical chemical composition data and historical microbial community data, where the low-quality sequences are sequences with lengths and base quality values below a preset value;

[0018] Perform standardization processing on the cleaned historical chemical composition data and historical microbial community data to obtain preprocessed historical chemical composition data and preprocessed historical microbial community data.

[0019] Optionally, obtaining the multi-dimensional data matrix includes:

[0020] Perform linear transformation or standardization processing on the sensory evaluation data to obtain processed sensory evaluation data;

[0021] Sort the preprocessed historical chemical composition data, the preprocessed historical microbial community data, the historical physical property data, and the processed sensory evaluation data to obtain the multi-dimensional data matrix.

[0022] Optionally, inputting the multi-modal data into a deep learning model to obtain sensory evaluation data includes:

[0023] Input the multi-modal data into the multi-modal fusion module, and the multi-modal fusion module fuses the multi-modal data according to the attention mechanism and outputs the fused data;

[0024] Input the fused data into the hybrid deep neural network for feature extraction, representation learning, and time series feature capture to obtain sensory evaluation data, where the sensory evaluation data includes predicted flavor intensity scores and flavor categories.

[0025] Optionally, the hybrid deep neural network includes a deep belief network, a recurrent neural network, and a gated recurrent unit;

[0026] The deep belief network includes a first hidden layer, a second hidden layer, and a third hidden layer. Among them, the first hidden layer is used to process the fused data to obtain a first feature vector, the second hidden layer is used to process the first feature vector to obtain a second feature vector, and the third hidden layer is used to process the second feature vector to obtain a third feature vector;

[0027] Input the third feature vector into the recurrent neural network to obtain the final hidden state, and based on the final hidden state, obtain the predicted flavor intensity score;

[0028] Input the third feature vector into the gated recurrent unit to obtain the predicted flavor category.

[0029] Optionally, training the deep learning model includes: using early stopping and model checkpoints when training the deep learning model to obtain the deep learning model.

[0030] Optionally, after preprocessing the historical chemical composition data and the historical microbial community data, it includes: performing transformations such as random translation, rotation, and scaling on the preprocessed historical chemical composition data, and adding normal distribution noise to the preprocessed historical microbial community data to obtain data-augmented data.

[0031] The beneficial effects of the present invention are: The present invention can achieve real-time, dynamic, and accurate prediction of flavors during the fermentation process, providing strong support for the quality control and flavor optimization of traditional fermented Trachinotus ovatus. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0033] Figure 1 It is a graph showing the change trend of the content of volatile compounds in fermented Trachinotus ovatus at different stages of the embodiments of the present invention;

[0034] Figure 2It is a graph of the microbial succession change of Trachinotus ovatus at different fermentation stages in the embodiments of the present invention. Among them, (a) is the α-diversity index of the microbial community, (b) is the microorganisms with the top 1% abundance in the D16 sample, (c) is the relative abundance of the microbial community composition at the phylum level, and (d) is the relative abundance of the microbial community composition at the genus level;

[0035] Figure 3 It is the sensory evaluation radar chart of the embodiments of the present invention;

[0036] Figure 4 It is the regression evaluation result of the predicted value and the true value of the initial model in the embodiments of the present invention;

[0037] Figure 5 It is the regression evaluation result of the predicted value and the true value of the optimized model in the embodiments of the present invention;

[0038] Figure 6 It is the importance ranking chart of important compounds of each sensory characteristic in the embodiments of the present invention. Among them, (a) is the importance ranking chart of important compounds with fruity aroma, (b) is the importance ranking chart of important compounds with floral aroma, (c) is the importance ranking chart of important compounds with fatty flavor, (d) is the importance ranking chart of important compounds with musty flavor, (e) is the importance ranking chart of important compounds with fishy smell, and (f) is the importance ranking chart of important compounds with putrid smell;

[0039] Figure 7 It is the flow chart of the dynamic flavor prediction method for traditional fermented Trachinotus ovatus based on multi-modal fusion deep learning in the embodiments of the present invention. Detailed implementation manners

[0040] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0041] To make the above objects, features, and advantages of the present invention more obvious and understandable, the present invention will be further described in detail below in conjunction with the drawings and specific implementation manners.

[0042] As Figure 7 shown, this embodiment provides a dynamic flavor prediction method for traditional fermented Trachinotus ovatus based on multi-modal fusion deep learning, including:

[0043] Obtain multi-modal data of the to-be-detected traditional fermented Trachinotus ovatus, where the multi-modal data includes chemical composition analysis data, microbial community data, and physical property data;

[0044] Input multimodal data into a deep learning model to obtain sensory evaluation data. Among them, the deep learning model is constructed based on a multimodal fusion module and a hybrid deep neural network, and is obtained by training based on a training set, which is a multi-dimensional data matrix including historical chemical composition data, historical microbial community data, historical physical property data, and corresponding sensory evaluation data.

[0045] Furthermore, obtaining the training set includes:

[0046] Sampling is carried out in a fermentation vessel to obtain collected samples;

[0047] Analyze the collected samples to obtain historical chemical composition data;

[0048] Use high-throughput sequencing technology to analyze the collected samples to obtain historical microbial community data;

[0049] Preprocess the historical chemical composition data and historical microbial community data to obtain preprocessed historical chemical composition data and preprocessed historical microbial community data;

[0050] Conduct sensory evaluation on the collected samples to obtain sensory evaluation data;

[0051] Integrate the preprocessed historical chemical composition data, preprocessed historical microbial community data, historical physical property data, and sensory evaluation data to obtain a multi-dimensional data matrix;

[0052] Obtain a training set according to the multi-dimensional data matrix.

[0053] Even further, preprocessing the historical chemical composition data and historical microbial community data includes:

[0054] Remove or correct obvious outliers from the historical chemical composition data, remove low-quality sequences and chimeric sequences from the historical microbial community data, and perform merging or ignoring processing on relative abundances below a preset value to obtain cleaned historical chemical composition data and historical microbial community data, where low-quality sequences are sequences with lengths and base quality values below the preset value;

[0055] Perform standardization processing on the cleaned historical chemical composition data and historical microbial community data to obtain preprocessed historical chemical composition data and preprocessed historical microbial community data.

[0056] Specifically, the acquisition of chemical composition analysis data includes: setting multiple sampling points in the fermentation container according to a certain spatial distribution (such as the top, middle, bottom, surrounding and center positions). After the fermentation starts, a suitable amount of oval pomfret samples are collected at specific time intervals using a sterile sampling tool. After pre-treatment of the collected samples, advanced analytical techniques (such as liquid chromatography-mass spectrometry (LC-MS), gas chromatography-mass spectrometry (GC-MS), etc.) are used to comprehensively detect the volatile and non-volatile flavor compounds therein, and obtain chemical composition data including the types and content information of flavor substances such as organic acids, amino acids, esters, aldehydes, etc.

[0057] The acquisition of microbial community data includes: collecting microorganisms in samples, using high-throughput sequencing technology (such as 16SrRNA sequencing, metagenomic sequencing, etc.) to analyze the structure and diversity of microbial communities, determining the dominant microbial species and their relative abundance in different fermentation stages, and obtaining microbial community data. Specific operations include extracting total DNA with Qiagen DNA mini kit, detecting purity and integrity with 1% agarose gel electrophoresis, constructing double-end PE libraries after DNA fragmentation with Covaris M220, sequencing on Illumina platform after bridge PCR, cutting sequences with fastp software, removing host DNA sequences with BWA alignment, splicing and assembling to retain the shortest length of 300pb, predicting open reading frames with MetaGene to obtain gene sets, constructing non-redundant genes with CD-HIT clustering, and statistically analyzing gene abundance information with SOAPaligner.

[0058] Sensory quantitative description analysis QDA: Organize a professional sensory evaluation team to conduct sensory evaluation of fermenting oval pomfret samples at specific time intervals. Use descriptive sensory analysis methods to evaluate and quantify appearance, smell, taste, mouthfeel, etc. to obtain sensory evaluation data. The evaluation team consists of 12 members (6 men and 6 women). The members have undergone systematic sensory training and have experience in quantitative description analysis. They refer to relevant standards to determine six flavor descriptors (floral, fruity, musty, fishy, ​​fat, and rancid) and reference standards. In a constant temperature (25°C) sensory analysis laboratory, the samples in the five fermentation stages are scored on a 0-10 intensity scale, and the final score is averaged.

[0059] Data cleaning includes: in chemical component analysis data, checking GC-MS and LC-MS spectrum peak area data, removing or correcting obvious abnormal values ​​(such as values ​​that are more than 5 times higher than the average peak area of ​​samples of the same type or less than 1 / 5 and are found to be caused by instrument or operation errors after review). In microbial community data, remove low-quality sequences (too short length, low base quality value, etc.) and chimera sequences, and appropriately handle (merge or ignore) microbial species data with extremely low relative abundance (less than 0.1% of the total number of sequences and low frequency in multiple samples).

[0060] Further, obtaining the multi-dimensional data matrix includes:

[0061] Performing linear transformation or normalization on the sensory evaluation data to obtain the processed sensory evaluation data;

[0062] Sorting the preprocessed historical chemical composition data, preprocessed historical microbial community data, historical physical property data, and processed sensory evaluation data to obtain the multi-dimensional data matrix.

[0063] Specifically, the normalization process includes: for the content of flavor compounds in the chemical composition data, using the min-max normalization method to map the values to the interval [0, 1]. In the microbial community data, dividing the relative abundance of each microorganism by the total to make it 1. The sensory evaluation data is linearly transformed or normalized according to its own range.

[0064] Data integration includes: constructing a multi-dimensional data matrix, arranging the chemical composition data (in the order of chemical categories and chromatogram peak elution order), microbial community data (in the taxonomic order), physical property data (such as pH value, conductivity, texture parameters, etc.), and sensory evaluation data (in the order of evaluation indicators such as odor, taste, and texture). Each data element is attached with detailed metadata information (sampling time, sampling point location, data measurement unit, data accuracy, etc.). Selecting the HDF5 format for storage, grouping by fermentation batches, stratifying by sampling time within batches, storing different-dimensional data in blocks for each layer, and performing compressed storage at the same time (selecting the appropriate compression algorithm according to the data type).

[0065] Further, inputting the multi-modal data into the deep learning model to obtain the sensory evaluation data includes: inputting the multi-modal data into the multi-modal fusion module, and the multi-modal fusion module fuses the multi-modal data according to the attention mechanism and outputs the fused data; inputting the fused data into the hybrid deep neural network for feature extraction, representation learning, and time series feature capture to obtain the sensory evaluation data, where the sensory evaluation data includes the predicted flavor intensity score and flavor category.

[0066] Specifically, in this embodiment, an innovative deep learning model is constructed, which includes a multi-modal fusion module and a hybrid deep neural network. The multi-modal fusion module, based on the attention mechanism, automatically learns the importance weights of the chemical composition, microbial community, physical property data, and sensory evaluation data in predicting flavor changes. The hybrid deep neural network combines the deep belief network (DBN) and the recurrent neural network (RNN, using the gated recurrent unit GRU). The DBN is used for feature extraction and representation learning, and the GRU captures time series features.

[0067] Even further, the hybrid deep neural network includes a deep belief network, a recurrent neural network, and a gated recurrent unit;

[0068] The deep belief network includes a first hidden layer, a second hidden layer, and a third hidden layer. Among them, the first hidden layer is used to process the fused data to obtain a first feature vector, the second hidden layer is used to process the first feature vector to obtain a second feature vector, and the third hidden layer is used to process the second feature vector to obtain a third feature vector;

[0069] Input the third feature vector into the recurrent neural network to obtain the final hidden state, and obtain the predicted flavor intensity score according to the final hidden state;

[0070] Input the third feature vector into the gated recurrent unit to obtain the predicted flavor category.

[0071] Furthermore, training the deep learning model includes: using the early stopping method and model checkpoints when training the deep learning model to obtain the deep learning model.

[0072] Furthermore, after preprocessing the historical chemical composition data and historical microbial community data, it includes: performing transformations such as random translation, rotation, and scaling on the preprocessed historical chemical composition data, and adding normal distribution noise to the preprocessed historical microbial community data to obtain data-augmented data.

[0073] Example 1:

[0074] The method of this embodiment will be further described below with reference to the accompanying drawings:

[0075] The traditional fermentation Trachinotus ovatus dynamic flavor prediction method based on multi-modal fusion deep learning includes:

[0076] (1) Multi-dimensional flavoromics data collection steps:

[0077] 1. Collection of flavor component analysis dimension data:

[0078] In the fermentation container (such as a fermentation tank or fermentation pool) of traditional fermentation Trachinotus ovatus, multiple sampling points are set according to a certain spatial distribution (such as positions at the top, middle, bottom, around, and center, etc.). At different time intervals after the start of fermentation (such as every 2 days, 4 days, 8 days, etc., determined according to the fermentation cycle), appropriate samples (such as 10 - 20g) are collected using sterile sampling tools.

[0079] The collected samples were immediately pre-treated. For the analysis of volatile flavor compounds, headspace solid-phase microextraction (HS-SPME) was first used to extract the volatile flavor compounds in fermented Trachinotus ovatus. 0.5 g of the fermented fish sample was placed in a 20 ml headspace vial. After adding 10 μL of the internal standard, it was incubated at 40 °C for 10 min. The SPME extraction head (DVB / CAR / PDMS, 50 / 30 μm × 1 cm) was aged at 270 °C for 10 min and then inserted into the headspace vial. It was adsorbed at 40 °C for 40 min. After the adsorption was completed, the SPME extraction head was transferred to the GC injection port and desorbed at 250 °C for 5 min.

[0080] The determination of volatile flavor substances in fermented Trachinotus ovatus was carried out by two-dimensional gas chromatography-time-of-flight mass spectrometry (GC×GC-TOF / MS). The system consisted of an Agilent 8890 gas chromatograph (Agilent Technologies, Palo Alto, CA, USA), a bipolar jet modulator and a LECO Pegasus BT 4D mass detector (LECO, St. Joseph, MI, USA). High-purity helium was used as the carrier gas with a constant flow rate of 1.0 ml / min. The initial temperature of the first-dimensional chromatographic column DB-Heavy Wax (30 m × 250 μm × 0.5 μm) was maintained at 40 °C for 5 min, then increased to 100 °C at a rate of 5 °C / min, increased to 120 °C at a rate of 2 °C / min, maintained for 3 min, and then increased to 250 °C at a rate of 6 °C / min and maintained for 5 min. The temperature programming of the second-dimensional chromatographic column Rxi-5Sil MS (2 m × 150 μm × 0.15 μm) was 5 °C higher than that of the first-dimensional chromatographic column. The modulator temperature was always 15 °C higher than the second-dimensional chromatographic column temperature, and the modulation period was 4.0 s. The temperature of the mass spectrometry transfer line was 250 °C, the ion source temperature was 250 °C, the acquisition rate was 200 spectra / s, the electron impact source was 70 eV, and the detector voltage was 1960 V. The mass spectrometry scanning range was m / z 35 - 550.

[0081] Experimental results and analysis:

[0082] The results of volatile compounds separated and identified based on HS-SPME and GC×GC-TOF-MS technologies are as Figure 1 shown. A total of 252 compounds were identified in the Trachinotus ovatus samples at five fermentation stages.

[0083] 2. Collection of dimensional data for microbial community analysis:

[0084] Total DNA was extracted from the cell pellet using the Qiagen DNA mini kit (Qiagen, Hilden, Germany) according to the manufacturer's instructions. After extraction, the purity and integrity of genomic DNA were detected by 1% agarose gel electrophoresis. After fragmenting the DNA (350 pb) using Covaris M220, a paired-end PE library was constructed, and after further bridge PCR, sequencing was performed using the Illumina platform. The fastp software (version 0.20.1) was used to trim the 3′ and 5′ ends of the sequences and remove reads with an average quality lower than 20 and a length shorter than 50 pb after quality trimming. The BWA (version 0.7.9a) was used to align the quality-controlled reads and remove the host DNA sequences to obtain high-quality sequences. The shortest length retained after sequence splicing and assembly was 300 pb. Then, MetaGene was used to predict the open reading frames in the contigs of the splicing results to obtain a gene set. The CD-HIT software was used to cluster the gene set (90% identity, 90% coverage) to construct non-redundant genes, and the SOAPaligner software was used to statistically analyze the abundance information of genes in each sample.

[0085] Experimental results and analysis:

[0086] As Figure 2 shown in (a), first, the species diversity and richness of the microbial community of fermented Trachinotus ovatus were evaluated by Simpson index, Shannon index, Chao index, Sobs index, and Ace index. The Sobs index directly reflects the number of species in the community, while the Chao index and Ace index reflect the richness of the bacterial flora; Figure 2 (b) reflects the top 1% of the abundant microorganisms, among which Staphylococcus equorum is 81% and occupies the main position. The bacterial flora composition was analyzed at the phylum and genus levels, as Figure 2 shown in (c)-(d). At the phylum level, Bacillota and Pseudomonadota are the two main phyla during the whole fermentation process. At the genus level, Staphylococcus and Mammaliicoccus are the dominant genera in the later stage of fermentation. They appear in the D4 sample and their abundance ratios continue to increase in the subsequent fermentation. Especially Staphylococcus, as the main dominant genus in the later stage of fermentation, accounts for 92.21%, 86.38%, and 82.85% in D8, D12, and D16, respectively.

[0087] 3. Data collection for sensory evaluation dimensions:

[0088] The sensory evaluation panel consisted of 12 members (6 males and 6 females). The panel members had received systematic sensory training beforehand and had experience in quantitative descriptive analysis. Each panel member voluntarily participated and had been informed about the preparation of experimental samples, evaluation methods, etc. Referring to relevant food standards and internal discussions among the sensory analysis team members, six flavor descriptors were determined, namely: floral flavor, fruity flavor, musty flavor, fishy smell, fatty flavor, putrid smell. The specific definitions and reference standards of each flavor descriptor are shown in Table 1. The sensory evaluators scored the Trachinotus ovatus samples at five fermentation stages on a scale of 0 - 10 (0: no odor perceived, 5: moderate odor, 10: strong odor) in a sensory analysis laboratory at a constant temperature (25°C). The final score for each aroma attribute was the average of the scores given by each member.

[0089] Table 1

[0090]

[0091] Experimental results and analysis:

[0092] The QDA method was used to analyze the aroma sensory differences and sensory characteristics among the Trachinotus ovatus at five fermentation stages. After discussion, experienced panel members used six descriptors, namely floral flavor, fruity flavor, musty flavor, fishy smell, fatty flavor, putrid smell, to analyze the aroma characteristics. The results of the sensory evaluation experiment are as Figure 3 shown. There were obvious differences in the sensory aspects of Trachinotus ovatus at different fermentation stages. As fermentation progressed, the fishy smell of Trachinotus ovatus was gradually weakened, and the fishy smell attribute of the Trachinotus ovatus samples on the 16th day was the lowest. The other aroma attributes generally showed an upward trend with the increase in the number of fermentation days, especially the fatty flavor and fruity flavor were the most obvious. This indicates that the fatty flavor and fruity flavor are the main sensory attributes of fermented Trachinotus ovatus. Among them, the Trachinotus ovatus products on the 16th day of fermentation had the highest comprehensive sensory score, the weakest fishy smell, and had the unique musty flavor, floral and fruity aroma, and fatty aroma of mature fermented Trachinotus ovatus.

[0093] (2) Data preprocessing and integration steps:

[0094] 1. Data cleaning:

[0095] In the chemical composition analysis data, check the peak area data of each sample in the GC-MS and LC-MS spectra. For peak area values that are significantly higher or lower than the normal range (determined based on multiple experiments and experience), they are judged as outliers and are either removed or corrected. For example, if a peak area value is more than 5 times or less than 1 / 5 of the average peak area value of the same type of sample, and it is confirmed to be caused by instrument or operation errors after recheck, then it is processed.

[0096] In microbial community data, check the quality of the sequencing data and remove low-quality sequences (such as those with too short length, low base quality value, etc.) and chimeric sequences. For microbial species data with extremely low relative abundance (such as less than 0.1% of the total number of sequences) and very low occurrence frequency in multiple samples, it can be regarded as noise data and processed appropriately (such as merging or ignoring).

[0097] 2. Standardization processing:

[0098] For the content data of flavor compounds in chemical composition data, the min-max normalization method is adopted to map the content values of each flavor compound in each sample to the interval [0,1]. For example, for the content x of a certain alcohol flavor compound, the normalized value x'=(x - min(x)) / (max(x)-min(x)), where min(x) and max(x) are the minimum and maximum content values of this alcohol flavor compound in all samples respectively.

[0099] In microbial community data, divide the relative abundance value of each microorganism in each sample by the sum of the relative abundances of all microorganisms to make the sum of the relative abundances of all microorganisms equal to 1, thus achieving standardization.

[0100] For sensory evaluation data, perform linear transformation or standardization processing according to their respective data ranges. For example, the sensory evaluation scores can be normalized according to the scoring scale (such as mapping the scores from 0 - 10 to the interval [0,1]).

[0101] 3. Data integration:

[0102] Data matrix construction method: When constructing a multi-dimensional data matrix, first arrange the chemical composition data, arranging according to the chemical categories of flavor compounds and the elution order on the chromatogram. For microbial community data, arrange it according to the taxonomic order of microorganisms (first bacteria, then fungi, and for bacteria, in the order of phylum, class, order, family, genus, species). The physical property data follows immediately, in the order of the pH value obtained by the pH meter and the texture parameters obtained by the texture analyzer, etc. Finally, there is the sensory evaluation data, arranged in the order of evaluation indicators such as odor, taste, and texture. Each data element is attached with detailed metadata information, including sampling time (accurate to minutes), sampling point location (represented by three-dimensional coordinates), data measurement unit (such as concentration unit mg / g, relative abundance unit unitless, etc.), and data precision (such as the minimum scale value of the measuring instrument).

[0103] Storage Format Selection and Optimization: Select the HDF5 (Hierarchical Data Format 5) format to store data. During the storage process, group the data according to fermentation batches, and then store them hierarchically according to the sampling time within each batch group. For each sampling time layer, store the data in different dimensions in chunks to improve the data reading and writing efficiency. For example, when querying all the data at a specific sampling point in the middle stage of a certain batch of fermentation, the data block at the corresponding batch group, sampling time layer, and sampling point position can be quickly located, reducing the data retrieval time. At the same time, compress the data for storage, and select an appropriate compression algorithm according to the data type (for example, use a lossless compression algorithm for a large number of numerical data in chemical composition data), reducing the storage space occupancy.

[0104] (III) Steps for Constructing the Deep Learning Model:

[0105] 1. Model Selection:

[0106] The constructed deep learning model includes a multi-modal fusion module and a hybrid deep neural network. The multi-modal fusion module is based on the attention mechanism and automatically learns the importance weights of chemical composition, microbial community, and sensory evaluation data in predicting flavor changes during the training process. In the hybrid deep neural network, the deep belief network (DBN) is used for feature extraction and representation learning of multi-dimensional data, which can automatically discover high-order features and hidden patterns in the data. The recurrent neural network (RNN) uses the gated recurrent unit (GRU) to capture the time series features of flavor data.

[0107] (1) Output Data of the Multi-modal Fusion Module:

[0108] Let the chemical composition data in the multi-dimensional data matrix be X chem , the microbial community data be X micro , the sensory evaluation data be X sensory , and the weights determined by the multi-modal fusion module be w 1 , w 2 , w 3 (w 1 + w 2 + w 3 = 1). Taking actual data as an example, assume w 1 = 0.4, w 2 = 0.35, w 3 = 0.25, the value of X chem after quantization and weighted calculation is 4.5, the value of X micro is 0.7, and the value of X sensory is 6.

[0109] The multi-modal fusion module outputs the fusion data X fusion The calculation formula is:

[0110] X fusion = w 1 ×X chem + w 2 ×X micro + w 3 ×X sensory

[0111] = 0.4×4.5 + 0.35×0.7 + 0.25×6 = 5.75

[0112] (2) Output data of the multimodal fusion module

[0113] The deep belief network (DBN) is set to have 3 hidden layers, with the number of nodes in each layer being 128, 64, and 32 respectively, and the input is X fusion = 5.75

[0114] The output of the first hidden layer (128 nodes) is H 1 (Assume the weight matrix W 1 , and the bias vector b 1 = 0, and the activation function f is sigmoid): H 1 = f(W 1 ×X fusion + b 1 ) = sigmoid(W 1 ×3.545 + 0), and after calculation, the partial values of H 1 are [0.18, 0.23, 0.12,..., 0.09] (the first eigenvector of the 128-dimensional vector set).

[0115] The output of the second hidden layer (64 nodes) is H 2 (Assume the weight matrix W 2 , and the bias vector b 2 = 0): H 2 = f(W 2 ×H 1 + b 2 ) = sigmoid(W 2 ×H 1 + 0), and assume that after calculation, the partial values of H 2 are (0.25, 0.31, 0.18,..., 0.11] (the 64-dimensional vector, i.e., the second eigenvector).

[0116] The output of the third hidden layer (32 nodes) is H 3 (Assume the weight matrix W 3 , and the bias vector b 3 = 0: H 3 = f(W 3 ×H 2 + b 3 ) = sigmoid(W3 ×H 2 +0), assuming that after calculation, H 3 The partial values are [0.32, 0.28, 0.22, …, 0.15] (a 32-dimensional vector, i.e., the third eigenvector).

[0117] (3) Output data of the recurrent neural network (taking the prediction of flavor intensity score regression task as an example):

[0118] For the recurrent neural network (RNN-GRU), assume the input sequence length T = 8, the GRU has 64 units, and the input is H 3 (a 32-dimensional vector, the actual values are [0.32, 0.28, 0.22, …, 0.15]).

[0119] At time step t = 1, update the gate z 1 (assuming the weight matrix W 2 , and the bias vector b z ), reset gate r 1 (assuming the weight matrix W r , and the bias vector b r ) and candidate hidden state (assuming the weight matrix W h , and the bias vector b h , and the initial hidden state h 0 is a vector of all 0s) calculate:

[0120] z 1 = σ(W z × [h 0 , H 3 (1)] + b z ) = σ(W z × [0, 0.32] + b z ), assuming that after calculation, z 1 = 0.28.

[0121] r 1 = σ(W r × [h 0 , H 3 (1)] + b r ) = σ(W r × [0, 0.32] + b r ), assuming that after calculation, r 1 = 0.22.

[0122]

[0123] Assuming that after calculation The partial values are [0.08, 0.12, 0.06, …, 0.03] (a 64-dimensional vector).

[0124] Calculate the hidden state at the current time step Obtain h 1 (64-dimensional vector)

[0125] Calculate the hidden states from t = 2 to t = 8 in sequence to obtain the final hidden state h 8 (Assume some values are [0.45, 0.38, 0.32, …, 0.22]).

[0126] Connect the fully connected layer (assuming the weight matrix W fc , and the bias b fc Predict the flavor intensity score y pred : y pred = W fc ×h 8 +b fc , and calculate y pred = 6.8.

[0127] (4) Output data of the recurrent neural network (taking the prediction of flavor category classification task as an example):

[0128] The input is also H 3 , and after being processed by GRU, the final hidden state h is obtained 8 (The calculation process is the same as above, h 8 = [0.52, 0.45, 0.36, …, 0.25]).

[0129] Assume 3 flavor categories (high fruit flavor, medium fruit flavor, low fruit flavor), and the softmax layer calculates where ( and have been initialized).

[0130] Calculate that the probability of high fruit flavor p 1 = 0.65, the probability of medium fruit flavor p 2 = 0.28, the probability of low fruit flavor p 3 = 0.07, and the model predicts the flavor category as high fruit flavor, and the probability distribution [0.65, 0.28, 0.07] is the predicted output.

[0131] Construct a model containing a multimodal fusion module and a hybrid deep neural network (DBN-GRU). The initial weights of the multimodal fusion module are randomly set within the range of [-0.1, 0.1]. The DBN part is set with 3 hidden layers, and the number of nodes in each layer is 128, 64, and 32 respectively. The weight initialization adopts the Xavier initialization method, and the bias is initialized to 0. The GRU part is set with 1 layer, containing 64 units, and its weights and biases also adopt appropriate initialization strategies (such as orthogonal initialization, etc.) to ensure good gradient flow characteristics of the model in the initial stage of training.

[0132] 2. Model Training:

[0133] 1) Divide the integrated multi-dimensional dataset into a training set, a validation set, and a test set according to the ratio of 70%:20%:10%.

[0134] 2) Training parameter optimization:

[0135] ① Learning rate adjustment strategy: Compare the three adaptive learning rate algorithms of Adagrad, Adadelta, and Adam. In the initial stage of training (the first 10 rounds), the model loss value drops relatively fast under the Adagrad algorithm. Its initial loss value is 2.5, and it drops to 1.8 after 10 rounds. However, in the later stage, around the 50th round, it fluctuates, and the loss value rises to 2.0. The loss value of the Adadelta algorithm drops relatively smoothly. The initial loss value is 2.6, it drops to 2.0 after 10 rounds, and it is 1.9 at the 50th round. The Adam algorithm has a relatively fast and stable drop in the loss value throughout the training process. The initial loss value is 2.5, it drops to 1.5 after 10 rounds, and it is 1.2 at the 50th round. Finally, select the Adam algorithm, set its initial learning rate to 0.001, and adopt an exponential decay strategy according to the performance of the validation set during the training process, decaying to 0.9 times the original every 30 rounds.

[0136] ② Batch size selection: Try different batch sizes in the range of 32 - 128. When the batch size is 32, the loss value fluctuates greatly during the model training process, and the average fluctuation amplitude reaches 0.3. When the batch size is 64, the loss value converges relatively fast and has good stability, and the average fluctuation amplitude is 0.1. When the batch size is 128, although the computing resources are fully utilized, the convergence speed slightly decreases, and the loss value after 100 rounds of training is still higher than the final loss value when the batch size is 64. Therefore, determine the batch size to be 64.

[0137] ③ Determination of the number of training rounds: By observing the change curve of the loss value on the validation set, it is found that around the 80th - 100th rounds, the loss value of the validation set tends to be stable and does not decrease significantly after 8 consecutive rounds of training. The specific data is that the loss value of the validation set is 0.8 at the 80th round, 0.78 at the 85th round, 0.77 at the 90th round, 0.78 at the 95th round, and 0.77 at the 100th round. Determine the optimal number of training rounds to be 100 rounds.

[0138] 3) Improvement measures for training strategies:

[0139] ① Early Stopping and Model Checkpoint: Set the tolerance threshold to stop training after the validation set loss value has not decreased for 5 consecutive rounds. During training, when this threshold is reached (around the 90th round), the model stops training and selects the checkpoint with the best performance on the validation set at this time as the final trained model. The validation set loss value was 0.77 at the 85th round, 0.76 at the 90th round, and 0.76 at the 95th round, triggering the early stopping mechanism. The final model is the model saved at the 90th round.

[0140] ② Data Augmentation Techniques: Perform random translations (with an amplitude range of ±0.1), rotations (with an angle range of ±5°), and scalings (with a scale range of 0.9 - 1.1) on the spectrogram data to generate new training data. Add normally distributed noise with a mean of 0 and a standard deviation of 0.001 to the microbial community data. After data augmentation, the loss value of the model on the training set decreases faster. Before data augmentation, the average loss value decreased by 0.05 per round in the first 30 rounds, and after augmentation, it decreased by 0.08 per round on average. The accuracy on the validation set also increased. The accuracy was 0.75 before augmentation and increased to 0.82 after augmentation.

[0141] ③ Adaptive Adversarial Training Mechanism: Construct a generator and a discriminator network, and let them compete with each other during training. At the beginning, the intensity of adversarial training is weak, with the loss weight of the generator set to 0.1 and the loss weight of the discriminator set to 0.9. As the model training progresses, gradually increase the intensity of adversarial training according to the performance on the validation set. At the 30th round, adjust the loss weight of the generator to 0.3 and the loss weight of the discriminator to 0.7; at the 60th round, adjust them to 0.5 and 0.5. When there are signs of overfitting in the model on the validation set (such as the accuracy no longer improving but the loss value continuing to decrease), appropriately reduce the intensity of adversarial training. At the 80th round, adjust back to 0.3 and 0.7. Through this adaptive adversarial training, when the model processes multi-modal data (chemical composition analysis data, microbial community data, and physical property data), it can better capture the complex relationships and features in the data. In the tasks of predicting flavor intensity scores and flavor categories, compared with when this mechanism is not used, the performance on the test set is better. When adaptive adversarial training is not adopted, the test set accuracy is 0.8, while after application, it has increased to 0.88, effectively improving the accuracy and reliability of the model in predicting the flavor of traditional fermented Trachinotus ovatus, providing strong support for realizing real-time, dynamic, and accurate prediction of flavor during the fermentation process, and playing a key role in the quality control and flavor optimization of traditional fermented foods.

[0142] ④Hyperparameter Optimization Based on Reinforcement Learning: Treat combinations of hyperparameters such as learning rate, batch size, number of nodes in the DBN hidden layer, and number of GRU units as actions, and consider the accuracy of the model on the validation set as the reward. Through the exploration of the reinforcement learning agent, a set of optimized hyperparameters is obtained: the learning rate is adjusted to 0.0005, the batch size remains 64, the number of nodes in the DBN hidden layer is adjusted to 100, 50, 25, and the number of GRU units is adjusted to 48. Retrain the model using the optimized hyperparameters, and the accuracy on the validation set increases from 0.82 to 0.88, and the accuracy on the test set increases from 0.8 to 0.92

[0143] 3. Model Evaluation and Optimization:

[0144] 1) Regression Task Evaluation (Taking the Prediction of Aroma Intensity Score as an Example):

[0145] ①Initial Model Evaluation Metrics: As Figure 4 shown, for the regression task of predicting the flavor intensity score, in addition to calculating the mean squared error (MSE) and mean absolute error (MAE), the root mean squared error (RMSE) and coefficient of determination (R 2 ) are also analyzed in depth. RMSE is obtained by taking the square root of MSE, and its unit is the same as the flavor intensity score, which more intuitively reflects the deviation degree between the predicted value and the true value. If the RMSE value is 0.5, it means that the predicted flavor intensity score on average differs from the true value by 0.5 points. R 2 is used to measure the fitting degree of the model to the data. The closer its value is to 1, the stronger the model's prediction ability for the flavor intensity score. By analyzing the changes in RMSE and R 2 at different training stages and on different datasets, the prediction accuracy and stability of the model are understood. The results show that the calculated mean squared error (MSE) is 0.8, the mean absolute error (MAE) is 0.6, the root mean squared error (RMSE) is approximately 0.89, and the coefficient of determination (R 2 ) is 0.7. By comparing the specific predicted values and true values of 10 samples in the test set (the true values are [4, 5, 6, 7, 8, 3, 4, 5, 6, 7], and the predicted values are [3.5, 4.8, 5.5, 6.2, 7.5, 2.8, 3.8, 4.5, 5.8, 6.5]), it can be seen that there is a certain deviation between the predicted values and the true values.

[0146] ②Optimized Model Evaluation Metrics: As Figure 5 shown, after the above series of improvements in training strategies and hyperparameter optimizations, the MSE is reduced to 0.4, the MAE is reduced to 0.35, the RMSE is approximately 0.63, and R 2Improved to 0.85. Comparing the new predicted values with the true values (the predicted values become [3.8, 4.9, 5.8, 6.8, 7.8, 3.2, 4.2, 4.8, 5.9, 6.8]), it can be found that the predicted values are closer to the true values, and the prediction accuracy of the model for the fruit flavor intensity score has been significantly improved. A smaller RMSE value means that the deviation between the predicted value and the true value is reduced. An R 2 value closer to 1 indicates a better fit of the model to the data.

[0147] 2) Extended analysis of classification task evaluation metrics: Taking the example of distinguishing high-fruit-flavor and low-fruit-flavor samples, assume that a fruit flavor intensity score greater than 5 is high-fruit-flavor and less than or equal to 5 is low-fruit-flavor. Calculate metrics such as mean squared error (MSE) and mean absolute error (MAE). where n is the number of samples in the test set, y i is the actual sensory evaluation score, is the score predicted by the model. For classification tasks such as predicting flavor categories, calculate metrics such as accuracy, recall, and F1 score.

[0148] ① Initial model evaluation metrics: The accuracy is 0.7, the recall is 0.65, and the F1 score is 0.67. This indicates that the initial model performs moderately in the classification task and there are certain misclassification situations.

[0149] ② After optimizing the model evaluation metrics, the accuracy is improved to 0.85, the recall is improved to 0.8, and the F1 score is improved to 0.82. This shows that the performance of the model in the classification task has been significantly improved, and it can more accurately distinguish high-fruit-flavor and low-fruit-flavor samples. According to Figure 6As shown in (a)-(f), the key compound corresponding to the "fruity aroma" is 2-heptanone; this compound has a fruity aroma similar to fresh apples and pears and is often used as a flavor additive to enhance fruit flavors in the food industry. The compound corresponding to the "floral aroma" is ethyl octanoate; ethyl octanoate is famous for its unique floral note and can mimic the aromas of roses, jasmine and other flowers, contributing to the flavor composition of Chinese liquor as an important ester. The key compound corresponding to the "fatty flavor" is 5-methyl-2-hexanone, which can mimic the aromas of foods such as meat or cream and enhance the richness of food flavors. △-Cadinene is a compound with a strong "musty smell" characteristic, which can mimic the unpleasant smell produced during the growth of mold in a humid environment. The key compound corresponding to the "fishy smell" is 2-acetyl-2-thiazoline; 2-acetyl-2-thiazoline is a compound with an obvious fishy smell, which can mimic the unique marine smell of fresh fish. In the food industry, this compound can be used to enhance the unique flavor of seafood and help mask unpleasant odors, thus improving the overall flavor quality of food. And (2E,4E)-2,4-octadienal, as a chemical marker of the "putrid smell", its production is closely related to the lipid oxidation process in food. The content of unsaturated fatty acids in Trachinotus ovatus is relatively high, and these fatty acids are prone to generate (2E,4E)-2,4-octadienal under the action of oxidation, thus giving the food a unique putrid smell.

[0150] IV. Steps for dynamic flavor monitoring and prediction:

[0151] 1. Real-time data collection and update - Optimization of data collection frequency and real-time processing technology:

[0152] Use a stream data processing framework (Apache Flink) to achieve real-time data processing. Establish an efficient data transmission channel between the data collection device (such as GC-MS, Illumina sequencing platform, etc.) and the processing platform, and the data is transmitted to the Flink cluster in the form of a stream. In the Flink cluster, utilize its parallel processing ability to perform parallel cleaning and standardization processing on each data stream (corresponding to different dimensional data). For example, for the data stream of chemical composition data, multiple processing nodes simultaneously perform cleaning and standardization operations on the chemical composition data collected at different sampling points. After processing, the data is promptly input into the trained deep learning model. To improve the data processing speed, configure a multi-core CPU and high-speed memory on the hardware, and optimize the execution parameters (such as parallelism, memory allocation, etc.) of the Flink job on the software to ensure that the newly collected data can be updated to the model for flavor prediction in the shortest time.

[0153] 2. Dynamic prediction and early warning - Determination of prediction thresholds and setting of early warning levels:

[0154] Details of the prediction threshold determination method: The prediction thresholds for different flavor indicators are determined by analyzing in detail the historical data and model training results of multiple batches of normal fermentation processes. For the content of key flavor substances, taking the ester compounds of Trachinotus ovatus as an example, analyze their content distribution during the normal fermentation process. Through statistical analysis, it is found that the average content is 5 mg / g and the standard deviation is 1 mg / g. According to quality control requirements and actual production experience, the lower threshold is set to the mean minus 1.5 times the standard deviation (i.e., 3.5 mg / g). When the model predicts that the content of this amino acid is lower than this threshold, it is considered that there may be a problem of insufficient umami. For the flavor intensity score, if the full score is set to 10 points, according to consumer acceptance and product quality standards, the lower threshold of the flavor intensity score is set to 7 points. When the predicted value is lower than this score, a warning is issued. During the determination of the threshold, the normal fluctuation range between different batches is fully considered, combined with statistical methods and the acceptable range in actual production to ensure the accuracy and timeliness of the warning.

[0155] Warning level setting and response strategy formulation: According to the deviation degree between the prediction result and the threshold, the warning level is divided into three levels: mild, moderate, and severe. When the predicted value of the flavor indicator is within the range of 90%-100% of the threshold (such as the predicted value of the key flavor substance content is above 90% of the threshold or the flavor intensity score is between 9-10 points), a mild warning is issued. At this time, the system prompts the operator to pay attention to the fermentation situation, and relevant data can be recorded in the background and simple data analysis can be carried out, such as checking the change trend of other flavor indicators in the recent period, but no immediate action is required. When the predicted value is within the range of 70%-90% of the threshold (such as the key flavor substance content is 70%-90% of the threshold or the flavor intensity score is between 7-9 points), a moderate warning is issued. At this time, it is recommended that the operator take preliminary adjustment measures. For example, for flavor problems related to temperature, if it is predicted that the flavor substance content is related to temperature and is moderately deviated from the threshold, the fermentation temperature can be slightly adjusted by ±2°C, and the subsequent changes in flavor indicators should be closely observed. At the same time, more detailed data analysis results and possible cause suggestions are displayed on the operation interface.

[0156] When the predicted value is lower than 70% of the threshold (for example, the content of key flavor substances is lower than 70% of the threshold or the flavor intensity score is lower than 7), a severe warning is issued. This indicates that the flavor has a serious abnormality and urgent measures need to be taken. For example, immediately stop the fermentation process and conduct a detailed laboratory analysis of the sample, including rechecking data in dimensions such as chemical composition and microbial community. At the same time, according to the specific flavor deficiency situation, specific flavor adjustment substances may need to be added, such as appropriate flavor enhancers or specific microbial inoculants. For the types and dosages of the added substances, they are implemented according to the pre-established emergency treatment plan, which is summarized based on a large number of experiments and historical data to ensure that the flavor abnormality can be corrected quickly and effectively and losses can be reduced. During the entire warning handling process, all operations and data changes are recorded in the system log for convenient subsequent tracing and problem analysis.

[0157] The embodiments described above are only descriptions of the preferred embodiments of the present invention and do not limit the scope of the present invention. Without departing from the design spirit of the present invention, various deformations and improvements made by those of ordinary skill in the art to the technical solutions of the present invention shall fall within the protection scope determined by the claims of the present invention.

Claims

1. A dynamic flavor prediction method for traditional fermented oval pomfret based on multimodal fusion deep learning, characterized in that: include: Acquiring multimodal data of the traditional fermented oval pomfret to be tested, wherein the multimodal data includes chemical composition analysis data, microbial community data, and physical property data; The multimodal data is input into a deep learning model to obtain sensory evaluation data, wherein the deep learning model is constructed based on a multimodal fusion module and a hybrid deep neural network, and is obtained based on training of a training set, and the training set is a multidimensional data matrix including historical chemical composition data, historical microbial community data, historical physical property data and corresponding sensory evaluation data.

2. The dynamic flavor prediction method of traditional fermented oval pomfret based on multimodal fusion deep learning according to claim 1, characterized in that: Acquiring the training set includes: Sampling is performed in the fermentation vessel to obtain the collection sample; Analyzing the collected samples to obtain the historical chemical composition data; Using high-throughput sequencing technology to analyze the collected samples to obtain the historical microbial community data; Preprocessing the historical chemical composition data and the historical microbial community data to obtain preprocessed historical chemical composition data and preprocessed historical microbial community data; Performing sensory evaluation on the collected samples to obtain sensory evaluation data; Integrate the pre-processed historical chemical composition data, the pre-processed historical microbial community data, the historical physical property data and the sensory evaluation data to obtain a multidimensional data matrix; The training set is obtained according to the multidimensional data matrix.

3. The dynamic flavor prediction method of traditional fermented oval pomfret based on multimodal fusion deep learning according to claim 2, characterized in that: Preprocessing the historical chemical composition data and the historical microbial community data includes: Eliminate or correct obvious abnormal values ​​from the historical chemical composition data, remove low-quality sequences and chimera sequences from the historical microbial community data, and merge or ignore relative abundances below a preset value to obtain cleaned historical chemical composition data and historical microbial community data, wherein the low-quality sequence is a sequence whose length and base quality value are lower than a preset value; The cleaned historical chemical composition data and historical microbial community data are standardized to obtain pre-processed historical chemical composition data and pre-processed historical microbial community data.

4. The method for predicting the dynamic flavor of traditional fermented oval pomfret based on multimodal fusion deep learning according to claim 2, characterized in that: Obtaining a multidimensional data matrix includes: Performing linear transformation or standardization processing on the sensory evaluation data to obtain processed sensory evaluation data; The pre-treated historical chemical composition data, the pre-treated historical microbial community data, the historical physical property data and the processed sensory evaluation data are sorted to obtain the multi-dimensional data matrix.

5. The method for predicting the dynamic flavor of traditional fermented oval pomfret based on multimodal fusion deep learning according to claim 1, characterized in that: Inputting the multimodal data into a deep learning model to obtain sensory evaluation data includes: Inputting the multimodal data into the multimodal fusion module, the multimodal fusion module fuses the multimodal data according to the attention mechanism, and outputs fused data; The fused data is input into the hybrid deep neural network for feature extraction, representation learning and time series feature capture to obtain the sensory evaluation data, wherein the sensory evaluation data includes a predicted flavor intensity score and a flavor category.

6. The method for predicting the dynamic flavor of traditional fermented oval pomfret based on multimodal fusion deep learning according to claim 5, characterized in that: The hybrid deep neural network includes a deep belief network, a recurrent neural network and a gated recurrent unit; The deep belief network includes a first hidden layer, a second hidden layer and a third hidden layer, wherein the first hidden layer is used to process the fused data to obtain a first eigenvector, the second hidden layer is used to process the first eigenvector to obtain a second eigenvector, and the third hidden layer is used to process the second eigenvector to obtain a third eigenvector; Inputting the third feature vector into the recurrent neural network to obtain a final hidden state, and obtaining a predicted flavor intensity score based on the final hidden state; The third feature vector is input into the gated recurrent unit to obtain a predicted flavor category.

7. The method for predicting the dynamic flavor of traditional fermented oval pomfret based on multimodal fusion deep learning according to claim 1, characterized in that: Training the deep learning model includes: using an early stopping method and a model checkpoint when training the deep learning model to obtain the deep learning model.

8. The method for predicting the dynamic flavor of traditional fermented oval pomfret based on multimodal fusion deep learning according to claim 2, characterized in that: The preprocessing of the historical chemical composition data and the historical microbial community data includes: performing random translation, rotation, scaling and other transformations on the preprocessed historical chemical composition data, adding normal distribution noise to the preprocessed historical microbial community data, and obtaining data after data enhancement.

Images