A multi-modal generative adversarial neural network modeling method and device for multi-dimensional sequence data

Abstract

The application discloses a kind of multi-mode generative adversarial network modeling method and device for multidimensional sequence data.It includes normalizing original observation data, obtaining formal uniform standard observation data, organizing examples for multi-mode generative adversarial network model training based on standard observation data, designing the structure and training mode of basic module, establishing multi-mode generative adversarial network model and other steps.This model is suitable for learning sequence data with missing partial index data, reducing the waste of incomplete data samples.At the same time, the model is trained in the feature space, reducing the complexity of data dimension and model training, avoiding the occurrence of overfitting phenomenon.In the training of the model, the multiple possible feature data at the same position are constantly updated, increasing the potential examples available for training.Compared with the traditional generative adversarial network, a large number of positive samples are needed for model training.

Description

Technical Field

[0001] This invention belongs to the field of computational methods and is a method and apparatus for modeling multi-mode generative adversarial networks for multi-dimensional sequence data. Background Technology

[0002] Sequences are a common form of information collection in life, such as natural language, speech, and time series. A sequence is composed of positions, each with multiple observable indicators, and these positions are correlated rather than independently distributed. Modeling multidimensional sequence data can solve most problems encountered in daily life and industrial production, such as trend analysis of stock market value, real-time monitoring of weather conditions, and yield prediction in fermentation processes. Therefore, establishing a method for building neural network models of multidimensional sequence data to learn the intrinsic mechanisms of sequence changes and predict observed indicators has become an important direction in sequence research within the field of artificial intelligence.

[0003] This patent describes the proposed modeling method using the simulation and prediction of bio-fermentation processes as an application scenario. A bio-fermentation process is a time series consisting of several time points. A set of fixed indicators is collected at each time point in the fermentation cycle; common indicators include transcriptomics, metabolomics, and fermentation process parameters, collectively referred to as system state data. In actual production, due to numerous uncontrollable factors such as operational errors, instrument malfunctions, and detection failures, real fermentation time series data inevitably contains missing time points or missing indicators. Furthermore, the system state changes during fermentation typically exhibit nonlinear characteristics; indicators are often coupled and correlated, and the number of data dimensions far exceeds the absolute number of samples.

[0004] Artificial neural networks can be used to perform deep modeling of complex, multidimensional nonlinear systems, obtaining highly accurate predictive models. Current time-series models first require preprocessing of missing data, learning from the system state at multiple consecutive time points to predict the next time point. This method typically requires a large number of examples for training, but in actual production, such a large and complete sample of data is often unavailable, thus limiting its industrial application value. Furthermore, the extraction of time-point feature data and the training of the time-series model are conducted in separate stages or frameworks, and the extracted feature data may not be the optimal representation for time-series modeling, thus failing to accurately model complex fermentation systems.

[0005] In summary, current methods for modeling multidimensional sequence data using artificial neural networks have the following shortcomings:

[0006] 1) In practical application scenarios, the number of sequence samples that can complete collection and detection of multiple indicators is small;

[0007] 2) In real samples, some sequence data with missing indicator data cannot be learned, resulting in data waste;

[0008] 3) The observation data has too many dimensions, which increases the computational complexity of the model and makes it easy to overfit.

[0009] 4) The feature extraction process, which is independent of the sequence model training, ignores the relationship between different sequence positions, and the obtained features may not be the optimal representation for sequence modeling.

[0010] 5) The model training process under the multi-stage framework is cumbersome and computationally complex, and it will cause error accumulation, resulting in poor model prediction performance.

[0011] To address the aforementioned problems, this patent proposes a novel multimodal generative adversarial neural network (GAN) for modeling multidimensional sequence data. This model consists of three sub-modules: a data generator (F), a generator (G), and a discriminator (D). Unlike traditional GANs, the proposed model does not use random noise as initial input. Instead, it employs multimodal generation, where each sequence position (e.g., a time point) corresponds to a data generator (F), which obtains feature data for each sequence position as input. The generator-discriminator hybrid module, composed of the data generator (F) and the discriminator (D), extracts the sequence position feature data and trains the discriminator. The generator-discriminator hybrid module, composed of the data generator (F), generator (G), and discriminator (D), generates the feature data for the next sequence position and trains the generator. By alternately training and learning the two hybrid structures, a sequence change mechanism learning model under optimal feature representation is obtained. This model not only achieves interpolation of missing values, optimal representation of feature data, and generator training within the same framework, but also introduces multiple possible feature representations for the same sequence position during training, increasing the number of potential examples that can be used for training and improving the model's accuracy.

[0012] Based on the above modeling method, a neural network model is constructed for the fermentation process. According to the current system state data, the system state after one unit of time can be accurately predicted. This can not only reduce the cost of industrial fermentation and increase product yield, but also guide the research and modification of engineered strains and obtain better industrial benefits. Summary of the Invention

[0013] This invention discloses a method and apparatus for multi-modal generative adversarial network (GAN) modeling of multi-dimensional sequence data, which can model the changes in multi-dimensional sequence data within a feature space using a GAN model. The technical solution provided by this invention is detailed below using time-series modeling of biological fermentation as an application scenario:

[0014] A method for modeling multi-modal generative adversarial networks for multi-dimensional sequence data includes the following steps:

[0015] 1) The original observation data is standardized and organized to obtain standardized observation data with a uniform format;

[0016] 2) An example of using canonical observation data for training multimodal generative adversarial network models;

[0017] 3) Design the structure and training mode of the basic modules, and establish a multi-modal generative adversarial network model;

[0018] 4) Use the training examples from step 2) to train the multimodal generative adversarial network model established in step 3) to obtain the model's parameter matrix;

[0019] 5) Using the training examples in step 2), evaluate the impact of the multi-modal generative adversarial network model structure designed in step 3) and the parameters updated during the training process in step 4) on the model accuracy, and select the optimal result under different combinations of structure and parameters as the final fermentation time series model.

[0020] 6) Based on the final fermentation timeline model, test cases are used to evaluate the accuracy of the model and to predict system state changes in real-world application scenarios.

[0021] The multimodal generative adversarial network model in step 3) has the following characteristics:

[0022] Its structure is a neural network containing three basic modules: a data generator (F), a generator (G), and a discriminator (D). Each time point corresponds to a data generator (F), which is a single-layer or multi-layer neural network structure with a fixed starting unit value. The data generator (F) and the discriminator (D) constitute the generation-discrimination chimera (FD), which is used for extracting feature data at each time point and training the discriminator (D). The data generator (F), the generator (G), and the discriminator (D) constitute the generation-discrimination chimera (FGD), which is used for generating feature data after each unit of time and training the generator. By alternating between the generation-discrimination chimera (FD) and the generation-discrimination chimera (FGD) modes, the fermentation time series model can be trained in the optimal feature space.

[0023] As a further improvement, the basic module structure of this invention is established as follows: each time point corresponds to a data generator (F), the input layer is a neuron with a fixed value, and the number of neurons in the output layer is the dimension of the feature data (to facilitate matrix operations in the neural network model, the data generators (F) at all time points will be integrated into a single mode that takes a scalar matrix as input for operation); the number of neurons in the input and output layers of the generator (G) is the dimension of the feature data; the number of neurons in the input layer of the discriminator (D) is the dimension of the feature data, and the number of neurons in the output layer is the dimension of the observed data. All model structures adopt a fully connected structure.

[0024] As a further improvement, step 1) of this invention specifically involves: obtaining multidimensional observation data from multiple batches and multiple time points, where each batch includes a set of time points, each time point's observation data includes a set of indicators, and each indicator's observation data is a specific value; organizing the multidimensional time series observation data into a four-tuple format, namely batch, time, indicator, and value; establishing a corresponding quantity matrix based on the total number of time points in the existing data, where the values ​​in column j and row j of the quantity matrix represent the input data of the data generator (F) at the j-th time point.

[0025] As a further improvement, the training examples organized in step 2) of this invention are divided into two groups according to different training stages. First, during the training of the generation-discrimination chimera module (FD), the data generator (F) at each time point and the corresponding system state observation data (represented in the form of quadruples) are used as one training example. After the network parameters of the chimera module are updated, feature data for all time points can be obtained; then, the generation-discrimination chimera module (FGD) is trained, and the data from two time points with a unit time interval are organized into one example, where the first time point uses the corresponding data generator (F), and the second time point uses the system state observation data (represented in the form of quadruples).

[0026] As a further improvement, the training method in step 4) of the present invention is specifically as follows:

[0027] 4.1) This model consists of two training steps: First, the generation-discrimination chimera (FD) calculates the model loss for all time points; then, the data generators (F) from the training examples are input into this training structure, and the network output data is obtained through forward propagation. The corresponding loss value is calculated using the network output data and the corresponding observation data. For example, T... a Time point data generator The input is fed into the fusion module to obtain network output data. With T a Observational data at time points Loss value can be obtainedF-D The formula is Based on the model loss value, the gradient is calculated through backpropagation and the network weight parameters (for convenience, network weight parameters and network bias parameters are collectively referred to as network weight parameters) in the (FD) chimeric module are updated using the gradient descent method.

[0028] 4.2) After updating the weight parameters of the data generator (F) and discriminator (D), the second step is performed: training the generator-discriminator (FGD) module. The network weight parameters of the discriminator (D) are fixed, and the loss value between the predicted value and the real data after a unit of time is calculated. For example, if T... a Time point data generator The input is fed into the (FGD) module to obtain network output data. With T a+1 Observational data at time points Loss value can be obtained F-G-D The formula is Based on the model loss value, the gradient is calculated through backpropagation, and the network weight parameters in the (FGD) splice module are updated using gradient descent. This process will update the parameters updated in the first step. The module's weight parameters are updated again to make its extracted features more suitable for time series modeling.

[0029] 4.3) Calculate the corresponding loss values ​​for the two chimeric modules (FD) and (FGD) in the model respectively, and determine whether the loss value of each chimeric module has converged. If they have converged, proceed to the next step; otherwise, continue with steps 4.1), 4.2), and 4.3) until the loss values ​​of the two chimeric modules have converged, and obtain the result model.

[0030] As a further improvement, step 5) of this invention specifically refers to optimizing the modeling process from the hyperparameter level. Specific hyperparameters include: feature dimension; structural parameters of network modules (number of nodes per layer in the data generator, generator, and discriminator networks, and number of hidden layers); learning rate of the gradient descent method; number of training data examples input for each training iteration; number of training iterations for a single splice module; number of iterations for alternating model training; constructing and training corresponding neural networks using different combinations of the above hyperparameters to obtain new resulting models; evaluating the fitting accuracy of the resulting models under different hyperparameter combinations to the observed data, and selecting the optimal model.

[0031] As a further improvement, the method of prediction using the constructed time series model in step 6) of this invention is as follows:

[0032] 6.1) The initial predicted T aThe observation data at each time point are standardized and organized. The network structure and parameters of the discriminator (D) and generator (G) in the optimal model obtained in step 5) are fixed. a Time point data generator Input into the (FD) chimera module, and T a Observational data at time points Loss value can be obtained F-D The formula is The gradient is calculated through backpropagation and then applied using the gradient descent method. The network weight parameters in the module are updated until the loss value converges.

[0033] 6.2) After completing the work on T a Time point data generator After the network parameters of the module are updated, they are input into the (FGD) chimera module to obtain T. a+1 Predictive data of system state at specific points in time

[0034] The present invention also discloses a multi-modal generative adversarial network modeling device for multi-dimensional sequence data, comprising the following units:

[0035] Acquisition Unit: Used to standardize and organize the raw observation data to obtain standardized observation data with uniform format.

[0036] Organizational unit: An example used to organize the acquired canonical observation data into a format suitable for training a multimodal generative adversarial network model.

[0037] Building blocks: used to design the structure and training modes of basic modules, and to build multi-modal generative adversarial network models.

[0038] Training unit: Used to train the established multimodal generative adversarial network model using training examples to obtain the model's parameter matrix.

[0039] Optimization Unit: Used to evaluate the impact of the designed multimodal generative adversarial network model structure and the parameters updated during the training process on the model accuracy using training examples, and select the optimal result under different combinations of structure and parameters as the final model.

[0040] Prediction Unit: Based on the final optimized model, it is trained using multidimensional observation data at any sequence position to obtain the data generator at that position; and it predicts the multidimensional observation data at the next sequence position, thus realizing the sequence prediction task with multidimensional parameters and multiple position intervals.

[0041] The designed multimodal generative adversarial neural network structure has the following characteristics:

[0042] The architecture primarily consists of three basic modules: a data generator (F), a generator (G), and a discriminator (D). Each sequence position (e.g., a time point) corresponds to a data generator. The generator-discriminator hybrid module, composed of the data generator (F) and the discriminator (D), extracts the sequence position feature data and trains the discriminator. The generator-discriminator hybrid module, composed of the data generator (F), generator (G), and discriminator (D), generates the feature data for the next sequence position and trains the generator. Through alternating training in these two modes, the learning of sequence change mechanisms can be achieved within the optimal feature space.

[0043] The beneficial effects of this invention are as follows:

[0044] 1) This model is suitable for learning from sequence data with missing indicators, reducing the waste of incomplete data samples. Furthermore, training the model in the feature space reduces data dimensionality and model training complexity, avoiding overfitting.

[0045] 2) Each sequence position corresponds to a data generator (F), which is jointly trained by the multidimensional observation data of the current position and the next position within the same framework. This associates the correlation between adjacent positions and improves the accuracy and fitting precision of the model.

[0046] 3) During model training, multiple possible feature data at the same location are continuously updated, increasing the number of potential examples that can be used for training. Compared with traditional generative adversarial networks, which require a large number of positive samples for model training, this model is suitable for application scenarios with small samples, such as biological fermentation.

[0047] 4) In the prediction task, based on multidimensional observation data at arbitrary sequence positions, a corresponding data generator (F) can be obtained. Through the calculation of the generator (G) in succession and the transformation of the discriminator (D), multidimensional observation data after multiple position intervals can be predicted.

[0048] 5) The multi-mode generative adversarial network structure and training method of the present invention can realize the prediction task of multi-dimensional parameters and multi-sequence position intervals within the system. Attached Figure Description

[0049] Figure 1 A schematic diagram illustrating the training steps of a multimodal generative adversarial network model;

[0050] Figure 2 A structural diagram of the basic module;

[0051] Figure 3 This is a schematic diagram of the structure of a multimodal generative adversarial network model. Detailed Implementation

[0052] The multimodal generative adversarial network model designed in this invention comprises three basic modules: a data generator (F), a generator (G), and a discriminator (D). It can be used to simulate and predict the changing mechanisms of biofermentation processes, guiding the optimization of industrial fermentation and the modification of engineered strains. The method disclosed in this invention models the process of a biofermentation system through the following steps. Figure 1 This is a schematic diagram of the model training steps.

[0053] 1. Organize the time series data into a quaternary organization format.

[0054] Indicators for bio-fermentation systems include indicators of the growth status of the microbial strain and fermentation process parameters. The growth status of the microbial strain can be characterized using various omics technologies (such as transcriptomics, proteomics, and metabolomics), while fermentation process parameters can be acquired in real-time using online monitoring equipment. The fermentation process is a time series consisting of multiple time points; therefore, this fermentation time series data can be organized into a four-part structure: batch, time, indicator, and value. Fermentation time series data is divided into training data and testing data. Training data is used to build the model, and testing data is used to evaluate the model's accuracy.

[0055] Assume that both training and testing data are metabolome data from multiple batches and time points measured during the fermentation of acarbose-based actinomycetes. Assume there are (M+U) batches (e.g., M=7, U=3), and each batch has (K+1) time points (e.g., K=10, each time point denoted as T1, T2, T3…T…). 11 ).

[0056] For (M+U) batches of actinomycete fermentation, samples at (K+1) time points corresponding to each batch were analyzed by mass spectrometry to determine the abundance of various compounds in each sample, i.e., fermentation metabolome data. q compounds related to acarbose synthesis were selected, and their abundance at each time point was extracted as observation data of the bio-fermentation system state at that time point. Compounds not detected by mass spectrometry were recorded as missing data in this batch of samples. Data for time points T3 of batch 2 and T2 of batch 3 were missing due to experimental failure in sample determination.

[0057] The (M+U) batches of actinomycete fermentation data are divided into training data and test data, with M batches as training data and U batches as test data. Therefore, there are a total of (M+U) batches of data, and each batch has (K+1) time points (T1, T2, T3…T…). K+1 Each time point has q-dimensional index data values, where the q-dimensional data of the p-th time point in the m-th batch is represented as follows:

[0058]

[0059] 2. Construct a multimodal generative adversarial neural network model

[0060] Figure 3 The multimodal generative adversarial neural network model designed for this patent consists of three basic modules: a data generator (F), a generator (G), and a discriminator (D). Each of these modules is a fully connected neural network composed of an input layer, hidden layers, and an output layer. A schematic diagram of the structure is shown below. Figure 2 As shown.

[0061] One of the features of this patent is its multimodal nature, where each time point corresponds to a data generator (F) with different neural network weight parameters. The input to the F module is a neuron with fixed values, the number of nodes in the output layer is equal to the dimension e of the feature data, the number of nodes in the hidden layer is s1, and the number of hidden layers is f1. To facilitate matrix operations in the neural network model, the data generators (F) at all time points are integrated into a mode that takes a scalar matrix as input for computation. For example, if the actinomycete fermentation training data has M batches, and each batch has (K+1) time points, then the data has a total of M(K+1) time points. Based on this result, a scalar matrix I can be constructed. Z Where Z = M(K+1). The values ​​in column j and row j of the scalar matrix represent the input data of the data generator (F) at the j-th time point. The input data of the data generator (F) at time point p in the m-th batch is represented as follows: The discriminator (D) has an input layer with nodes equal to the feature data dimension e, hidden layers with nodes s², a hidden layer length of f², and an output layer with nodes equal to the observation data dimension r. The generator (G) has an input and output layer with nodes equal to e, hidden layers with nodes s³, and a hidden layer length of f³.

[0062] The model comprises two types of inter-modules, participating in different training modes: a generator-discriminator inter-module consisting of a data generator (F) and a discriminator (D), and a generator-discriminator inter-module consisting of a data generator (F), a generator (G), and a discriminator (D). The network weight parameters of the D and G modules are respectively expressed as W... D W G This is represented. The multi-modal F module, due to its input mode of a scalar matrix, can also be represented using a single W module. F This is represented as follows. The initial weight parameters of the F, D, and G modules in the training structure are set to the same random values:

[0063] 3. Training cases for the tissue origination-discrimination chimerism module (FD)

[0064] Using M batches of actinomycete fermentation data as training data, the data generator (F) and the observation data of multidimensional indicators (quaternary tissue form) at any time point of any batch are organized into an example, T. a This indicates that at any given time point, the instance training data is organized as follows:

[0065] Batch 1 example set D1:

[0066] Batch 2 example set D2:

[0067] Batch 3 example set D3:

[0068] Batch m example set D m :

[0069] Where 4≤m≤M-1, m∈N *

[0070] Batch M example set D M :

[0071] Then, the examples from each batch are summarized, merged, and reorganized as follows:

[0072]

[0073] 4. Calculate the loss value of the occurrence-discrimination chimera module (FD).

[0074] Data collection F-D : Data generator at time points in the middle The input is fed into the FD interlocking module to obtain the output data. Where a≤K+1. With T a Multidimensional observation data at time points The loss value can be calculated. F-D The calculation formula is: For system state parameters with missing observations, the loss value is set to 0.

[0075] 5. Update the network weight parameters of the FD chimera module through backpropagation.

[0076] Based on the loss function defined in step 4, the FD chimera module uses the backpropagation algorithm to calculate the gradient and uses the SGD stochastic gradient descent method to update the weight parameters of the data generator (F) and decoder (D) multiple times in a loop.

[0077] The specific gradient calculation formula is as follows:

[0078]

[0079]

[0080] 6. Training cases for the Generative-Discriminatory Chimera Module (FGD)

[0081] After completing the loss calculation of the FD splicing module and updating the network weight parameters, the network structure of the discriminator (D) is fixed, and then the generator-discriminator pattern is trained.

[0082] Organize the data generator (F) at any time point within the same batch, along with the observed data of multidimensional indicators after a unit of time, into an example, T. a Let T represent any point in time, where T K+1 If there is no corresponding observation data after a unit of time at a given time point, it is denoted as Y. NA The instance training data is organized as follows:

[0083] Batch 1 example set D G1 :

[0084] Batch 2 example set D G2 :

[0085] Batch 3 example set D G3 :

[0086] Batch m example set D Gm :

[0087] Where 4≤m≤M-1, m∈N *

[0088] Batch M example set D GM :

[0089] Then, the examples from each batch are summarized, merged, and reorganized as follows:

[0090]

[0091] 7. Calculate the loss value of the generative-discriminatory chimera module (FGD).

[0092] Data collection F-G-D : Data generator at time points in the middle The input is fed into the FGD interlocking module to obtain the output data. Where a≤K+1. With T a+1Observational data at time points The loss value can be calculated. F-G-D The calculation formula is: All batches T K+1 The feature loss values ​​at each time point were all set to 0.

[0093] 8. Update the network weight parameters of the FGD chimeric module through backpropagation.

[0094] Based on the loss function defined in step 7, the FGD chimera module uses the backpropagation algorithm to calculate the gradient and uses the SGD stochastic gradient descent method to update the network weight parameters of the data generator (F) and the generator (G).

[0095] The specific gradient calculation formula is as follows:

[0096]

[0097]

[0098] 9. Dual-mode alternating training optimization

[0099] Gradient calculation is performed using the above method, and the network weight parameters of the two chimeric modules FD and FGD are updated alternately. After completion, the current training ends and the next training cycle begins.

[0100] The number of alternating training cycles is set to H. The number of times the weight parameters of the FD chimera module are updated in each training cycle is N, and the number of times the weight parameters of the FGD chimera module are updated is V. In the t-th training cycle, the initial weight parameters of the F, D, and G modules are the weight parameters updated in the (t-1)-th training cycle. The training process is as follows:

[0101] During the h-th alternation training, in the n-th training of the occurrence-discrimination chimerism module, the update gradients of the F and D modules are respectively: Where α represents the learning rate of the SGD gradient descent method; Let F and D represent the updated weight parameters of the F and D modules respectively after the (n-1)th training iteration. This represents the updated weight parameters of the corresponding module after the nth training iteration.

[0102] During the training of the generative-discriminative chimera module, the update gradients of modules F and G in the v-th training iteration are as follows: Where α represents the learning rate of the SGD gradient descent method; Let F and G represent the updated weight parameters of modules F and G respectively after the (v-1)th training iteration. This represents the updated weight parameters of the corresponding module after the v-th training iteration.

[0103] 10. Obtain the final model after all training structure losses have converged.

[0104] After each iteration of training in step 9, all training examples are input into the model. Following the method in step 4, the loss value of all data for the FD chimera module can be obtained; following the method in step 7, the loss value of all data for the FGD chimera module can be obtained. The specific calculation is as follows:

[0105] Calculate the loss value of the FD splicing module, i.e.

[0106] Calculate the loss value of the FG chimera module, i.e.

[0107] Determine Loss F-D and Loss F-G-D If all convergences have been achieved, proceed to the next step; otherwise, continue with the iterative optimization calculation in step 9 until the loss occurs. F-D and loss F-G-D All converge. This is denoted as the overall loss of the model converging after H alternating training cycles.

[0108] 11. Adjust the hyperparameters of the model to optimize the resulting model.

[0109] After obtaining the model through training in step 10, adjust the model's hyperparameters: feature dimension e, network module structural parameters (number of hidden layer nodes s1 and number of hidden layers f1 in the data generator; number of hidden layer nodes s2 and number of hidden layers f2 in the discriminator; number of hidden layer nodes s3 and number of hidden layers f3 in the generator); learning rate α for the SGD stochastic gradient descent method; number of alternating training cycles H; number of updates to the weight parameters of a single splicing module in each training cycle (number of updates N for the FD splicing module and number of updates V for the FGD splicing module); and number of training examples used in a single training cycle. After adjusting the combination of hyperparameters, repeat steps 4-10 until the network loss difference converges, then terminate training.

[0110] 12. Implement time-series data prediction and verify prediction accuracy.

[0111] Based on the optimal combination obtained after hyperparameter optimization in step 11, the final fermentation time series model is obtained, which can be used to predict time series data.

[0112] The fermentation data of U batches of actinomycetes described in the first step of the application case are used as test data, with each batch having K+1 time points (T1, T2, T3…T…). K+1Each time period has q indicators. The test data is organized according to the quaternary organization method in step 1, and the system state data at two time points separated by a unit time interval within the same batch is grouped into an example, T. a and T a+1 Let T represent adjacent time points in the example, where T is the time point. K+1 Since there is no corresponding system status data after a given unit of time at any given time point, it is not included in the test dataset. The instance test data is organized as follows:

[0113] Batch 1 example set D1:

[0114] Batch 2 example set D2:

[0115] Batch 3 example set D3:

[0116] Batch m example set D m :

[0117] Where 4≤m≤U-1, m∈N *

[0118] Batch example set D U :

[0119] Then, the examples from each batch are summarized, merged, and reorganized as follows:

[0120]

[0121] The total number of test cases in Test is denoted as T.

[0122] First, T a Time point data generator and system status data The training of the F module is completed through the FD chimera module, and then the FGD chimera module outputs the time T after a unit of time. a+1 Predictive system state data And perform an accuracy assessment:

[0123] The model's prediction accuracy is:

[0124] Furthermore, continuous computation via a tandem generator (G) can be used for accurate prediction of state changes in biofermentation systems over multiple unit time intervals.

[0125] The present invention also discloses a multi-modal generative adversarial network modeling device for multi-dimensional sequence data, comprising the following units:

[0126] Acquisition Unit: Used to standardize and organize the raw observation data to obtain standardized observation data with uniform format.

[0127] Organizational unit: An example used to organize the acquired canonical observation data into a format suitable for training a multimodal generative adversarial network model.

[0128] Building blocks: used to design the structure and training modes of basic modules, and to build multi-modal generative adversarial network models.

[0129] Training unit: Used to train the established multimodal generative adversarial network model using training examples to obtain the model's parameter matrix.

[0130] Optimization Unit: Used to evaluate the impact of the designed multimodal generative adversarial network model structure and the parameters updated during the training process on the model accuracy using training examples, and select the optimal result under different combinations of structure and parameters as the final model.

[0131] Prediction Unit: Based on the final optimized model, it is trained using multidimensional observation data at any sequence position to obtain the data generator at that position; and it predicts the multidimensional observation data at the next sequence position, thus realizing the sequence prediction task with multidimensional parameters and multiple position intervals.

[0132] The designed multimodal generative adversarial neural network structure has the following characteristics:

[0133] The architecture primarily consists of three basic modules: a data generator (F), a generator (G), and a discriminator (D). Each sequence position (e.g., a time point) corresponds to a data generator. The generator-discriminator hybrid module, composed of the data generator (F) and the discriminator (D), extracts the sequence position feature data and trains the discriminator. The generator-discriminator hybrid module, composed of the data generator (F), generator (G), and discriminator (D), generates the feature data for the next sequence position and trains the generator. Through alternating training in these two modes, the learning of sequence change mechanisms can be achieved within the optimal feature space.

[0134] The above is not intended to limit the specific embodiments of this patent. It should be noted that those skilled in the art can make various changes, modifications, additions, or substitutions without departing from the essential scope of this invention, and these improvements and refinements should also be considered within the scope of protection of this invention.

Claims

1. A method for modeling multi-modal generative adversarial networks for multi-dimensional sequence data, characterized in that, Includes the following steps: 1) The original observation data is standardized and organized to obtain standardized observation data with a uniform format; 2) An example of using canonical observation data for training multimodal generative adversarial network models; 3) Design the structure and training mode of the basic modules, and establish a multi-modal generative adversarial network model; 4) Use the training examples from step 2) to train the multimodal generative adversarial network model established in step 3) to obtain the model's parameter matrix; 5) Using the training examples in step 2), evaluate the impact of the multi-modal generative adversarial network model structure designed in step 3) and the parameters updated during the training process in step 4) on the model accuracy, and select the optimal result under different combinations of structure and parameters as the final fermentation time series model. 6) Based on the final fermentation timeline model, test cases are used to evaluate the accuracy of the model and to predict system state changes in real-world application scenarios. The multimodal generative adversarial network model in step 3) has the following characteristics: Its structure is a neural network containing three basic modules: a data generator F, a generator G, and a discriminator D. Each time point corresponds to a data generator F, which is a single-layer or multi-layer neural network structure with a fixed starting unit. The data generator F and the discriminator D constitute the generation-discrimination chimera module FD, which is used for extracting feature data at each time point and training the discriminator D. The data generator F, the generator G, and the discriminator D constitute the generation-discrimination chimera module FGD, which is used for generating feature data after a unit time and training the generator. Through alternating training of the generation-discrimination chimera module FD and the generation-discrimination chimera module FGD, the fermentation time series model can be trained in the optimal feature space. The basic module structure is established as follows: each time point corresponds to a data generator F, the input layer is a neuron with a fixed value, and the number of neurons in the output layer is the dimension of the feature data; the number of neurons in the input and output layers of the generator G is the dimension of the feature data; the number of neurons in the input layer of the discriminator D is the dimension of the feature data, and the number of neurons in the output layer is the dimension of the observed data. All model structures adopt a fully connected structure.

2. The method for multi-modal generative adversarial network modeling of multidimensional sequence data according to claim 1, characterized in that, In step 1), the method of normalizing and organizing the raw data is as follows: multiple batches of multidimensional observation data at multiple time points are obtained by detection. Each batch includes a set of time points, and the observation data at each time point includes a set of indicators. The observation data of each indicator is a specific value. The multidimensional time series observation data is organized into a four-tuple format, namely batch, time, indicator, and value. A corresponding quantity matrix is ​​established based on the total number of time points of the existing data. The values ​​in the j-th column and j-th row of the quantity matrix represent the input data of the data generator F at the j-th time point.

3. The method for modeling multi-modal generative adversarial networks for multi-dimensional sequence data according to claim 1, characterized in that, The training examples in step 2) are organized as follows: they are divided into two groups according to different training stages. First, during the training of the generation-discrimination chimera module FD, the data generator F and the corresponding system state observation data at each time point are represented as a quadruple and used as one training example. After the network parameters of the chimera module are updated, the feature data of all time points are obtained. Then, the generation-discrimination chimera module FGD is trained, and the data of two time points with a time interval of one unit are organized into one example, where the first time point uses the corresponding data generator F and the second time point uses the system state observation data, represented as a quadruple.

4. The method for modeling multi-mode generative adversarial networks for multi-dimensional sequence data according to claim 1, characterized in that, The training method in step 4) is specifically as follows: 4.1) First, the generation-discrimination chimera module FD calculates the model loss for all time points; the data generator F from the training examples is input into the training structure, and the network output data is obtained through forward propagation. The corresponding loss value is calculated from the network output data and the corresponding observation data; based on the model loss value, the gradient is calculated through backpropagation and the network weight parameters in the FD chimera module are updated using the gradient descent method. 4.2) After updating the weight parameters of the data generator F and discriminator D, the generator-discriminator hybrid module FGD is trained. The network weight parameters of the discriminator D are fixed, and the loss value between the predicted value and the real data after a unit time is calculated. Based on the model loss value, the gradient is calculated through backpropagation, and the network weight parameters in the FGD hybrid module are updated using gradient descent. This process will affect the updated parameters in 4.1). The module's weight parameters are updated again to make its extracted features more suitable for time series modeling; 4.3) Calculate the corresponding loss values ​​for the two chimeric modules FD and FGD in the model respectively, and determine whether the loss values ​​of each chimeric module have converged. If they have converged, proceed to step 5); otherwise, continue with steps 4.1), 4.2), and 4.3) until the loss values ​​of the two chimeric modules have converged, and obtain the result model.

5. The method for modeling multi-modal generative adversarial networks for multi-dimensional sequence data according to claim 1, characterized in that, Step 5) specifically refers to optimizing the modeling process from the hyperparameter level. Specific hyperparameters include: feature dimension; structural parameters of network modules, including the number of nodes per layer in the data generator, generator, and discriminator networks, and the number of hidden layers; the learning rate of the gradient descent method; the number of training data examples input for each training iteration; the number of training iterations for a single chimeric module; the number of iterations for alternating model training; constructing and training corresponding neural networks using different combinations of the above hyperparameters to obtain new resulting models; evaluating the fitting accuracy of the resulting models under different hyperparameter combinations to the observed data, and selecting the optimal model.

6. The method for modeling multi-modal generative adversarial networks for multi-dimensional sequence data according to claim 1, characterized in that, The method for prediction using the time series model in step 6) is as follows: 6.1) The initial predicted T a The observation data at each time point are standardized and organized. The network structure and parameters of the discriminator D and generator G in the optimal model obtained in step 5) are fixed, and T is then... a Time point data generator The network output data is obtained by inputting into the FD chimera module, and the output data is then compared with T. a The loss value is calculated based on the observation data at each time point, the gradient is calculated through backpropagation, and the gradient descent method is applied. The network weight parameters in the module are updated until the loss value converges; 6.2) After completing the work on T a Time point data generator After the network parameters of the module are updated, they are input into the FGD fusion module to obtain T. a+1 Predictive data on the system state at specific points in time.

7. A multi-modal generative adversarial network modeling device for multi-dimensional sequence data, characterized in that, Includes the following units: Acquisition Unit: Used to standardize and organize raw observation data to obtain standardized observation data with uniform format; Organizational unit: An example used to organize the acquired canonical observation data into a format suitable for training a multimodal generative adversarial network model; Building blocks: used to design the structure and training modes of basic modules, and to build multimodal generative adversarial network models; Training unit: Used to train the established multimodal generative adversarial network model using training examples to obtain the model's parameter matrix; Optimization Unit: Used to evaluate the impact of the designed multimodal generative adversarial network model structure and the parameters updated during the training process on the model accuracy using training examples, and select the optimal result under different combinations of structure and parameters as the final model. Prediction Unit: Based on the final optimized model, it is trained using multidimensional observation data at any sequence position to obtain the data generator at that position; and it predicts the multidimensional observation data for the next sequence position, thus realizing the sequence prediction task for multidimensional parameters and multiple position intervals. The multimodal generative adversarial neural network structure described above has the following characteristics: The structure mainly consists of three basic modules: a data generator F, a generator G, and a discriminator D. Each sequence position corresponds to a data generator. The generator-discriminator hybrid module, composed of the data generator F and the discriminator D, can extract the sequence position feature data and train the discriminator. The generator-discriminator hybrid module, composed of the data generator F, the generator G, and the discriminator D, can generate the feature data of the next sequence position and train the generator. Through alternating training in the two modes, it is possible to learn the sequence change mechanism in the optimal feature space. The basic module structure is established as follows: each time point corresponds to a data generator F, the input layer is a neuron with a fixed value, and the number of neurons in the output layer is the dimension of the feature data; the number of neurons in the input and output layers of the generator G is the dimension of the feature data; the number of neurons in the input layer of the discriminator D is the dimension of the feature data, and the number of neurons in the output layer is the dimension of the observed data. All model structures adopt a fully connected structure.

Images