Multimodal information fusion method, system, terminal and storage medium for protein representation learning

Abstract

The multimodal information fusion method, system, terminal and storage medium for protein representation learning provided in the present application use the strategy of early extraction, mid-term fusion and late prediction to enable each single-modal model to fully extract the high-level semantic information of its own modality before fusion, and then use a feedforward neural network to predict the task in the later stage; at the same time, a multimodal fusion module is proposed, which can perform fine-grained interaction on the different modal information of each layer of the network during mid-term fusion, so as to better fuse and pass on the multimodality; in the last layer of the feature extractor in the late prediction stage, the fused multimodal embedding and the previous single-modal embedding are spliced ​​together as the representation of the protein itself, so as to retain the original information of the single modality to the greatest extent.

Description

Technical Field

[0001] The present application belongs to the field of medical data processing technology, and specifically relates to a multimodal information fusion method, system, terminal and storage medium for protein characterization learning. Background Art

[0002] Protein representation learning is a very important research topic in the field of bioinformatics. It plays a critical role in predicting protein-protein interactions, protein-drug interactions, protein-gene interactions, etc. A good data representation should be able to cover the information of the object itself in all aspects, so that the reasoning process of downstream tasks has more available feature support.

[0003] In the computational research of proteins, proteins need to be converted into data that can be processed by computers. Before the raw data is input into the model, features need to be extracted. This process is called representation learning. A good representation learning is of great help in improving the performance of downstream tasks. The representation learning of proteins can be divided into single-modal representation and multi-modal representation.

[0004] In single-mode, learning mainly targets sequence and structure features respectively. Protein sequences are similar to text sequences, and can be modeled by borrowing techniques from the NLP field. In the past, some studies used CNN to perform one-dimensional convolution on protein sequences, extracting sequence features of proteins for subsequent tasks; some studies also used RNN models that excel in time series data, and also achieved good results. Recently, many people have tried Transformer, which has made breakthrough progress in the fields of NLP and CV, to pre-train large-scale protein sequences and achieve better results in downstream tasks. Compared with the sequence mode, the structural mode of proteins is also crucial to understanding the protein itself. There are fewer studies on modeling protein structures than sequences. Some studies convert 3D protein structures into images, and then use CNN to extract features to represent proteins. Other studies flatten 3D structures into adjacency matrices of amino acid nodes, and then use graph neural network algorithms to model them.

[0005] In multimodality, the key is how to fuse unimodal information. Most studies use different feature extractors to extract unimodal information, and then concatenate or add up the embeddings of different modalities to obtain new embeddings as multimodal representations. Some people input the embeddings into a new interactive network after concatenation or addition, such as Transformer, to obtain an interactive embedding.

[0006] Many current multimodal fusion methods simply concatenate or add up the representations of single modalities. This method cannot learn the interaction information between modalities in a fine-grained manner, and the resulting representation vector will lose a lot of information. Some studies consider learning the interaction between modalities. They concatenate the data of the two modalities in the initial embedding layer of the original data, and then pass it into the encoding layer of the Transformer to learn the relationship between tokens. However, this method fuses the modalities in the early stage, resulting in each modality not fully extracting high-level semantic information before merging with other modalities, and performing unsatisfactory in subsequent tasks. In addition, almost all studies extract multimodal representations and use them directly downstream, but no matter how well the multimodal features are learned, there will always be single-modal information lost in the transmission process. Summary of the invention

[0007] In view of this, it is necessary to provide a multimodal information fusion method for protein characterization learning that can retain the original information of a single modality to the greatest extent in order to address the defects of the existing technology.

[0008] To solve the above problems, this application adopts the following technical solutions:

[0009] One of the purposes of this application is to provide a multimodal information fusion method for protein characterization learning, comprising the following steps:

[0010] Preprocessing of open source protein data;

[0011] Dividing the protein dataset into a training set, a validation set, and a test set;

[0012] constructing a unimodal feature extractor, wherein the unimodal feature extractor is used as a feature extractor for protein sequences;

[0013] Constructing a multimodal fusion module, wherein the multimodal fusion module updates the amino acid token embedding of the unimodal feature extractor so that the unimodal has information of multiple models and serves as the input of the unimodal feature extractor;

[0014] Building a learning model based on the multimodal fusion module;

[0015] The training set is used to train the learning model, the validation set is used to measure the effect of the learning model, and the parameters with the best performance are selected as the parameters of the learning model. The test set is used to independently test the generalization ability of the learning model.

[0016] In some of the embodiments, the step of preprocessing the open source protein data specifically includes the following steps:

[0017] The sequence data of the protein is extracted from the open source protein data set, the sequence consists of 20 English letters, and the 20 English letters represent 20 amino acids, and the 3D structure of the protein is converted into an adjacency matrix diagram.

[0018] In some of the embodiments, the step of constructing a unimodal feature extractor specifically includes:

[0019] The unimodal feature extractor is a pre-trained Transformer model.

[0020] In some of the embodiments, the step of constructing a multimodal fusion module specifically includes the following steps:

[0021] The sequence feature matrix and the structural feature matrix are averaged and pooled, and the feature vector of each amino acid is given a representative value. The formula is as follows, where

[0022]

[0023]

[0024] They represent the sequence feature matrix and structural feature matrix before inputting the multimodal module, respectively, where D seq Represents the characteristic dimension of each amino acid in the sequence, D struc Represents the characteristic dimension of each amino acid in the structure, L seq and L struc They represent the amino acid lengths in the sequence and structure, but they are actually equal, i.e. L seq =L struc =L;

[0025] The pooled vectors of the sequence and structure are concatenated and then transformed into a vector containing multimodal information through a fully connected network. The formula is as follows:

[0026] M comp =W[M seq ,M struc ]+b

[0027] in, Let D comp =(L seq +L struc ) / 5;

[0028] Compress the multimodal information into vector M comp Re-dividing the data into different modes to calibrate the single-mode information, the diversion process is to introduce the fully connected transformation layer of each mode respectively, the formula is as follows:

[0029] T seq =Wseq M comp +b seq ,

[0030] T struc =W struc M comp +b struc ,

[0031] The modal vector of the shunt transformation is activated through the activation function as a gate switch to limit the contribution of each amino acid to the overall task. The specific formula is as follows:

[0032]

[0033]

[0034] Where σ refers to the sigmoid function, ⊙ refers to the Hadamard product;

[0035] After multiplying with the activated gated vector, the reconstructed unimodal vector is obtained as the input of the next layer of unimodal feature extractor.

[0036] In some embodiments, the step of constructing a learning model based on the multimodal fusion module specifically includes the following steps:

[0037] Add a special token named [cls] to the original input of protein sequence and structure. The [cls] of the sequence is placed at the front of the entire sequence, and the [cls] of the structure establishes a virtual full connection with all amino acids.

[0038] The original protein data was processed by N e The early unimodal feature extractor of the layer, the sequence passes through the encoding layer of the Transformer model, the structure passes through the graph attention network layer, and the output result represents the unimodal vector representation of the extracted high-level semantics;

[0039] Inserting the multimodal fusion module to perform mid-term fusion;

[0040] After mid-term fusion, the single modality is calibrated by multimodal information and continues through N l Layer feature extractor, further calibrated feature mining;

[0041] The [cls] vectors of the two modalities mined by the calibrated feature are concatenated, then passed through a feed-forward neural network, and then concatenated with the [cls] vector obtained by the early single-modal feature extractor;

[0042] After passing through the second feedforward neural network, the learning model is obtained.

[0043] In some embodiments, after completing the step of building a learning model based on the multimodal fusion module, the following steps are also included:

[0044] Auxiliary loss is added to update the parameters of the learning model.

[0045] The second purpose of this application is to provide a multimodal information fusion system for protein characterization learning, comprising:

[0046] Data processing unit: used to pre-process open source protein data;

[0047] Classification unit: used to divide the protein dataset into a training set, a validation set and a test set;

[0048] A unimodal feature extractor construction unit: used to construct a unimodal feature extractor, wherein the unimodal feature extractor is used as a feature extractor for protein sequences;

[0049] A multimodal fusion module construction unit: used to construct a multimodal fusion module, wherein the multimodal fusion module updates the amino acid token embedding of the single-modal feature extractor so that the single-modal has information of multiple models and serves as the input of the single-modal feature extractor;

[0050] A learning model building unit: used to build a learning model based on the multimodal fusion module;

[0051] Training unit: The training set is used to train the learning model, the validation set is used to measure the effect of the learning model, and the parameters with the best performance are selected as the parameters of the learning model. The test set is used to independently test the generalization ability of the learning model.

[0052] The third object of the present application is to provide a terminal, comprising: the terminal comprises a processor and a memory coupled to the processor, wherein:

[0053] The memory stores program instructions for implementing the multimodal information fusion method for protein characterization learning;

[0054] The processor is used to execute the program instructions stored in the memory to control multimodal information fusion.

[0055] A fourth object of the present application is to provide a storage medium storing program instructions executable by a processor, wherein the program instructions are used to execute the multimodal information fusion method for protein characterization learning.

[0056] This application adopts the above technical solution to achieve the following effects:

[0057] The multimodal information fusion method, system, terminal and storage medium for protein representation learning provided in the present application use the strategy of early extraction, mid-term fusion and late prediction to enable each single-modal model to fully extract the high-level semantic information of its own modality before fusion, and then use a feedforward neural network to predict the task in the later stage; at the same time, a multimodal fusion module is proposed, which can perform fine-grained interaction on the different modal information of each layer of the network during mid-term fusion, so as to better fuse and pass on the multimodality; in the last layer of the feature extractor in the late prediction stage, the fused multimodal embedding and the previous single-modal embedding are spliced ​​together as the representation of the protein itself, so as to retain the original information of the single modality to the greatest extent.

[0058] In addition, the multimodal information fusion method, system, terminal and storage medium for protein characterization learning provided by the present application predict a result for the feature extraction networks of different layers in the later prediction stage when designing the loss function, which serves as an auxiliary loss of the final loss. The introduction of the auxiliary loss can help the model converge faster and achieve better performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0060] Figure 1 A flowchart of the steps of a multimodal information fusion method for protein characterization learning provided in an embodiment of the present application.

[0061] Figure 2 This is an adjacency matrix diagram of the protein provided in the examples of the present application.

[0062] Figure 3 Schematic diagram of the multimodal fusion module provided in an embodiment of the present application.

[0063] Figure 4 A schematic diagram of a learning module provided in an embodiment of the present application.

[0064] Figure 5 It is a schematic diagram of the structure of a multimodal information fusion method for protein characterization learning provided in an embodiment of the present application.

[0065] Figure 6 A schematic diagram of the terminal structure provided in an embodiment of the present application.

[0066] Figure 7A schematic diagram of the structure of the storage medium provided in an embodiment of the present application. DETAILED DESCRIPTION

[0067] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0068] In the description of the present application, it should be understood that the terms "upper", "lower", "horizontal", "inside", "outside", etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present application.

[0069] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0070] See also Figure 1 , which is a flowchart of the steps of the multimodal information fusion method for protein characterization learning provided in this application, comprising the following steps:

[0071] Step S110: pre-processing the open source protein data.

[0072] In this embodiment, in the open source protein data, these data sets have various tasks, including predicting protein fluorescence, protein secondary structure, protein remote homology and protein stability. Protein sequence data is extracted from these data sets, the sequence consists of 20 English letters (representing 20 amino acids), and the 3D structure of the protein is converted into an adjacency matrix graph, also known as a contact map.

[0073] like Figure 2 , which indicates whether the amino acids are in contact in space. The white part indicates contact, and the black part indicates no contact.

[0074] Step 120: Divide the protein dataset into a training set, a validation set, and a test set.

[0075] It can be understood that the processed data set is divided into training, validation and test sets. The training set is used to adjust the model parameters to fit the target, the validation set is used to select the optimal parameters, and the test set is used to evaluate the final effect of the model.

[0076] Step S130: constructing a unimodal feature extractor, wherein the unimodal feature extractor is used as a feature extractor for protein sequences.

[0077] In this embodiment, the present application uses the pre-trained Transformer model TAPE as a feature extractor for protein sequences. The pre-training strategy allows the model to have prior information before training, which can play a positive role in the reasoning of the model, can capture the amino acid association of the entire sequence, and supports parallelization. For protein structure, the nature of its topological structure is suitable for solving with graph algorithms. This application uses an effective graph neural network, specifically the graph attention network GAT, which also uses the attention mechanism to capture the relationship between neighbor nodes and target nodes.

[0078] Step S140: construct a multimodal fusion module, which updates the amino acid token embedding of the unimodal feature extractor so that the unimodal contains information of multiple models and serves as the input of the unimodal feature extractor.

[0079] See also Figure 3 In the steps of building a multimodal fusion module, there are four main steps: pooling, compression, diversion, and reconstruction, which specifically include the following steps:

[0080] Step 1: Average pooling is performed on the sequence feature matrix and the structural feature matrix, and the feature vector of each amino acid obtains a representative value. The formula is as follows, where

[0081]

[0082]

[0083] Assumptions They represent the sequence feature matrix and structural feature matrix before inputting the multimodal module, respectively, where D seq Represents the characteristic dimension of each amino acid in the sequence, D struc Represents the characteristic dimension of each amino acid in the structure, L seq and L struc They represent the amino acid lengths in the sequence and structure, respectively, but they are actually equal, i.e., L seq =L struc =L.

[0084] Step 2: Concatenate the sequence and structure pooling vectors, and then transform them into vectors containing multimodal information through a fully connected network. The formula is as follows:

[0085] M comp =W[M seq ,M struc ]+b

[0086] in, Let D comp =(L seq +L struc ) / 5.

[0087] It can be understood that the above steps can limit the size of the model and improve its generalization ability. This step is the process of multimodal information interaction and compression.

[0088] Step 3: Compress the multimodal information into vector M comp Re-dividing the data into different modes to calibrate the single-mode information, the diversion process is to introduce the fully connected transformation layer of each mode respectively, the formula is as follows:

[0089] T seq =W seq M comp +b seq ,

[0090] T struc =W struc M comp +b struc ,

[0091] Step 4: Activate the modal vector transformed by the shunt through the activation function as a gate switch to limit the contribution of each amino acid to the overall task. The specific formula is as follows:

[0092]

[0093]

[0094] Where σ refers to the sigmoid function and ⊙ refers to the Hadamard product.

[0095] Step 5: After multiplying with the activated gated vector, the reconstructed unimodal vector is obtained as the input of the next layer of unimodal feature extractor.

[0096] It can be understood that the present application uses a calibration reconstruction method in the step of constructing a multimodal fusion module. Specifically, it uses multimodal information interaction to update the amino acid token embedding of each single modality, so that the single modality that may have ambiguous information can be guided by multi-model information, making it clearer during pattern recognition.

[0097] Step S150: constructing a learning model based on the multimodal fusion module.

[0098] See also Figure 4 , is a schematic diagram of the principle of building a learning model based on the multimodal fusion module, which specifically includes the following steps:

[0099] Step S151: Add a special token named [cls] to the original input of the protein sequence and structure. The [cls] of the sequence is placed at the front of the entire sequence, and the [cls] of the structure establishes a virtual full connection with all amino acids.

[0100] It can be understood that the purpose of introducing [cls] is to allow [cls] to represent the entire modality and participate in subsequent predictions.

[0101] Step S152: The original protein data is processed by N e The early unimodal feature extractor of the layer passes through the encoding layer of the Transformer model, and the structure passes through the graph attention network layer. The output result represents the unimodal vector representation of the extracted high-level semantics.

[0102] Step S153: inserting the multimodal fusion module to perform mid-term fusion.

[0103] It can be understood that when entering the multimodal fusion stage, that is, the mid-term fusion stage, on the basis of the previous early extraction, each layer adds the interaction between modalities and inserts Figure 3 The multimodal fusion network described has undergone N m layer.

[0104] Step S154: After mid-term fusion, the single modality is calibrated by multi-modality information, and then N l Layer feature extractor for further calibrated feature mining.

[0105] Step S155: The [cls] vectors of the two modes of feature mining after calibration are concatenated, and then passed through a feedforward neural network, and then concatenated with the [cls] vector obtained by the early single-modal feature extractor.

[0106] It is understandable that since the concatenated vector is relatively fragmented, the concatenated feature vector can be passed through a learnable feedforward neural network to obtain a more holistic feature vector, and the prediction result is more accurate.

[0107] It is understandable that since some unimodal information may be lost during the multimodal information transmission process, the information can be completed after being spliced ​​with the unimodal vector.

[0108] Step S156: Pass through the second feedforward neural network to obtain a learning model.

[0109] It can be understood that the multimodal fusion strategy provided in the embodiment of the present application, through the strategies of early extraction, mid-term fusion, and late prediction, can enable the model to learn unimodal and multimodal information more fully; the multimodal representation obtained in the later stage is not directly used for prediction, but is added to the early unimodal representation, so that the unimodal information lost during the network transmission process can be supplemented in the end.

[0110] In some embodiments, after completing the step of building a learning model based on the multimodal fusion module, the following steps are also included:

[0111] Step S157: Add auxiliary loss to update the parameters of the learning model.

[0112] It can be understood that since the parameters of the main network are large and the model is complex, it will be difficult to converge during training. Therefore, in each feature extraction layer in the later prediction stage of this application, the results will be output to predict the final target. The loss obtained in this way is used as an auxiliary loss and added to the main loss to update the model parameters.

[0113] Step S160: The training set is used to train the learning model, the validation set is used to measure the effect of the learning model, and the parameters with the best performance are selected as the parameters of the learning model. The generalization ability of the learning model is independently tested using the test set.

[0114] See also Figure 5, which is a structural diagram of a multimodal information fusion system for protein characterization learning provided by the present application, including: a data processing unit 110: used for preprocessing open source protein data; a classification unit 120: used for dividing the protein data set into a training set, a validation set and a test set; a unimodal feature extractor construction unit 130: used for constructing a unimodal feature extractor, the unimodal feature extractor being used as a feature extractor for protein sequences; a multimodal fusion module construction unit 140: used for constructing a multimodal fusion module, the multimodal fusion module updates the amino acid token embedding of the unimodal feature extractor so that the unimodal has information of multiple models and serves as the input of the unimodal feature extractor; a learning model construction unit 150: used for constructing a learning model based on the multimodal fusion module; a training unit 160: the training set trains the learning model, the validation set measures the effect of the learning model, and selects the parameters with the best performance as the parameters of the learning model, and uses the test set to independently test the generalization ability of the learning model. Its detailed implementation method has been described in the above method description of the present application, and will not be repeated here.

[0115] See also Figure 6 , is a schematic diagram of a terminal structure of an embodiment of the present application. The terminal 50 includes a processor 51 and a memory 52 coupled to the processor 51.

[0116] The memory 52 stores program instructions for implementing the multimodal information fusion method for protein characterization learning.

[0117] The processor 51 is used to execute program instructions stored in the memory 52 to control the multimodal information fusion.

[0118] The processor 51 may also be referred to as a CPU (Central Processing Unit). The processor 51 may be an integrated circuit chip having the ability to process signals. The processor 51 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0119] See also Figure 7, which is a schematic diagram of the structure of the storage medium of the embodiment of the present application. The storage medium of the embodiment of the present application stores a program file 61 that can implement all the above methods, wherein the program file 61 can be stored in the above storage medium in the form of a software product, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to execute all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, or terminal devices such as a computer, a server, a mobile phone, and a tablet.

[0120] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A multimodal information fusion method for protein representation learning, characterized in that: The steps include: Preprocessing of open source protein data; Dividing the protein dataset into a training set, a validation set, and a test set; constructing a unimodal feature extractor, wherein the unimodal feature extractor is used as a feature extractor for protein sequences; Constructing a multimodal fusion module, the multimodal fusion module updates the amino acid token embedding of the single-modal feature extractor so that the single modality carries information of multiple models and serves as the input of the single-modal feature extractor; Building a learning model based on the multimodal fusion module; The training set trains the learning model, the validation set measures the effect of the learning model and selects the parameters with the best performance as the parameters of the learning model, and the test set is used to independently test the generalization ability of the learning model; The steps of constructing a multimodal fusion module specifically include the following steps: The sequence feature matrix and the structural feature matrix are averaged and pooled, and the feature vector of each amino acid is given a representative value. The formula is as follows, where They represent the sequence feature matrix and structural feature matrix before inputting the multimodal module, respectively, where D seq Represents the characteristic dimension of each amino acid in the sequence, D struc Represents the characteristic dimension of each amino acid in the structure, L seq and L struc They represent the amino acid lengths in the sequence and structure, but they are actually equal, i.e. L seq =L struc =L; The pooled vectors of the sequence and structure are concatenated and then transformed into a vector containing multimodal information through a fully connected network. The formula is as follows: M comp =W[M seq ,M struc ]+b in, Let D comp =(L seq +L struc ) / 5; Compress the multimodal information into vector M comp Re-dividing the data into different modes to calibrate the single-mode information, the diversion process is to introduce the fully connected transformation layer of each mode respectively, the formula is as follows: T seq =W seq M comp +b seq , T struc =W struc M comp +b struc , The modal vector of the shunt transformation is activated through the activation function as a gate switch to limit the contribution of each amino acid to the overall task. The specific formula is as follows: Where σ refers to the sigmoid function, ⊙ refers to the Hadamard product; After multiplying with the activated gated vector, the reconstructed unimodal vector is obtained as the input of the next layer of unimodal feature extractor.

2. The multimodal information fusion method for protein representation learning according to claim 1, characterized in that: The steps of preprocessing open source protein data include the following: The sequence data of the protein is extracted from the open source protein data set, the sequence consists of 20 English letters, and the 20 English letters represent 20 amino acids, and the 3D structure of the protein is converted into an adjacency matrix diagram.

3. The multimodal information fusion method for protein representation learning according to claim 2, characterized in that: The steps of constructing a unimodal feature extractor specifically include: The unimodal feature extractor is a pre-trained Transformer model.

4. The multimodal information fusion method for protein representation learning according to claim 3, characterized in that: The step of constructing a learning model based on the multimodal fusion module specifically includes the following steps: Add a special token named [cls] to the original input of protein sequence and structure. The [cls] of the sequence is placed at the front of the entire sequence, and the [cls] of the structure establishes a virtual full connection with all amino acids. The original protein data was processed by N e The early unimodal feature extractor of the layer, the sequence passes through the encoding layer of the Transformer model, the structure passes through the graph attention network layer, and the output result represents the unimodal vector representation of the extracted high-level semantics; Inserting the multimodal fusion module to perform mid-term fusion; After mid-term fusion, the single modality is calibrated by multimodal information and continues through N l Layer feature extractor, further calibrated feature mining; The [cls] vectors of the two modalities mined by the calibrated feature are concatenated, then passed through a feedforward neural network, and then concatenated with the [cls] vector obtained by the early single-modal feature extractor; After passing through the second feedforward neural network, the learning model is obtained.

5. The multimodal information fusion method for protein representation learning according to claim 4, characterized in that: After completing the step of building a learning model based on the multimodal fusion module, the following steps are also included: Auxiliary loss is added to update the parameters of the learning model.

6. A multimodal information fusion system for protein representation learning, characterized in that: Adopting the multimodal information fusion method for protein characterization learning according to any one of claims 1 to 5; The system includes: Data processing unit: used to pre-process open source protein data; Classification unit: used to divide the protein dataset into a training set, a validation set and a test set; A unimodal feature extractor construction unit: used to construct a unimodal feature extractor, wherein the unimodal feature extractor is used as a feature extractor for protein sequences; A multimodal fusion module construction unit: used to construct a multimodal fusion module, wherein the multimodal fusion module updates the amino acid token embedding of the single-modal feature extractor so that the single-modal has information of multiple models and serves as the input of the single-modal feature extractor; A learning model building unit: used to build a learning model based on the multimodal fusion module; Training unit: The training set is used to train the learning model, the validation set is used to measure the effect of the learning model, and the parameters with the best performance are selected as the parameters of the learning model. The test set is used to independently test the generalization ability of the learning model.

7. A terminal, characterized in that: include: The terminal includes a processor and a memory coupled to the processor, wherein: The memory stores program instructions for implementing the multimodal information fusion method for protein characterization learning according to any one of claims 1 to 5; The processor is used to execute the program instructions stored in the memory to control multimodal information fusion.

8. A storage medium, characterized in that: The method stores program instructions executable by a processor, wherein the program instructions are used to execute the multimodal information fusion method for protein characterization learning according to any one of claims 1 to 5.

Images