A protein multimodal joint modeling method and system based on cross-modal alignment

Abstract

The present invention discloses a protein multimodal joint modeling method and system based on cross-modal alignment, which aims to achieve unified representation and processing of protein text descriptions, sequence information and structural features, so as to improve the accuracy and generalization ability of complex protein analysis and prediction tasks and open research tasks. The present invention first collects and pre-processes protein multimodal data to generate a standardized representation, uses a geometric graph neural network and a protein sequence model to extract features, passes them to a large language model after alignment by a projection module, and then projects and fuses them to generate unified invariant features, and finally combines equivariant features to complete the protein analysis and prediction task. The present invention can effectively integrate the multimodal information of proteins, achieve efficient alignment and fusion of cross-modal features, improve the understanding and prediction capabilities of complex protein properties, and provide strong support for precise bioinformatics analysis and biomedical research.

Description

Technical Field

[0001] The present application relates to the field of bioinformatics, and specifically to a protein multimodal joint modeling method and system based on cross-modal alignment. Background Art

[0002] Proteins are key molecules in life science research, and their functions and behaviors are often determined by both their sequence and three-dimensional structure. Each protein possesses information in multiple modalities, including sequence information based on UniProt (Universal Protein Database) files and three-dimensional structural information based on PDB (Program Database) files. Numerous mature models have been developed for these diverse data modalities. For example, geometric graph neural networks excel at processing protein three-dimensional structures, while protein sequence models excel at parsing and understanding sequence information. However, a mature framework that can uniformly process this multimodal information remains lacking.

[0003] In recent years, large language models have achieved tremendous success, demonstrating their powerful capabilities in text understanding and generation. These models are not only capable of processing natural language but also effectively integrating and leveraging scientific knowledge. However, large language models have limitations in understanding and processing three-dimensional spatial data, which limits their potential for application in protein analysis.

[0004] The problems of protein analysis and prediction are often complex and diverse. Establishing a unified framework that can simultaneously process multimodal information is of great significance for the comprehensive analysis of protein function.

[0005] This paper explores the potential of integrating large-scale language models, geometric graph deep learning models, and protein sequence models in the protein domain. By aligning the multimodal representations of these models, a unified framework is proposed to better address complex protein analysis and prediction problems. Specifically, this framework aims to organically combine protein sequence information with three-dimensional structure information, leveraging the strengths of each model to provide a more comprehensive protein description. Through model alignment, these models can work together in protein analysis tasks, improving the accuracy and reliability of protein analysis predictions.

[0006] This invention not only fills a gap in existing multimodal protein analysis frameworks but also provides a powerful tool for complex protein research and open questions. This innovative framework is expected to promote the in-depth development of protein science research and have broad applications in fields such as biomedicine. Summary of the Invention

[0007] In response to the current problem of lack of a mature framework for unified processing of multimodal information in the field of protein analysis, the present invention proposes a protein multimodal joint modeling method and system based on cross-modal alignment.

[0008] The object of the present invention is achieved through the following technical solutions:

[0009] The present invention provides a protein multimodal joint modeling method based on cross-modal alignment, comprising the following steps:

[0010] Step 1: Collect and preprocess multimodal protein data to generate standardized representations;

[0011] Step 2: Extract invariant feature representations of protein 3D structure using geometric graph neural networks and equivariant feature representation , and process the protein sequence information through the protein sequence model to obtain feature representation ;

[0012] Step 3: Represent the features and Input projection module to generate feature representations aligned with the input space of the large language model and ;

[0013] Step 4: Represent the aligned features and The text description information of the protein is passed to the large language model to output the semantic feature representation ;

[0014] Step 5: Represent the semantic features through the projection module Projection back to the invariant feature representation The space of Projected to the invariant feature representation space, and finally fusion 、 and Generating unified invariant feature representations ;

[0015] Step 6: Combine invariant feature representations and 3D equivariant feature representation , complete the analysis and prediction tasks of proteins.

[0016] Furthermore, step 1 includes:

[0017] Collect multimodal protein data. Each protein's data includes sequence information and text description information from the UniProt database, as well as a 3D graph structure from the PDB database. Extract protein sequence information S from the UniProt file, convert the content other than sequence information in the UniProt file into a natural language description that can be understood by the large language model, namely text description information T, and extract the protein's 3D structure information G from the PDB file. Preprocess the sequence information, text description information, and 3D structure information to ensure consistency and usability of the input format.

[0018] Find the relevant UniProt ID from the annotation of the PDB entry to achieve pairing and linking of the three-dimensional structure information, sequence and text description information of the same protein; for proteins that lack UniProt entries, obtain the sequence information of the protein from the PDB entry, and the corresponding text description information is missing; for proteins that lack PDB entries, use structure prediction tools to predict the three-dimensional structure of the protein; in the subsequent processing process, give higher weight to modal data that are clearly present in the database.

[0019] Furthermore, step 1 further includes:

[0020] The protein information described in text uses a unified modular language structure: "The protein structure [protein ID] has a sequence length of [number] amino acids, involving the following chains: [chain], the protein is named [protein name], and comes from the organism [organism]"; other feature descriptions of the protein are added to the text description, including statistical information and 3D structure information. For 3D structure information, a node in the protein structure diagram is selected as the anchor point to describe the relative positions of other nodes to this node; a concise description of the task is provided in the text description, allowing the large language model to quickly identify the task objectives and release domain-specific knowledge.

[0021] Furthermore, step 2 includes:

[0022] Input the three-dimensional structure information G of the protein obtained in step 1 into the geometric graph neural network to extract the invariant feature representation of the protein and equivariant feature representation ; Input the protein sequence information S obtained in step 1 into the protein sequence model to obtain the feature representation .

[0023] Furthermore, step 3 includes:

[0024] Use the trained projection module to represent the features obtained in step 2 and Projecting to the input space of the large language model to generate feature representations aligned with the input space of the large language model and ; The projection module is used to ensure that the aligned features can interact with the text description information T in the same semantic space.

[0025] Furthermore, step 4 includes:

[0026] For the text description information T obtained in step 1, the word segmenter and embedding layer of the large language model are used to obtain the corresponding word embedding features. Then, according to the specific task, the obtained word embedding features are combined with the feature representation obtained in step 3. and Splicing is performed to form joint features and input into the large language model, and semantic feature representation is obtained by combining the knowledge of the large language model .

[0027] Furthermore, step 5 includes:

[0028] Use the trained projection module to represent the semantic features obtained in step 4 Projected back to space, and at the same time Projected to the corresponding space, integrating the three aligned features 、 and Generating unified invariant feature representations .

[0029] Furthermore, step 6 includes:

[0030] Combined with the feature representation obtained in step 5 And the three-dimensional equivariant feature representation obtained in step 2 , complete the analysis and prediction task of protein; for the invariant task of protein function prediction, only a unified invariant feature representation is used For prediction, the core goal is to predict the three-dimensional coordinates of the task, while combining the feature representation and 3D equivariant feature representation Make predictions.

[0031] The present invention also provides a protein multimodal joint modeling system based on cross-modal alignment for implementing the above method, comprising:

[0032] Data preprocessing module, used to collect and preprocess multimodal data of proteins;

[0033] Feature extraction module, used to extract three-dimensional structural features, sequence features and isovariant features from multimodal protein data and generate preliminary feature representation;

[0034] A cross-modal feature alignment module is used to align the extracted protein features with the input space of the large language model;

[0035] Multimodal semantic fusion module, used to fuse multimodal features of proteins;

[0036] The reverse projection and feature fusion module is used to fuse the semantic features generated by the large language model with the feature representations generated by the geometric graph neural network and the protein sequence model to generate a unified invariant feature representation;

[0037] The task-adaptive prediction module is used to complete protein analysis and prediction tasks.

[0038] The present invention also provides a protein multimodal joint modeling device based on cross-modal alignment, comprising a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, the above-mentioned protein multimodal joint modeling method based on cross-modal alignment is implemented.

[0039] The present invention provides a method for multimodal protein joint modeling based on cross-modal alignment, filling a gap in existing multimodal protein analysis frameworks and providing a powerful tool for complex protein research and open questions. By aligning geometric graph neural networks, protein sequence models, and large language models, the present invention enables their collaborative work in protein analysis tasks. When applied to protein analysis tasks, the present invention significantly improves the accuracy and reliability of protein analysis predictions. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 This is a flowchart of the protein multimodal joint modeling method based on cross-modal alignment proposed in the present invention;

[0041] Figure 2 Schematic diagram of the structure of the protein multimodal joint modeling system based on cross-modal alignment provided by the present invention;

[0042] Figure 3 It is a structural schematic diagram of the protein multimodal joint modeling device based on cross-modal alignment provided by the present invention. DETAILED DESCRIPTION

[0043] Exemplary embodiments will be described in detail herein, examples of which are illustrated in the accompanying drawings. In the following description, when referring to the drawings, like numbers in different figures represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all possible embodiments consistent with the present invention. Rather, they are merely examples of systems and methods consistent with certain aspects of the present invention, as detailed in the appended claims.

[0044] The terms used in this invention are for the purpose of describing specific embodiments only and are not intended to limit the invention. The singular forms "a," "the," and "the" used in this invention and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0045] It should be understood that although the terms "first," "second," "third," etc. may be used in the present invention to describe various information, such information should not be limited to these terms. These terms are merely used to distinguish information of the same type from one another. For example, first information may also be referred to as second information, and similarly, second information may also be referred to as first information, without departing from the scope of the present invention. Depending on the context, the term "if" as used herein may be interpreted as "when," "when," or "in response to determining."

[0046] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in combination with specific implementation cases and with reference to the accompanying drawings.

[0047] See also Figure 1 The present invention proposes a protein multimodal joint modeling method based on cross-modal alignment, the main steps of which include:

[0048] Step 1: Collect and preprocess multimodal protein data to generate standardized representations.

[0049] Specifically, multimodal protein data is collected. Each protein's data includes sequence information and textual descriptions from the UniProt database, as well as 3D graph structures from the PDB database. Protein sequence information S is extracted from the UniProt file. The non-sequence information in the UniProt file is converted into a clear and detailed natural language description T (i.e., textual description information) that can be understood by the large language model. The protein's 3D structure information G is then extracted from the PDB file. Sequence information, textual description information, and 3D structure information are preprocessed to ensure consistency and usability of the input formats.

[0050] It should be noted that the amino acid sequence itself is very long and is therefore excluded from the text description information.

[0051] Find the relevant UniProt ID from the annotation of the PDB entry, thereby achieving paired links between the three-dimensional structure information, sequence, and text description information of the same protein.

[0052] For proteins that lack UniProt entries, the sequence information of the protein is obtained from the PDB entry, and the corresponding text description information is missing; for proteins that lack PDB entries, structure prediction tools such as AlphaFold and RoseTTAFold are used to predict the three-dimensional structure of the protein.

[0053] In the subsequent processing, higher weight is given to the modal data that clearly exists in the database.

[0054] The protein information described in text uses a unified modular language structure, similar to: "The protein structure [protein ID] has a sequence length of [number] amino acids, involving the following chains: [chain], the protein is named [protein name], and is derived from the organism [organism]"; regular expressions or natural language processing technology can be used to convert non-sequence information into structured natural language descriptions.

[0055] It should be noted that other characteristic descriptions, statistical information and even 3D structural information of the protein can be added to the text description. For 3D structural information, a node in the protein structure diagram can be selected as an anchor point to describe the relative positions of other nodes to this node.

[0056] A concise description of the task can be provided in the text description to help the Large Language Model (LLM) quickly identify the task goal and also help the LLM release domain-specific knowledge.

[0057] The anchor point here refers to the key reference node in the structure, which can be determined based on functional site positioning or structural key node selection method. Functional sites include key amino acid residues (such as conserved sequence regions) and catalytic active centers, etc. Structural key nodes are generally obtained through geometric analysis methods.

[0058] Although geometric graph neural networks also process 3D structural information, due to the difference in abstraction levels, complementarity between multimodal information is formed. Through semantic, global, and noise-resistant information complementarity, multi-granularity knowledge fusion and cross-modal error correction mechanisms are formed.

[0059] Use bioinformatics tools such as Biopython to process PDB files, extract atomic coordinates, bond structures, and spatial relationships, and convert them into graph-structured representations.

[0060] Step 2: Based on the protein multimodal data obtained in step 1, a geometric graph neural network is used to extract the invariant feature representation of the protein three-dimensional structure. and equivariant feature representation , and process the protein sequence information through the protein sequence model to obtain feature representation .

[0061] Specifically, the three-dimensional structure information G of the protein obtained in step 1 is input into the geometric graph neural network, which can be any suitable equivariant model in the field and must meet the E(3) symmetry requirement to extract the invariant feature representation of the protein. and equivariant feature representation ; Input the protein sequence information S obtained in step 1 into the protein sequence model, such as the ESM (Evolutionary Scale Modeling) model, to obtain the feature representation .

[0062] It is easy to understand that the E(3) symmetry requirement means that when the model processes geometric data in three-dimensional space (such as protein structure), it must satisfy the symmetry invariance or equivariance of the three-dimensional Euclidean group (Euclidean group E(3)).

[0063] Adjust the hyperparameters of the geometric graph neural network, such as the number of layers and node feature dimensions, to optimize feature extraction.

[0064] Step 3: Represent the features and Input projection module to generate feature representations aligned with the large language model input space and .

[0065] Specifically, use the trained projection module to represent the feature obtained in step 2 and Projecting to the input space of the large language model to generate feature representations aligned with the input space of the large language model and The projection module can be a multilayer perceptron (MLP) or other suitable mapping models to ensure that the aligned features can interact with the text description information T in the same semantic space.

[0066] It should be noted that training the projection module can be carried out in two steps. The first step is to train the projection module separately to ensure that it can retain the key information of the original data. The second step is to combine the various projection modules and the large language model training to maximize the complementarity between the modules.

[0067] Step 4: Align the features and The information such as the text description of the protein is passed to the large language model to output the semantic feature representation .

[0068] Specifically, for the protein text description information T obtained in step 1, the word segmenter and embedding layer of the large language model are used to obtain the corresponding word embedding features. Then, according to the specific task, the obtained word embedding features are combined with the feature representation obtained in step 3. and Splice in an appropriate way to form a joint feature input, input the joint feature into the large language model, and combine the knowledge of the large language model to obtain the semantic feature representation Based on the specific task requirements, task-specific prompts can be designed to guide the large language model to generate more accurate semantic features. For example, for the ligand binding site prediction task, the prompt word is: "Identify possible ligand binding sites based on protein feature information." For the molecular dynamics simulation task, the prompt word is: "Predict the trajectory of the next F frames based on the previous trajectory information."

[0069] Adjust the input layer of the large language model so that it can handle the joint feature input. The specific adjustment process is: after projection and The dimension is consistent with the embedding dimension of LLM, and the gated attention mechanism is used to represent the feature and Splicing with text embedding features not only keeps the spliced ​​dimension unchanged, but also effectively fuses feature information from different sources.

[0070] Design specific fusion strategies, such as weighted summation or attention mechanism, to optimize the feature fusion effect.

[0071] Step 5: Use the projection module to Projected back to space, and at the same time Projected to the corresponding space, and finally fusion 、 and Generating unified invariant feature representations .

[0072] Specifically, the trained projection module is used to represent the semantic features obtained in step 4 Projected back to space, and at the same time Projected to the corresponding space, integrating the three aligned features 、 and Generating unified invariant feature representations .

[0073] Step 6, Combine and 3D equivariant feature representation , complete the analysis and prediction tasks of proteins.

[0074] Specifically, combined with the feature representation obtained in step 5 And the three-dimensional equivariant feature representation obtained in step 2 , complete the analysis and prediction task of protein. For invariant tasks such as protein function prediction, only a unified invariant feature representation is used For the task of predicting three-dimensional coordinates, the core goal is to combine feature representation and 3D equivariant feature representation Make predictions.

[0075] It should be noted that during the entire multimodal model training process, the parameters of the LLM and protein sequence model are always frozen, which significantly reduces computational cost and time overhead. The entire training process is divided into the following three stages.

[0076] In stage 1, the geometric graph neural network is trained separately and then the parameters of the geometric graph neural network are frozen.

[0077] Phase 2: Design the projection module to represent the features obtained in step 2 and Projecting to the input space of the large language model to generate feature representations aligned with the input space of the large language model and , the semantic features obtained in step 4 are represented as Projected back to space, and obtain the features , and at the same time Projected to the corresponding space, and obtain the features The projection module is trained using the contrastive learning loss function so that the projected features and and aligned in the same space, where for the same protein, and as well as and is a positive sample pair, for different proteins, and other proteins as well as and other proteins is a negative sample pair.

[0078] and The contrast loss is defined as:

[0079]

[0080] and The contrast loss is defined as:

[0081]

[0082] The total contrast loss is:

[0083]

[0084] in, Represents the feature vector and The similarity measure between Represents the feature vector and The similarity measure between them is usually calculated using cosine similarity. is the temperature parameter, and N is the number of negative samples.

[0085] Stage 3: Training and integrating three modal features for different downstream tasks 、 and Here, a task-aware readout function is used to calculate the attention weight between each modality feature and the task-specific query, and then the three modality features are weighted to obtain .

[0086] The protein multimodal joint modeling system based on cross-modal alignment provided by the present invention is described below. The protein multimodal joint modeling system based on cross-modal alignment described below and the protein multimodal joint modeling method based on cross-modal alignment described above can be referenced to each other.

[0087] Figure 2 The schematic diagram of the structure of the protein multimodal joint modeling system based on cross-modal alignment provided by the present invention is as follows: Figure 2 As shown, the present invention provides a protein multimodal joint modeling system based on cross-modal alignment, the system comprising:

[0088] Data preprocessing module, used to collect and preprocess multimodal data of proteins;

[0089] Feature extraction module, used to extract three-dimensional structural features, sequence features and isovariant features from multimodal protein data and generate preliminary feature representation;

[0090] A cross-modal feature alignment module is used to align the extracted protein features with the input space of the large language model;

[0091] Multimodal semantic fusion module, used to fuse multimodal features of proteins;

[0092] The reverse projection and feature fusion module is used to fuse the semantic features generated by the large language model with the feature representations generated by the geometric graph neural network and the protein sequence model to generate a unified invariant feature representation;

[0093] The task-adaptive prediction module is used to complete protein analysis and prediction tasks.

[0094] This embodiment effectively aligns protein multimodal data and effectively improves the reliability of protein model analysis and prediction.

[0095] Corresponding to the aforementioned embodiment of the protein multimodal joint modeling method based on cross-modal alignment, the present invention also provides an embodiment of a protein multimodal joint modeling device based on cross-modal alignment.

[0096] See also Figure 3 An embodiment of the present invention provides a protein multimodal joint modeling device based on cross-modal alignment, comprising a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it is used to implement the protein multimodal joint modeling method based on cross-modal alignment in the above embodiment.

[0097] The embodiment of the protein multimodal joint modeling device based on cross-modal alignment provided by the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capabilities in which it is located reading the corresponding computer program instructions in the non-volatile memory into the memory for execution. From the hardware level, if Figure 3 As shown, it is a hardware structure diagram of any device with data processing capability where a protein multimodal joint modeling device based on cross-modal alignment provided by the present invention is located, except Figure 3 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiment is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0098] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0099] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present invention. A person of ordinary skill in the art can understand and implement the present invention without inventive work.

[0100] Corresponding to the aforementioned embodiment of the protein multimodal joint modeling method based on cross-modal alignment, an embodiment of the present invention also provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the protein multimodal joint modeling method based on cross-modal alignment in the above embodiment is implemented.

[0101] The computer-readable storage medium may be an internal storage unit of any device with data processing capabilities described in any of the aforementioned embodiments, such as a hard disk or memory. The computer-readable storage medium may also be any device with data processing capabilities, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit of any device with data processing capabilities and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and may also be used to temporarily store data that has been output or is to be output.

[0102] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

[0103] The above embodiments are intended only to illustrate the design concepts and features of the present invention. Their purpose is to enable those skilled in the art to understand the contents of the present invention and implement them accordingly. The scope of protection of the present invention is not limited to the above embodiments. Therefore, any equivalent changes or modifications made based on the principles and design concepts disclosed in the present invention are within the scope of protection of the present invention.

Claims

1. A protein multimodal joint modeling method based on cross-modal alignment, characterized by: The following steps are involved: Step 1: Collect and preprocess multimodal protein data to generate standardized representations; Step 2: Extract invariant feature representations of protein 3D structure using geometric graph neural networks and equivariant feature representation , and process the protein sequence information through the protein sequence model to obtain feature representation ; Step 3: Represent the features and Input projection module to generate feature representations aligned with the large language model input space and ; Step 4: Represent the aligned features and The text description information of the protein is passed to the large language model to output the semantic feature representation ; Step 5: Represent the semantic features through the projection module Projection back to the invariant feature representation The space of Projected to the invariant feature representation space, and finally fusion 、 and Generating unified invariant feature representations ; Step 6: Combine invariant feature representations and 3D equivariant feature representation , complete the analysis and prediction tasks of proteins.

2. The protein multimodal joint modeling method based on cross-modal alignment according to claim 1, characterized in that: Step 1 includes: Collect multimodal protein data. Each protein's data includes sequence information and text description information from the UniProt database, as well as a 3D graph structure from the PDB database. Extract protein sequence information S from the UniProt file, convert the content other than sequence information in the UniProt file into a natural language description that can be understood by the large language model, namely text description information T, and extract the protein's 3D structure information G from the PDB file. Preprocess the sequence information, text description information, and 3D structure information to ensure consistency and usability of the input format. Find the relevant UniProt ID from the annotation of the PDB entry to achieve pairing and linking of the three-dimensional structure information, sequence and text description information of the same protein; for proteins that lack UniProt entries, obtain the sequence information of the protein from the PDB entry, and the corresponding text description information is missing; for proteins that lack PDB entries, use structure prediction tools to predict the three-dimensional structure of the protein; in the subsequent processing process, give higher weight to modal data that are clearly present in the database.

3. The protein multimodal joint modeling method based on cross-modal alignment according to claim 1, characterized in that: Step 1 further includes: The text description of protein information uses a unified modular language structure: "The protein structure [protein ID] has a sequence length of [number] amino acids and involves the following chains: [chain], the protein is named [protein name], and it originates from the organism [organism]"; other feature descriptions of the protein are added to the text description, including statistical information and 3D structure information. For the 3D structure information, a node in the protein structure diagram is selected as the anchor point to describe the relative positions of other nodes with respect to the node; a concise description of the task is provided in the text description, allowing the large language model to quickly identify the task objectives and release domain-specific knowledge.

4. The protein multimodal joint modeling method based on cross-modal alignment according to claim 2, characterized in that: Step 2 includes: Input the three-dimensional structure information G of the protein obtained in step 1 into the geometric graph neural network to extract the invariant feature representation of the protein and equivariant feature representation ; Input the protein sequence information S obtained in step 1 into the protein sequence model to obtain the feature representation .

5. The protein multimodal joint modeling method based on cross-modal alignment according to claim 2, characterized in that: Step 3 includes: Use the trained projection module to represent the features obtained in step 2 and Projecting to the input space of the large language model to generate feature representations aligned with the input space of the large language model and ; The projection module is used to ensure that the aligned features can interact with the text description information T in the same semantic space.

6. The protein multimodal joint modeling method based on cross-modal alignment according to claim 2, characterized in that: Step 4 includes: For the text description information T obtained in step 1, the word segmenter and embedding layer of the large language model are used to obtain the corresponding word embedding features. Then, according to the specific task, the obtained word embedding features are combined with the feature representation obtained in step 3. and Splicing is performed to form joint features and input into the large language model, and semantic feature representation is obtained by combining the knowledge of the large language model .

7. The protein multimodal joint modeling method based on cross-modal alignment according to claim 1, characterized in that: Step 5 includes: Use the trained projection module to represent the semantic features obtained in step 4 Projected back to space, and at the same time Projected to the corresponding space, integrating the three aligned features 、 and Generating unified invariant feature representations .

8. The protein multimodal joint modeling method based on cross-modal alignment according to claim 1, characterized in that: Step 6 includes: Combined with the feature representation obtained in step 5 And the three-dimensional equivariant feature representation obtained in step 2 , complete the analysis and prediction task of protein; for the invariant task of protein function prediction, only a unified invariant feature representation is used For prediction, the core goal is to predict the three-dimensional coordinates of the task, while combining the feature representation and 3D equivariant feature representation Make predictions.

9. A protein multimodal joint modeling system based on cross-modal alignment that implements the method of claim 1, characterized in that: include: Data preprocessing module, used to collect and preprocess multimodal data of proteins; Feature extraction module for extracting invariant feature representations of 3D structures from multimodal protein data and equivariant feature representation , sequence features, generate preliminary feature representation ; Cross-modal feature alignment module, used to represent features and Input projection module to generate feature representations aligned with the input space of the large language model and ; Multimodal semantic fusion module, used to represent the aligned features and The text description information of the protein is passed to the large language model to output the semantic feature representation ; The reverse projection and feature fusion module is used to represent the semantic features through the projection module. Projection back to the invariant feature representation The space of Projected to the invariant feature representation space, and finally fusion 、 and Generate unified invariant feature representation; The task-adaptive prediction module is used to complete protein analysis and prediction tasks.

10. A protein multimodal joint modeling device based on cross-modal alignment, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the processor executes the executable code, the protein multimodal joint modeling method based on cross-modal alignment according to any one of claims 1 to 8 is implemented.

Images