Cross-modal model training method and device, electronic equipment and storage medium
The cross-modal model training method improves the accuracy of feature extraction by encoding protein and text features, determining sub-contrastive losses, and updating the model parameters, addressing the alignment issues in existing methods.
Patent Information
- Application Number
- CN202411311038.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-19
- Publication Date
- 2024-12-06
AI Technical Summary
Existing methods fail to accurately align protein data and text data, which affects the accuracy of feature extraction and downstream prediction tasks.
The method involves a cross-modal model training device, which includes a feature extraction module, a cross-modal model, and a cross-modal model, a cross-modal model training device, an electronic device, and a computer-readable storage medium, which can improve the accuracy of feature extraction, and a computer program product, which can improve the accuracy of feature extraction.
The method provides a computer program product, which can improve the accuracy of feature extraction by implementing the cross-modal model training method, which can improve the accuracy of feature extraction by encoding the first protein feature and the first text feature, and determining the first sub-contrastive loss and the second sub-contrastive loss, and updating the parameters of the cross-modal model.
Smart Images

Figure CN119091971A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to artificial intelligence technology, and in particular to a training method, device, electronic device and storage medium for a cross-modal model. Background Art
[0002] In the related art, in the task of predicting protein properties based on deep learning, the deep learning model is usually trained in a supervised training method so that the deep learning model can predict the corresponding properties of the protein based on the protein sequence. Labeled proteins account for a minority of all known proteins. A protein encoder can be pretrained in an unsupervised manner, and the pretrained protein encoder is fine-tuned. The fine-tuned protein encoder can be used for protein property prediction. However, the alignment ability of protein data and text annotations in the related art is not high, which limits the accuracy of deep learning models in feature extraction of unknown proteins, thereby affecting the accuracy of protein property prediction.
[0003] In the related technology, there is currently no better way to improve the accuracy of feature extraction using cross-modal models. Summary of the invention
[0004] The embodiments of the present application provide a cross-modal model training method, device, electronic device, computer-readable storage medium and computer program product, which can improve the accuracy of feature extraction of the cross-modal model of protein and text.
[0005] The technical solution of the embodiment of the present application is implemented as follows:
[0006] The present application provides a method for training a cross-modal model, the method comprising:
[0007] Based on the sample protein data and the sample annotation data, calling the cross-modal model to perform feature extraction processing to obtain a first protein feature of the sample protein data and a first text feature of the sample annotation data;
[0008] Encoding the first protein feature and the first text feature respectively to obtain a second protein feature and a second text feature;
[0009] determining a first sub-contrastive loss based on the first protein feature and the second text feature, and determining a second sub-contrastive loss based on the second protein feature and the first text feature;
[0010] Determining a total loss function corresponding to the cross-modal model based on the first sub-contrastive loss and the second sub-contrastive loss;
[0011] Parameters of the cross-modal model are updated based on the total loss function to obtain the trained cross-modal model.
[0012] The present application provides a cross-modal model training device, including:
[0013] A feature extraction module, configured to call a cross-modal model to perform feature extraction processing based on the sample protein data and the sample annotation data, so as to obtain a first protein feature of the sample protein data and a first text feature of the sample annotation data;
[0014] The feature extraction module is further used to encode the first protein feature and the first text feature respectively to obtain a second protein feature and a second text feature;
[0015] a model training module, configured to determine a first sub-contrast loss based on the first protein feature and the second text feature, and to determine a second sub-contrast loss based on the second protein feature and the first text feature;
[0016] The model training module is further used to determine a total loss function corresponding to the cross-modal model based on the first sub-contrast loss and the second sub-contrast loss;
[0017] The model training module is also used to update the parameters of the cross-modal model based on the total loss function to obtain the trained cross-modal model.
[0018] An embodiment of the present application provides an electronic device, the electronic device comprising:
[0019] A memory for storing computer executable instructions or computer programs;
[0020] The processor is used to implement the cross-modal model training method provided in the embodiment of the present application when executing the computer executable instructions or computer program stored in the memory.
[0021] An embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions or a computer program, which is used to implement the cross-modal model training method provided by the embodiment of the present application when executed by a processor.
[0022] An embodiment of the present application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, the cross-modal model training method provided by the embodiment of the present application is implemented.
[0023] The embodiments of the present application have the following beneficial effects:
[0024] By encoding the first protein feature and the first text feature again, the second protein feature and the second text feature are obtained. The contrastive learning loss is determined based on the first protein feature and the second text feature, and the second protein feature and the first text feature, respectively. The sample difference is characterized by the contrastive learning loss, so that the feature distance between the features extracted by the cross-modal model for similar objects is closer, and the feature distance between the features extracted for different objects is farther, thereby improving the cross-modal model's ability to align features of different modalities. The total loss function combines the losses obtained by comparing protein features to text and text features to protein features. The parameters of the cross-modal model are updated based on the total loss function to obtain a trained cross-modal model, which can improve the accuracy of feature extraction. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] Figure 1 It is a schematic diagram of an application mode of the training method of the cross-modal model provided in an embodiment of the present application;
[0026] Figure 2 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;
[0027] Figure 3A It is a first flow chart of the cross-modal model training method provided in an embodiment of the present application;
[0028] Figure 3B It is a second flow chart of the cross-modal model training method provided in an embodiment of the present application;
[0029] Figure 3C It is a third flow chart of the cross-modal model training method provided in an embodiment of the present application;
[0030] Figure 3D It is a fourth flow chart of the cross-modal model training method provided in an embodiment of the present application;
[0031] Figure 3E It is a fifth flow chart of the cross-modal model training method provided in an embodiment of the present application;
[0032] Figure 3F It is a sixth flow chart of the cross-modal model training method provided in an embodiment of the present application;
[0033] Figure 3G is a seventh flow chart of the cross-modal model training method provided in an embodiment of the present application;
[0034] Figure 4 is a training principle diagram of a cross-modal model provided in an embodiment of the present application;
[0035] Figure 5It is an eighth flow chart of the cross-modal model training method provided in an embodiment of the present application;
[0036] Fig. 6A is a first structural diagram of a cross-modal model provided in an embodiment of the present application;
[0037] Figure 6B It is a second structural diagram of the cross-modal model provided in an embodiment of the present application.
[0038] It should be pointed out that the above-mentioned "first" and "second" are only used to distinguish different solutions, and do not represent the degree of superiority or inferiority of the solutions or the priority in the implementation process. DETAILED DESCRIPTION
[0039] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.
[0040] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.
[0041] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.
[0042] It should be pointed out that the collection and processing of relevant data in this application (for example, protein-related data) should be strictly in accordance with the requirements of relevant laws and regulations when applied in examples, and the informed consent or separate consent of the personal information subject should be obtained, and subsequent data use and processing should be carried out within the scope of authorization of laws and regulations and the personal information subject.
[0043] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.
[0044] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.
[0045] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.
[0046] 1) Protein: A biological macromolecule composed of at least one polypeptide chain combined in a specific way. A polypeptide chain is formed by amino acids combined in a certain order, and each polypeptide chain has twenty to hundreds of amino acid residues (-R). Protein is the scaffold and main substance that constitutes human tissues and organs.
[0047] 2) Amino acids: They are the basic units that make up proteins. Amino acids are linked into peptide chains through dehydration condensation; various amino acid residues are arranged in a certain order. The amino acid sequence of a protein is encoded by the corresponding gene. In addition to the 20 basic amino acids encoded by the genetic code, some amino acid residues in proteins can also be modified post-translationally to change their chemical structure, thereby activating or regulating the protein.
[0048] 3) Protein sequence: refers to the order of arrangement of amino acid residues in a protein molecule, also known as the amino acid sequence.
[0049] 4) Contrastive Learning: It is an unsupervised learning method that learns feature representations by comparing data points. In this learning framework, the model is trained to bring together (i.e. reduce the feature distance) the feature representations of similar data points, while pushing apart (i.e. increase the distance) the feature representations of dissimilar data points. Contrastive learning is often used to train deep learning models, especially in the fields of computer vision and natural language processing. In contrastive learning, positive and negative pairs are usually constructed. Positive pairs include data points from the same category or with similar features, while negative pairs include data points from different categories or with dissimilar features.
[0050] 5) Transformer model: It is a deep neural network model based on the self-attention mechanism, which is widely used in various tasks in the field of natural language processing, such as text classification, machine translation, and question-answering systems. This model can convert input sequences into output sequences while retaining important information in the input sequence. Since the transformer model performs well in processing long texts, it has been widely used in the field of Chinese natural language processing. Compared with traditional recurrent neural networks (RNNs) and convolutional neural networks (CNNs), the Transformer model can be calculated in parallel to speed up training. It is widely used in tasks such as natural language processing, speech recognition, and image generation.
[0051] 6) Bidirectional Encoder Representations from Transformers (BERT): A pre-trained language representation model, hereinafter referred to as BERT. The purpose of BERT is to provide deep bidirectional representation of text data. Deep bidirectional representation can be used for various natural language processing (NLP) tasks such as text classification, question answering, named entity recognition, etc.
[0052] 7) Momentum Encoder: An encoder that encodes according to the learning process of data augmentation and contrastive loss in self-supervised contrastive learning. The goal of contrastive learning is to compare the contrastive loss between the two positive samples and the negative samples generated after data augmentation of other input samples. The momentum encoder uses a fixed-size queue (memory bank) to store features. The size of this queue is set to K, and the features in the queue are randomly initialized at the beginning. After the training starts, the features generated for the sample are added to the queue, and the last sample in the queue is removed from the queue. Then, the latest feature generated by the encoder is compared with a random sample in the queue. In this way, the momentum encoder can use K negative samples for calculation each time the loss is calculated without increasing the memory burden of the GPU.
[0053] 8) Modality: Each source or form of information can be called a modality. For example, the media of information include voice, image, text, etc.; the sources of information include various sensors, such as radar, infrared, accelerometer, etc. Each of the above can be called a modality. The modalities involved in the embodiments of the present application are protein data and text data. Protein data is represented in the form of protein sequences, which are formed by the arrangement of multiple amino acids.
[0054] 9) Multilayer Perceptron (MLP): It is a feed-forward artificial neural network consisting of at least three layers of nodes: input layer, output layer, and at least one hidden layer. The nodes (or "neurons") in each layer are interconnected in a fully connected manner. In MLP, each node receives input from the previous layer and produces an output, which is a linear combination of the input signal plus a bias term, and then transformed by a nonlinear activation function.
[0055] In the related art, a deep learning model for protein property prediction is obtained through a two-stage training. The training stage includes pre-training for the protein encoder and supervised training for the deep learning model as a whole. However, the accuracy of feature extraction of proteins and texts obtained by the pre-training model is not high, which in turn affects the accuracy of downstream protein property prediction and the generalization of the model.
[0056] The embodiments of the present application provide a cross-modal model training method, a cross-modal model training device, an electronic device, a computer-readable storage medium, and a computer program product, which can improve the accuracy of feature extraction of a cross-modal model of protein and text.
[0057] The following describes an exemplary application of the electronic device provided in the embodiments of the present application. The electronic device provided in the embodiments of the present application can be implemented as a terminal device, such as a laptop computer, a tablet computer, a desktop computer, a set-top box, a smart TV, a vehicle terminal, a virtual reality (VR) device, an augmented reality (AR) device, and other types of terminals, and can also be implemented as a server. The following describes an exemplary application when the electronic device is implemented as a terminal device or a server.
[0058] refer to Figure 1 , Figure 1 : is a schematic diagram of an application mode of a cross-modal model training method provided in an embodiment of the present application; for example, Figure 1 The server 200, network 300, terminal device 400 and database 500 are involved. The terminal device 400 is connected to the server 200 via the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.
[0059] For example, a large amount of sample protein data and sample annotation data are stored in the database 500 , the terminal device 400 may be a computer or a laptop computer or a mobile phone used by a technician, and the server 200 is a server of a protein performance detection platform.
[0060] In some embodiments, the technician sends a training request through the terminal device 400, and the training request sent by the terminal device 400 is transmitted to the server 200 through the network 300. The server 200 obtains the sample protein data and the sample annotation data from the database 500 according to the training request, and calls the training method of the cross-modal model provided in the embodiment of the present application to train the cross-modal model. Further, the server 200 fine-tunes the trained cross-modal model into a cross-modal model for protein property prediction, and the trained model is stored in the server 200. The technician sends the protein data to be processed to the server 200 through the terminal device 400, and the server 200 calls the trained cross-modal model to perform property prediction processing on the protein data to be processed, obtains the protein property prediction result, and feeds the protein property prediction result back to the terminal device 400, and the technician can perform related work according to the protein property.
[0061] In some embodiments, the training method of the cross-modal model of the embodiment of the present application can also be applied in the following application scenarios: protein screening, the cross-modal model obtained by training the cross-modal model training method provided in the embodiment of the present application, and the cross-modal model is fine-tuned into a protein property prediction model, and the properties of proteins with unknown functions are predicted based on the protein property prediction model. The property prediction results are, for example, catalytic reactions (as enzymes), structural support (such as collagen), transport (such as hemoglobin), immunity (such as antibodies), signal transduction (such as receptor proteins), etc. Researchers can screen out the desired proteins based on the property prediction results. Protein screening is an important biotechnology method that can play a key role in many fields, such as drug development, agricultural crop improvement, gene function research, and food processing.
[0062] The embodiments of the present application can be implemented through database technology. In short, a database can be regarded as an electronic file cabinet where electronic files are stored. Users can add, query, update, delete, etc. data in the files. The so-called "database" is a collection of data that is stored together in a certain way, can be shared with multiple users, has as little redundancy as possible, and is independent of the application program.
[0063] A database management system (DBMS) is a computer software system designed for managing databases. It generally has basic functions such as storage, retrieval, security, and backup. Database management systems can be classified according to the database model they support, such as relational, XML (Extensible Markup Language); or according to the type of computer they support, such as server clusters, mobile phones; or according to the query language used, such as Structured Query Language (SQL), XQuery; or according to performance focus, such as maximum scale, maximum operating speed; or other classification methods. Regardless of the classification method used, some DBMS can cross categories, for example, supporting multiple query languages at the same time.
[0064] In some embodiments, server 200 may be a server cluster composed of multiple servers, for example, server 200 includes a training server and a protein property prediction server. The training server is used to call the training method of the cross-modal model provided in the embodiment of the present application to obtain a trained cross-modal model. The trained cross-modal model is used to predict protein properties. The training server sends the trained cross-modal model to the protein property prediction server. The protein property prediction server stores the trained cross-modal model and, upon receiving at least one of the protein sequence to be processed and the protein annotation text, performs protein property prediction to obtain a prediction result.
[0065] In some embodiments, the server may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or may provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and cloud servers. The electronic device may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto. The terminal device and the server may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.
[0066] See also Figure 2 , Figure 2 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may be Figure 1 Server 200 in Figure 2The server 200 shown includes: at least one processor 410, a memory 450, and at least one network interface 420. The various components in the server 200 are coupled together via a bus system 440. It is understood that the bus system 440 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, the bus system 440 is not described in detail. Figure 2 Various buses are labeled as bus system 440 .
[0067] Processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.
[0068] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices that are physically remote from the processor 410.
[0069] The memory 450 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 450 described in the embodiments of the present application is intended to include any suitable type of memory.
[0070] In some embodiments, memory 450 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.
[0071] Operating system 451, including system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;
[0072] A network communication module 452, used to reach other electronic devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 include: Bluetooth, Wireless Compatibility Certification (WiFi), and Universal Serial Bus (USB), etc.;
[0073] In some embodiments, the device provided in the embodiments of the present application can be implemented in software. Figure 2 A training device 455 of a cross-modal model stored in a memory 450 is shown, which may be software in the form of a program and a plug-in, including the following software modules: a feature extraction module 4551, a model training module 4552. These modules are logical, and therefore may be arbitrarily combined or further split according to the functions implemented. The functions of each module will be described below.
[0074] In some embodiments, the terminal device or server can implement the training method of the cross-modal model provided in the embodiment of the present application by running a computer program. For example, the computer executable instructions can be microprogram-level commands, machine instructions, or software instructions. The computer program can be a native program or software module in the operating system; in short, the above-mentioned computer executable instructions can be instructions in any form, and the above-mentioned computer program can be an application, module, or plug-in in any form.
[0075] The training method of the cross-modal model provided in the embodiment of the present application will be explained in combination with the exemplary application and implementation of the server provided in the embodiment of the present application.
[0076] Below, the training method of the cross-modal model provided in the embodiment of the present application is described. As mentioned above, the electronic device for implementing the training method of the cross-modal model in the embodiment of the present application can be a terminal device or a server, or a combination of the two. Therefore, the execution subject of each step will not be repeatedly described below.
[0077] It should be noted that in the example of training processing of the cross-modal model below, protein sequences and texts are used as examples. Based on the understanding of the following text, those skilled in the art can apply the training method of the cross-modal model provided in the embodiment of the present application to the processing of other modal data, for example: cross-modality between three-dimensional structure graph data of proteins and text data; cross-modality between protein-related knowledge graph data and protein sequences.
[0078] See also Figure 3A , Figure 3A is a flow chart of the training method of the cross-modal model provided in the embodiment of the present application, which will be combined with Figure 3A The steps shown are explained. Figure 3A The execution entity of the step is Figure 1 Server 200 in.
[0079] In step 301, based on the sample protein data and the sample annotation data, a cross-modal model is called to perform feature extraction processing to obtain a first protein feature of the sample protein data and a first text feature of the sample annotation data.
[0080] For example, the sample annotation data is used to annotate the sample protein, and the sample protein data includes the amino acid sequence that constitutes the sample protein. The cross-modal model includes a protein encoder and a text encoder, the protein encoder is used to extract features from the protein data, and the text encoder is used to extract features from the annotation data. In an embodiment of the present application, cross-modality refers to the cross-modality between protein and text modalities, and the protein data can be a protein sequence or a protein structure diagram. The protein sequence is used as an example for explanation below. The feature extraction processing of the sample protein data and the sample annotation data can be parallel or serial. For example: first extract the features of the protein data through the protein encoder, and then extract features from the sample annotation data through the text encoder. Alternatively, the protein encoder and the text encoder perform corresponding feature extraction processing separately and simultaneously.
[0081] In some embodiments, reference Figure 3B , Figure 3B is a second flow chart of the cross-modal model training method provided in the embodiment of the present application. Step 301 can be performed by Figure 3B Steps 3011 to 3015 are implemented as described in detail below.
[0082] In step 3011, the cross-modal model is called to perform the following processing: based on the first vocabulary, feature mapping processing is performed on each sample protein sequence in the sample protein data to obtain a first embedding vector for each amino acid in each sample protein sequence.
[0083] Here, the sample protein sequence includes a plurality of amino acids, and the first vocabulary includes a mapping relationship between the amino acids and the first embedding vector.
[0084] For example, the cross-modal model includes a protein encoder and a text encoder, and step 3011 refers to step 3013 being performed by the protein encoder. A protein sequence is a sequence composed of multiple amino acid residues, and the order in which the amino acid residues are sorted in the protein sequence is related to the properties of the protein. The protein sequence is expressed as a character sequence, and the expression of the amino acids in the protein sequence is usually expressed in three-letter abbreviations. These abbreviations represent the 20 standard amino acids that make up the protein. Specifically, each amino acid has a specific three-letter code, for example: Lysine is expressed as LYS or Lys. The first vocabulary stores the mapping relationship between the expression of the amino acid and the first embedding vector. The mapping process can be implemented in the following way: query the first vocabulary according to the expression of the amino acid to obtain the first embedding vector corresponding to the amino acid.
[0085] In step 3012, position encoding is performed on the first embedding vector of each amino acid according to the order of each amino acid in the sample protein sequence to obtain the amino acid feature of each amino acid.
[0086] For example, position encoding refers to the process of converting the position information of amino acids in a protein sequence into a numerical representation. The position of the amino acid in the protein sequence affects the properties of the protein, so position encoding is performed on the amino acid so that the extracted features can characterize the position information of the amino acids in the protein. The specific methods of position encoding can be direct position encoding and fixed window encoding. In the embodiment of the present application, direct position encoding is used as an example to illustrate, and a unique numerical index is assigned to each amino acid position to obtain the position encoding of the first embedded vector of each amino acid. For example, the position encoding of the first amino acid in the sequence is 1, the second is 2, and so on. The first embedded vector marked with the position encoding is used as the amino acid feature.
[0087] In step 3013, for each sample protein sequence, amino acid features of a plurality of amino acids included in the sample protein sequence are combined into a first protein feature.
[0088] For example, according to the order in which the amino acids appear in the protein sequence, the amino acid features are combined into a first protein feature. The first protein feature is represented as a feature vector formed by combining multiple embedding vectors. It is assumed that a protein encoder that performs protein feature extraction processing converts each amino acid in the protein sequence into a vector of a fixed size, called an amino acid embedding. In order to preserve the order information in the sequence, the protein encoder encodes the position of each amino acid to ensure that the model can understand the position of the amino acid in the sequence.
[0089] In step 3014, based on the second vocabulary, feature mapping processing is performed on each sample annotation text in the sample annotation data to obtain a second embedding vector for each character in each sample annotation text.
[0090] Here, the second vocabulary includes mapping relationships between characters and second embedding vectors.
[0091] For example, the sample annotation data is in the form of text, for example: This is [IQ motif and SEC7 domain protein 11, from rat], belongs to [BRAG family], contains [962 amino acids], has 1 IQ domain, 1 SEC7 domain, 1 PH domain. The above text is an annotation of a certain protein, characterizing the domains and sources contained in this protein, the total number of amino acids, and other information. By querying the second vocabulary, each character in the text is mapped to a second embedding vector.
[0092] In step 3015, for each sample annotation text, the second embedding vector of each character in the sample annotation text is combined into a first text feature.
[0093] For example, according to the order in which the characters appear in the sample annotation text, the second embedding vectors are combined into a first text feature. The first text feature is characterized as a feature vector formed by combining multiple embedding vectors.
[0094] In the embodiment of the present application, by using different encoders to perform feature extraction processing on data of different modalities, feature extraction processing can be performed in a targeted manner to improve the accuracy of feature extraction. Furthermore, when the cross-modal model is applied to the protein property prediction process, the cross-modal model can more accurately characterize the characteristics of the protein and the characteristics of the text, and accurately predict the properties of unknown proteins.
[0095] Continue to refer Figure 3A In step 302, the first protein feature and the first text feature are encoded respectively to obtain a second protein feature and a second text feature.
[0096] For example, the first protein feature and the first text feature are respectively encoded again to obtain the second protein feature and the second text feature. The encoding process here is a process of converting the first protein feature and the first text feature into low-dimensional features.
[0097] In some embodiments, step 302 can be implemented by: mapping each amino acid feature in the first protein feature to a numerical vector representation, and using the numerical vector representation as the second protein feature; and performing dimensionality reduction on the second embedding vector corresponding to each character in the first text feature to obtain the second text feature.
[0098] For example, the above mapping and dimensionality reduction processing apply the classification embedding (CLS) principle. Taking the first text feature as an example, a vector representation is assigned to each character in the first text feature, and the vector representation can capture the semantic and contextual information of each word. Each embedding vector in the first text feature will be combined into a single vector, which is used to represent the semantic information of the entire text. Similarly, each amino acid feature is mapped to a single numerical vector representation to obtain a second protein feature, which is also a single vector.
[0099] In step 303, a first sub-contrastive loss is determined based on the first protein feature and the second text feature, and a second sub-contrastive loss is determined based on the second protein feature and the first text feature.
[0100] For example, the first protein feature is compared with the second text feature, and the second protein feature is compared with the first text feature to determine the corresponding contrast loss. The first sub-contrast loss and the second sub-contrast loss can be determined simultaneously, or both can be determined in a serial execution manner.
[0101] In some embodiments, reference Figure 3G , Figure 3G 303 is a seventh flow chart of the cross-modal model training method provided in the embodiment of the present application. Step 303 can be implemented through steps 3031 to 3033, which are described in detail below.
[0102] In step 3031, the true similarity between the following parameters is determined: the sample protein sequence corresponding to the first protein feature and the sample annotation text corresponding to the first text feature.
[0103] For example, the true similarity is determined based on the entities described by the sample protein sequence and the sample annotation text, and the value of the true similarity is 1 or 0. That is, when the sample protein sequence and the protein described by the sample annotation text are the same protein, the true similarity is equal to 1, and conversely, when the sample protein sequence and the protein described by the sample annotation text are not the same protein, the true similarity is equal to 0.
[0104] In step 3032, a first similarity between the first protein feature and the second text feature is determined, and a second similarity between the second protein feature and the first text feature is determined.
[0105] For example, the similarity used in contrastive learning can be represented by the following formula (2.1) and formula (2.2):
[0106]
[0107] in, is the first similarity between the first protein feature and the second text feature of the protein; is the second similarity between the first text feature of the text and the second protein feature; S represents the vector representation (first protein feature) extracted by the protein encoder from the protein sequence. A represents the vector representation (first text feature) extracted by the text encoder from the text. S m A represents the representation of the protein sequence extracted using the momentum encoder (second protein feature). m represents the text representation (second text feature) extracted by the momentum encoder. τ represents a scalar tensor used to adjust the dynamic range of the similarity score between protein and text features. It is a learnable parameter with an initial value set to 0.07. s represents the function for obtaining similarity, which is specifically represented by cosine similarity.
[0108] In step 3033, the first cross entropy loss between the true similarity and the first similarity is used as the first sub-contrast loss, and the first cross entropy loss between the true similarity and the second similarity is used as the second sub-contrast loss.
[0109] For example, after calculating the corresponding similarity, record y a2s (A) and y s2a (S) is the true similarity, represented by a one-hot code. Positive sample pairs are represented by 1, and negative sample pairs are represented by 0. The first sub-contrast loss is represented by H(y s2a (S), p s2a (S)), the second sub-contrast loss is denoted as H(y a2s (A), p a2s (A)). Where H represents the cross entropy loss function.
[0110] In an embodiment of the present application, by determining the contrast loss in feature extraction of the cross-modal model, the contrast loss helps to ensure that representations from the same modality are close to each other in the feature space, while representations of different modalities are far away from each other, which helps the model learn more consistent and discriminative cross-modal representations. By re-encoding text features and protein features, negative samples can be constructed, and the contrast loss can be determined based on the features obtained by the re-encoding, which can save the computing resources required in the process of obtaining the contrast loss and training the cross-modal model.
[0111] Continue to refer Figure 3A In step 304, based on the first sub-contrast loss and the second sub-contrast loss, a total loss function corresponding to the cross-modal model is determined.
[0112] For example, the total loss function may be the average of the first sub-contrast loss and the second sub-contrast loss, or the total loss function may be the sum of the contrast loss and the matching loss.
[0113] In some embodiments, reference Figure 3C , Figure 3C is a third flow chart of the cross-modal model training method provided in the embodiment of the present application. Step 304 can be performed by Figure 3C Steps 3041 to 3043 are implemented as described in detail below.
[0114] In step 3041, the total contrast loss of the cross-modal model is determined based on the first sub-contrast loss and the second sub-contrast loss.
[0115] In some embodiments, step 3041 can be implemented by: determining a first mean of each first sub-contrast loss; determining a second mean of each second sub-contrast loss; and taking the average of the first mean and the second mean as the total contrast loss of the cross-modal model. Step 3041 can be represented by the following formula (3).
[0116]
[0117] Where D represents the entire data set. APC is the total contrast loss.
[0118] In step 3042, the matching loss of the cross-modal model is determined based on the sample feature pairs.
[0119] Here, the sample feature pair includes a first protein feature and a first text feature.
[0120] For example, a sample feature pair can be a combination of any first protein feature and a first text feature. The matching loss function (Annotation-Protein Matching) helps to improve the performance and reliability of deep learning models in protein function prediction tasks by taking the matching degree between annotations and protein sequences as the training target. The matching loss function can be represented by the cross entropy loss between the actual similarity between two features in the sample feature pair and the predicted similarity.
[0121] In some embodiments, reference Figure 3D , Figure 3D is a fourth flow chart of the cross-modal model training method provided in the embodiment of the present application. Step 3042 can be performed by Figure 3D Steps 30421 to 30424 are implemented as described in detail below.
[0122] In step 30421, the first protein feature and the first text feature in the sample feature pair are fused to obtain a fused feature.
[0123] For example, the fusion process is to combine two features into one feature. The fusion can be achieved by combining the first protein feature and the first text feature into a feature sequence (S, A) through an encoder, and encoding the feature sequence into a feature vector. f ME It's an encoder.
[0124] In step 30422, classification processing is performed based on the fused features to obtain a first probability that the sample feature pair is a positive sample.
[0125] Here, when the sample protein sequence corresponding to the first protein feature and the sample annotation text corresponding to the first text feature are used to characterize the same protein, the sample feature pair is a positive sample. The classification process can be implemented by an additional multilayer perceptron, which is not included in the cross-modal model and can be an additional setting.
[0126] In step 30423, the true probability that the sample feature pair is a positive sample is determined.
[0127] For example, the true probability ranges from 1 to 0. When two features in a sample feature pair represent the same protein, the true probability is 1. Otherwise, when two features in a sample feature pair represent the same protein, the true probability is 0.
[0128] In step 30424, the third cross entropy loss between the first probability and the true probability of each sample feature pair is determined, and the average value of the third cross entropy loss is used as the matching loss of the cross-modal model.
[0129] For example, the probability that a pair of sample feature pairs belongs to a positive sample pair is expressed as p APM (A, S), the real label is y APM The matching loss is expressed as the following formula (4).
[0130] L APM =E (S,A)~D H(y APM (A, S), p APM (A, S))(4)
[0131] Among them, L APM is the matching loss.
[0132] Continue to refer Figure 3C , in step 3043, the total contrast loss and the matching loss are combined into a total loss function.
[0133] For example, the total loss function may be the sum of the total contrast loss and the matching loss, or the total loss function may be the weighted sum of the total contrast loss and the matching loss. The specific weight value may be determined according to the application scenario.
[0134] In the embodiments of the present application, by comprehensively updating the model by contrasting the learning loss and the matching loss, the ability to align cross-modal models can be improved, thereby improving the accuracy of the model in downstream protein property prediction tasks.
[0135] Continue to refer Figure 3A In step 305, the parameters of the cross-modal model are updated based on the total loss function to obtain a trained cross-modal model.
[0136] In some embodiments, step 305 can be implemented in the following manner: performing back-propagation processing on the cross-modal model based on the total loss function to determine the gradient descent direction of the total loss function; iteratively updating the parameters of the cross-modal model according to the gradient descent direction to obtain the trained cross-modal model.
[0137] For example, the gradient descent direction is the direction that makes the value of the loss function decrease the fastest. The learning rate, number of iterations and sample batch size involved in the iterative update process can be set according to the actual application scenario. The trained cross-modal model can extract features from at least one of protein data and text data, or extract features from both at the same time to obtain fused features.
[0138] In some embodiments, the cross-modal model includes a protein encoder, a text encoder, and a momentum encoder; the protein encoder is used to extract features from protein data, the text encoder is used to extract features from annotation data, and the momentum encoder is used to encode features output by the protein encoder and the text encoder. Fig. 6A It is a first structural diagram of the cross-modal model provided in an embodiment of the present application; the cross-modal model 600 includes: a protein encoder 601, a text encoder 602, a momentum encoder 603, and a classifier 604. In the process of training the cross-modal model to extract features of protein and text cross-modality, there is no need to train the classifier 604. The momentum encoder 603 encodes the features output by the protein encoder 601 and the text encoder 602 to obtain the encoded features. Based on the encoded features, the features output by the protein encoder 601 and the text encoder 602, contrastive learning can be performed to determine the contrastive learning loss of the cross-modal model.
[0139] In some embodiments, when the parameters of the cross-modal model are iteratively updated according to the direction of gradient descent, the weight parameters of the momentum encoder are determined; based on preconfigured hyperparameters, the weight parameters are linearly transformed to obtain updated parameters; the weight parameters of the momentum encoder are replaced based on the updated parameters to obtain an updated momentum encoder, wherein the updated parameters are used as the weight parameters in the next update process.
[0140] For example, when the momentum encoder updates the parameters of the text encoder and the protein encoder in the cross-modal model, the parameters are also updated accordingly. The linear conversion process can be represented by the following formula (1):
[0141] f t =mf t-1 +(1-m)f(1)
[0142] Among them, m is a pre-configured hyperparameter, f tis the weight parameter of the momentum encoder obtained in the current round of iteration, f t-1 is the weight parameter corresponding to the momentum encoder obtained in one round of iteration. In practical applications, the number of momentum encoders can be two, that is, a first momentum encoder is set for the text encoder to encode the first text feature; a second momentum encoder is set for the protein encoder to encode the first protein feature. The features encoded by the momentum encoder are different from the original features, so that negative samples are introduced in the training process, and a large amount of additional video memory is not occupied, saving the computing resources required in the training process.
[0143] In some embodiments, the cross-modal model is applied to protein property prediction, and the cross-modal model may only include a protein encoder, a text encoder, and a classifier in the training and application of the protein property prediction task. Figure 6B , Figure 6B 600 is a second structural diagram of a cross-modal model provided in an embodiment of the present application. The cross-modal model 600 used in the classification task includes: a protein encoder 601, a text encoder 602, and a classifier 604. Protein data is input into the protein encoder 601, annotation data is input into the text encoder 602, and the classifier 604 outputs a classification result based on protein features and text features. The following is an explanation of the classification task performed by the trained cross-modal model.
[0144] In some embodiments, reference Figure 3E , Figure 3E is a fifth flow chart of the cross-modal model training method provided in the embodiment of the present application. After step 305, further execute Figure 3E Steps 3061 to 3063 are described in detail below.
[0145] In step 3061, a first sample set is obtained.
[0146] Here, the first sample set includes a plurality of first sample pairs, and the actual probability that the sample protein associated with each first sample pair has a preconfigured property. The first sample pair includes: a sample protein sequence and a sample annotation text of the sample protein.
[0147] For example, the properties of a protein include biological properties, structural properties, chemical or physical properties of the protein. Taking the stability of a protein as an example, the actual probability ranges from 0 to 1, and a higher actual probability indicates a stronger stability of the protein. The actual stability of a protein can be obtained through experiments.
[0148] In step 3062, based on the multiple first sample pairs, the trained cross-modal model is called to perform classification processing to obtain a second probability that the sample protein associated with each first sample pair has a preconfigured property.
[0149] For example, the cross-modal model extracts features from the protein sequence and the text in the first sample pair, respectively, and the classifier in the cross-modal model performs classification processing based on the feature extraction results to obtain a second probability that the sample protein has a preconfigured property. The second probability is a predicted probability. The classification processing is implemented by the classifier in the cross-modal model, and the classifier can be a multi-layer perceptron.
[0150] In step 3063, based on the cross entropy loss between each second probability and the corresponding actual probability, the parameters of the cross-modal model are updated to obtain an updated cross-modal model.
[0151] For example, the method of obtaining the cross entropy loss can refer to step 303 and step 304 above, which will not be repeated here. The method of updating the parameters of the cross-modal model can be back propagation processing, which can refer to step 305 above, which will not be repeated here. Steps 3061 to 3063 train the model through supervised learning, which can improve the accuracy of the model for specific data classification.
[0152] In some embodiments, after step 3063, the following processing is also performed: based on the protein sequence to be detected, the updated cross-modal model is called to perform classification processing to obtain the predicted probability that the protein associated with the protein sequence to be detected has a preconfigured property; wherein the strength of the preconfigured property is positively correlated with the predicted probability, and the type of the preconfigured property includes at least one of the following: biological properties, structural properties, chemical properties, and physical properties.
[0153] For example, the value range of the prediction probability is 0 to 1. The higher the prediction probability, the higher the possibility that the protein to be detected has the corresponding property, or the higher the intensity of the corresponding property.
[0154] In some embodiments, reference Figure 3F , Figure 3F is a sixth flow chart of the cross-modal model training method provided in the embodiment of the present application. After step 305, further execute Figure 3F Steps 3071 to 3073 are described in detail below.
[0155] In step 3071, a second sample set is obtained.
[0156] Here, the second sample set includes a plurality of second sample pairs and a sample label value of a sample protein associated with each second sample pair. The second sample pair includes: a sample protein sequence and a sample annotation text corresponding to the sample protein.
[0157] For example, when the sample protein is located at a preconfigured position, the sample label value is 1, and when the sample protein is located at a position other than the preconfigured position, the sample label value is 0.
[0158] In step 3072, based on the plurality of second sample pairs, the trained cross-modal model is called to perform classification processing to obtain a third probability that the sample protein associated with each second sample pair is located at a preconfigured position.
[0159] For example, the third probability is a predicted probability, and the prediction process for the preconfigured position may be a binary classification process, and the classification process is implemented by a classifier in the cross-modal model. The classifier may be a multi-layer perceptron.
[0160] In step 3073, based on the cross entropy loss between each third probability and the corresponding sample label value, the parameters of the cross-modal model are updated to obtain an updated cross-modal model.
[0161] For example, the method of obtaining the cross entropy loss can refer to step 303 and step 304 above, which will not be repeated here. The method of updating the parameters of the cross-modal model can be back propagation processing, which can refer to step 305 above, which will not be repeated here. Steps 3071 to 3073 train the model through supervised learning, which can improve the accuracy of the model for specific data classification.
[0162] In some embodiments, after step 3073, the following processing is performed: based on the protein sequence to be detected, the updated cross-modal model is called to perform classification processing to obtain the predicted position of the protein associated with the protein sequence to be detected in the cell; wherein the type of predicted position includes at least one of the following: cell membrane, cell fluid.
[0163] The training method of the cross-modal model provided in the embodiment of the present application can realize the pre-training processing of the cross-modal model, so that the cross-modal model has higher accuracy in the downstream protein property prediction task. The cross-modal model trained in the embodiment of the present application can be applied in the following application scenarios: 1. Drug development. The model trained in the embodiment of the present application can predict the properties of the biological macromolecular proteins involved in the drug development process, and researchers can study the corresponding drugs according to the properties of the protein. 2. Food processing. The model trained in the embodiment of the present application can predict the physical and chemical properties of the protein in the food, so that researchers can improve the processing or storage methods of the food.
[0164] In the embodiment of the present application, the first protein feature and the first text feature are encoded again to obtain the second protein feature and the second text feature, and the contrastive learning loss is determined based on the first protein feature and the second text feature, the second protein feature and the first text feature, respectively. The sample difference is characterized by the contrastive learning loss, so that the feature distance between the features extracted by the cross-modal model for similar objects is closer, and the feature distance between the features extracted for different objects is farther, thereby improving the alignment ability of the cross-modal model for different modal features. The total loss function combines the losses obtained by comparing protein features to text and text features to protein features. The parameters of the cross-modal model are updated based on the total loss function to obtain the trained cross-modal model, which can improve the accuracy of feature extraction. Through the training method of the cross-modal model provided in the embodiment of the present application, the function of training protein property prediction on the basis of the trained cross-modal model can improve the accuracy of protein property prediction.
[0165] Below, an exemplary application of the cross-modal model training method of an embodiment of the present application in an actual application scenario will be described.
[0166] In the related art, in the task of predicting protein properties based on deep learning, the proteins in the data set are labeled with attribute labels according to the properties of the proteins, such as the location of the protein in the cell, stability and thermal stability. The labeled protein data is used to train deep learning models, such as multi-layer perceptron (MLP), convolutional neural network (CNN) and Transformer. When receiving the protein sequence, the model will output the properties of these proteins. The training method of the above model is usually in the context of supervised learning, which can be to randomly initialize a protein encoder and then start training. However, labeled proteins account for a minority of all known proteins, so the related art also pre-trains a protein encoder in an unsupervised manner, and then performs supervised tasks based on the pre-trained protein encoder. In addition to pre-training, the related art also improves the understanding ability of the protein encoder by aligning the protein and the text. In contrastive learning, due to the limitation of the graphics processing unit (Graphics Processing Unit) memory of the computing device, it is not necessarily possible to fully utilize the existing negative samples, which in turn affects the accuracy of feature extraction of the protein encoder in the pre-training stage.
[0167] The embodiment of the present application provides a training method for a cross-modal model, involving an alignment framework of protein modality and text modality based on contrastive learning, with the goal of using this alignment framework to improve the ability of the protein encoder to recognize specific properties of proteins. During the alignment training process, the modality of the text will act as a certain supervisory signal to drive the protein encoder to obtain the information of the text, so that in the downstream protein property prediction task, the aligned protein encoder will have better performance.
[0168] The following is an explanation of the cross-modal model training method provided in the embodiment of the present application in conjunction with the accompanying drawings. Figure 5 , Figure 5 8 is a schematic diagram of the eighth flow chart of the cross-modal model training method provided in the embodiment of the present application. Figure 1 The server 200 in the Figure 5 The steps are explained below.
[0169] In step 501, a cross-modal model is called to perform feature extraction on sample protein data and sample annotation data to obtain a first protein feature and a first text feature.
[0170] For example, the cross-modal model includes a protein encoder and a text encoder. The protein encoder can be an evolutionary scale model (ESM2) or a protein pre-trained language model (ProtBERT). ProtBERT is a model for protein sequence encoding. It is a pre-trained model based on the transformer model architecture and is specifically designed to process protein sequence data. ProtBERT can capture the functional and structural information of proteins by learning language patterns in protein sequences, which is crucial for protein representation learning. This model is commonly used in the fields of bioinformatics and computational biology.
[0171] The text encoder can be a pre-trained language model for scientific text (SciBERT). SciBERT is a pre-trained language model specifically for scientific text. It is developed based on the transformer model architecture and is designed to capture semantic and grammatical features in scientific text to improve the understanding and processing of scientific literature. SciBERT has learned language patterns and knowledge unique to the scientific field by pre-training on a large amount of scientific literature, which makes it perform well in processing related tasks in fields such as biomedical research, physics, chemistry, etc. SciBERT can provide better initialization parameters for downstream tasks, accelerate convergence, and improve generalization performance, avoiding overfitting on small data sets.
[0172] For ease of understanding, the following is described in conjunction with the accompanying drawings. Figure 4 , Figure 4 is a training principle diagram of the cross-modal model provided in the embodiment of the present application. The training of the cross-modal model includes two stages: feature extraction and alignment. Figure 4 In the protein and text encoder 411, the cross-modal model includes a self-attention mechanism 401 and a language model 402. The protein sequence is input into the self-attention mechanism 401 model to extract protein features (first protein features). The attribute annotation text is input into the language model 402 to extract text features (first text features).
[0173] Self-attention, also known as intra-attention, is a mechanism that allows the model to focus on different position information within a sequence when processing sequence data. By encoding proteins through the self-attention mechanism, the output features can represent the position information of amino acids in the protein. The position information of amino acids is associated with the properties of proteins, and then feature extraction of protein sequences based on the self-attention mechanism can increase the accuracy of feature extraction for proteins.
[0174] In step 502, the first protein feature and the first text feature are encoded by a momentum encoder to obtain a second protein feature and a second text feature.
[0175] Continue to refer Figure 4 , the protein feature is input into the first forward feedback layer 403 for feature extraction, the classification embedding 405 performs momentum encoding on the output of the first forward feedback layer 403, and the result of momentum encoding (the second protein feature) is added to the key queue 406. The classification embedding 404 performs momentum encoding on the text feature, and the result of momentum encoding (the second text feature) is also added to the key queue 406. Based on the momentum encoding result, text features, and protein features in the key queue, the contrastive learning loss function L can be determined. APC .
[0176] For example, feed forward refers to the transmission process of input data through each layer of the network, starting from the input layer, passing through the hidden layer, and finally reaching the output layer. In this process, the output of each layer is the input of the next layer, the information flow is unidirectional, and there is no feedback loop between layers. Feed forward can be applied to feature extraction processing. In a feed forward neural network, each neuron processes the input data through a weight matrix and an activation function to extract and transform features. This process is repeated in each layer, allowing the network to learn higher-level features from the original input. Classification Embedding (CLSEmb), this embedding is a method of converting text data into a numerical representation that can be understood by a machine, and is used for text classification tasks. Classification embedding assigns a vector representation to each word in the text, and the vector representation can capture the semantic and contextual information of each word. The embedding vectors of the entire text are combined into a single vector, which is used to represent the semantic information of the entire text. In the embodiment of the present application, in order to increase the total amount of negative samples in the contrastive learning process, two additional momentum encoders f are set in addition to the original protein and text encoders f m , the momentum encoder update formula is represented by the following formula (1):
[0177] f t =mf t-1 +(1-m)f(1)
[0178] Among them, m is the hyperparameter that needs to be set. For these two momentum encoders, the momentum update method is used to update the two encoders. The features extracted by the momentum encoder will be regarded as negative samples and stored in a dynamic queue. Figure 4 A queue of keys.
[0179] In the embodiment of the present application, negative samples are introduced through momentum coding without occupying a large amount of additional video memory, thereby saving computing resources required during the training process.
[0180] In step 503, a contrastive learning loss of a cross-modal model is determined based on the first protein feature, the first text feature, the second protein feature, and the second text feature.
[0181] For example, in multimodal contrastive learning, the similarity of two modal data is calculated. If the two modal data describe the same entity, the expected output similarity will be set to 1, otherwise it will be 0. The specific formula of similarity is represented by the following formula (2.1) and formula (2.2):
[0182]
[0183] in, is the similarity between the vector representation of the protein and the text representation extracted by the momentum encoder; is the similarity between the vector representation of the text and the representation of the protein sequence extracted by the momentum encoder; S represents the vector representation (first protein feature) extracted from the protein sequence by the protein encoder. A represents the vector representation (first text feature) extracted from the text by the text encoder. S m A represents the representation of the protein sequence extracted using the momentum encoder (second protein feature). m represents the text representation (second text feature) extracted by the momentum encoder. τ represents a scalar tensor used to adjust the dynamic range of the similarity score between protein and text features. It is a learnable parameter with an initial value set to 0.07. s represents the function for obtaining similarity, which is specifically represented by cosine similarity.
[0184] After calculating the corresponding similarity, record y a2s (A) and y s2a (S) is the true similarity, represented by a one-hot code. Positive sample pairs will be represented as 1, and negative sample pairs will be represented as 0. The final annotation-protein contrast learning loss function APC (Annotation-Protein Contrastive) can be expressed by the following calculation formula (3).
[0185]
[0186] Among them, H represents the cross entropy loss function and D represents the entire dataset.
[0187] In the embodiment of the present application, by selecting a pre-trained encoder for alignment training, the convergence of the alignment can be accelerated.
[0188] In step 504, a plurality of sample pairs are determined based on the sample protein data and the sample annotation data, and a matching loss of a cross-modal model is determined according to the features of each sample pair.
[0189] For example, the present embodiment also introduces a pairing task to additionally align protein sequences and text modalities. The pairing task can be implemented by a multimodal encoder. The multimodal encoder f ME , which can receive protein and text data at the same time and fuse the data of these two modalities into one vector This vector can be used to determine whether the received protein and text data belong to the same pair. It should be noted here that both the multimodal encoder and the protein encoder adopt the architecture of the converter model, so the parameters of the self-attention layer of the two encoders in the embodiment of the present application are shared, which can reduce the pressure on the graphics processor video memory, thereby increasing the batch size to a certain extent, thereby improving the efficiency of contrastive learning.
[0190] vector It will be input into an additional multi-layer perceptron (MLP), which will output a 0-1 value representing the probability p that this pair of samples belongs to the positive sample pair. APM (A, S), the real label is y APM (A, S) is used to represent it.
[0191] Continue to refer Figure 4 , the text features and protein features constituting the sample pair are converted through the cross attention mechanism 407 to obtain a splicing vector, and the conversion result is predicted through the second forward feedback layer 408 to obtain the probability that the sample pair belongs to a positive sample. Based on the probability that the sample pair belongs to a positive sample and the actual label of the sample pair, the matching loss function L is determined. APM The second forward feedback layer 408 plays a different role than the first forward feedback layer. Cross-Attention is an attention mechanism used on neural networks, which allows the model to focus on and fuse the relevant information between two sequences when processing sequence data from two different sources. When the cross-attention mechanism is applied to multimodal tasks, different types of information can be combined.
[0192] For example, the cross attention mechanism 407 and the second forward feedback layer 408 constitute a multimodal encoder. Similar to the contrast loss, the embodiment of the present application uses cross entropy to calculate the protein annotation matching loss function L of the matching task. APM (Annotation-Protein Matching), expressed as the following formula (4).
[0193] L APM =E (S,A)~D H(y APM (A, S), p APM (A, S))(4)
[0194] In step 505, the cross-modal model is trained based on contrastive learning loss and matching loss to obtain a trained cross-modal model.
[0195] For example, during the training process, the embodiment of the present application simply adds the two loss functions to obtain the final loss function, and uses the gradient descent method to optimize the entire model. In the embodiment of the present application, the hyperparameters involved in the process of training the model by comparative learning include: learning rate, sample batch size, weight decay, training rounds, momentum, and sequence length (QueueSize). The above hyperparameters can be set according to the actual application scenario. For example: the learning rate is set to 0.00003. A smaller learning rate can make the model adjust the parameters more carefully during the training process, and can make the model converge more stably during the training process, thereby reducing the risk of overfitting.
[0196] In some embodiments, the total loss function may also be a weighted summation result between the contrastive learning loss and the matching loss. The weight values used in the weighted summation process may be determined according to the application scenario.
[0197] In the embodiment of the present application, the momentum update method is used to update the negative samples in contrastive learning. Matching learning is introduced to determine whether a pair of text descriptions and protein sequences is a positive pair. The model is updated as a whole through contrastive learning loss and matching loss, which improves the alignment ability of cross-modal models and thus improves the accuracy of the model in downstream protein property prediction tasks.
[0198] In step 506, fine-tuning is performed based on the trained cross-modal model to obtain a protein property prediction model.
[0199] For example, the training of the protein property prediction model includes two stages, a pre-training stage and a fine-tuning stage, wherein steps 501 to 505 correspond to the pre-training stage. The fine-tuning process may be a supervised training method.
[0200] The following examples illustrate two types of protein property prediction tasks. One type of task is the location prediction task of two proteins in the cell. The two tasks are binary classification and ten-class classification. The binary classification task is to judge the possible locations of the protein in the cell (cell wall or free state) according to the properties of the protein. The other type of task is the protein property prediction task (ten-classification). In the data set of this type of task, certain properties of the protein are ranked. For example, in the task of protein stability prediction, several proteins are ranked according to the judged protein stability, and then the correlation coefficient is calculated based on the predicted ranking and the actual protein stability ranking. The higher the correlation coefficient, the better the model of the embodiment of the present application can understand the properties of the protein.
[0201] The training method of the cross-modal model provided in the embodiment of the present application can be used in the design process of proteins to quickly screen proteins and reduce the high cost and long cycle of wet experiments. In the process of protein design, the protein language model is used to design multiple protein sequences, and the protein property prediction model is used to predict the properties of the generated proteins, and screen these proteins according to the properties to obtain proteins that meet the requirements. The prediction task of the protein may include: predicting the possible position of the generated protein in the cell (located in the cell wall or free state), the stability of the protein, and the biological properties of other proteins. The biological properties of proteins include: 1. The function of the protein, such as catalytic reaction (as an enzyme), structural support (such as collagen), transportation (such as hemoglobin), immunity (such as antibodies), signal transmission (such as receptor proteins), etc.; 2. The structure of the protein.
[0202] The following illustrates the beneficial effects of the model trained by the cross-modal model training method provided in the embodiment of the present application through comparison with related technologies.
[0203] For example, the prediction of the location of a protein in a cell is measured by accuracy. Bin stands for Binary classification, which determines whether the type of protein appears on the cell membrane or whether the protein is free and hydrolyzable. Sub stands for Subceller, which is a very specific task. Subceller is used to describe the location of a substance or structure in different parts or organelles inside a cell. In cell biology, many biological molecules and structures can be classified as subcellular structures according to their location in the cell. For example, proteins can be located on the cell membrane, in the cytoplasm, in the nucleus, or distributed in other specific organelles, such as mitochondria, endoplasmic reticulum, Golgi apparatus, etc.
[0204] In the embodiment of the present application, the test task is performed with the ten types of tasks given above, and the two protein encoders based on ESM2 and ProtBERT are aligned respectively. The properties of the proteins predicted in the test task include: β-lactamase, AAV, thermal stability, catalytic ability and stability. Through the above test tasks, it can be concluded that compared with the scheme of the related art, the method of the embodiment of the present application has been improved in various indicators, and the average improvement in all tasks is 2.67%. ProtST is a method of alignment in the related art, but ProtST does not introduce momentum encoders and matching tasks as in the embodiment of the present application. Compared with ProtST, the embodiment of the present application can further improve the efficiency of alignment, so that the protein encoder has a stronger ability to predict protein properties. In addition, compared with the model trained from scratch, that is, the related technology without pre-training, in the embodiment of the present application, by setting a pre-training stage during the training process, the ability to predict protein properties is greatly improved compared to the method without a pre-training stage.
[0205] The following continues to describe the exemplary structure of the cross-modal model training device 455 provided in the embodiment of the present application implemented as a software module. In some embodiments, Figure 2 As shown, the software modules in the training device 455 of the cross-modal model stored in the memory 450 may include: a feature extraction module 4551, which is used to call the cross-modal model to perform feature extraction processing based on the sample protein data and the sample annotation data to obtain the first protein feature of the sample protein data and the first text feature of the sample annotation data; the feature extraction module 4551 is also used to encode the first protein feature and the first text feature respectively to obtain the second protein feature and the second text feature; the model training module 4552 is used to determine the first sub-contrast loss based on the first protein feature and the second text feature, and to determine the second sub-contrast loss based on the second protein feature and the first text feature; the model training module 4552 is also used to determine the total loss function corresponding to the cross-modal model based on the first sub-contrast loss and the second sub-contrast loss; the model training module 4552 is also used to update the parameters of the cross-modal model based on the total loss function to obtain the trained cross-modal model.
[0206] In some embodiments, the model training module 4552 is used to determine the true similarity between the following parameters: a sample protein sequence corresponding to a first protein feature and a sample annotation text corresponding to a first text feature; determine a first similarity between the first protein feature and the second text feature, and determine a second similarity between the second protein feature and the first text feature; use a first cross entropy loss between the true similarity and the first similarity as a first sub-contrast loss; use the first cross entropy loss between the true similarity and the second similarity as a second sub-contrast loss.
[0207] In some embodiments, the model training module 4552 is used to determine the total contrast loss of the cross-modal model based on the first sub-contrast loss and the second sub-contrast loss; determine the matching loss of the cross-modal model based on the sample feature pair, wherein the sample feature pair includes a first protein feature and a first text feature; and combine the total contrast loss and the matching loss into a total loss function.
[0208] In some embodiments, the model training module 4552 is used to determine a first mean of each first sub-contrast loss; determine a second mean of each second sub-contrast loss; and take the average of the first mean and the second mean as the total contrast loss of the cross-modal model.
[0209] In some embodiments, the model training module 4552 determines the matching loss of the cross-modal model based on the sample feature pairs, including: fusing the first protein feature and the first text feature in the sample feature pair to obtain a fused feature; performing classification processing based on the fused feature to obtain a first probability that the sample feature pair is a positive sample, wherein when the sample protein sequence corresponding to the first protein feature and the sample annotation text corresponding to the first text feature are used to characterize the same protein, the sample feature pair is a positive sample; determining the true probability that the sample feature pair is a positive sample; determining the third cross entropy loss between the first probability and the true probability of each sample feature pair, and taking the average value of the third cross entropy loss as the matching loss of the cross-modal model.
[0210] In some embodiments, the feature extraction module 4551 is used to call the cross-modal model to perform the following processing: based on the first vocabulary, feature mapping processing is performed on each sample protein sequence in the sample protein data to obtain a first embedding vector for each amino acid in each sample protein sequence, wherein the sample protein sequence includes multiple amino acids, and the first vocabulary includes a mapping relationship between the amino acid and the first embedding vector; according to the order of each amino acid in the sample protein sequence, the first embedding vector of each amino acid is positionally encoded to obtain an amino acid feature of each amino acid; for each sample protein sequence, the amino acid features of the multiple amino acids included in the sample protein sequence are combined into a first protein feature; based on the second vocabulary, feature mapping processing is performed on each sample annotation text in the sample annotation data to obtain a second embedding vector for each character in each sample annotation text, wherein the second vocabulary includes a mapping relationship between the character and the second embedding vector; for each sample annotation text, the second embedding vector of each character in the sample annotation text is combined into a first text feature.
[0211] In some embodiments, the feature extraction module 4551 is used to map each amino acid feature in the first protein feature into a numerical vector representation, and use the numerical vector representation as the second protein feature; and perform dimensionality reduction on the second embedding vector corresponding to each character in the first text feature to obtain a second text feature.
[0212] In some embodiments, the model training module 4552 is used to perform back-propagation processing on the cross-modal model based on the total loss function to determine the gradient descent direction of the total loss function; iteratively update the parameters of the cross-modal model according to the gradient descent direction to obtain the trained cross-modal model.
[0213] In some embodiments, the cross-modal model includes a protein encoder, a text encoder, and a momentum encoder; the protein encoder is used to extract features from protein data, the text encoder is used to extract features from annotation data, and the momentum encoder is used to encode features output by the protein encoder and the text encoder.
[0214] In some embodiments, the model training module 4552 is used to determine the weight parameters of the momentum encoder when iteratively updating the parameters of the cross-modal model according to the direction of gradient descent; based on preconfigured hyperparameters, the weight parameters are linearly transformed to obtain updated parameters; based on the updated parameters, the weight parameters of the momentum encoder are replaced to obtain an updated momentum encoder, wherein the updated parameters are used as the weight parameters in the next update process.
[0215] In some embodiments, the model training module 4552 is used to update the parameters of the cross-modal model based on the total loss function to obtain a trained cross-modal model, and then obtain a first sample set, wherein the first sample set includes multiple first sample pairs and the actual probability that the sample protein associated with each first sample pair has a pre-configured property, and the first sample pairs include: a sample protein sequence and a sample annotation text of the sample protein; based on the multiple first sample pairs, call the trained cross-modal model for classification processing to obtain a second probability that the sample protein associated with each first sample pair has a pre-configured property; based on the cross entropy loss between each second probability and the corresponding actual probability, update the parameters of the cross-modal model to obtain an updated cross-modal model.
[0216] In some embodiments, the model training module 4552 is used to update the parameters of the cross-modal model based on the cross-entropy loss between each second probability and the corresponding actual probability, and after obtaining the updated cross-modal model, call the updated cross-modal model based on the protein sequence to be detected for classification processing to obtain the predicted probability that the protein associated with the protein sequence to be detected has a pre-configured property; wherein the strength of the pre-configured property is positively correlated with the predicted probability, and the type of the pre-configured property includes at least one of the following: biological properties, structural properties, chemical properties, and physical properties.
[0217] In some embodiments, the model training module 4552 is used to update the parameters of the cross-modal model based on the total loss function to obtain the trained cross-modal model, and then obtain a second sample set, wherein the second sample set includes multiple second sample pairs and sample label values of sample proteins associated with each second sample pair, and the second sample pairs include: sample protein sequences and sample annotation texts corresponding to the sample proteins; based on the multiple second sample pairs, call the trained cross-modal model for classification processing to obtain a third probability that the sample protein associated with each second sample pair is located at a preconfigured position; based on the cross entropy loss between each third probability and the corresponding sample label value, update the parameters of the cross-modal model to obtain an updated cross-modal model.
[0218] In some embodiments, the model training module 4552 is used to update the parameters of the cross-modal model based on the cross entropy loss between each third probability and the corresponding sample label value, and after obtaining the updated cross-modal model, call the updated cross-modal model based on the protein sequence to be detected to perform classification processing to obtain the predicted position of the protein associated with the protein sequence to be detected in the cell; wherein the type of predicted position includes at least one of the following: cell membrane, cell fluid.
[0219] The embodiment of the present application provides a computer program product, which includes a computer program or a computer executable instruction, and the computer program or the computer executable instruction is stored in a computer-readable storage medium. The processor of the electronic device reads the computer program or the computer executable instruction from the computer-readable storage medium, and the processor executes the computer program or the computer executable instruction, so that the electronic device executes the cross-modal model training method described in the embodiment of the present application.
[0220] The present application embodiment provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions or computer programs are stored. When the computer-executable instructions or computer programs are executed by a processor, the processor will be caused to execute the cross-modal model training method provided by the present application embodiment, for example, Figure 3A The training method of the cross-modal model is shown.
[0221] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or it may be various devices including one or any combination of the above memories.
[0222] In some embodiments, computer executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.
[0223] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file storing other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).
[0224] As an example, computer executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed at multiple sites and interconnected by a communication network.
[0225] In summary, through the embodiments of the present application, the first protein feature and the first text feature are encoded again to obtain the second protein feature and the second text feature. The contrastive learning loss is determined based on the first protein feature and the second text feature, the second protein feature and the first text feature, respectively. The sample difference is characterized by the contrastive learning loss, so that the feature distance between the features extracted by the cross-modal model for similar objects is closer, and the feature distance between the features extracted for different objects is farther, thereby improving the alignment ability of the cross-modal model for different modal features. The total loss function combines the losses obtained by comparing protein features to text and text features to protein features. The parameters of the cross-modal model are updated based on the total loss function to obtain the trained cross-modal model, which can improve the accuracy of feature extraction. Through the training method of the cross-modal model provided in the embodiments of the present application, the function of training protein property prediction on the basis of the trained cross-modal model can improve the accuracy of protein property prediction.
[0226] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.
Claims
1. A cross-modal model training method, characterized in that: The method comprises: Based on the sample protein data and the sample annotation data, calling the cross-modal model to perform feature extraction processing to obtain a first protein feature of the sample protein data and a first text feature of the sample annotation data; Encoding the first protein feature and the first text feature respectively to obtain a second protein feature and a second text feature; determining a first sub-contrastive loss based on the first protein feature and the second text feature, and determining a second sub-contrastive loss based on the second protein feature and the first text feature; Determining a total loss function corresponding to the cross-modal model based on the first sub-contrastive loss and the second sub-contrastive loss; Parameters of the cross-modal model are updated based on the total loss function to obtain the trained cross-modal model.
2. The method according to claim 1, characterized in that The determining of a first sub-contrast loss based on the first protein feature and the second text feature, and the determining of a second sub-contrast loss based on the second protein feature and the first text feature, comprises: determining a true similarity between the following parameters: a sample protein sequence corresponding to the first protein feature and a sample annotation text corresponding to the first text feature; determining a first similarity between the first protein feature and the second text feature, and determining a second similarity between the second protein feature and the first text feature; Using a first cross entropy loss between the true similarity and the first similarity as the first sub-contrast loss; A first cross entropy loss between the true similarity and the second similarity is used as the second sub-contrast loss.
3. The method according to claim 1, characterized in that: The determining, based on the first sub-contrast loss and the second sub-contrast loss, a total loss function corresponding to the cross-modal model includes: Determining a total contrast loss of the cross-modal model based on the first sub-contrast loss and the second sub-contrast loss; Determining a matching loss of the cross-modal model based on a sample feature pair, wherein the sample feature pair includes the first protein feature and the first text feature; The total contrast loss and the matching loss are combined into the total loss function.
4. The method according to claim 3, characterized in that The determining, based on the first sub-contrast loss and the second sub-contrast loss, a total contrast loss of the cross-modal model includes: determining a first mean of each of the first sub-contrast losses; determining a second mean of each of said second sub-contrast losses; An average of the first mean and the second mean is taken as the total contrast loss of the cross-modal model.
5. The method according to claim 3, characterized in that: The determining the matching loss of the cross-modal model based on the sample feature pair includes: fusing the first protein feature and the first text feature in the sample feature pair to obtain a fused feature; Performing classification processing based on the fusion feature to obtain a first probability that the sample feature pair is a positive sample, wherein when the sample protein sequence corresponding to the first protein feature and the sample annotation text corresponding to the first text feature are used to characterize the same protein, the sample feature pair is a positive sample; Determine the true probability that the sample feature pair is a positive sample; A third cross entropy loss between the first probability and the true probability of each of the sample feature pairs is determined, and an average value of the third cross entropy losses is used as the matching loss of the cross-modal model.
6. The method according to claim 1, characterized in that The method of calling a cross-modal model to perform feature extraction based on the sample protein data and the sample annotation data to obtain a first protein feature of the sample protein data and a first text feature of the sample annotation data includes: Call the cross-modal model to perform the following processing: Based on the first vocabulary, feature mapping processing is performed on each of the sample protein sequences in the sample protein data to obtain a first embedding vector for each amino acid in each of the sample protein sequences, wherein the sample protein sequence includes a plurality of amino acids, and the first vocabulary includes a mapping relationship between the amino acids and the first embedding vector; According to the order of each amino acid in the sample protein sequence, position encoding is performed on the first embedding vector of each amino acid to obtain an amino acid feature of each amino acid; For each of the sample protein sequences, combining the amino acid features of the plurality of amino acids included in the sample protein sequence into a first protein feature; Based on the second vocabulary, performing feature mapping processing on each sample annotation text in the sample annotation data to obtain a second embedding vector for each character in each sample annotation text, wherein the second vocabulary includes a mapping relationship between the character and the second embedding vector; For each of the sample annotation texts, the second embedding vector of each of the characters in the sample annotation text is combined into a first text feature.
7. The method according to claim 1, characterized in that The encoding process is performed on the first protein feature and the first text feature respectively to obtain a second protein feature and a second text feature, including: Mapping each amino acid feature in the first protein feature into a numerical vector representation, and using the numerical vector representation as the second protein feature; Perform dimension reduction on the second embedding vector corresponding to each character in the first text feature to obtain the second text feature.
8. The method according to any one of claims 1 to 7, characterized in that: The step of updating the parameters of the cross-modal model based on the total loss function to obtain the trained cross-modal model includes: Performing back-propagation processing on the cross-modal model based on the total loss function to determine a gradient descent direction of the total loss function; Iteratively update the parameters of the cross-modal model according to the gradient descent direction to obtain the trained cross-modal model.
9. The method according to claim 8, characterized in that The cross-modal model includes a protein encoder, a text encoder and a momentum encoder; the protein encoder is used to extract features from protein data, the text encoder is used to extract features from annotation data, and the momentum encoder is used to encode features output by the protein encoder and the text encoder.
10. The method according to claim 9, characterized in that When iteratively updating the parameters of the cross-modal model according to the gradient descent direction, the method further includes: Determining weight parameters of the momentum encoder; Based on preconfigured hyperparameters, linear transformation is performed on the weight parameters to obtain updated parameters; The weight parameters of the momentum encoder are replaced based on the update parameters to obtain the updated momentum encoder, wherein the update parameters are used as weight parameters in the next update process.
11. The method according to any one of claims 1 to 7, characterized in that: After updating the parameters of the cross-modal model based on the total loss function to obtain the trained cross-modal model, the method further includes: Acquire a first sample set, wherein the first sample set includes a plurality of first sample pairs, an actual probability that a sample protein associated with each first sample pair has a preconfigured property, and the first sample pairs include: a sample protein sequence and a sample annotation text of the sample protein; Based on the multiple first sample pairs, calling the trained cross-modal model to perform classification processing to obtain a second probability that the sample protein associated with each of the first sample pairs has a preconfigured property; Based on the cross entropy loss between each of the second probabilities and the corresponding actual probabilities, the parameters of the cross-modal model are updated to obtain the updated cross-modal model.
12. The method according to claim 11, characterized in that After updating the parameters of the cross-modal model based on the cross entropy loss between each of the second probabilities and the corresponding actual probabilities to obtain the updated cross-modal model, the method further includes: Based on the protein sequence to be detected, calling the updated cross-modal model to perform classification processing to obtain a predicted probability that the protein associated with the protein sequence to be detected has the preconfigured property; The strength of the preconfigured property is positively correlated with the predicted probability, and the type of the preconfigured property includes at least one of the following: biological property, structural property, chemical property and physical property.
13. The method according to any one of claims 1 to 7, characterized in that: After updating the parameters of the cross-modal model based on the total loss function to obtain the trained cross-modal model, the method further includes: Acquire a second sample set, wherein the second sample set includes a plurality of second sample pairs, and a sample label value of a sample protein associated with each second sample pair, and the second sample pair includes: a sample protein sequence and a sample annotation text corresponding to the sample protein; Based on the plurality of second sample pairs, calling the trained cross-modal model to perform classification processing to obtain a third probability that the sample protein associated with each second sample pair is located at a preconfigured position; Based on the cross entropy loss between each of the third probabilities and the corresponding sample label value, the parameters of the cross-modal model are updated to obtain the updated cross-modal model.
14. The method according to claim 13, characterized in that After updating the parameters of the cross-modal model based on the cross entropy loss between each of the third probabilities and the corresponding sample label value to obtain the updated cross-modal model, the method further includes: Based on the protein sequence to be detected, calling the updated cross-modal model to perform classification processing to obtain the predicted position of the protein associated with the protein sequence to be detected in the cell; The type of the predicted location includes at least one of the following: cell membrane and cell fluid.
15. A cross-modal model training device, characterized in that: The device comprises: A feature extraction module, configured to call a cross-modal model to perform feature extraction processing based on the sample protein data and the sample annotation data, so as to obtain a first protein feature of the sample protein data and a first text feature of the sample annotation data; The feature extraction module is further used to encode the first protein feature and the first text feature respectively to obtain a second protein feature and a second text feature; a model training module, configured to determine a first sub-contrast loss based on the first protein feature and the second text feature, and to determine a second sub-contrast loss based on the second protein feature and the first text feature; The model training module is further used to determine a total loss function corresponding to the cross-modal model based on the first sub-contrast loss and the second sub-contrast loss; The model training module is also used to update the parameters of the cross-modal model based on the total loss function to obtain the trained cross-modal model.
16. An electronic device, characterized in that: The electronic device comprises: A memory for storing computer executable instructions or computer programs; A processor, used to implement the cross-modal model training method described in any one of claims 1 to 14 when executing computer executable instructions or computer programs stored in the memory.
17. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that: When the computer executable instructions or computer program are executed by a processor, the cross-modal model training method described in any one of claims 1 to 14 is implemented.
18. A computer program product comprising computer executable instructions or a computer program, characterized in that When the computer executable instructions or computer program are executed by a processor, the training method of the cross-modal model described in any one of claims 1 to 14 is implemented.
Citation Information
Cited By
Multi-modal protein sequence generation method based on deep learning
CN119943134A
Protein multi-modal joint modeling method and system based on cross-modal alignment
CN120260664A
