Molecular multi-modal large language model construction method and device based on parameter space alignment

Abstract

This invention belongs to the interdisciplinary field of artificial intelligence and cheminformatics, and provides a method, inference method, and apparatus for constructing a molecular multimodal large language model based on parameter space alignment. The method includes: constructing an instance set; constructing a molecular multimodal large language model, including an adaptive weight generator, a graph neural network encoder, and a large language model; training the molecular multimodal large language model using the instance set to update the parameters of the adaptive weight generator; the graph neural network encoder extracting node-level feature representations of the molecular structure diagram; the adaptive weight generator obtaining the low-rank weight update amount of the target weight component in the large language model based on the node-level feature representation, and injecting the low-rank weight update amount into the original weights of the target weight component; and the large language model after injecting the low-rank weight update amount generating an actual text response based on the input training text command. By integrating the molecular structure into the parameter layer of the large language model, deep perception of the molecular structure is achieved without changing the length of the input sequence.

Description

Technical Field

[0001] This invention relates to the interdisciplinary field of artificial intelligence and cheminformatics, and in particular to a method, inference method and apparatus for constructing molecular multimodal large language models based on parameter space alignment. Background Technology

[0002] In recent years, the deep integration of artificial intelligence (AI) technology with chemistry, materials science, and pharmaceutical science has catalyzed the rise of scientific intelligence. In key tasks such as drug molecule discovery, molecular property prediction, and retrosynthetic planning, models not only need powerful natural language processing capabilities to understand research instructions, but also a deep perception of the two-dimensional topological structure or three-dimensional spatial configuration of molecules. Therefore, constructing multimodal molecular language models capable of simultaneously processing molecular structure data and text data has become a research hotspot in the field of cheminformatics.

[0003] In related technologies, mainstream molecular multimodal large language model construction schemes, such as LLaMo (large language model molecular assistant), InstructMol (molecular instruction fine-tuning model), and PRESTO (progressive pre-trained model), typically employ an input space alignment paradigm. Specifically, molecular graph features are extracted using a pre-trained graph encoder, and then forced to be mapped into one-dimensional sequential graph tokens using a linear projection layer. Subsequently, these graph tokens are concatenated with text instruction tokens at the input layer to form the input sequence for the large language model. Although the aforementioned spatial alignment paradigm has been extensively studied in the multimodal field, this method has the following drawbacks: (1) High computational complexity: Linearizing complex molecular graphs into long sequence tokens will cause the input sequence to expand significantly. LLM (Large Language Model) must consume a lot of computational resources to try to reconstruct multi-hop chemical relationships in the fragmented sequence through self-attention mechanism, which greatly increases the cost of inference and training.

[0004] (2) Fragmentation of structural information: Straightening a molecular diagram with complex branches and ring structures into a one-dimensional sequence causes irreversible collapse of structural information in the original complex long-range topological relationships, ring structures and nonlinear branches of the molecule, which can easily lead to a weakening of the model's structural perception.

[0005] (3) Context window pollution: A large number of graph tokens occupy the limited context window, compressing the space for text instructions and reasoning processes, and limiting the model's ability to process long texts or complex instructions. When dealing with scientific tasks that rely heavily on global topological awareness (such as multi-step synthesis path planning or reaction mechanism analysis), large language models are prone to logical breaks due to insufficient perception of structural information or missing context, and may even generate results that violate basic chemical common sense. Summary of the Invention

[0006] This application aims to at least solve the technical problems existing in the prior art and provide a method, reasoning method and device for constructing molecular multimodal large language models based on parameter space alignment.

[0007] In a first aspect, this application provides a method for constructing a molecular multimodal large language model based on parameter space alignment. The method includes: constructing an instance set, where each instance includes a molecular structure diagram, training text instructions, and a target text response corresponding to the training text instructions based on the molecular structure diagram; constructing a molecular multimodal large language model, wherein the molecular multimodal large language model includes an adaptive weight generator, a pre-trained parameter-frozen graph neural network encoder, and a large language model; training the molecular multimodal large language model using the instance set, updating the parameters of the adaptive weight generator during training; wherein the graph neural network encoder is used to extract node-level feature representations of the molecular structure diagram; the adaptive weight generator obtains the low-rank weight update amount of at least one target weight component in one or more decoder layers of the large language model based on the node-level feature representations, and injects the low-rank weight update amount of the target weight component into the original weights of the target weight component; the large language model after injecting the low-rank weight update amount generates an actual text response based on the input training text instructions.

[0008] Secondly, this application provides a reasoning method, the method comprising: acquiring a molecular structure diagram and a user's query text command for the molecular structure diagram; inputting the molecular structure diagram and the query text command into a molecular multimodal large language model to obtain an actual text response to the query text command, wherein the molecular multimodal large language model is obtained according to the molecular multimodal large language model construction method based on parameter space alignment described in the first aspect of this application; wherein, in the molecular multimodal large language model: a graph neural network encoder is used to extract node-level feature representations of the molecular structure diagram; an adaptive weight generator, based on the node-level feature representations, acquires the low-rank weight update amount of at least one target weight component in one or more decoder layers of the large language model, and injects the low-rank weight update amount of the target weight component into the original weight of the target weight component; the large language model after injecting the low-rank weight update amount generates an actual text response based on the query text command.

[0009] Thirdly, this application provides a computer device including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method described in the first or second aspect of this application.

[0010] Fourthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method described in the first or second aspect of this application.

[0011] Fifthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the first or second aspect of this application.

[0012] In a sixth aspect, this application provides an electronic device, characterized in that the electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the steps of the method described in the first or second aspect of this application.

[0013] The beneficial technical effects of this application are as follows: (1) Instead of converting complex molecular graphs into long sequence tokens, the graph neural network encoder extracts the node-level feature representation of the molecular structure graph. Then, the adaptive weight generator obtains the low-rank weight update of at least one target weight component in one or more decoder layers of the large language model based on the node-level feature representation. The low-rank weight update of the target weight component is injected into the original weight of the target weight component. This realizes the parameter space modulation of the large language model guided by the molecular structure graph. Then, the large language model after injecting the low-rank weight update generates the actual text response based on the input training text command or query text command. This can effectively enhance the overall perception ability of the molecular multimodal large language model of complex molecular topology, alleviate the problem of large computational overhead and fragmentation of structural information caused by the input space alignment method in related technologies, and no longer generate a large number of graph tokens that occupy a limited context window like related technologies, thus improving the accuracy of the actual text response output by the large language model. (2) During the training of the molecular multimodal large language model using the instance set, the parameters of the large language model and the graph neural network encoder are frozen, and only the parameters of the adaptive weight generator are updated and optimized. This is used to learn to convert molecular structural features into the parameter space modulation signal of the optimal large language model, thereby achieving efficient parameter fine-tuning and significantly reducing training costs. While reducing training overhead, it ensures that the model can quickly adapt to molecular-related tasks, taking into account both training efficiency and model performance. (3) The integration of molecular structure is transferred from the input sequence layer to the parameter layer of the large language model, so as to achieve deep perception of molecular structure without changing the length of the text input sequence. Attached Figure Description

[0014] Figure 1 This is a flowchart illustrating a preferred embodiment of the present invention for constructing a molecular multimodal large language model based on parameter space alignment. Figure 2This is a schematic diagram of the structure of a molecular multimodal large language model in a preferred embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of an adaptive weight generator in a preferred embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of each decoding layer in a stacked Transformer decoding layer according to a preferred embodiment of the present invention; Figure 5 This is a flowchart illustrating the reasoning method in a preferred embodiment of the present invention; Figure 6 This is a schematic diagram of the structure of an electronic device in a preferred embodiment of the present invention. Detailed Implementation

[0015] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0016] This application provides a method for constructing a molecular multimodal large language model based on parameter space alignment and an inference method. The execution subject of the above method includes, but is not limited to, at least one of the following electronic devices that can be configured to execute the method provided in the embodiments of this application: a server, a terminal, or a computer. In other words, the method for constructing a molecular multimodal large language model based on parameter space alignment and the inference method can be executed by software or hardware installed on a terminal device or a server device. The software can be a blockchain platform. The server includes, but is not limited to, a single server, a server cluster, a cloud server, or a cloud server cluster. The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, and big data and artificial intelligence platforms.

[0017] This invention provides a method for constructing a molecular multimodal large language model based on parameter space alignment. In a preferred embodiment, such as... Figure 1 As shown, the method includes: Step A1: Construct an instance set, where each instance in the instance set includes a molecular structure diagram. Training text instructions Target text response based on molecular structure diagram text commands An instance can be represented as .

[0018] In this embodiment, the molecular structure diagram adopts the molecular topology diagram representation commonly used in the field of chemistry. .in, The set of nodes in a molecular structure diagram is equivalent to the set of atoms in a molecule, with one atom corresponding to one node. An edge set represents the set of edges in a molecular structure diagram. An edge represents a chemical bond between the atoms at its two ends, and the edge set represents the set of topological connections between atoms in a molecule.

[0019] For example, based on the molecular basic information in publicly available chemical databases (such as QM9, PubChem, and USPTO), data such as atom types, bonding types, and atomic connection relationships are extracted to generate a two-dimensional molecular structure diagram of the corresponding molecule. For example, if the N-carbamoylglycine molecule in the database is selected, the nodes of its molecular structure diagram are C, N, O, and H atoms, and the edges are the single / double bond connections between each atom, fully presenting the chain-like, branched, and other topological structures of the molecule.

[0020] In this embodiment, training text instructions are designed in natural language form according to the downstream molecular task type. The instructions are natural language texts that only describe the task requirements and may not contain any molecular structure-related symbols. They serve as the text sequence input for the large language model.

[0021] For example, the training text instructions are not limited to: "Predict the main product of the chemical reaction corresponding to this molecule"; "Give the relative molecular mass value of this molecule"; "Analyze the types of functional groups contained in this molecule"; "Recommend a suitable catalyst for the reaction in which this molecule participates"; "Deduce possible reactants based on this product molecule".

[0022] In this embodiment, authoritative data / conclusions from publicly available chemical databases, professional chemical literature, and chemical engineering handbooks are extracted as the corresponding target text responses for the aforementioned training text commands. Taking the N-carbamoylglycine molecule as an example: for the training text command "Analyze the types of functional groups contained in this molecule", the target text response is "This molecule contains carbamoyl, carboxyl, and amino groups"; for the training text command "Give the relative molecular mass value of this molecule", the corresponding target text response is "The relative molecular mass of this molecule is 118.09"; for the training text command "Predict the main product of the esterification reaction of this molecule with ethanol", based on the known reaction formula of this molecule in the database, the target text response is "The main product of the esterification reaction of this molecule with ethanol is N-carbamoylglycine ethyl ester".

[0023] Step A2: Construct a molecular multimodal large language model, which includes an adaptive weight generator, a pre-trained parameter-frozen graph neural network encoder, and a large language model.

[0024] In this embodiment, a graph neural network (GNN), commonly used in cheminformatics, is selected as the graph neural network encoder, such as a graph attention network (GAT). The graph neural network encoder can be pre-trained using large-scale public molecular graph datasets (such as QM9 and ZINC-15). During training, a decoding head is used. After training is completed, all parameters of the graph neural network encoder are frozen and do not participate in any training or updates of the subsequent molecular multimodal large language model.

[0025] For example, the decoder head includes a globally average pooling layer and a single fully connected layer connected in sequence. The globally average pooling layer aggregates the node-level feature representations of all nodes in the molecular structure diagram output by the graph neural network encoder into graph-level features of the entire molecular structure diagram. The graph-level features are then mapped by the fully connected layer to predicted values ​​of attributes such as molecular molar mass and charge distribution. The predicted values ​​are compared with the corresponding true attribute values, and the error is calculated using the MSE loss function. Backpropagation also only updates the parameters of the graph neural network encoder and decoder head until the value of the MSE loss function is less than or equal to a preset function threshold.

[0026] In this embodiment, the mainstream large language model LLaMA2 mentioned in the disclosure is selected as the basic large language model. A publicly available pre-trained version with 7B parameters is selected. This model has been pre-trained on massive general and scientific text corpora and has good natural language understanding and generation capabilities. After acquisition, the backbone network of the large language model is kept in a fixed state so that it does not perform gradient updates during the fine-tuning stage. All parameters are frozen, and only the weight injection interface is reserved to receive the low-rank weight update amount generated by the subsequent adaptive weight generator.

[0027] In this embodiment, the large language model is not limited to a general-purpose large language model or a molecular domain-specific / multimodal large language model, such as LLaMA2 (a publicly pre-trained version with 7B parameters), InstructMol, PRESTO, LLaMo (a molecular multimodal adapted version, based on LLaMA extensions), T5Chem (a molecular domain adapted version based on the T5 large language model), Mol-Instruction (a molecular instruction fine-tuning large language model), HIGHT (a molecular multimodal large language model), and UniMoT (a molecular multimodal large language model). These large language models have been pre-trained on massive amounts of general and scientific text corpora and possess excellent natural language understanding and generation capabilities. After acquisition, all parameters of the backbone network of the large language model are frozen, with only a weight injection interface reserved to receive low-rank weight updates generated by the subsequent adaptive weight generator.

[0028] In this embodiment, the internal structure of the molecular multimodal large language model is as follows: Figure 2 As shown, the specific functions of each module are as follows: An adaptive weight generator obtains the low-rank weight update of at least one target weight component in one or more decoder layers of a large language model based on node-level feature representations, and injects the low-rank weight update of the target weight component into the original weight of the target weight component. This can be understood as the adaptive weight generator performing cross-attention interaction with the node-level feature representations through learnable molecular query vectors to achieve feature distillation, and obtaining the low-rank weight update of at least one target weight component in one or more decoder layers of a large language model based on the distilled features.

[0029] After injecting low-rank weight updates, the large language model performs multimodal reasoning based on the input training text instructions, generating actual text responses including molecular structure sequences, text descriptions, or attribute prediction values.

[0030] Specifically, the graph neural network encoder iteratively updates the node state through a message passing mechanism, after... After layer propagation, the final node embedding set and node-level feature representation are obtained. , shape , Represents a node In the The feature vectors of the layer, which encapsulate the topological structure information of the molecule, are one-dimensional vectors with a feature dimension of . . This represents the number of nodes in a molecular structure diagram, which is the total number of atoms in the molecule. It is a two-dimensional matrix, with the number of rows equal to the number of nodes in the molecular structure diagram. The number of columns is the feature dimension. .

[0031] The adaptive weight generator obtains the low-rank weight update of at least one target weight component in one or more decoder layers of a large language model based on node-level feature representation, and injects the low-rank weight update of the target weight component into the original weight of the target weight component.

[0032] Generally, large language models contain multiple consecutively stacked decoder layers. Each decoder layer includes a multi-head self-attention module and a feedforward neural network module. One or more decoder layers are selected from these consecutively stacked layers, and a target weight component is determined from the weight components of the selected decoder layer. Let the determined weight component be... The target weight components can be the same or different depending on the decoder layer. It is a positive integer.

[0033] Preferably, the target weight component includes the query projection matrix of the multi-head self-attention module in the decoder layer. Key projection matrix Value projection matrix and output projection matrix and the feedforward neural network in the decoder layer projection matrix At least one of the five matrices.

[0034] It should be noted that the multi-head self-attention module is the core of large language models for understanding textual context and establishing semantic relationships, and it queries the projection matrix. Key projection matrix Value projection matrix The attention weights are determined by Wo, which in turn determines the final output of the attention features. The feedforward neural network is responsible for performing non-linear transformations on the attention output features, which is crucial for enhancing text semantic expression. Therefore, the five types of matrices mentioned above are the most critical weight components in the decoder layer that influence the model output, and their number is fixed (maximum 5 per layer). This eliminates the need to modulate all weights in the decoder layer (such as layer normalization auxiliary parameters and bias terms), requiring only minor adjustments to a few parameters and reducing computational costs. Selecting these five types of weight components allows molecular structure information to be directly injected into the core semantic understanding and feature transformation stages of the large language model, avoiding the perception weakening problem caused by molecular information only being transmitted in peripheral modules. Different molecular tasks have different requirements for model capabilities (e.g., "numerical prediction of molecular attributes" requires accurate feature transformation, while "chemical reaction description" requires strong contextual association). Therefore, target weight components can be selected based on the functional differences of these five types of weight components. If it is necessary to strengthen contextual association, the query projection matrix can be selected as the primary choice. Key projection matrix Value projection matrix If feature representation needs to be optimized, the output projection matrix can be selected as the primary focus. and feedforward neural networks projection matrix .

[0035] For example, the specific structure of the adaptive weight generator is as follows: it includes a global pooling layer and two fully connected layers connected sequentially. The global pooling layer aggregates the node-level feature representations into a global feature vector of the molecular graph. The two fully connected layers perform feature transformation and dimension matching on the global feature vector of the molecular graph sequentially, generating the low-rank weight update amounts for each target weight component. Only the parameters in the two fully connected layers are learnable parameters of the adaptive weight generator.

[0036] In this embodiment, the adaptive weight generator injects the low-rank weight update amount of the target weight component into the original weight of the target weight component in a manner not limited to: The element-wise multiplication method performs element-wise multiplication between the low-rank weight update of the target weight component and the original weight of the target weight component, and achieves parameter space update through proportional modulation. Alternatively, a row-by-row weighted fusion method can be adopted, using the low-rank weight update amount of the target weight component as the row-by-row weight coefficient of the original weight, and making weighted adjustments to each row feature of the original weight. Alternatively, a low-rank matrix concatenation method can be used, where the low-rank weight update is used as a low-rank supplementary matrix and concatenated with the original weights along the column dimension, and then the concatenated matrix is ​​restored to the original feature dimension through linear projection.

[0037] In this embodiment, preferably, the original weights of the target weight component and the update amounts of the low-rank weights of the target weight component are added element-wise, and the calculation formula is as follows: in, This represents the weight matrix of the target weight component after injection. row and number Column elements. The original weight matrix representing the target weight component. The row and number Column elements. This represents the low-rank weight update amount of the target weight component. The row and number Column elements. Element-level additive injection reduces computational and memory overhead, meeting the inference speed requirements of large molecular tasks and ensuring stable convergence during training. Original weight matrix. It is in a frozen state and does not participate in parameter updates.

[0038] Step A3: Train the molecular multimodal large language model using the instance set, and update the parameters of the adaptive weight generator during training.

[0039] In this embodiment, the molecular multimodal large language model is iteratively trained using an instance set based on a preset loss function until the preset maximum number of training iterations is reached, or the value of the loss function is less than a preset loss threshold. During iterative training, the parameters of the graph neural network encoder and the large language model are frozen, and only the parameters of the adaptive weight generator are updated through backpropagation.

[0040] In this iterative training, the molecular structure diagram of the instance and the training text instructions are input into the molecular multimodal large language model to obtain the actual text response. The loss function is the cross-entropy loss function calculated based on the actual text response and the target text response, or, the loss function is to maximize the log-likelihood probability of the generated actual text response (sequence), in which case the loss function is defined as: in, Includes only the trainable parameters of the adaptive weight generator ( ), minimizing this loss function, the target text response is expressed as , represents the number of words in the target text response or the length of the target text response, respectively represent the first word, the second word, …, the th word in the target text response.

[0041] represents the temporary weight of the large language model after weight update (i.e., the original weight + the generated by the adaptive weight generator), the first t - 1 tokens (a < t, such as the first to the t - 1th tokens) of the generated target text response, and the input training text instruction (such as 'Analyze the functional groups of this molecule'), under these three conditions, the probability value of the large language model generating the tth token ( ) in the target text response.

[0042] In a preferred embodiment, as Figure 3 shown, the adaptive weight generator includes: Stacked Transformer decoder layers, used to perform feature distillation on the node - level feature representation using a preset learnable molecular query vector, and output a context - aware molecular query vector ]>, represents the number of target weight components. Among them, each row component in the molecular query vector corresponds to a target weight component and serves as the molecular query sub - vector of the corresponding target weight component. Exemplarily, the molecular query sub - vector of the target weight component is expressed as .

[0043] A fully - connected layer, which projects and reshapes the molecular query sub - vector of the target weight component into a low - rank matrix. The weight matrix of the fully - connected layer is expressed as , using the fully - connected layer to project the molecular query sub - vector of the target weight component and reshape it into the low - rank matrix of the target weight component, represents the action of reshaping into a low - rank. , is the rank (Rank) of the low - rank matrix factorization, is the feature dimension of the weight matrix of the target weight component inside the large language model.

[0044] A shared projection unit, configured with a learnable shared projection matrix, expressed as The shared projection matrix is ​​a globally shared linear projection matrix used to project features from the low-rank space back to the original weight dimension of the large language model. At the start of training, the shared projection matrix is ​​initialized as a shared, learnable linear projection matrix.

[0045] The weight generation unit multiplies the low-rank matrix of the target weight component with the shared projection matrix to obtain the low-rank weight update amount of the target weight component. This is achieved by calculating the low-rank matrix of the target weight component. With shared projection matrix The product of these factors represents the parameter space update for synthesizing a specific molecule. ,Right now .

[0046] For the target weight component Its low-rank weight update amount is: ; in, This design aims to achieve efficient low-rank updates in large language models with large parameter spaces. Represents the target weight component A low-rank matrix.

[0047] In this implementation, the molecular query vectors extracted by distillation from the stacked Transformer decoding layers accurately capture molecular topological information and correspond one-to-one with each target weight component of the model, laying the foundation for the subsequent accurate generation of low-rank weight updates. The fully connected layer projects and reshapes the molecular query sub-vectors into a low-rank matrix, realizing low-rank weight updates for large language models, significantly reducing computational and memory overhead in the parameter space, and adapting to the need for efficient parameter fine-tuning. The shared projection matrix is ​​used by all target weight components, reducing the number of trainable parameters in the adaptive weight generator, making model training more efficient, and reducing the possibility of overfitting. The low-rank weight updates generated by multiplying the low-rank matrix and the shared projection matrix can accurately convert molecular structural features into model parameter modulation signals, allowing the model to implicitly perceive molecular structure and improve structure perception capabilities.

[0048] The adaptive weight generator's modules work together to achieve efficient conversion of molecular structure features into model parameter updates, making parameter space alignment more efficient and ensuring the integrity of molecular structure information transmission, thus avoiding structural information fragmentation.

[0049] In this embodiment, during training, all trainable parameters of the adaptive weight generator include the learnable query vector. Fully connected layer weights Shared projection matrix And all weight parameters in the stacked Transformer decoding layers.

[0050] In this embodiment, more preferably, such as Figure 4 As shown, each decoding layer in the stacked Transformer decoding layers includes: The self-attention module performs self-attention calculations on learnable molecular query vectors or molecular query vectors output from the previous Transformer decoding layer based on the self-attention mechanism to obtain self-attention features. The self-attention module allows molecular query vectors to achieve internal information interaction first, strengthens the contextual association of the vectors themselves, and lays a good foundation for subsequent fusion of molecular structure features.

[0051] The cross-attention module uses self-attention features as queries and node-level features as keys and values ​​to perform cross-attention calculations to obtain cross-attention features. The cross-attention module accurately integrates the node-level topological features of the molecular graph into the query vector, so that the output query vector fully carries molecular structure information.

[0052] The feedforward neural network transforms the cross-attention features to obtain the molecular query vector output by the decoding layer. The feedforward neural network further transforms the fused features to enhance their expressive power, making the query vector output by each layer more suitable for the generation of subsequent low-rank weight updates.

[0053] In this embodiment, the progressive processing method of the three-layer structure of self-attention module, cross-attention module and feedforward neural network allows the final distilled query vector to deeply capture the global molecular structure information, ensuring the accuracy of molecular structure perception during subsequent parameter space alignment, realizing the complete transmission of molecular structure information and avoiding the fragmentation of structural information.

[0054] This invention discloses a reasoning method, in a preferred embodiment, such as... Figure 5 As shown, the method includes: Step B1: Obtain the molecular structure diagram and the user's query text command for the molecular structure diagram; Step B2: Input the molecular structure diagram and query text command into the molecular multimodal large language model to obtain the actual text response of the query text command. The molecular multimodal large language model is obtained according to the above-mentioned molecular multimodal large language model construction method based on parameter space alignment. In the molecular multimodal large language model: Graph neural network encoders are used to extract node-level feature representations of molecular structure diagrams; The adaptive weight generator obtains the low-rank weight update amount of at least one target weight component in one or more decoder layers of a large language model based on node-level feature representation, and injects the low-rank weight update amount of the target weight component into the original weight of the target weight component. After injecting low-rank weight updates, the large language model generates actual text responses based on query text commands.

[0055] In this embodiment, the query text command is a natural language description text. For example, the training text command is not limited to: "Predict the main product of the chemical reaction corresponding to this molecule"; "Give the relative molecular mass value of this molecule"; "Analyze the types of functional groups contained in this molecule"; "Recommend a suitable catalyst for the reaction in which this molecule participates"; "Deduce possible reactants based on this product molecule".

[0056] In this embodiment, the detailed processing procedure for the molecular multimodal large language model can be referred to the implementation method of the above-described method for constructing a molecular multimodal large language model based on parameter space alignment.

[0057] Using the reasoning method provided in this application, let... The length of the token is the sequence of query text instructions in natural language input. This refers to the number of tokens after linearizing the molecular structure diagram in related technologies. The inference method provided in this application avoids sequence expansion, significantly reduces the computational complexity of the attention mechanism, and makes it more efficient. Reduced to .

[0058] The technical solution of this invention has been validated in various molecular-related tasks, covering application scenarios such as molecular structure understanding, cross-modal generation, and quantitative reasoning. These tasks cover different scenarios of understanding molecular structure and properties, examining the model's ability to deeply understand changes in molecular structure and chemical reaction mechanisms. The experiment was validated under a typical large language model architecture, and three representative baseline models were selected for comparison: (1) Domain expert models: such as Chemformer, Molecule Transformers, and other Transformer models designed specifically for chemical tasks; (2) Molecular-specific LLMs: such as T5Chem; (3) Input space-based molecular multimodal LLMs: including InstructMol, PRESTO, LLaMo, HIGHT, etc. These models represent the highest level of current multimodal molecular large models, mainly using the method of concatenating graph tokens to the input sequence.

[0059] Evaluation employed standard metrics for each task domain. For molecule generation tasks, ExactMatch was used to measure the consistency between predicted and actual structures, Validity was used to measure whether generated molecules conformed to chemical rules, and Tanimoto similarity was used to measure molecular structural similarity. For text generation tasks such as generating molecular properties, BLEU, ROUGE-L, and METEOR metrics were used to evaluate the quality and fluency of the generated text. For numerical regression tasks such as predicting molecular properties, Mean Absolute Error (MAE) and Coefficient of Determination (CDE) were used. To assess the accuracy of numerical predictions.

[0060] Experimental data show that the method proposed in this invention significantly outperforms existing mainstream technologies in core task metrics. In the MOL2MOL task (see Table 1), the adoption of a parameter space alignment mechanism avoids the fragmentation of structural information caused by input graph tokens, resulting in an improved accurate matching rate of 0.697 for positive reaction prediction. Simultaneously, it maintains near 100% chemovalidity. This demonstrates that directly injecting molecular structural information into the parameter space avoids the structural fragmentation problem caused by serialization, thereby improving the model's reasoning ability for complex chemical transformations. In the Mol2Text task (see Table 2), this invention achieves a BLEU-4 score of 0.486, indicating that the model can accurately convert encoded structural features into natural language. In the Mol2Num task (see Table 3), the average error of QM9 attribute prediction is only 0.0038, demonstrating the superiority of the parameter modulation method in handling quantitative structure-activity relationship (QSAR). Regarding computational efficiency, a comparative analysis of the processing overhead of different models when handling molecules of varying sizes revealed a significant positive correlation between inference time and computational cost in traditional input space alignment models and the increase in the number of molecules with atoms, with a correlation coefficient approaching 1.0. This is primarily attributed to the quadratic computational complexity of the self-attention mechanism as sequence length increases. In contrast, this invention injects structural information into the parameter space, eliminating the need for additional graph tokens at the input, thereby decoupling inference computational cost from molecular size and maintaining a constant computational overhead when processing large molecules.

[0061] Table 1 Performance Comparison of MOL2MOL Tasks

[0062] Table 2 Comparison of Molecular Description Task Performance

[0063] Table 3 Comparison of Molecular Property Prediction Performance

[0064] Therefore, the molecular multimodal large language model construction method proposed in this invention, based on parameter space alignment, effectively enhances the model's overall perception of complex molecular topologies through parameter space modulation guided by molecular graphs, alleviating the problems of high computational overhead and fragmented structural information caused by input space alignment methods. This method improves the accuracy of molecular structure reasoning and the chemical rationality of generated results while reducing the computational complexity of reasoning, providing a solution that balances performance and efficiency for the application of multimodal large models in the scientific field.

[0065] The present invention discloses a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above-described method for constructing or inferring a molecular multimodal large language model based on parameter space alignment.

[0066] The present invention also discloses a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-mentioned method or reasoning method for constructing or inferring a molecular multimodal large language model based on parameter space alignment provided by the present invention. The computer program product should be understood as a software product that mainly implements its solution through a computer program, such as a program product integrated in the cloud or a software library.

[0067] The present invention also discloses a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the above-described method for constructing or inferring a molecular multimodal large language model based on parameter space alignment.

[0068] The present invention also discloses an electronic device, in a preferred embodiment of which the electronic device includes at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, such that the at least one processor can execute the method for constructing or reasoning a molecular multimodal large language model based on parameter space alignment provided by the present invention.

[0069] like Figure 6 The diagram shown is a structural schematic of an electronic device for a molecular multimodal large language model construction method or inference method based on parameter space alignment, according to an embodiment of the present invention. The electronic device may include a processor 10, a memory 11, a communication bus 12, and a communication interface 13. It may also include a computer program stored in the memory 11 and executable on the processor 10, such as a program for a molecular multimodal large language model construction method or inference method based on parameter space alignment.

[0070] In some embodiments, the processor 10 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 10 is the control unit of the electronic device, connecting various components of the entire electronic device through various interfaces and lines. It executes programs or modules stored in the memory 11 (e.g., executing methods for constructing or inferring molecular multimodal large language models based on parameter space alignment), and calls data stored in the memory 11 to perform various functions of the electronic device and process data.

[0071] The memory 11 includes at least one type of readable storage medium, including flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 11 can be an internal storage unit of an electronic device, such as a portable hard drive. In other embodiments, the memory 11 can be an external storage device of the electronic device, such as a plug-in portable hard drive, SmartMediaCard (SMC), SecureDigital (SD) card, FlashCard, etc. Furthermore, the memory 11 can include both internal and external storage units of the electronic device. The memory 11 can be used not only to store application software and various types of data installed on the electronic device, such as code for a molecular multimodal large language model construction method or inference method program based on parameter space alignment, but also to temporarily store data that has been output or will be output.

[0072] The communication bus 12 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. The bus is configured to enable communication between the memory 11 and at least one processor 10, etc.

[0073] Communication interface 13 is used for communication between the aforementioned electronic device and other devices, including a network interface and a user interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a Wi-Fi interface, Bluetooth interface, etc.), typically used to establish communication connections between the electronic device and other electronic devices. The user interface may be a display, an input unit (such as a keyboard), or optionally, a standard wired or wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen, etc. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the electronic device and to display a visual user interface.

[0074] Figure 6 Only electronic devices with components are shown; it will be understood by those skilled in the art that... Figure 6 The structure shown does not constitute a limitation on the electronic device and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0075] For example, although not shown, the electronic device may also include a power supply (such as a battery) to power various components. Preferably, the power supply can be logically connected to at least one processor 10 via a power management device, thereby enabling functions such as charging management, discharging management, and power consumption management. The power supply may also include one or more DC or AC power supplies, recharging devices, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The electronic device may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be elaborated further here.

[0076] It should be understood that the embodiments are for illustrative purposes only and are not limited to this structure in the scope of the patent application.

[0077] Furthermore, if the modules / units integrated into the electronic device are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile. For example, a computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0078] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. A method for constructing a molecular multimodal large language model based on parameter space alignment, characterized in that, The method includes: Construct an instance set, where each instance includes a molecular structure diagram, training text instructions, and target text responses corresponding to the text instructions based on the molecular structure diagram. A molecular multimodal large language model is constructed, which includes an adaptive weight generator, a pre-trained parameter-frozen graph neural network encoder, and a large language model. A molecular multimodal large language model is trained using an instance set, and the parameters of the adaptive weight generator are updated during training. Among them, the graph neural network encoder is used to extract node-level feature representations of molecular structure diagrams; The adaptive weight generator obtains the low-rank weight update amount of at least one target weight component in one or more decoder layers of a large language model based on node-level feature representation, and injects the low-rank weight update amount of the target weight component into the original weight of the target weight component. After injecting low-rank weight updates, the large language model generates actual text responses based on the input training text instructions.

2. The method for constructing a molecular multimodal large language model based on parameter space alignment according to claim 1, characterized in that, The target weight component includes at least one of the following five matrices: the query projection matrix, key projection matrix, value projection matrix, and output projection matrix of the multi-head self-attention module in the decoder layer, and the projection matrix of the feedforward neural network in the decoder layer.

3. The method for constructing a molecular multimodal large language model based on parameter space alignment according to claim 1 or 2, characterized in that, The adaptive weight generator includes: The stacked Transformer decoding layer is used to perform feature distillation on the node-level feature representation using a pre-defined learnable molecular query vector, and output a context-aware molecular query vector. Each row component in the molecular query vector corresponds to a target weight component and serves as the molecular query sub-vector of the corresponding target weight component. The fully connected layer projects and reshapes the molecular query subvectors of the target weight component into a low-rank matrix; A shared projection unit is configured with a learnable shared projection matrix, wherein the shared projection matrix is ​​a linear projection matrix that is globally shared by all target weight components; The weight generation unit multiplies the low-rank matrix of the target weight component with the shared projection matrix to obtain the low-rank weight update amount of the target weight component.

4. The method for constructing a molecular multimodal large language model based on parameter space alignment according to claim 3, characterized in that, Each decoding layer in a stacked Transformer decoding layer includes: The self-attention module performs self-attention calculations on the learnable molecular query vector or the molecular query vector output by the previous Transformer decoding layer based on the self-attention mechanism to obtain self-attention features; The cross-attention module uses self-attention features as queries and node-level features as keys and values ​​to perform cross-attention calculations to obtain cross-attention features. The feedforward neural network transforms the cross-attention features to obtain the molecular query vector output by the decoding layer.

5. The method for constructing a molecular multimodal large language model based on parameter space alignment according to claim 1, 2, or 4, characterized in that, Inject the low-rank weight update amount of the target weight component into the original weight of the target weight component, including: Perform element-wise addition between the original weights of the target weight component and the low-rank weight update amount of the target weight component.

6. A reasoning method, characterized in that, The method includes: Retrieve molecular structure diagrams and user-defined query text commands for molecular structure diagrams; Input a molecular structure diagram and a query text command into a molecular multimodal large language model to obtain the actual text response of the query text command, wherein the molecular multimodal large language model is obtained according to any one of claims 1-5; In the molecular multimodal large language model: Graph neural network encoders are used to extract node-level feature representations of molecular structure diagrams; The adaptive weight generator obtains the low-rank weight update amount of at least one target weight component in one or more decoder layers of a large language model based on node-level feature representation, and injects the low-rank weight update amount of the target weight component into the original weight of the target weight component. After injecting low-rank weight updates, the large language model generates actual text responses based on query text commands.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1-6.

8. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the method described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the method described in any one of claims 1-6.

10. An electronic device, characterized in that, The electronic device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the method as described in any one of claims 1-6.

Images