Training method and device of language understanding model for molecular processing, electronic equipment, storage medium and computer program product

By constructing a molecular vocabulary list and molecular training set, and updating the language understanding model for word segmentation and training, the problem of low understanding of molecular generation processing in the existing technology is solved, and the accuracy of molecular structural formula generation is improved.

CN119007870APending Publication Date: 2024-11-22TENCENT TECHNOLOGY (SHENZHEN) CO LTD
View PDF 0 Cites 3 Cited by

Patent Information

Application Number
CN202410306003.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-03-15
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

In the prior art, there is a low degree of understanding of molecular generation processing based on molecular description text, which affects the accuracy of molecular formulas and lacks an effective method to improve the accuracy of molecular generation processing.

Method used

By constructing a molecular vocabulary list, updating the initial language understanding model, making it have word segmentation ability, and training the model based on the molecular training set to generate molecular structural formulas corresponding to the molecular description text, improving the model's understanding and generation accuracy of the molecular structural formulas.

Benefits of technology

The ability and generation accuracy of the pre-trained language understanding model to understand molecular structural formulas is improved, and the accuracy of determining molecular structural formulas based on text is enhanced.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119007870A_ABST
    Figure CN119007870A_ABST
Patent Text Reader

Abstract

The invention provides a language understanding model training method and device for molecular processing, electronic equipment, a storage medium and a computer program product. The method comprises the steps of obtaining a molecular word list, wherein the molecular word list comprises a plurality of molecular lexical elements; updating the initial language understanding model based on the molecular word list to obtain a pre-training language understanding model, the pre-training language understanding model being used for splitting the molecular structural formula to obtain molecular lexical elements of the molecular structural formula; a molecular training set is obtained, the molecular training set comprises a sample molecular description text and a sample molecular structural formula, and the sample molecular description text is used for describing molecular components of the sample compound; and training the pre-training language understanding model based on the molecule training set to obtain a target language understanding model, the target language understanding model being used for generating a molecular structural formula of the compound molecule corresponding to the target molecule description text. According to the invention, the accuracy of molecule generation processing can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to artificial intelligence technology, and in particular to a training method, device, electronic device, storage medium and computer program product for a language understanding model for molecular processing. Background Art

[0002] Natural language processing (NLP) is an important direction in the fields of computer science and artificial intelligence. It studies various theories and methods that can enable effective communication between people and computers using natural language. Natural language processing involves natural language, which is the language people use in daily life and is closely related to linguistic research; it also involves important technologies for model training in the fields of computer science, mathematics, and artificial intelligence. The pre-trained model is developed from the Large Language Model (LLM) in the field of natural language processing. After fine-tuning, the large language model can be widely used in downstream tasks. Natural language processing technology usually includes text processing, semantic understanding, machine translation, robot question answering, knowledge graphs and other technologies.

[0003] In the related art, the molecular formula is obtained by calling the sequence-to-sequence model based on the molecular description text, but the molecular formula output by the above method cannot be modified twice; the molecular generation processing based on the large language model has a multi-round dialogue function, but the understanding of the molecular language is not high, which affects the accuracy of the generated molecular formula. In the related art, there is no better way to improve the accuracy of the molecular generation processing. Summary of the invention

[0004] The embodiments of the present application provide a training method, device, electronic device, storage medium and computer program product for a language understanding model for molecular processing, which can improve the accuracy of molecular generation processing.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] The present application embodiment provides a method for training a language understanding model for molecular processing, the method comprising:

[0007] Acquire a molecular vocabulary, wherein the molecular vocabulary includes: a plurality of molecular words, and one molecular word is used to represent a type of molecular substructure;

[0008] Based on the molecular vocabulary, an initial language understanding model is updated to obtain a pre-trained language understanding model, wherein the pre-trained language understanding model is used to split the molecular structure formula to obtain the molecular word units of the molecular structure formula;

[0009] Acquire a molecular training set, wherein the molecular training set includes: sample molecular description text and sample molecular structural formula, wherein the sample molecular description text is used to describe the molecular components of the sample compound;

[0010] The pre-trained language understanding model is trained based on the molecular training set to obtain a target language understanding model, wherein the target language understanding model is used to generate a molecular structural formula of a compound molecule corresponding to a target molecule description text.

[0011] The present application embodiment provides a training device for a language understanding model for molecular processing, comprising:

[0012] A data acquisition module is used to acquire a molecular vocabulary, wherein the molecular vocabulary includes: a plurality of molecular words, and one molecular word is used to represent a type of molecular substructure;

[0013] A model training module, used for updating an initial language understanding model based on the molecular vocabulary to obtain a pre-trained language understanding model, wherein the pre-trained language understanding model is used for splitting the molecular structure formula to obtain the molecular word units of the molecular structure formula;

[0014] The data acquisition module is used to acquire a molecular training set, wherein the molecular training set includes: sample molecular description text and sample molecular structural formula, wherein the sample molecular description text is used to describe the molecular components of the sample compound;

[0015] The model training module is used to train the pre-trained language understanding model based on the molecular training set to obtain a target language understanding model, wherein the target language understanding model is used to generate a molecular structural formula of a compound molecule corresponding to a target molecule description text.

[0016] An embodiment of the present application provides an electronic device, the electronic device comprising:

[0017] A memory for storing computer executable instructions or computer programs;

[0018] The processor is used to implement the training method of the language understanding model for molecular processing provided in the embodiment of the present application when executing the computer executable instructions or computer program stored in the memory.

[0019] An embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions or a computer program, which is used to implement the training method of the language understanding model for molecular processing provided in the embodiment of the present application when executed by a processor.

[0020] An embodiment of the present application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, the training method of the language understanding model for molecular processing provided in the embodiment of the present application is implemented.

[0021] The embodiments of the present application have the following beneficial effects:

[0022] Each molecular word in the molecular vocabulary is used to characterize a molecular substructure. The parameters of the initial language understanding model are updated based on the molecular vocabulary, so that the pre-trained language understanding model can perform word segmentation on molecular structural formulas according to the molecular substructures, thereby improving the pre-trained language understanding model's ability to understand molecular structural formulas. The pre-trained language understanding model is trained based on the molecular training set, so that the pre-trained language understanding model can be used to determine molecular structural formulas based on text, thereby improving the accuracy of determining molecular structural formulas based on descriptive texts. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a schematic diagram of an application mode of a training method for a language understanding model for molecular processing provided in an embodiment of the present application;

[0024] Figure 2 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0025] Figure 3A It is a first flow chart of a method for training a language understanding model for molecular processing provided in an embodiment of the present application;

[0026] Figure 3B It is a second flow chart of the training method of the language understanding model for molecular processing provided in an embodiment of the present application;

[0027] Figure 3C 3 is a schematic diagram of a third flow chart of a method for training a language understanding model for molecular processing provided in an embodiment of the present application;

[0028] Figure 3D It is a fourth flow chart of the training method of the language understanding model for molecular processing provided in an embodiment of the present application;

[0029] Figure 3E It is a fifth flow chart of the training method of the language understanding model for molecular processing provided in an embodiment of the present application;

[0030] Figure 3F 6 is a schematic diagram of a training method for a language understanding model for molecular processing provided in an embodiment of the present application;

[0031] Figure 47 is a schematic diagram of a seventh flow chart of a method for training a language understanding model for molecular processing provided in an embodiment of the present application;

[0032] Figure 5A It is a schematic diagram of the word segmentation principle provided in the embodiment of the present application;

[0033] Figure 5B It is a schematic diagram of an application scenario provided by an embodiment of the present application;

[0034] Fig. 6A It is a parameter table provided in the embodiment of the present application;

[0035] Figure 6B It is the comparison result between the model trained by the embodiment of the present application and the model of the related technology provided by the embodiment of the present application.

[0036] It should be pointed out that the above-mentioned "first" and "second" are only used to distinguish different solutions, and do not represent the degree of superiority or inferiority of the solutions or the priority in the implementation process. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0038] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0039] If similar descriptions of "first / second" appear in the application documents, the following instructions are added. In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0040] It should be pointed out that the relevant data collection and processing in this application should be strictly in accordance with the requirements of relevant national laws and regulations when applied in examples, and the informed consent or separate consent of the personal information subject should be obtained, and subsequent data use and processing should be carried out within the scope of authorization of laws and regulations and the personal information subject.

[0041] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0042] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0043] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0044] 1) Vocabulary: The vocabulary used in NLP usually refers to a collection of all possible words and their corresponding information. The corresponding information of a word can include the word's definition, part of speech, meaning, pronunciation, and possible variant forms. The vocabulary is created to support various natural language processing tasks, such as word segmentation, part of speech tagging, named entity recognition, and sentiment analysis. For example, the Jieba part-of-speech comparison table provides the corresponding part of speech of Chinese characters in Pinyin and the corresponding part-of-speech tagging rules.

[0045] 2) Chemical formula: A formula that uses element symbols and numbers to represent the composition of a substance is called a chemical formula. One substance can be represented by only one chemical formula.

[0046] 3) Structural formula is a chemical formula that uses element symbols and short lines to represent the arrangement and bonding of atoms in a compound (or single substance) molecule. It is a simple way to describe the structure of a molecule. The structural formula can completely draw the chemical bonds between each atom in the molecule.

[0047] 4) Molecular formula. The chemical formula of a molecular crystal is called a molecular formula. Molecular formula is a chemical formula that uses element symbols to represent the molecular composition and relative molecular mass of a pure substance (single substance, compound). The chemical formula is a formula that uses element symbols to represent the composition of a substance. Molecular formula is different from the simplest formula. For compounds, their molecular formula is an integer multiple of the simplest formula, or the relative molecular mass is an integer multiple of the simplest formula. The simplest formula is the same as the molecular formula only when the relative molecular mass and the simplest formula are the same. In this case, the simplest formula is the molecular formula.

[0048] 5) The radical of a molecule. The uncharged part remaining after removing some atoms or atomic groups from a compound molecule is called a radical. A radical is a type of atomic group.

[0049] 6) Molecular substructure, the part of a compound molecule that is used to form a complete molecular structure. In the embodiment of the present application, the molecular substructure can be the structure of a base in a molecule.

[0050] 7) Simplified Molecular Input Line Entry System (SMILES) is a specification that uses ASCII strings to clearly describe molecular structures, using a string of characters to describe a three-dimensional chemical structure.

[0051] 8) Byte Pair Encoder (BPE), also known as digram coding, is mainly used for data compression. The algorithm is described as an iterative process in which the most common pair of characters in a string is replaced by a character that does not appear in the string.

[0052] The embodiments of the present application provide a training method for a language understanding model for molecular processing, a training device for a language understanding model for molecular processing, an electronic device, a computer-readable storage medium, and a computer program product, which can improve the accuracy of molecular generation processing.

[0053] The following describes an exemplary application of the electronic device provided in the embodiments of the present application. The electronic device provided in the embodiments of the present application can be implemented as a terminal device, such as a laptop computer, a tablet computer, a desktop computer, a set-top box, a smart TV, a vehicle terminal, a virtual reality (VR) device, an augmented reality (AR) device, and other types of terminals, and can also be implemented as a server. The following describes an exemplary application when the electronic device is implemented as a terminal device or a server.

[0054] refer to Figure 1 , Figure 1 Schematic diagram of the application mode of the training method of the language understanding model for molecular processing provided in the embodiment of the present application; for example, Figure 1 The server 200, network 300, terminal device 400 and database 500 are involved. The terminal device 400 is connected to the server 200 via the network 300, and the network 300 can be a wide area network or a local area network, or a combination of the two.

[0055] In some embodiments, the user may be a technician, the terminal device 400 may be a mobile phone or computer used by the technician, the server 200 may be a server of a drug development platform, and the database 500 stores a large amount of natural language corpus and molecule-related data (for example: molecular structure formula, molecular description text).

[0056] For example, in response to receiving a training request sent by the terminal device 400, the server 200 obtains a molecular training set and a molecular vocabulary from the database 500, calls the training method of the language understanding model for molecular processing provided in the embodiment of the present application, and trains to obtain a target language understanding model. The user can enter a molecular description text in the terminal device 400, and the molecular description text is used to describe the components of the molecule. In response to receiving the molecular description text sent by the terminal device 400, the server 200 calls the target language understanding model based on the molecular description text for predictive processing, obtains the molecular structure corresponding to the molecular description text, and sends the molecular structure to the terminal device 400 for the user to conduct molecular research.

[0057] For example, Figure 1 In the example, the trained target language understanding model is deployed in the server 200, and the user sends the corresponding target molecule description text to the server 200 through the terminal device 400 to generate the corresponding molecular structure formula. The trained target language understanding model can be deployed in the terminal device 400. When the terminal device 400 is offline, the terminal device 400 determines the molecular structure formula of the molecule corresponding to the target molecule description text based on the locally stored data and the target language understanding model.

[0058] In some embodiments, the training method of the language understanding model for molecular processing of the embodiment of the present application can also be applied in the following application scenarios: drug research. During the drug research process, the R&D personnel input the text describing the drug molecules into the target language understanding model trained by the training method of the language understanding model for molecular processing provided by the embodiment of the present application to obtain the drug molecular formula. The R&D personnel can design and study the drug molecules based on the drug molecular formula, thereby improving the R&D efficiency and reducing the drug R&D cost.

[0059] In some embodiments, the server 200 may be composed of multiple servers, for example: a training server and a molecular processing server, wherein the training server is used for execution, and the molecular processing server is used for storing the trained target language understanding model, and based on the target molecule description text, calling the target language understanding model to determine the molecular structure formula corresponding to the molecule description text.

[0060] In some embodiments, the server 200 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. The terminal and the server may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.

[0061] The embodiment of the present application can be implemented through blockchain technology. The target language understanding model of the embodiment of the present application can be uploaded to the blockchain for storage, and the reliability of the target language understanding model can be guaranteed by the consensus algorithm. Blockchain is a new application mode of computer technologies such as distributed data storage, point-to-point transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods, each of which contains information about a batch of data, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, the platform product service layer, and the application service layer.

[0062] The embodiments of the present application can be implemented through database technology. In short, a database can be regarded as an electronic file cabinet where electronic files are stored. Users can add, query, update, delete, etc. data in the files. The so-called "database" is a collection of data that is stored together in a certain way, can be shared with multiple users, has as little redundancy as possible, and is independent of the application program.

[0063] A database management system (DBMS) is a computer software system designed for managing databases. It generally has basic functions such as storage, retrieval, security, and backup. Database management systems can be classified according to the database model they support, such as relational, XML (Extensible Markup Language); or according to the type of computer they support, such as server clusters, mobile phones; or according to the query language used, such as Structured Query Language (SQL), XQuery; or according to performance focus, such as maximum scale, maximum operating speed; or other classification methods. Regardless of the classification method used, some DBMS can cross categories, for example, supporting multiple query languages ​​at the same time.

[0064] The embodiments of the present application can also be implemented through cloud technology. Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on the cloud computing business model application. It can form a resource pool, which is used on demand and is flexible and convenient. Cloud computing technology will become an important support. The background services of the technical network system require a large amount of computing and storage resources, such as video websites, picture websites and more portal websites. With the high development and application of the Internet industry, as well as the promotion of search services, social networks, mobile commerce and open collaboration, each item may have its own hash code identification mark in the future, and all need to be transmitted to the background system for logical processing. Data of different levels will be processed separately, and all kinds of industry data require strong system backing support, which can only be achieved through cloud computing.

[0065] The embodiments of the present application can also be implemented through a large language model. A large language model (LLM) is a machine learning model designed to understand and generate human language. They are trained on large amounts of text data and can perform a wide range of tasks, including text summarization, translation, sentiment analysis, etc. Large language models are characterized by their large scale and billions of parameters that help them learn complex patterns in language data, usually based on a deep learning architecture. A large language model refers to a deep learning model trained using a large amount of text data, containing billions or even larger parameters, which can be used to generate natural language text and understand the meaning of natural language text. Through training, the model can learn the statistical laws and semantic associations of language to build a huge language knowledge base, thereby simulating human language understanding and generation capabilities. A large language model has the following characteristics:

[0066] Learning ability: Through training on massive amounts of text data, large language models can learn a wealth of language knowledge and expressions, including grammar, semantics, and common expression habits.

[0067] Pattern recognition: Large language models can recognize common text patterns and semantic associations, such as the co-occurrence relationship between words, the logical structure and semantic roles of sentences, etc.

[0068] Contextual understanding: Large language models can capture contextual information in text, understand the impact of previous text on subsequent text, and generate appropriate responses based on the context.

[0069] Generation capability: The large language model can generate relevant natural language text based on the input information, including answering questions, generating articles, and conducting conversations.

[0070] Resolving ambiguity: Despite the polysemy and ambiguity of language, large language models resolve ambiguity through contextual information and language regularities, providing more accurate and appropriate text generation or understanding.

[0071] The application scenarios of large language models are very broad. They can be applied to intelligent customer service, intelligent question and answer, natural language generation, advertising recommendation, games and other fields. They can improve the efficiency and accuracy of human-computer interaction and enhance user experience.

[0072] See also Figure 2 , Figure 2 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may be Figure 1 Server 200, Figure 2 The server 200 shown includes: at least one processor 410, a memory 450, and at least one network interface 420. The various components in the server 200 are coupled together via a bus system 440. It is understood that the bus system 440 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, the bus system 440 is not described in detail. Figure 2 Various buses are labeled as bus system 440 .

[0073] Processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0074] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices that are physically remote from the processor 410.

[0075] The memory 450 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 450 described in the embodiments of the present application is intended to include any suitable type of memory.

[0076] In some embodiments, memory 450 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0077] Operating system 451, including system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0078] A network communication module 452, used to reach other electronic devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 include: Bluetooth, Wireless Compatibility Certification (WiFi), and Universal Serial Bus (USB), etc.;

[0079] In some embodiments, the device provided in the embodiments of the present application can be implemented in software. Figure 2 A training device 455 of a language understanding model for molecular processing stored in a memory 450 is shown, which can be software in the form of a program and a plug-in, including the following software modules: a data acquisition module 4551 and a model training module 4552. These modules are logical, so they can be arbitrarily combined or further split according to the functions implemented. The functions of each module will be explained below.

[0080] In some embodiments, a terminal device or a server can implement the training method of the language understanding model for molecular processing provided by the embodiment of the present application by running a computer program. For example, a computer executable instruction can be a command, a machine instruction or a software instruction at a microprogram level. A computer program can be a native program or a software module in an operating system; it can be a local (Native) application (APPlication, APP), that is, a program that needs to be installed in an operating system to run, such as a chat APP or an instant messaging APP; it can also be a small program that can be embedded in any APP, that is, a program that can be run only by downloading it to a browser environment. In short, the above-mentioned computer executable instructions can be instructions in any form, and the above-mentioned computer program can be an application, a module or a plug-in in any form.

[0081] The training method of the language understanding model for molecular processing provided by the embodiment of the present application will be explained in combination with the exemplary application and implementation of the server provided by the embodiment of the present application.

[0082] Next, the training method of the language understanding model for molecular processing provided by the embodiment of the present application is described. As mentioned above, the electronic device for implementing the training method of the language understanding model for molecular processing of the embodiment of the present application can be the server described above. Therefore, the execution subject of each step will not be repeatedly described below.

[0083] It should be noted that the examples of molecular processing below are illustrated using small molecule drugs as an example. Those skilled in the art, based on their understanding of the following, can apply the training method of the language understanding model for molecular processing provided in the embodiments of the present application to the processing of other types of molecules, such as proteins.

[0084] See also Figure 3A , Figure 3A is a flow chart of a training method for a language understanding model for molecular processing provided in an embodiment of the present application, which will be combined with Figure 3A The steps shown are explained below. Figure 3A The main body of the steps is described using the server as an example.

[0085] In step 301, a molecular vocabulary is obtained.

[0086] Here, the molecular vocabulary includes: a plurality of molecular words, and a molecular word is used to represent a type of molecular substructure.

[0087] For example, the molecular vocabulary is used to configure the initial language model. The initial language understanding model in the embodiment of the present application can be a large language model, such as the LLaMA-7B-Chat model. The large language model includes a decoder (Transformer Decoder) structure of the transformer model, pre-training normalization (Pre-normalization), a gated activation function (SwiGLU) layer, and a RoPE rotary position encoder. Among them, the SwigLU activation function is a combination of a Swish function and a gated activation function, and the Swish function is a linear function used for gating.

[0088] In some embodiments, taking the execution subject as a server as an example, the initial language understanding model can be stored in the server as the execution subject, for example, loaded from the storage space of the server into the memory. The initial language understanding model can also be stored in a database or an external server, and the server as the execution subject obtains the initial language understanding model through the application programming interface (API) of the external server.

[0089] Molecular substructure refers to the part used to make up the molecular structure. Each molecular substructure has a corresponding chemical meaning. Molecular substructure can be the structure of the molecular group. For example: the structure of benzene ring, carboxyl group, nitrate group, etc. The molecular group is a part of the atom or atomic group with special properties in the compound molecule. The functional group of an organic compound is the group that determines the main properties of the substance, such as the hydroxyl group (-OH) of alcohol and the carboxyl group (-COOH) of carboxylic acid.

[0090] In some embodiments, reference Figure 3B , Figure 3B This is a second flow chart of the training method of the language understanding model for molecular processing provided in an embodiment of the present application. Figure 3A Step 301 can be Figure 3B Steps 3011 to 3016 are implemented as described in detail below.

[0091] In step 3011, a plurality of candidate molecular structural formulas are obtained.

[0092] For example, candidate molecular structures can be obtained from public molecular datasets.

[0093] In step 3012, the character string corresponding to each candidate molecular structure formula is obtained.

[0094] For example, the string is used to describe the molecular structure formula, and the string can be characterized according to the Simplified Molecular Linear Input Specification (SMILES). For example, the string of the Simplified Molecular Linear Input Specification of the benzoic acid molecule is: Cl=CC=C(C=C1)C(=O)O.

[0095] In step 3013, word units are extracted from the character strings corresponding to each candidate molecular structure formula to obtain a plurality of candidate molecular word units.

[0096] For example, the word unit extraction process can be implemented by training the byte pair encoding algorithm (BPE) for word segmentation, calling the byte pair encoding algorithm to perform word segmentation according to the chemical meaning of the molecular structure to obtain candidate molecular word units of the molecular substructure. Each candidate molecular word unit represents a substructure in the molecule.

[0097] In some implementations, reference Figure 3C , Figure 3C is a third flow chart of the method for training a language understanding model for molecular processing provided by an embodiment of the present application. Step 3013 can be performed by Figure 3C Steps 30131 to 30137 are implemented as described in detail below.

[0098] In step 30131, character splitting processing is performed on each character string to obtain first character sequences corresponding to multiple candidate molecular structure formulas.

[0099] For example, the first character sequence includes multiple characters. The "first" in the first character sequence is only used to distinguish the character sequences that appear in the processing of different stages, and does not specifically refer to a certain type. Taking the string "Cl=CC=C(C=C1)C(=O)O" as an example, the following first character sequence is obtained by splitting:

[0100] [C,1,=,C,C,=,C,(,C,=,C,1,),C,(,=,O,),O]

[0101] In step 30132, the frequency of occurrence of each character in the first character sequence is determined.

[0102] For example, the statistical range of the occurrence frequency can be to divide multiple first character sequences into a batch for statistics, for example: 10 character sequences are counted as the same batch. The occurrence frequency is the number of times a character appears within the statistical range. For example: [C, 1, =, C, C, =, C, (, C, =, C, 1,), C, (, =, O,), O], the occurrence frequency of the carbon atom C is 7 times.

[0103] In step 30133, the frequency of occurrence of consecutive byte pairs is determined based on the frequency of occurrence of each character.

[0104] For example, consecutive byte pairs include multiple characters that appear adjacently in order.

[0105] For example: In [C,1,=,C,C,=,C,(,C,=,C,1,),C,(,=,O,),O], the frequency of occurrence of carbon atom C is 7 times, the frequency of occurrence of character 1 is 2, and the frequency of occurrence of the continuous byte pair "C1" is 2. The frequency of occurrence of a continuous byte pair is the frequency of occurrence of the character with the lowest frequency in the continuous byte pair.

[0106] In step 30134, the characters in the consecutive byte pairs with the maximum occurrence frequency are grouped into subwords.

[0107] For example, as mentioned above, the statistical range of the maximum occurrence frequency is for multiple first character sequences, and the characters in the continuous byte pairs with the maximum occurrence frequency in the multiple first character sequences are combined into subwords.

[0108] In step 30135, the frequency of occurrence of each subword in the first character sequence is determined.

[0109] For example, if a subword consists of characters of consecutive byte pairs, the occurrence frequency of the subword is the occurrence frequency of the consecutive byte pairs.

[0110] In step 30136, the following process is iteratively performed: two adjacent subwords with the maximum occurrence frequency are combined into a new subword, and the occurrence frequency of each new subword in the first character sequence is determined.

[0111] For example, the granularity of the subword includes characters and byte pairs consisting of multiple characters, then step 30134 can be used as the initial round, and step 30136 iterates the 2nd to Nth rounds, where N is a positive integer.

[0112] In step 30137, in response to the maximum occurrence frequency being equal to a preconfigured value and the new sub-word satisfying the condition of representing a molecular structure with chemical significance, the new sub-word is used as a candidate molecular word-element.

[0113] For example, the preconfigured value is 1. The types of molecular structures with chemical meanings include: the structure of the molecular base or the molecular structure. For example, if the new sub-word corresponds to the molecular structure of oxygen, the new sub-word is used as a candidate molecular word. For another example, if the new sub-word is represented by the carboxyl group of a molecule, the new sub-word is used as a candidate molecular word.

[0114] For example, the character string of each molecular structure is split to obtain a character sequence. The occurrence frequency of each character in the character sequence is counted, and the occurrence frequency of consecutive byte pairs is determined. The consecutive byte pairs with the highest frequency are selected to form a new subword, and step 30136 is repeated until the maximum occurrence frequency is equal to a preconfigured value (for example: 1), and the new subword meets the condition of representing a molecular structure with chemical significance.

[0115] In an embodiment of the present application, the word elements in the molecular vocabulary are screened by combining the chemical meaning of the molecular structure with the byte pair encoding method, so that the word elements in the molecular vocabulary can conform to the chemical meaning of the molecular structure, and the language understanding model can have the function of splitting the molecular structure into molecular substructures, thereby improving the accuracy of predicting the molecular structure.

[0116] In step 3014, multiple candidate molecular word units are deduplicated.

[0117] For example, in an embodiment of the present application, the first character sequence of the candidate molecular structure formula is processed in batches to extract candidate molecular words. There may be duplications between the candidate molecular words obtained by statistics between each batch. The duplicate candidate molecular words are removed, and the retained candidate molecular words are different from each other.

[0118] In step 3015, in response to the number of candidate molecular word-grams remaining after the deduplication process is greater than a preconfigured number, the candidate molecular word-grams remaining after the deduplication process are combined into a candidate word list.

[0119] For example, the preconfigured number may be 1000. The preconfigured number may be determined before obtaining the candidate molecular structural formulas. For example, if the vocabulary size of the expected subwords is determined (eg, 1000), then candidate molecular structural formulas much larger than 1000 are obtained for word unit extraction.

[0120] In step 3016, the candidate vocabulary is merged with the natural language vocabulary preset for the pre-trained language understanding model to obtain a molecular vocabulary.

[0121] For example, the natural language vocabulary includes multiple natural language tokens. Based on the natural language vocabulary, the pre-trained language understanding model can perform word segmentation processing on text using natural language grammar. If the vocabulary size of the pre-trained language understanding model is initially configured to be 32,000, the candidate vocabulary (size is 1,000) is merged with the initially configured vocabulary, and the size of the new molecular vocabulary is 33,000.

[0122] Continue to refer Figure 3A , in step 302, the initial language understanding model is updated based on the molecular vocabulary to obtain a pre-trained language understanding model.

[0123] For example, the pre-trained language understanding model is used to split the molecular structure formula to obtain the molecular words of the molecular structure formula; the initial language understanding model has the function of segmenting natural language, and there is a configured vocabulary in the configuration item of the initial language understanding model. Step 302 can be implemented in the following way: replace the configured vocabulary in the initial language understanding model based on the molecular vocabulary to obtain the pre-trained language understanding model.

[0124] Continue to refer Figure 3A , in step 303, a molecular training set is obtained.

[0125] Here, the molecular training set includes: sample molecular description text and sample molecular structural formula, and the sample molecular description text is used to describe the molecular components of the sample compound.

[0126] For example, the molecular components include, but are not limited to, molecular substructures, elements constituting the molecule, and bases constituting the molecule. Sample molecular description text, for example: the molecule is a compound containing a benzene nucleus, wherein the benzene nucleus carries a carboxyl group. In some embodiments, the molecule may be a drug molecule, and the molecular description text may also include the source and method of molecule extraction.

[0127] In some embodiments, the molecular training set includes multiple pieces of sample data; each piece of sample data includes: a piece of sample molecule description text and a sample molecule structure formula corresponding to the sample molecule description text.

[0128] Step 303 can be implemented in the following manner: obtaining a sample molecule description text; performing the following processing for each sample molecule description text: determining a character string of a sample molecule structure formula corresponding to the sample molecule description text based on the molecular components described by the sample molecule description text; obtaining an instruction template to be configured, wherein the parameters to be configured in the instruction template to be configured include: instruction, input, and output; configuring the instruction of the instruction template to be configured to indicate the following: based on the sample molecule description text, calling a pre-trained language understanding model to generate a character string of the sample molecule structure formula; configuring the sample molecule description text as the input in the instruction template to be configured, and configuring the character string of the sample molecule structure formula as the output in the instruction template to be configured; using the configured instruction template to be configured as sample data, wherein the instruction is used to trigger the execution of a processing of calling a pre-trained language understanding model to generate a character string of the sample molecule structure formula based on the sample molecule description text when the sample data is read; combining the sample data corresponding to each sample molecule description text to obtain a molecule training set.

[0129] The following is explained with reference to specific examples. The sample molecule description text can be in Chinese, English or other languages. In the embodiments of this application, Chinese is used as an example for explanation. The source of the sample molecule description text can be a molecular database, which stores a large amount of popular science content related to molecules, such as molecular structural formulas, molecular description texts, etc. The sample molecule description text is obtained, and based on the mapping relationship between the sample molecule description text and the corresponding molecular structural formula string, the sample molecule structural formula string is retrieved from the molecular database. Obtain the instruction template to be configured, for example:

[0130] {"command":"***","input":

[0131] "***",

[0132] "Output":"***"}

[0133] Among them, *** refers to the content that is not filled in the instruction template to be configured. In a specific implementation, in the unfilled instruction template to be configured, the position where *** is located is blank.

[0134] The following is an example of sample data after configuration is completed, for example:

[0135] {"Command":"Generate a molecule based on this description","Input":

[0136] "The molecule is a tetracyclic triterpene, which is 4,4,8-trimethyl-1,14-diene, substituted with an oxy group at position 3, an acetoxy group at position 7 and a furan-3-yl group at position 17. It is isolated from plants of the Meliaceae family and has anti-malarial and anti-tumor activities. It acts as an anti-tumor agent, anti-malarial agent and plant metabolite. It is an acetate, a cyclic terpene ketone, a furanoid, a limonene and a tetracyclic triterpene.",

[0137] "Output":"CC(=O)O[C@@H]1C[C@@H]2[C@](C=CC(=O)C2(C)C)([C@@H]3[C@@]1(C4=CC[C@H]([C@@]4(CC3)C)C5=COC=C5)C)C"}

[0138] Among them, C@@H refers to carbon oxides. @ means "and", which refers to a mixture of two substances.

[0139] Continue to refer Figure 3A In step 304, the pre-trained language understanding model is trained based on the molecular training set to obtain a target language understanding model.

[0140] For example, the target language understanding model is used to generate a molecular structure formula of a compound molecule corresponding to the target molecule description text.

[0141] In some embodiments, reference Figure 3D , Figure 3D This is a fourth flow chart of the training method for a language understanding model for molecular processing provided in an embodiment of the present application; step 304 can be implemented through steps 3041 to 3043, which are described in detail below.

[0142] In step 3041, a pre-trained language understanding model is called based on the sample molecule description text to perform molecular prediction processing to obtain a first predicted molecular structure formula of the sample compound.

[0143] For example, the sample molecular description text is used as the input part in the sample data. When the instruction in the sample data is read, the sample data calls the pre-trained language understanding model to perform molecular prediction processing on the sample molecular description text to obtain the first predicted molecular structure of the sample compound. The molecular prediction processing process includes: converting the sample molecular description text into text features, predicting the probability that the sample molecular description text belongs to different molecular structures based on the text features, and selecting the molecular structure with the highest probability as the first predicted molecular structure corresponding to the sample molecular description text.

[0144] In some embodiments, reference Figure 3E , Figure 3Eis a fifth flow chart of the method for training a language understanding model for molecular processing provided by an embodiment of the present application. Step 3041 can be performed by Figure 3E Steps 30411 to 30414 are implemented as described in detail below.

[0145] In step 30411, the pre-trained language understanding model is called based on the sample molecular description text to perform the following processing: the sample molecular description text of the sample compound is segmented to obtain molecular word units included in the sample molecular description text.

[0146] For ease of understanding, the following explanation of the word segmentation of the sample molecular description text of the sample compound is combined with the accompanying drawings. Figure 5A , Figure 5A It is a schematic diagram of the word segmentation principle provided in the embodiment of the present application; Figure 5A In the benzoic acid molecule, the word segmentation is performed using the word segmentation vocabulary and the natural language vocabulary. For the molecular structure of the benzoic acid molecule C1=CC=C(C=C1)C(=O)O, the word segmentation is performed based on the natural language vocabulary, and the sequence composed of each character in the molecular structure of the benzoic acid molecule can be obtained, which is represented as:

[0147] [C,1,=,C,C,=,C,(,C,=,C,1,),C,(,=,O,),O]

[0148] Based on the molecular word list segmentation, the sub-molecular structural formulas corresponding to the benzene ring (C1=CC=C(C=C1)) and the carboxyl group (C(=O)O) in the benzoic acid molecule can be obtained as word units.

[0149] In step 30412, feature extraction processing is performed on the molecular words included in the sample molecular description text to obtain the first text feature of the sample molecular description text.

[0150] In some embodiments, step 30412 can be implemented in the following manner: perform feature extraction processing on the molecular words included in the sample molecular description text to obtain a first embedding vector; determine the third character sequence corresponding to the molecular word; obtain the second weight value of the second embedding vector of each character in the third character sequence; determine the average value of each second weight value as the first weight value of the first embedding vector of the molecular word; determine the product of each first embedding vector and the corresponding first weight value, and combine each product in turn into the first text feature of the sample molecular description text.

[0151] For example, continue to use the carboxyl group as an example to illustrate, perform feature extraction on the molecular word "carboxyl (C(=O)O)", obtain the first embedding vector of the word, and split the molecular word "carboxyl (C(=O)O)" into the third character sequence, and obtain the following third character sequence:

[0152] [C,(,=,O,),O]

[0153] Determine the second embedding vector corresponding to each character in the third character sequence, the second embedding vector is the embedding vector of each character obtained by extracting features from the third character sequence based on a natural language vocabulary, and the second weight value corresponding to each character is also determined according to the natural language feature extraction method. The average value of the second weight value of each character is used as the first weight value of the first embedding vector. Multiply the first weight value corresponding to each molecular word in the benzoic acid molecule by the first embedding vector, sort the products corresponding to the benzene ring and the carboxyl group into a vector sequence, and obtain the first text feature of the benzoic acid molecule.

[0154] In step 30413, probability prediction processing is performed based on the first text feature to obtain a second character sequence with the highest prediction probability corresponding to the first text feature.

[0155] For example, character prediction processing is performed for different types of characters based on the first text feature, and the character with the highest probability is used as the first character in the second character sequence. Character prediction processing is performed for different types of characters based on the first character in the second character sequence and the first text feature, and the character with the highest probability is used as the second character in the second character sequence, and so on. Each prediction processing is performed based on the predicted characters and the first text feature. In response to the number of characters included in the second character sequence reaching a preconfigured number, the prediction processing is stopped to obtain the final second character sequence.

[0156] In step 30414, the second character sequence is decoded to obtain a character string of the first predicted molecular structure formula, and the character string of the first predicted molecular structure formula is converted to obtain the first predicted molecular structure formula.

[0157] For example, the character string of the first predicted molecular structure formula is a character string based on the molecular linear input specification. Based on the mapping relationship between the characters in the vocabulary and the character string of the molecular substructure, the second character sequence is mapped to a character string representing the first predicted molecular structure formula; the character string of the first predicted molecular structure formula can be a character string in a simplified molecular linear input specification form, and the character string is converted into the first predicted molecular structure formula.

[0158] In the embodiment of the present application, the word segmentation processing of the molecular structure character string is performed based on the molecular vocabulary. Compared with the word segmentation scheme based on the natural language method, it can improve the model's understanding of the molecular structure formula and avoid breaking up the groups in the molecular structure to cause inaccurate prediction results.

[0159] Continue to refer Figure 3DIn step 3042, based on the difference between the sample molecular structure formula and the first predicted molecular structure formula, a loss function of the pre-trained language understanding model is determined.

[0160] For example, the difference between the structural formulas can be the similarity between the molecules, or the difference between the corresponding prediction probabilities. In the embodiment of the present application, the cross entropy loss corresponding to the difference between the corresponding prediction probabilities is used as an example for explanation.

[0161] In some embodiments, step 3042 can be implemented by: determining a first prediction probability corresponding to the sample molecular structure formula, and a second prediction probability corresponding to the first predicted molecular structure formula, wherein the first prediction probability represents the probability of obtaining the sample molecular structure formula based on the sample molecular description text, and the second prediction probability represents the probability of obtaining the first predicted molecular structure formula based on the sample molecular description text; based on each first prediction probability and each second prediction probability, determining the cross-entropy loss function of the pre-trained language understanding model.

[0162] For example, the cross entropy loss L can be expressed as the following formula (1):

[0163] L=∑ i logP(t i |t1, ..., t i-1 ;θ) (1)

[0164] Where t is the input molecular description text and θ is the parameter of the pre-trained language understanding model. i ) refers to the first predicted probability corresponding to the molecular description text of the i-th input.

[0165] In step 3043, the parameters of the pre-trained language understanding model are iteratively updated based on the loss function to obtain a target language understanding model.

[0166] For example, the principle of updating parameters may be an Adaptive Moment Estimation (Adam) algorithm.

[0167] In some embodiments, step 3043 can be implemented in the following manner: back-propagating the loss function (Back Forward) to obtain the gradient of each network layer of the pre-trained language understanding model; determining the mean of each gradient and the variance of each gradient; updating the initial first learning rate based on the mean and variance to obtain the second learning rate; based on the second learning rate and the gradient of each network layer, iteratively updating the parameters of the pre-trained language understanding model until the iteration round is equal to the iteration round threshold, and using the pre-trained language understanding model after the parameter update as the target language understanding model.

[0168] For example, the back propagation process can also be called backward calculation. The method for determining the gradient of each network layer of the pre-trained language understanding model can be an exponential decay averaging method, and the mean (first-order moment) of each gradient and the estimated value of the variance (second-order moment) of each gradient are determined based on the exponential decay averaging method. The iteration round threshold can be 10 times. In the process of iterative parameter update processing of the pre-trained language understanding model, a dropout coefficient can also be introduced. The dropout coefficient represents the random discarding of some neural units in the network during training, which is used to reduce overfitting and enhance the generalization ability of the network.

[0169] In an embodiment of the present application, the parameters of the pre-trained language understanding model are updated by means of adaptive moment estimation, thereby enhancing the generalization ability of the pre-trained language understanding model, achieving faster convergence speed and smaller fluctuation amplitude, thereby improving the accuracy of the trained model in predicting molecular structures.

[0170] In some embodiments, the mean and variance may be corrected respectively to reduce the impact caused by the estimation bias, and the initial first learning rate may be updated based on the corrected mean and variance to obtain the second learning rate.

[0171] In some embodiments, after step 304, reference Figure 3F , Figure 3F This is a sixth flow chart of the training method for the language understanding model for molecular processing provided in the embodiment of the present application. Execute step 305 to step 306, which are described in detail below.

[0172] In step 305, in response to an input operation for a molecule description text, based on the target molecule description text input by the input operation, the target language understanding model is called to perform the following processing: feature extraction processing is performed on the target molecule description text to obtain a second text feature.

[0173] For example, the principle of step 305 can refer to the above Figure 3E Steps 30411 to 30412 are not described here. When the execution subject is a server, the target molecule description text can be input by a user in a terminal device, and the terminal device sends the target molecule description text to the server. When the execution subject is a terminal device, the target molecule description text calls the target language understanding model stored locally in the terminal device for feature extraction.

[0174] In step 306, a molecular prediction process is performed based on the second text feature to obtain a second predicted molecular structure formula.

[0175] For example, the principle of step 306 can refer to the above Figure 3E Step 30413 is not repeated here.

[0176] In some embodiments, after step 306, the following processing is performed: data search processing is performed based on the target molecule description text to obtain the actual molecular structure formula corresponding to the target molecule description text; in response to the second predicted molecular structure formula being different from the actual molecular structure formula, the predicted molecular structure formula and the actual molecular structure formula are used as new training samples; based on the new training samples, training processing is performed on the target language understanding model to obtain an updated target language understanding model, wherein the training processing includes incrementally updating the parameters of the target language understanding model.

[0177] For example, the data search process can be implemented by calling a search engine to perform a keyword search on the Internet, or by searching a public molecule database based on the target molecule description text.

[0178] In the embodiment of the present application, incremental training can save more time and computing resources compared to full training. By means of incremental training, the target language understanding model is optimized in real time, and the accuracy of molecular structure prediction by the target language understanding model can be maintained during the application process.

[0179] In an embodiment of the present application, each molecular word in the molecular vocabulary is used to characterize a molecular substructure, and the parameters of the initial language understanding model are updated based on the molecular vocabulary, so that the pre-trained language understanding model can have the ability to perform word segmentation on molecular structural formulas according to the molecular substructures, thereby improving the pre-trained language understanding model's ability to understand molecular structural formulas. The pre-trained language understanding model is trained based on a molecular training set, so that the pre-trained language understanding model can be used to determine molecular structural formulas based on text, thereby improving the accuracy of determining molecular structural formulas based on descriptive texts.

[0180] Below, an exemplary application of the training method of the language understanding model for molecular processing in an embodiment of the present application in an actual application scenario will be described.

[0181] In the related art, in the field of text-based small molecule drug generation, there are mainly two types of algorithm implementations. One is based on the traditional sequence to sequence (Sequence To Sequence, Seq2Seq) algorithm implementation, such as the Recurrent Neural Network (RNN), Text-to-Text Transfer Transformer (T5) model and other model implementations. A representative work is the Molecules Text-to-Text Transfer Transformer (MolT5) model; the other is based on the large language model implementation of the Generative Pre-Trained Transformer (GPT) model class, such as MolReGPT, which can be implemented based on ChatGPT. The Generative Pre-Trained Transformer model is referred to as the GPT model below.

[0182] For sequence-to-sequence algorithms, the algorithm input is a molecular description text, and the algorithm output is a string of the Simplified Molecular Input Line Entry System (SMILES). Sequence-to-sequence algorithms lack the function of multi-round dialogue. After one algorithm input and output, it cannot be input again based on historical data. When the technician uses it again, he cannot provide feedback to the model, making it difficult to effectively use historical data. Although algorithms based on large language models support multi-round dialogue, the large language model algorithms in related technologies are built based on natural language vocabulary, without considering molecular language. The large language model does not have a high degree of understanding of the molecular SMILES structure, resulting in low accuracy of the generated molecular structure.

[0183] In the present application embodiment, in order to improve the existing large model's ability to understand molecular language, a molecular vocabulary is constructed based on a large-scale molecular data set, and the molecular vocabulary is merged with the original natural language vocabulary to form a new vocabulary. At the same time, based on the molecular generation data, an instruction-based molecular generation data set is constructed, and the final molecular large model (MolGPT) is obtained by fine-tuning the large model on the molecular generation data set. Compared to other models of related art, the accuracy of the language understanding model generation molecule of the present application embodiment is improved.

[0184] The following is an explanation of the training method of the language understanding model for molecular processing provided in the embodiment of the present application in conjunction with the accompanying drawings. Figure 4 , Figure 4This is the seventh flow chart of the training method of the language understanding model for molecular processing provided in an embodiment of the present application.

[0185] In step 401, a molecular dataset is constructed.

[0186] In an embodiment of the present application, an instruction-based molecule generation (Molecule Generation) data set is constructed, a large language model for molecule processing is configured based on a molecule vocabulary (Molecule Vocabulary), and training is performed on the molecule data set. After training, the large molecular model obtained can be used for molecule generation.

[0187] For example, the molecular data set in the embodiment of the present application is an instruction-based data set, and each data in the molecular data set is represented as an instruction form, wherein each data includes an instruction, an input, and an output. Among them, the content of the instruction includes: "Generate a molecule based on this description"; the input is a molecular description text, and the molecular description text is a text used to describe the molecular components. The types of molecular components include: elements that make up the molecule, and substructures in the molecule. When the molecule is a drug, the molecular description text may also include a method and source for extracting the substance corresponding to the molecule. The output is a character string representing the molecular structure. The character string of the molecular structure can be represented by simplifying the molecular linear input specification.

[0188] For example, the following is an example of a piece of data in a molecular dataset:

[0189] {"Command":"Generate a molecule based on this description","Input":

[0190] "The molecule is a tetracyclic triterpene, which is 4,4,8-trimethyl-1,14-diene, substituted with an oxy group at position 3, an acetoxy group at position 7 and a furan-3-yl group at position 17. It is isolated from plants of the Meliaceae family and has anti-malarial and anti-tumor activities. It acts as an anti-tumor agent, anti-malarial agent and plant metabolite. It is an acetate, a cyclic terpene ketone, a furanoid, a limonene and a tetracyclic triterpene.",

[0191] "Output":"CC(=O)O[C@@H]1C[C@@H]2[C@](C=CC(=O)C2(C)C)([C@@H]3[C@@]1(C4=CC[C@H]([C@@]4(CC3)C)C5=COC=C5)C)C"}

[0192] Among them, C@@H refers to carbon oxides. @ means "and", referring to a mixture of two substances. The above molecular description text can also be in English, for example: The molecule is a tetracyclic triterpenoidthat is 4,4,8-trimethylandrosta-1,14-diene substituted by an oxo group atposition 3,an acetoxy group at position 7and a furan-3-yl group at position17.Isolated from Azadirachta indica,it exhibits antiplasmodial and antimeplastic activities.It has a role as an antineoplastic agent,anantiplasmodial drug and a plant metabolite.It is an acetate ester,a cyclicterpene ketone,a member of furans,a limonoid and a tetracyclic triterpenoid.

[0193] In step 402, a molecular vocabulary is constructed.

[0194] For example, the embodiment of the present application collected 10 million molecular simplified molecular linear input standard string data from the ZINC database, trained word segmentation based on the byte pair encoder BPE (Byte Pair Encoder) algorithm, set the word list size to 1000, and finally the word segmentation algorithm generated a word list containing 1000 molecular tokens.

[0195] For example, the following explains the principle of word segmentation based on molecular structure formula. Figure 5A , Figure 5A It is a schematic diagram of the word segmentation principle provided in the embodiment of the present application; Figure 5A In the benzoic acid molecule, the word segmentation is performed using the word segmentation vocabulary and the natural language vocabulary. For the molecular structure of the benzoic acid molecule C1=CC=C(C=C1)C(=O)O, the word segmentation is performed based on the natural language vocabulary, and the sequence composed of each character in the molecular structure of the benzoic acid molecule can be obtained, which is represented as:

[0196] [C,1,=,C,C,=,C,(,C,=,C,1,),C,(,=,O,),O]

[0197] Based on the molecular word list segmentation, the molecular structural formulas corresponding to the benzene ring (C1=CC=C(C=C1)) and the carboxyl group (C(=O)O) in the benzoic acid molecule can be obtained.

[0198] according to Figure 5A As can be seen from the example, segmenting the SMILES string of a molecular structure formula based on a natural language vocabulary usually divides the SMILES string of the molecular structure formula into multiple fragmented characters, which ignore the fragment structure in the molecule, such as the functional group structure; while using a molecular vocabulary for word segmentation, the molecule can be divided into multiple chemically meaningful substructures, which is conducive to the model learning and understanding the molecular structure. A chemically meaningful substructure may refer to the structure of a base in a molecule. The embodiment of the present application merges the generated segmentation vocabulary with the original natural language vocabulary of the large model to form a new vocabulary, and uses the new vocabulary as the molecular vocabulary to configure the large language model.

[0199] In step 403, a large language model is configured based on the molecular vocabulary, and the configured large language model is trained based on the molecular dataset.

[0200] For example, the large language model (Large Language Model Meta AI, LLaMA) in the embodiment of the present application can be the LLaMA-7B-Chat model, the initial configuration of the model vocabulary size is 32000, plus the newly added molecular vocabulary (size is 1000) in the embodiment of the present application, the new vocabulary size is 33000. For the initial weight of the newly added molecular vocabulary, the embodiment of the present application is based on the original natural language vocabulary weight, and the average method is used for initialization to determine the weight value of the word in the molecular vocabulary.

[0201] Determining the weight value of a token in a molecular vocabulary can be achieved in the following way: for a token in a molecular vocabulary, use a natural language vocabulary to perform word segmentation, average the weights corresponding to the embedding features of the sub-tokens obtained by the word segmentation, and use it as the weight of the token in the word segmentation vocabulary. For example: for the string of the benzoic acid molecular structure (Cl=CC=C(C=C1)C(=O)O). Convert each character in the benzoic acid molecular formula into a corresponding sub-token, and the weight value for the token carboxyl (C(=O)O) is the average of the weight values ​​corresponding to the characters [C(=O)O].

[0202] For model training, the embodiment of the present application adopts the supervised fine-tuning (Supervised Fine-Tuning, SFT) method for training, the goal is to improve the ability of the large model in molecular generation. In addition, in order to increase the training speed and retain the original ability of the large model as much as possible, the embodiment of the present application adopts low-rank adaptation (Low-Rank Adaptation, LoRA) for training. During the model training process, the loss (Loss) will be calculated between the predicted string obtained by the forward calculation and the true label (Label). The true label is also the true molecular string. The loss type can be a cross entropy loss. The cross entropy loss L can be represented by the following formula (1):

[0203] L=∑ i logP(t i |t1, ..., t i-1 ;θ) (1)

[0204] Where t is the input molecular description text and θ is the model parameter. i ) refers to the predicted probability corresponding to the molecular description text of the i-th input.

[0205] The loss is calculated through back-propagation, and the gradient of each network layer in the model is calculated, and the adaptive moment estimation (Adaptive Moment Estimation, Adam) algorithm is used to update the neural network model parameters. During the entire training process, the embodiment of the present application trained a total of 10 epochs.

[0206] The adaptive moment estimation algorithm can be regarded as a root mean square back propagation algorithm with a momentum term in essence. On the one hand, it uses the momentum method to accumulate the historical gradients of the parameters to make better use of historical information. On the other hand, it uses the first-order moment estimation and second-order moment estimation of the gradient to dynamically adjust the learning rate of each parameter, thereby achieving faster convergence speed and smaller fluctuation amplitude.

[0207] refer to Fig. 6A , Fig. 6AIt is a parameter table provided in the embodiment of the present application; the types of parameters include: learning rate, batch size, low-rank adaptive rank (Lora rank), low-rank adaptive fine-tuning coefficient (Lora alpha), low-rank adaptive loss coefficient (Lora dropout), weight decay coefficient (Weightdecay), and iteration rounds (Epochs). Among them, the low-rank adaptive loss coefficient represents the random discarding of some neural units in the network during training, which is used to reduce overfitting and enhance the generalization ability of the network.

[0208] The training method of the language understanding model for molecular processing provided in the embodiment of the present application can be used in the process of drug development. Using a large model to generate small molecule drugs can increase the speed of new candidate drug discovery and reduce research and development costs. In the new drug development process, the model trained using the embodiment of the present application can help drug development tasks design candidate molecules more efficiently.

[0209] In some embodiments, the main processes of drug development include, in chronological order: target identification and validation, compound screening and lead discovery, preclinical study, and clinical trials. In the drug development process, after target identification and validation, it is necessary to find candidate drug small molecules. In artificial intelligence-driven drug design (AI-Driven Drug Design, AIDD) technology, a molecule generation algorithm is generally used to generate candidate drug small molecules. At present, the molecule generation algorithm has developed into a text-based molecule generation method, which can further facilitate the use of pharmaceutical chemistry experts. Pharmaceutical chemistry experts can describe the required candidate drug molecules in natural language. For the generation of candidate drug molecules, pharmaceutical chemistry experts can adopt the training method of the language understanding model for molecular processing provided in the embodiment of the present application. The trained model can help R&D personnel design drug molecules through dialogue, greatly improve R&D efficiency, and reduce drug R&D costs. Reference Figure 5B , Figure 5B It is a schematic diagram of an application scenario provided by an embodiment of the present application;

[0210] The user may be a research and development personnel, who inputs a molecular description text 501 into a model 502 trained in the embodiment of the present application through a terminal device, and the model 502 generates a molecular structure formula 503 based on the molecular description text 501. The content of the molecular description text 501 is: "The molecule is a compound comprising a benzene ringcore carrying a carboxylic acid substituent. The molecule is a compound comprising a benzene ringcore, wherein the benzene ringcore carries a carboxyl group", and the output molecular structure formula is C1=CC=C(C=C1)C(=O)O.

[0211] As an example, the following is a description of the beneficial effects of the training method for the language understanding model for molecular processing provided by the embodiment of the present application in combination with data. Figure 6B , Figure 6B It is the comparison result between the model trained by the embodiment of the present application and the model of the related technology provided by the embodiment of the present application. Figure 6B The evaluation indicators related to the effect of the model trained by the embodiment of the present application are compared with those of the existing models in the related technology. The types of existing models in the related technology include: recurrent neural network (RNN), transformer model (Transformer), molecular text-to-text transfer transformer model (MolT5), and molecular ReGPT model.

[0212] Figure 6B The following evaluation indicators are included:

[0213] Bilingual Evaluation Understudy (BLEU) is used to evaluate the quality of machine translation.

[0214] Exact characterizes the accuracy of the model in generating molecular structures based on text.

[0215] Levenshtein distance, also known as edit distance, refers to the minimum number of edit operations required to convert one string into another. Allowed edit operations include replacing one character with another, inserting a character, and deleting a character.

[0216] Molecular Access System (MACCS) molecular fingerprint is a binary fingerprint used to represent molecular structure information. It is used to measure the similarity between molecules. RDK molecular fingerprint is a commonly used type of molecular fingerprint in RDKit, which can be used for drug similarity and virtual screening. Morgan molecular fingerprints are circular fingerprints and also belong to topological fingerprints.

[0217] The "effectiveness" in "Validity" means that the output content of the model satisfies the molecular grammar, and the "effectiveness" is the probability that the output content satisfies the grammar corresponding to the molecular structure.

[0218] The embodiments of the present application are tested on the industry's authoritative Benchmark dataset ChEBI-20 and compared with the work in the related art. Figure 6B It can be seen from the table that the model trained by the embodiment of the present application is superior to the model of the related technology in many indicators.

[0219] The following is a description of an exemplary structure of a training device 455 for a language understanding model for molecular processing provided by an embodiment of the present application implemented as a software module. In some embodiments, Figure 2 As shown, the software modules in the training device 455 of the language understanding model for molecular processing stored in the memory 450 may include: a data acquisition module 4551, used to acquire a molecular vocabulary, wherein the molecular vocabulary includes: multiple molecular words, and one molecular word is used to represent a type of molecular substructure; a model training module 4552, used to update the initial language understanding model based on the molecular vocabulary to obtain a pre-trained language understanding model, wherein the pre-trained language understanding model is used to split the molecular structure formula to obtain the molecular word of the molecular structure formula; the data acquisition module 4551 is used to acquire a molecular training set, wherein the molecular training set includes: sample molecular description text and sample molecular structure formula, and the sample molecular description text is used to describe the molecular components of the sample compound; the model training module 4552 is used to train the pre-trained language understanding model based on the molecular training set to obtain a target language understanding model, wherein the target language understanding model is used to generate a molecular structure formula of a compound molecule corresponding to the target molecular description text.

[0220] In some embodiments, the data acquisition module 4551 is used to obtain multiple candidate molecular structural formulas; obtain the character string corresponding to each candidate molecular structural formula, wherein the character string is used to describe the molecular structural formula; perform word unit extraction processing on the character string corresponding to each candidate molecular structural formula to obtain multiple candidate molecular word units; perform deduplication processing on the multiple candidate molecular word units; in response to the number of candidate molecular word units remaining after the deduplication processing being greater than a preconfigured number, combining the candidate molecular word units remaining after the deduplication processing into a candidate vocabulary; and merging the candidate vocabulary with the natural language vocabulary preset for the pre-trained language understanding model to obtain a molecular vocabulary.

[0221] In some embodiments, the data acquisition module 4551 is used to perform character splitting processing on each character string to obtain a first character sequence corresponding to multiple candidate molecular structures, wherein the first character sequence includes multiple characters; determine the occurrence frequency of each character in the first character sequence; determine the occurrence frequency of consecutive byte pairs based on the occurrence frequency of each character, wherein the consecutive byte pairs include multiple adjacent characters; combine the characters in the consecutive byte pairs with the maximum occurrence frequency into subwords; determine the occurrence frequency of each subword in the first character sequence; iteratively perform the following processing: combine two adjacent subwords with the maximum occurrence frequency into a new subword, and determine the occurrence frequency of each new subword in the first character sequence; in response to the maximum occurrence frequency being equal to a preconfigured value and the new subword satisfying the condition of representing a molecular structure with chemical significance, use the new subword as a candidate molecular word.

[0222] In some embodiments, the model training module 4552 is used to replace the configured vocabulary in the initial language understanding model based on the molecular vocabulary to obtain a pre-trained language understanding model.

[0223] In some embodiments, the molecular training set includes multiple segments of sample data; each sample data includes: a segment of sample molecule description text and a sample molecule structure formula corresponding to the sample molecule description text; a data acquisition module 4551 acquires the sample molecule description text; performs the following processing for each sample molecule description text: based on the molecular components described in the sample molecule description text, determines the string of the sample molecule structure formula corresponding to the sample molecule description text; acquires the instruction template to be configured, wherein the parameters to be configured in the instruction template to be configured include: instructions, inputs, and outputs; configures the instructions of the instruction template to be configured to indicate the following: based on the sample molecule description text, calls a pre-trained language understanding model to generate a string of the sample molecule structure formula; configures the sample molecule description text as the input in the instruction template to be configured, and configures the string of the sample molecule structure formula as the output in the instruction template to be configured; uses the configured instruction template to be configured as sample data, wherein the instructions are used to trigger the execution of the processing of calling the pre-trained language understanding model to generate a string of the sample molecule structure formula based on the sample molecule description text when the sample data is read; combines the sample data corresponding to each sample molecule description text to obtain a molecular training set.

[0224] In some embodiments, the model training module 4552 is used to call a pre-trained language understanding model to perform molecular prediction processing based on the sample molecular description text to obtain a first predicted molecular structure formula of the sample compound; based on the difference between the sample molecular structure formula and the first predicted molecular structure formula, determine the loss function of the pre-trained language understanding model; based on the loss function, iteratively update the parameters of the pre-trained language understanding model to obtain a target language understanding model.

[0225] In some embodiments, the model training module 4552 is used to call the pre-trained language understanding model based on the sample molecule description text to perform the following processing:

[0226] The sample molecular description text of the sample compound is segmented to obtain the molecular word elements included in the sample molecular description text; the molecular word elements included in the sample molecular description text are extracted to obtain the first text feature of the sample molecular description text; probability prediction processing is performed based on the first text feature to obtain a second character sequence with the highest prediction probability corresponding to the first text feature; the second character sequence is decoded to obtain a character string of a first predicted molecular structure formula; the character string of the first predicted molecular structure formula is converted to obtain the first predicted molecular structure formula.

[0227] In some embodiments, the model training module 4552 is used to perform feature extraction processing on the molecular words included in the sample molecular description text to obtain a first embedding vector; determine a third character sequence corresponding to the molecular word; obtain a second weight value of the second embedding vector of each character in the third character sequence; determine an average value of each second weight value as the first weight value of the first embedding vector of the molecular word; determine the product of each first embedding vector and the corresponding first weight value, and combine each product in turn into a first text feature of the sample molecular description text.

[0228] In some embodiments, the model training module 4552 is used to determine a first prediction probability corresponding to a sample molecular structure formula, and a second prediction probability corresponding to the first predicted molecular structure formula, wherein the first prediction probability represents the probability of obtaining the sample molecular structure formula based on the sample molecular description text, and the second prediction probability represents the probability of obtaining the first predicted molecular structure formula based on the sample molecular description text; based on each first prediction probability and each second prediction probability, determine the cross-entropy loss function of the pre-trained language understanding model.

[0229] In some embodiments, the model training module 4552 is used to perform back-propagation processing on the loss function to obtain the gradient of each network layer of the pre-trained language understanding model; determine the mean of each gradient and the variance of each gradient; update the initial first learning rate based on the mean and variance to obtain the second learning rate; based on the second learning rate and the gradient of each network layer, iteratively update the parameters of the pre-trained language understanding model until the iteration round is equal to the iteration round threshold, and use the pre-trained language understanding model after the parameter update as the target language understanding model.

[0230] In some embodiments, the model training module 4552 is used to train the pre-trained language understanding model based on the molecular training set to obtain the target language understanding model, and then, in response to an input operation for the molecular description text, based on the target molecular description text input by the input operation, call the target language understanding model to perform the following processing:

[0231] The target molecule description text is subjected to feature extraction processing to obtain a second text feature; and a molecule prediction processing is performed based on the second text feature to obtain a second predicted molecular structure formula.

[0232] In some embodiments, the model training module 4552 is used to perform molecular prediction processing based on the second text feature to obtain the second predicted molecular structure formula, and then perform data search processing based on the target molecule description text to obtain the actual molecular structure formula corresponding to the target molecule description text; in response to the second predicted molecular structure formula being different from the actual molecular structure formula, use the predicted molecular structure formula and the actual molecular structure formula as new training samples; and perform training processing on the target language understanding model based on the new training samples to obtain an updated target language understanding model, wherein the training processing includes incrementally updating the parameters of the target language understanding model.

[0233] The embodiment of the present application provides a computer program product, which includes a computer program or a computer executable instruction, and the computer program or the computer executable instruction is stored in a computer-readable storage medium. The processor of the electronic device reads the computer program or the computer executable instruction from the computer-readable storage medium, and the processor executes the computer program or the computer executable instruction, so that the electronic device executes the training method of the language understanding model for molecular processing described in the embodiment of the present application.

[0234] The present application embodiment provides a computer-readable storage medium storing computer executable instructions, wherein the computer executable instructions or computer programs are stored. When the computer executable instructions or computer programs are executed by a processor, the processor will be caused to execute the training method of the language understanding model for molecular processing provided by the present application embodiment, for example, Figure 3A A method for training a language understanding model for molecular processing is shown.

[0235] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or it may be various devices including one or any combination of the above memories.

[0236] In some embodiments, computer executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0237] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file storing other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0238] As an example, computer executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed at multiple sites and interconnected by a communication network.

[0239] In summary, each molecular word in the molecular vocabulary of the embodiment of the present application is used to characterize the molecular substructure, and the parameters of the initial language understanding model are updated based on the molecular vocabulary, so that the pre-trained language understanding model can have the ability to perform word segmentation on the molecular structural formula according to the molecular substructure, thereby improving the pre-trained language understanding model's understanding ability for molecular structural formulas. The pre-trained language understanding model is trained based on the molecular training set, so that the pre-trained language understanding model can be used to determine the molecular structural formula based on the text, thereby improving the accuracy of determining the molecular structural formula based on the descriptive text.

[0240] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A method for training a language understanding model for molecular processing, characterized in that: The method comprises: Acquire a molecular vocabulary, wherein the molecular vocabulary includes: a plurality of molecular words, and one molecular word is used to represent a type of molecular substructure; Based on the molecular vocabulary, an initial language understanding model is updated to obtain a pre-trained language understanding model, wherein the pre-trained language understanding model is used to split the molecular structure formula to obtain the molecular word units of the molecular structure formula; Acquire a molecular training set, wherein the molecular training set includes: sample molecular description text and sample molecular structural formula, wherein the sample molecular description text is used to describe the molecular components of the sample compound; The pre-trained language understanding model is trained based on the molecular training set to obtain a target language understanding model, wherein the target language understanding model is used to generate a molecular structural formula of a target compound molecule corresponding to a target molecule description text.

2. The method according to claim 1, characterized in that The step of obtaining a molecular vocabulary comprises: Obtain multiple candidate molecular structural formulas; Obtaining a character string corresponding to each of the candidate molecular structural formulas, wherein the character string is used to describe the molecular structural formula; Performing word unit extraction processing on the character string corresponding to each of the candidate molecular structural formulas to obtain a plurality of candidate molecular word units; Performing deduplication processing on the multiple candidate molecular words; In response to the number of the candidate molecular word units remaining after the deduplication process is greater than a preconfigured number, combining the candidate molecular word units remaining after the deduplication process into a candidate word list; The candidate vocabulary is merged with a natural language vocabulary preset for the pre-trained language understanding model to obtain a molecular vocabulary.

3. The method according to claim 2, characterized in that The word unit extraction process is performed on the character string corresponding to each of the candidate molecular structural formulas to obtain a plurality of candidate molecular word units, including: Performing character splitting processing on each of the character strings to obtain first character sequences corresponding to the plurality of candidate molecular structure formulas, wherein the first character sequence includes a plurality of characters; determining the frequency of occurrence of each character in the first character sequence; Determining the occurrence frequency of consecutive byte pairs based on the occurrence frequency of each of the characters, wherein the consecutive byte pairs include a plurality of adjacent characters; Grouping the characters in the consecutive byte pairs with the maximum occurrence frequency into subwords; determining the frequency of occurrence of each of the subwords in the first character sequence; Iteratively performing the following processing: combining two adjacent subwords with the maximum occurrence frequency into a new subword, and determining the occurrence frequency of each new subword in the first character sequence; In response to the maximum occurrence frequency being equal to a preconfigured value and the new sub-word satisfying a condition of representing a molecular structure having chemical significance, the new sub-word is used as a candidate molecular word-element.

4. The method according to claim 1, characterized in that: The updating of the initial language understanding model based on the molecular vocabulary to obtain a pre-trained language understanding model includes: The configured vocabulary in the initial language understanding model is replaced based on the molecular vocabulary to obtain a pre-trained language understanding model.

5. The method according to claim 1, characterized in that The molecular training set includes multiple sections of sample data; each of the sample data includes: a section of sample molecule description text and a sample molecule structure formula corresponding to the sample molecule description text; The step of obtaining a molecular training set comprises: Get sample molecule description text; The following processing is performed for each sample molecule description text: Based on the molecular components described by the sample molecule description text, determining a character string of a sample molecule structure formula corresponding to the sample molecule description text; Obtaining a to-be-configured instruction template, wherein the parameters to be configured in the to-be-configured instruction template include: instructions, inputs, and outputs; The instructions of the instruction template to be configured are configured to indicate the following: based on the sample molecule description text, calling the pre-trained language understanding model to generate a character string of the sample molecule structure formula; The sample molecule description text is configured as the input in the instruction template to be configured, and the character string of the sample molecule structure formula is configured as the output in the instruction template to be configured; The configured instruction template to be configured is used as sample data, wherein the instruction is used to trigger the execution of the processing of calling the pre-trained language understanding model to generate a character string of the sample molecule structure formula based on the sample molecule description text when the sample data is read; The sample data corresponding to each sample molecule description text are combined to obtain the molecule training set.

6. The method according to claim 1, characterized in that The step of training the pre-trained language understanding model based on the molecular training set to obtain a target language understanding model includes: Based on the sample molecule description text, the pre-trained language understanding model is called to perform molecular prediction processing to obtain a first predicted molecular structure formula of the sample compound; Determining a loss function of the pre-trained language understanding model based on a difference between the sample molecular structure formula and the first predicted molecular structure formula; The pre-trained language understanding model is iteratively updated with parameters based on the loss function to obtain a target language understanding model.

7. The method according to claim 6, characterized in that The calling of the pre-trained language understanding model to perform molecular prediction processing based on the sample molecular description text of the sample compound to obtain a first predicted molecular structure formula includes: Based on the sample molecule description text, the pre-trained language understanding model is called to perform the following processing: Performing word segmentation processing on the sample molecule description text of the sample compound to obtain the molecule word elements included in the sample molecule description text; Performing feature extraction processing on the molecular word units included in the sample molecular description text to obtain a first text feature of the sample molecular description text; Performing probability prediction processing based on the first text feature to obtain a second character sequence with the highest prediction probability corresponding to the first text feature; Decoding the second character sequence to obtain a character string of the first predicted molecular structure formula; The character string of the first predicted molecular structure formula is converted to obtain the first predicted molecular structure formula.

8. The method according to claim 7, characterized in that The step of performing feature extraction processing on the molecular word element included in the sample molecular description text to obtain a first text feature of the sample molecular description text includes: Performing feature extraction processing on the molecular word units included in the sample molecular description text to obtain a first embedding vector; Determine a third character sequence corresponding to the molecular word element; Obtain a second weight value of a second embedding vector for each of the characters in the third character sequence; Determine an average of each of the second weight values ​​as a first weight value of the first embedding vector of the molecular word unit; The product of each of the first embedding vectors and the corresponding first weight value is determined, and each of the products is sequentially combined into a first text feature of the sample molecule description text.

9. The method according to claim 6, characterized in that The determining of the loss function of the pre-trained language understanding model based on the difference between the sample molecular structure formula and the first predicted molecular structure formula includes: Determine a first prediction probability corresponding to the sample molecular structure formula, and a second prediction probability corresponding to the first predicted molecular structure formula, wherein the first prediction probability represents the probability of obtaining the sample molecular structure formula based on the sample molecular description text, and the second prediction probability represents the probability of obtaining the first predicted molecular structure formula based on the sample molecular description text; Based on each of the first prediction probabilities and each of the second prediction probabilities, a cross entropy loss function of the pre-trained language understanding model is determined.

10. The method according to claim 6, characterized in that The iteratively performing parameter updating processing on the pre-trained language understanding model based on the loss function to obtain a target language understanding model includes: Performing back-propagation processing on the loss function to obtain the gradient of each network layer of the pre-trained language understanding model; determining a mean of each of the gradients and a variance of each of the gradients; The initial first learning rate is updated based on the mean and the variance to obtain a second learning rate; Based on the second learning rate and the gradient of each of the network layers, the pre-trained language understanding model is iteratively updated with parameters until the iteration round is equal to the iteration round threshold, and the pre-trained language understanding model after the parameter update is used as the target language understanding model.

11. The method according to any one of claims 1 to 10, characterized in that: After training the pre-trained language understanding model based on the molecular training set to obtain a target language understanding model, the method further includes: In response to an input operation for a target molecule description text, based on the target molecule description text input by the input operation, the target language understanding model is called to perform the following processing: Performing feature extraction processing on the target molecule description text to obtain a second text feature; Molecular prediction processing is performed based on the second text feature to obtain a second predicted molecular structure formula.

12. The method according to claim 11, characterized in that After performing molecular prediction processing based on the second text feature to obtain a second predicted molecular structure formula, the method further includes: Performing data search processing based on the target molecule description text to obtain the actual molecular structure formula corresponding to the target molecule description text; In response to the second predicted molecular structure formula being different from the actual molecular structure formula, taking the predicted molecular structure formula and the actual molecular structure formula as new training samples; The target language understanding model is trained based on the new training samples to obtain an updated target language understanding model, wherein the training includes incrementally updating parameters of the target language understanding model.

13. A training device for a language understanding model for molecular processing, characterized in that: The device comprises: A data acquisition module is used to acquire a molecular vocabulary, wherein the molecular vocabulary includes: a plurality of molecular words, and one molecular word is used to represent a type of molecular substructure; A model training module, used for updating an initial language understanding model based on the molecular vocabulary to obtain a pre-trained language understanding model, wherein the pre-trained language understanding model is used for splitting the molecular structure formula to obtain the molecular word units of the molecular structure formula; The data acquisition module is used to acquire a molecular training set, wherein the molecular training set includes: sample molecular description text and sample molecular structural formula, wherein the sample molecular description text is used to describe the molecular components of the sample compound; The model training module is used to train the pre-trained language understanding model based on the molecular training set to obtain a target language understanding model, wherein the target language understanding model is used to generate a molecular structural formula of a compound molecule corresponding to a target molecule description text.

14. An electronic device, characterized in that: The electronic device comprises: A memory for storing computer executable instructions or computer programs; A processor, for implementing the training method of a language understanding model for molecular processing as described in any one of claims 1 to 12 when executing the computer executable instructions or computer program stored in the memory.

15. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that: When the computer executable instructions or computer program are executed by a processor, the training method of the language understanding model for molecular processing according to any one of claims 1 to 12 is implemented.

16. A computer program product comprising computer executable instructions or a computer program, characterized in that When the computer executable instructions or computer program are executed by a processor, the training method of the language understanding model for molecular processing according to any one of claims 1 to 12 is implemented.

Citation Information

Cited By

  • Interstellar molecule data cleaning method, device and system and storage medium

    CN119339836A

  • Data cleaning method

    CN119884611A

  • Multi-granularity, multi-view and multi-task incremental multi-language text recognition method and system

    CN121303120A