Learning device, method for operating learning device, program for operating learning device, drug development assistance device, method for operating drug development assistance device, program for operating drug development assistance device

Abstract

Provided is a learning device for a substance language model to which sequence information for amino acid residues constituting an amino-acid-derived substance contained in a drug is applied, the learning device comprising a processor, the processor performing, on the substance language model, first pre-learning for estimating masked amino acid residues in reference sequence information for amino acid residues constituting a reference substance, and second pre-learning for estimating a structure to which the amino acid residues belong.

Description

Learning device, operating method of learning device, operating program of learning device, drug development support device, operating method of drug development support device, and operating program of drug development support device

[0001] The technology disclosed herein relates to a learning device, an operating method for a learning device, an operating program for a learning device, a drug development support device, an operating method for a drug development support device, and an operating program for a drug development support device.

[0002] Recently, pharmaceuticals such as biopharmaceuticals, peptide drugs, and nucleic acid drugs have been attracting attention due to their high efficacy and few side effects. For example, biopharmaceuticals use proteins such as interferon and antibodies as their active ingredients.

[0003] In the field of natural language processing (NLP), language models such as BERT (Bidirectional Encoder Representations from Transformers) using a transformer encoder have been attracting attention. The language model undergoes pre-training through MLM (Masked Language Modeling). MLM is a so-called fill-in-the-blank problem in which a learning input sentence in which a portion of a word or phrase is masked is asked to estimate what word or phrase will be inserted in the masked portion. After this pre-training, the language model undergoes fine tuning (hereinafter abbreviated as FT (Fine-Tuning)) according to the desired natural language processing task. FT refers to additional training of a language model after pre-training using training data that is different from the training data used in pre-training and that corresponds to the desired natural language processing task.

[0004] Various technologies have been proposed that use language models to promote the development of biopharmaceuticals by treating sequence information of amino acid residues that make up proteins as sentences. For example, Kaiyuan Gao et al., "Pre-training Antibody Language Models for Antigen-Specific Computational Antibody Design," Knowledge Discovery and Data Mining, 4 August 2023 (hereinafter referred to as Non-Patent Document 1), discloses a language model that performs the task of estimating the structure of complementarity determining regions (CDRs), which are antigen-binding sites and determine the characteristics of antibodies. In Non-Patent Document 1, in an MLM, in addition to sequence information of amino acid residues constituting an antibody, in which amino acid residues in the complementarity determining regions (CDR-H1, CDR-H2, CDR-H3, etc.) are selectively masked, information indicating to which complementarity determining region the masked amino acid residues belong is included in the input data for learning. Then, the estimation results of the masked amino acid residues are output to a language model. Note that a language model to which sequence information of amino acid residues constituting a protein such as an antibody is applied is called a protein language model.

[0005] Structural information indicating which higher-order structure (secondary structure and / or tertiary structure) of an antibody each amino acid residue belongs to, such as a loop structure formed by a complementarity-determining region or a parallel or antiparallel structure of a polypeptide chain connected by a loop structure, is useful for improving the accuracy of task estimation using a protein language model. In Non-Patent Document 1, information indicating which complementarity-determining region a masked amino acid residue belongs to is included in the training input data, which is an example of structural information. Therefore, at first glance, it appears that structural information has been incorporated into the protein language model. However, since the protein language model only outputs the estimated results of the masked amino acid residues, training related to the structural information has not been sufficiently performed. Therefore, the training method described in Non-Patent Document 1 may not be expected to improve the accuracy of task estimation using a protein language model.

[0006] One embodiment of the technology of the present disclosure provides a learning device, an operating method of a learning device, an operating program of a learning device, a pharmaceutical development support device, an operating method of a pharmaceutical development support device, and an operating program of a pharmaceutical development support device, which are capable of improving the estimation accuracy of a substance language model to which sequence information of amino acid residues that constitute amino acid-derived substances such as proteins is applied.

[0007] The learning device of the present disclosure is a learning device for a substance language model to which sequence information of amino acid residues constituting substances derived from amino acids contained in pharmaceuticals is applied, and includes a processor, which performs first pre-learning on the substance language model to estimate masked amino acid residues in reference sequence information of amino acid residues constituting a reference substance, and second pre-learning to estimate the structure to which the amino acid residues belong.

[0008] After the first and second pre-training processes are completed, the processor preferably performs fine tuning on the material language model according to the desired task.

[0009] Preferably, the task is a task of estimating information about the prescription of a stock solution of a pharmaceutical product.

[0010] Preferably, the processor performs a second pre-training on the masked amino acid residues.

[0011] Preferably, the processor sets the mask across the boundary between two adjacent structures.

[0012] The processor preferably performs the first pre-learning and the second pre-learning in parallel.

[0013] Preferably, the substance is one of a protein, a peptide, and a nucleic acid.

[0014] Preferably, the protein is an antibody.

[0015] Preferably, the structure comprises a structure in the variable region of an antibody.

[0016] The structure preferably includes a structure in which atoms constituting the amino acid residues of the antibody are represented by positional information in a three-dimensional conformation.

[0017] Preferably, the structure includes framework regions or complementarity determining regions having amino acid residues of an antibody.

[0018] The structure preferably comprises a chain, loop or turn structure comprising amino acid residues of an antibody.

[0019] The method of operating a learning device disclosed herein is a method of operating a learning device for a substance language model to which sequence information of amino acid residues constituting a substance derived from amino acids contained in a pharmaceutical is applied, and includes performing, on the substance language model, a first pre-learning process to estimate masked amino acid residues in reference sequence information of amino acid residues constituting a reference substance, and a second pre-learning process to estimate a structure to which the amino acid residues belong.

[0020] The operating program of the learning device disclosed herein is an operating program of a learning device for a substance language model to which sequence information of amino acid residues constituting substances derived from amino acids contained in pharmaceuticals is applied, and causes a computer to execute processing including performing, on the substance language model, first pre-learning to infer masked amino acid residues from reference sequence information of amino acid residues constituting a reference substance, and second pre-learning to infer the structure to which the amino acid residues belong.

[0021] The drug development support device of the present disclosure is a drug development support device that causes a substance language model to which sequence information of amino acid residues that constitute a substance derived from amino acids contained in a drug is applied to make an inference regarding a substance, and includes a processor. The processor uses a substance language model that has undergone first pre-learning to infer masked amino acid residues in reference sequence information of amino acid residues that constitute a reference substance, and second pre-learning to infer a structure to which the amino acid residues belong, inputs target sequence information of amino acid residues that constitute a substance to be inferred into the substance language model, and causes the substance language model to output an inference result.

[0022] The method of operating the drug development support device of the present disclosure is a method of operating the drug development support device that causes a substance language model to which sequence information of amino acid residues constituting a substance derived from amino acids contained in a drug is applied to make an inference regarding a substance, and includes using a substance language model that has undergone first pre-learning to infer masked amino acid residues in reference sequence information of amino acid residues constituting a reference substance and second pre-learning to infer a structure to which the amino acid residues belong, and inputting target sequence information of amino acid residues constituting a substance to be inferred into the substance language model and outputting an inference result from the substance language model.

[0023] The operating program of the drug development support device of the present disclosure is an operating program of the drug development support device that causes a substance language model to be applied with sequence information of amino acid residues that constitute substances derived from amino acids contained in drugs, to make inferences about substances, and causes a computer to execute processes including: using a substance language model that has undergone first pre-learning to infer masked amino acid residues in reference sequence information of amino acid residues that constitute a reference substance, and second pre-learning to infer structures to which the amino acid residues belong; inputting target sequence information of amino acid residues that constitute a substance to be inferred into the substance language model; and outputting inference results from the substance language model.

[0024] According to the technology of the present disclosure, it is possible to provide a learning device, an operating method for a learning device, an operating program for a learning device, a pharmaceutical development support device, an operating method for a pharmaceutical development support device, and an operating program for a pharmaceutical development support device, which are capable of improving the estimation accuracy of a substance language model to which sequence information of amino acid residues that constitute amino acid-derived substances such as proteins is applied.

[0025] 1 is a diagram showing a drug development support system. FIG. 2 is a diagram showing target sequence information. FIG. 3 is a diagram showing the basic structure of an antibody. FIG. 4 is a diagram showing the detailed configuration of a variable domain. FIG. 5 is a diagram showing an example sequence of amino acid residues in a variable domain. FIG. 6 is a block diagram showing computers constituting a learning device, a drug development support device, and an operator terminal. FIG. 7 is a block diagram showing a processing unit of a CPU of the learning device. FIG. 8 is a diagram showing a past data group. FIG. 9 is a diagram showing an overview of the processing of a generation unit. FIG. 10 is a diagram showing mask processing for reference sequence information. FIG. 11 is a diagram showing a first training data group. FIG. 12 is a diagram showing a second training data group. FIG. 13 is a diagram showing an overview of the processing of a pre-training unit. FIG. 14 is a diagram showing first pre-training and second pre-training. FIG. 15 is a block diagram showing a processing unit of a CPU of a drug development support device. FIG. 16 is a diagram showing the processing of an estimation unit. FIG. 17 is a block diagram showing an information input screen. FIG. 18 is a diagram showing an estimation result display screen. A flowchart showing the processing procedure of a learning device. A flowchart showing the processing procedure of a drug development support device. A diagram showing a peptide drug or a nucleic acid drug using a peptide or a nucleic acid as an amino acid-derived substance.

[0026] As shown in FIG. 1 , a pharmaceutical development support system 10 is a system for supporting the development of a biopharmaceutical 11 and includes a learning device 12, a pharmaceutical development support device 13, and an operator terminal 14. The learning device 12 generates a protein language model 15 and transmits the generated protein language model 15 to the pharmaceutical development support device 13. The protein language model 15 is a BERT-based language model. The protein language model 15 estimates information useful for promoting the development of the biopharmaceutical 11 by treating sequence information of amino acid residues constituting a protein, which is the active ingredient of the biopharmaceutical 11, as a sentence. In this example, the protein is an antibody 16T. The information useful for promoting the development of the biopharmaceutical 11 is information regarding an appropriate prescription (also referred to as a formulation prescription) for a preservative solution to stabilize the quality of the biopharmaceutical 11. The biopharmaceutical 11 is an example of a "pharmaceutical" according to the technology disclosed herein. The protein language model 15 is an example of a "substance language model" according to the technology disclosed herein. The antibody 16T is an example of an "amino acid-derived substance," "protein," and "target substance" according to the technology disclosed herein. In the following description, antibody 16T may be simply referred to as antibody 16 unless a distinction is particularly required.

[0027] The drug development support device 13 and the operator terminal 14 are connected via a network 17. The operator terminal 14 is installed at a pharmaceutical company developing the biopharmaceutical 11 or at an organization contracted by a pharmaceutical company to develop the biopharmaceutical 11, i.e., a contract research organization (CRO). The operator terminal 14 is operated by an operator OP involved in the development of the biopharmaceutical 11 at the pharmaceutical company or contract research organization. The network 17 is, for example, a wide area network (WAN) such as the Internet or a public communication network. Note that while only one operator terminal 14 is connected to the drug development support device 13 in FIG. 1 , in reality, multiple operator terminals 14 from multiple pharmaceutical companies or contract research organizations are connected to the drug development support device 13.

[0028] The operator terminal 14 transmits an estimation request 18 to the drug development support device 13. The estimation request 18 is a request for causing the drug development support device 13 to estimate information regarding an appropriate prescription for a preservation solution of the biopharmaceutical 11. The estimation request 18 includes target sequence information 19T of the amino acid residues that make up the antibody 16T. The target sequence information 19T is identified through experiments. Although not shown in the figure, the estimation request 18 also includes a terminal ID (identification data) and the like for uniquely identifying the operator terminal 14 that transmitted the estimation request 18.

[0029] When receiving the estimation request 18, the pharmaceutical development support device 13 uses the protein language model 15 to estimate information regarding an appropriate prescription for a preservative solution of the biopharmaceutical 11. The resulting prescription estimation result 20 is then delivered to the operator terminal 14 that sent the estimation request 18. When receiving the prescription estimation result 20, the operator terminal 14 makes the prescription estimation result 20 available for viewing by the operator OP. The prescription estimation result 20 is an example of an "estimation result" according to the technology of the present disclosure.

[0030] As an example, as shown in Figure 2, the target sequence information 19T describes the order of peptide bonds of the amino acid residues that make up antibody 16T, from the amino terminus to the carboxyl terminus, using single-letter alphabetic abbreviations that represent the amino acid residues. Since antibody 16T contains approximately 450 amino acid residues, the target sequence information 19T also contains a string of approximately 450 letters. Examples of abbreviations include "E" for glutamic acid, "L" for leucine, and "G" for glycine. Such an amino acid residue sequence is also called a primary structure.

[0031] 3, the antibody 16 basically has four polypeptide chains, i.e., two identical heavy chains HC (Heavy Chains) and two identical light chains LC (Light Chains). The antibody 16 has a configuration in which the two heavy chains HC and the two light chains LC are bound by disulfide bonds DB (Disulfide Bonds), and has a symmetrical Y-shape.

[0032] The heavy chain HC is composed of a variable domain VH (Variable domain, Heavy Chain) and constant domains CH (Constant domain, Heavy Chain) 1, CH2, and CH3. The light chain LC is composed of a variable domain VL (Variable domain, Light Chain) and constant domain CL (Constant domain, Light Chain). The constant domains CH1 and CH2 of the heavy chain HC are connected by a hinge region HR (Hinge Region).

[0033] The variable domain VH contains three complementarity-determining regions: CDR-H (Complementary Determining Region, Heavy Chain) 1, CDR-H2, and CDR-H3. The variable domain VL also contains three complementarity-determining regions: CDR-L (Complementary Determining Region, Light Chain) 1, CDR-L2, and CDR-L3. These complementarity-determining regions, CDR-H1 to CDR-H3 and CDR-L1 to CDR-L3, are antigen-binding sites and are also called hypervariable regions.

[0034] The region composed of the variable domains VH and VL and the constant domains CH1 and CL is the fragment antigen-binding region FabR. The region composed of the constant domains CH2 and CH3 and a part of the hinge region HR is the fragment crystallizable region FcR. The region composed of the variable domains VH and VL is the fragment variable region FvR. The region of the fragment variable region FvR excluding the complementarity-determining regions CDR-H1 to CDR-H3 and CDR-L1 to CDR-L3 is called the framework region FW. The fragment variable region FvR is an example of a "variable region" according to the technology of the present disclosure. As is well known, the fragment antigen-binding region FabR and the fragment crystallizable region FcR can be produced by papain digestion. Similarly, the fragment variable region FvR can be produced by pepsin digestion.

[0035] As an example, as shown in FIG. 4 , the variable domain VH has nine polypeptide chains A, B, C1, C2, C3, D, E, F, and G, and the three complementarity-determining regions CDR-H1, CDR-H2, and CDR-H3. Each polypeptide chain A to G has an antiparallel structure in which two adjacent polypeptide chains, such as polypeptide chain A and polypeptide chain B, polypeptide chain C1 and polypeptide chain C2, and polypeptide chain D and polypeptide chain E, are aligned in opposite directions. The complementarity-determining regions CDR-H1, CDR-H2, and CDR-H3 have a loop structure connecting the two polypeptide chains. The complementarity-determining region CDR-H1 connects polypeptide chain B and polypeptide chain C1. The complementarity-determining region CDR-H2 connects polypeptide chain C2 and polypeptide chain C3. The complementarity-determining region CDR-H3 connects polypeptide chain F and polypeptide chain G. Although not shown, the variable domain VL also has a structure similar to that of the variable domain VH.

[0036] Figure 5 shows an example of the amino acid residue sequence in the higher-order structure of the variable domain VH. Numbers such as "1," "15," "85," and "128" added to the left or right of the amino acid residue sequence indicate the position of each amino acid residue in the variable domain VH. The numbers indicating the position of each amino acid residue can be assigned using amino acid residue numbering schemes such as IMGT (The International Immunogenetics Information System), Kabat, AHo, and Chothia.

[0037] 6 , the computers that make up the learning device 12, the drug development support device 13, and the operator terminal 14 basically have the same configuration, and include a storage 25, a memory 26, a CPU (Central Processing Unit) 27, a communication unit 28, a display 29, and an input device 30. These are interconnected via a bus line 31.

[0038] The storage 25 is a hard disk drive built into the computers constituting the learning device 12, the pharmaceutical development support device 13, and the operator terminal 14, or connected via a cable or network. Alternatively, the storage 25 is a disk array consisting of multiple hard disk drives. The storage 25 stores control programs such as an operating system, various application programs (hereinafter referred to as APs (Application Programs)), and various data associated with these programs. Note that a solid state drive may be used instead of a hard disk drive.

[0039] The memory 26 is a work memory for the CPU 27 to execute processing. The CPU 27 loads programs stored in the storage 25 into the memory 26 and executes processing in accordance with the programs. In this way, the CPU 27 comprehensively controls each part of the computer. The CPU 27 is an example of a "processor" according to the technology of the present disclosure. The memory 26 may be built into the CPU 27.

[0040] The communication unit 28 is a network interface that controls the transmission of various information via the network 17, etc. The display 29 displays various screens. The various screens are equipped with operation functions using a GUI (Graphical User Interface). The computers that make up the learning device 12, the pharmaceutical development support device 13, and the operator terminal 14 accept input of operation instructions from the input device 30 via the various screens. The input device 30 is a keyboard, a mouse, a touch panel, a microphone for voice input, etc.

[0041] In the following explanation, the parts of the computer that make up the learning device 12 (storage 25 and CPU 27) are distinguished by adding the suffix "A" to their symbols, the parts of the computer that make up the pharmaceutical development support device 13 (storage 25 and CPU 27) are distinguished by adding the suffix "B" to their symbols, and the parts of the computer that make up the operator terminal 14 (storage 25, CPU 27, display 29, and input device 30) are distinguished by adding the suffix "C" to their symbols.

[0042] 7, an operating program 35 is stored in the storage 25A of the learning device 12. The operating program 35 is an AP for causing a computer to function as the learning device 12. In other words, the operating program 35 is an example of an "operating program for a learning device" according to the technology of the present disclosure. The storage 25A also stores a protein language model 15, a past data group 36, a learning data group 37, and the like.

[0043] When the operating program 35 is started, the CPU 27A of the computer constituting the learning device 12 works in cooperation with the memory 26, etc. to function as a read / write (hereinafter abbreviated as RW (Read Write)) control unit 40, a generation unit 41, a pre-learning unit 42, and an FT unit 43.

[0044] The RW control unit 40 controls the storage of various data in the storage 25A and the reading of various data from the storage 25A. For example, the RW control unit 40 reads the past data group 36 from the storage 25A and outputs the read past data group 36 to the generation unit 41. The RW control unit 40 also reads a first training data group 371 of the training data group 37 from the storage 25A and outputs the read first training data group 371 to the pre-training unit 42. Similarly, the RW control unit 40 reads a second training data group 372 of the training data group 37 from the storage 25A and outputs the read second training data group 372 to the FT unit 43. The RW control unit 40 also reads the protein language model 15 from the storage 25A and outputs the protein language model 15 to the pre-training unit 42 or the FT unit 43.

[0045] The generation unit 41 generates a training data group 37, which is a collection of training data for training the protein language model 15, from the past data group 36. The generation unit 41 outputs the generated training data group 37 to the RW control unit 40. The RW control unit 40 stores the training data group 37 in the storage 25A.

[0046] The pre-training unit 42 performs pre-training on the protein language model 15 using the first training data group 371. The pre-training unit 42 outputs the protein language model 15 after pre-training to the RW control unit 40. The RW control unit 40 stores the protein language model 15 after pre-training in the storage 25A. In the following description, the protein language model 15 before pre-training will be referred to as the protein language model 15A, and the protein language model 15 after pre-training will be referred to as the protein language model 15B.

[0047] The FT unit 43 performs FT on the protein language model 15B using the second training data group 372. The FT unit 43 outputs the protein language model 15B after FT to the RW control unit 40. The RW control unit 40 stores the protein language model 15B after FT in the storage 25A. In the following description, the protein language model 15B after FT will be referred to as the protein language model 15C. This protein language model 15C is transmitted from the learning device 12 to the pharmaceutical development support device 13. In addition to these processing units 40 to 43, the CPU 27A also includes an instruction receiving unit that receives various operation instructions from the input device 30.

[0048] As an example, as shown in Figure 8, the past data group 36 is a collection of past data 45 for multiple biopharmaceuticals 11 developed in the past. Each piece of past data 45 is identified by the antibody ID of the antibody 16R contained in the biopharmaceutical 11 developed in the past. The past data 45 may be obtained from a public database of biopharmaceuticals 11, or may be obtained from a biopharmaceutical 11 developed in the past by a pharmaceutical company or a contract research organization.

[0049] The past data 45 includes reference sequence information 19R, reference structure information 46R, and reference prescription information 47R. Like the target sequence information 19T, the reference sequence information 19R describes the order of peptide bonds of the amino acid residues constituting antibody 16R from the amino terminus to the carboxyl terminus using single-letter alphabetic abbreviations representing the amino acid residues. The reference structure information 46R is information indicating the structure to which the amino acid residues constituting antibody 16R belong (see FIG. 10 ). The structure here refers to any of the polypeptide chains A to G (antiparallel structures) of the fragment variable region FvR and the complementarity determining regions CDR-H1 to CDR-H3 and CDR-L1 to CDR-L3 (loop structures). In other words, the structure includes the structure of the fragment variable region FvR of antibody 16. In other words, the structure includes a structure represented by positional information in a three-dimensional conformation of atoms constituting the amino acid residues of antibody 16 (the above-mentioned polypeptide chains A to G, and the complementarity determining regions CDR-H1 to CDR-H3, CDR-L1 to CDR-L3). Furthermore, the structure includes a framework region FW having the amino acid residues of antibody 16, or the complementarity determining regions CDR-H1 to CDR-H3, CDR-L1 to CDR-L3. The structure also includes secondary structures such as chain structures having the amino acid residues of antibody 16 (the above-mentioned polypeptide chains A to G, also referred to as strand structures) or loop structures (the above-mentioned complementarity determining regions CDR-H1 to CDR-H3, CDR-L1 to CDR-L3). The secondary structure may be a turn structure in addition to or instead of the chain structure and / or loop structure. The reference structure information 46R is generated using a numbering scheme such as the above-mentioned IMGT. The reference prescription information 47R is information on the prescription adopted for the preservation solution of the biopharmaceutical 11, and here, the hydrogen ion exponent (pH (Potential Of Hydrogen) value) of the preservation solution is shown as an example.

[0050] 9 , the generation unit 41 first randomly divides the multiple pieces of past data 45 in the past data group 36 into a first past data group 361 and a second past data group 362. In this case, the generation unit 41 divides the multiple pieces of past data 45 in the past data group 36 so that the first past data group 361 has a larger number of pieces of past data 45 than the second past data group 362. For example, 80% of the multiple pieces of past data 45 in the past data group 36 are divided into the first past data group 361 and the remaining 20% ​​are divided into the second past data group 362. Note that the past data 45 in the first past data group 361 and the past data 45 in the second past data group 362 may partially overlap.

[0051] The generation unit 41 generates a first learning data group 371 from the first past data group 361. More specifically, the generation unit 41 generates the first learning data group 371 by performing a masking process on the past data 45 of the first past data group 361. The generation unit 41 also generates a second learning data group 372 from the second past data group 362. The first learning data group 371 and the second learning data group 372 constitute the learning data group 37.

[0052] As an example, as shown in Table 50 in FIG. 10 , masking is a process in which each alphabet in the reference sequence information 19R is considered as a single token and the alphabet is masked. The masking is performed according to preset masking conditions. The masking conditions are such that, for example, 15% of the amino acid residues in the reference sequence information 19R are masked. Furthermore, the masking conditions are such that a mask is set across the boundary between two adjacent structures. Setting a mask across the boundary between two adjacent structures means setting a mask on at least two amino acid residues present at the boundary between the two adjacent structures. FIG. 10 shows an example in which a mask is set across the boundary between polypeptide chain A and polypeptide chain B, the boundary between complementarity determining region CDR-H1 and polypeptide chain C1, and the boundary between complementarity determining region CDR-H3 and polypeptide chain G.

[0053] 11 , the first learning data 551 constituting the first learning data group 371 includes the reference sequence information 19R after the masking process and the mask setting information 56. The mask setting information 56 is information in which a pair of the reference sequence information 19R masked by the masking process and the structure to which the masked amino acid residue belongs is registered. The mask setting information 56 is information that serves as the answer to the pre-learning.

[0054] 12 , second learning data 552 constituting second learning data group 372 includes reference sequence information 19R and reference prescription information 47R. Unlike first learning data 551, reference sequence information 19R of second learning data 552 is information that has not been subjected to masking. Reference prescription information 47R is information that provides the answer to FT.

[0055] 13 , as an example, the pre-training unit 42 performs first and second pre-training in parallel using a first training data group 371. In the first pre-training, masked amino acid residues are estimated. In the second pre-training, the structure to which the masked amino acid residues belong is estimated. By performing the first and second pre-training in parallel in this manner, the pre-training unit 42 converts the protein language model 15A into the protein language model 15B.

[0056] As an example, as shown in FIG. 14 , the pre-training unit 42 inputs the masked reference sequence information 19R from the first training data 551 into the protein language model 15A, causing the protein language model 15A to output a residue inference result 60 and a structure inference result 61. The residue inference result 60 is the result of inferring the identity of the masked amino acid residue in the reference sequence information 19R. The structure inference result 61 is the result of inferring the identity of the structure to which the masked amino acid residue in the reference sequence information 19R belongs. In this way, the pre-training unit 42 performs training in which, in addition to the MLM, the structure to which the amino acid residue belongs is considered as an entity label in entity recognition for the natural language processing task. Based on the residue inference result 60, the structure inference result 61, and the mask setting information 56, the pre-training unit 42 performs loss calculation for the protein language model 15A using a loss function. Then, the pre-training unit 42 updates the values ​​of various parameters of the protein language model 15A according to the results of the loss calculation, and updates the protein language model 15A according to the update settings.

[0057] The pre-training unit 42 repeatedly performs the above series of processes, including inputting the masked reference sequence information 19R to the protein language model 15A, outputting the residue estimation results 60 and the structure estimation results 61 from the protein language model 15A, calculating the loss, setting the update, and updating the protein language model 15A, while changing the first training data 551, until the estimation accuracy of the residue estimation results 60 and the structure estimation results 61 reaches a predetermined level. Alternatively, the pre-training unit 42 ends the above series of processes when it has been repeated a predetermined number of times, regardless of the estimation accuracy of the residue estimation results 60 and the structure estimation results 61.

[0058] 15 , the FT unit 43 inputs reference sequence information 19R from the second training data 552 to the protein language model 15B and causes the protein language model 15B to output a training prescription estimation result 20L. The FT unit 43 performs loss calculation for the protein language model 15B using a loss function based on the training prescription estimation result 20L and the reference prescription information 47R. Then, the FT unit 43 updates the values ​​of various parameters of the protein language model 15B according to the results of the loss calculation and updates the protein language model 15B according to the update settings.

[0059] The FT unit 43 repeatedly performs the above series of processes, including inputting the reference sequence information 19R to the protein language model 15B, outputting the training prescription estimation result 20L from the protein language model 15B, calculating the loss, updating the setting, and updating the protein language model 15B, while changing the second training data 552, until the estimation accuracy of the training prescription estimation result 20L reaches a predetermined level. Alternatively, the FT unit 43 terminates the above series of processes after a predetermined number of repetitions, regardless of the estimation accuracy of the training prescription estimation result 20L. The protein language model 15B whose estimation accuracy of the training prescription estimation result 20L has thus reached a predetermined level, or the protein language model 15B for which the above series of processes has been repeated a predetermined number of times, is transmitted from the learning device 12 to the drug development support device 13 as the protein language model 15C after FT.

[0060] The reference sequence information 19R is converted into vector data in the protein language model 15. The vector data is data in which the alphabet of the reference sequence information 19R is expressed as a multidimensional, for example, 64-dimensional vector.

[0061] 16 , an operating program 65 is stored in storage 25B of drug development support device 13. Operating program 65 is an AP for causing a computer to function as drug development support device 13. In other words, operating program 65 is an example of an "operating program for a drug development support device" according to the technology of the present disclosure. Storage 25B also stores a protein language model 15C, target sequence information 19T, and the like.

[0062] When the operating program 65 is started, the CPU 27B of the computer constituting the pharmaceutical development support device 13 works in cooperation with the memory 26 etc. to function as a request receiving unit 70, a RW control unit 71, an estimation unit 72, and a screen distribution control unit 73.

[0063] The request receiving unit 70 receives various requests from the operator terminal 14. In particular, the request receiving unit 70 receives an estimation request 18 from the operator terminal 14. When the estimation request 18 is received, the request receiving unit 70 outputs the target array information 19T included in the estimation request 18 to the RW control unit 71. In addition, the request receiving unit 70 outputs the terminal ID of the operator terminal 14 included in the estimation request 18 to the screen distribution control unit 73.

[0064] The RW control unit 71 controls the storage of various data in the storage 25B and the reading of various data from the storage 25B. For example, the RW control unit 71 stores the protein language model 15C transmitted from the learning device 12 in the storage 25B. The RW control unit 71 stores the target sequence information 19T from the request receiving unit 70 in the storage 25B. The RW control unit 71 also reads the target sequence information 19T from the storage 25B and outputs the read target sequence information 19T to the estimation unit 72. The RW control unit 71 also reads the protein language model 15C from the storage 25B and outputs the read protein language model 15C to the estimation unit 72.

[0065] 17 as an example, the estimation unit 72 inputs the target sequence information 19T to the protein language model 15C and causes the protein language model 15C to output a prescription estimation result 20. The estimation unit 72 outputs the prescription estimation result 20 to the screen delivery control unit 73. Here, the prescription estimation result 20 is the hydrogen ion exponent of the preservative solution.

[0066] The screen delivery control unit 73 controls the delivery of various screens to the operator terminal 14. Specifically, the screen delivery control unit 73 delivers and outputs various screens to the operator terminal 14 that has sent the various requests in the form of screen data for web delivery created using a markup language such as XML (Extensible Markup Language). At this time, the screen delivery control unit 73 identifies the operator terminal 14 that has sent the various requests based on the terminal ID from the request receiving unit 70. Note that instead of XML, other data description languages ​​such as JSON (Javascript (registered trademark) Object Notation) may be used.

[0067] The various screens include an information input screen 80 (see FIG. 19) for inputting the target sequence information 19T, and an estimation result display screen 85 (see FIG. 20) for displaying the prescription estimation result 20. In addition to these processing units 70 to 73, the CPU 27B also includes an instruction receiving unit that receives various operation instructions from the input device 30.

[0068] As an example, as shown in Figure 18, an estimated AP 75 is stored in the storage 25C of the operator terminal 14. The estimated AP 75 is installed in the operator terminal 14 by the operator OP. The estimated AP 75 is an AP for estimating information regarding an appropriate prescription for a preservative solution of the biopharmaceutical 11. When the estimated AP 75 is activated, the CPU 27C of the operator terminal 14 functions as a browser control unit 77 in cooperation with the memory 26 and the like. The browser control unit 77 controls the operation of a dedicated web browser for the estimated AP 75.

[0069] The browser control unit 77 reproduces various screens based on various screen data from the drug development support device 13 and displays the reproduced various screens on the display 29C. The browser control unit 77 also accepts various operation instructions input by the operator OP from the input device 30C via the various screens. The browser control unit 77 transmits various requests, including the estimation request 18, to the drug development support device 13 in response to the operation instructions.

[0070] When the estimation AP 75 is started, an information input screen 80 shown in Fig. 19 as an example is displayed on the display 29C under the control of the browser control unit 77. The information input screen 80 is provided with an input box 81 for the target sequence information 19T. The input box 81 allows the user to write the target sequence information 19T or drop a file of the target sequence information 19T.

[0071] The operator OP inputs the desired target sequence information 19T into the input box 81, and then selects the estimate button 82. When the estimate button 82 is selected, the browser control unit 77 generates an estimation request 18 including the target sequence information 19T input into the input box 81, and transmits the generated estimation request 18 to the drug development support device 13.

[0072] Furthermore, when the pharmaceutical development support device 13 estimates information regarding an appropriate prescription for a preservative solution of the biopharmaceutical 11, an estimation result display screen 85 shown in FIG. 20 is displayed on the display 29C under the control of the browser control unit 77. The estimation result display screen 85 displays the prescription estimation result 20, or more precisely, a message representing the prescription estimation result 20. In this way, the prescription estimation result 20 is presented to the operator OP in the form of screen data distribution.

[0073] A sequence information display button 86 is provided at the top of the estimation result display screen 85. When the sequence information display button 86 is selected, a display screen for the target sequence information 19T is popped up. Furthermore, a save button 87 and an OK button 88 are provided at the bottom of the estimation result display screen 85. When the save button 87 is selected, the target sequence information 19T and the prescription estimation result 20 are stored in association with each other in the storage 25C of the operator terminal 14. When the OK button 88 is selected, the display of the estimation result display screen 85 is cleared.

[0074] Next, the operation of the above configuration will be described with reference to the flowcharts shown in Figures 21 and 22 as an example. When the operating program 35 is launched in the learning device 12, the CPU 27A of the learning device 12 functions as the RW control unit 40, the generation unit 41, the pre-learning unit 42, and the FT unit 43, as shown in Figure 7. When the operating program 65 is launched in the drug development support device 13, the CPU 27B of the drug development support device 13 functions as the request receiving unit 70, the RW control unit 71, the estimation unit 72, and the screen distribution control unit 73, as shown in Figure 16. When the estimation AP 75 is launched in the operator terminal 14, the CPU 27C of the operator terminal 14 functions as the browser control unit 77, as shown in Figure 18.

[0075] 21 , in the learning device 12, as shown in FIGS. 9 to 12 , the generation unit 41 generates a learning data group 37 from the past data group 36 (step ST100). Specifically, a first learning data group 371 is generated, which is composed of a plurality of first learning data 551 including reference sequence information 19R in which a mask is set across the boundary between two adjacent structures and mask setting information 56. Also, a second learning data group 372 is generated, which is composed of a plurality of second learning data 552 including reference sequence information 19R and reference prescription information 47R. The learning data group 37 is output from the generation unit 41 to the RW control unit 40 and stored in the storage 25A under the control of the RW control unit 40.

[0076] The RW control unit 40 reads the protein language model 15A and the first training data group 371 from the storage 25A and outputs the read protein language model 15A and the first training data group 371 to the pre-training unit 42. As shown in FIGS. 13 and 14 , the pre-training unit 42 pre-trains the protein language model 15A using the first training data group 371 (step ST110). More specifically, first pre-training, which estimates masked amino acid residues, and second pre-training, which estimates structures to which the masked amino acid residues belong, are performed in parallel. The pre-trained protein language model 15B is output from the pre-training unit 42 to the RW control unit 40 and stored in the storage 25A under the control of the RW control unit 40.

[0077] The RW control unit 40 reads the protein language model 15B and the second training data group 372 from the storage 25A and outputs the read protein language model 15B and second training data group 372 to the FT unit 43. As shown in FIG. 15 , the FT unit 43 performs FT on the protein language model 15B using the second training data group 372 (step ST120). The protein language model 15C after FT is output from the FT unit 43 to the RW control unit 40 and stored in the storage 25A under the control of the RW control unit 40. The protein language model 15C is then transmitted from the learning device 12 to the drug development support device 13 and stored in the storage 25B of the drug development support device 13.

[0078] 19 is displayed on the display 29C of the operator terminal 14 under the control of the browser control unit 77. When the operator OP inputs desired target sequence information 19T into an input box 81 on the information input screen 80 and selects an estimate button 82, an estimate request 18 is sent from the browser control unit 77 to the drug development support device 13. As shown in FIG. 1, the estimate request 18 includes the target sequence information 19T, the terminal ID of the operator terminal 14, and the like.

[0079] 22 , in the pharmaceutical development support device 13, the request receiving unit 70 receives the estimation request 18 (YES in step ST200). The target sequence information 19T included in the estimation request 18 is output from the request receiving unit 70 to the RW control unit 71 and stored in the storage 25B under the control of the RW control unit 71 (step ST210). In addition, the terminal ID of the operator terminal 14 included in the estimation request 18 is output from the request receiving unit 70 to the screen distribution control unit 73.

[0080] The RW control unit 71 reads the target sequence information 19T from the storage 25B (step ST220). The RW control unit 71 outputs the target sequence information 19T to the estimation unit 72. The RW control unit 71 also reads the protein language model 15C from the storage 25B and outputs the read protein language model 15C to the estimation unit 72.

[0081] 17, in the estimation unit 72, the target sequence information 19T is input to the protein language model 15C. As a result, the prescription estimation result 20 is output from the protein language model 15C (step ST230). The prescription estimation result 20 is output from the estimation unit 72 to the screen delivery control unit 73.

[0082] The screen distribution control unit 73 generates screen data for the estimation result display screen 85 shown in Fig. 20 based on the prescription estimation result 20. Under the control of the screen distribution control unit 73, the screen data for the estimation result display screen 85 is distributed to the operator terminal 14 that is the sender of the estimation request 18 (step ST240).

[0083] In the operator terminal 14, under the control of the browser control unit 77, the screen data of the estimation result display screen 85 is reproduced, and the reproduced estimation result display screen 85 is displayed on the display 29C. In this way, the prescription estimation result 20 is presented to the operator OP.

[0084] As described above, the CPU 27A of the learning device 12 includes a pre-learning unit 42. The pre-learning unit 42 performs a first pre-learning process on the protein language model 15 to estimate masked amino acid residues in the reference sequence information 19R of the amino acid residues constituting the antibody 16R, and a second pre-learning process to estimate the structure to which the amino acid residues belong. Therefore, compared to the technique described in Non-Patent Document 1, which simply includes information indicating which complementarity-determining region the masked amino acid residues belong to in the learning input data, the learning process is more sufficient for learning the structural information of the antibody 16. Therefore, the estimation accuracy of the protein language model 15 can be improved compared to the technique described in Non-Patent Document 1.

[0085] 15 , after the first and second pre-learning are completed, the FT unit 43 performs FT according to a desired task on the protein language model 15. This makes it possible to obtain a protein language model 15 with high estimation accuracy that can handle the desired task.

[0086] As shown in FIG. 17 , the task is to estimate information about the prescription of a preservation solution for the biopharmaceutical 11, in this case, the pH of the preservation solution. To stably maintain the quality of the biopharmaceutical 11, it is important to ensure that the prescription of the preservation solution is appropriate for the biopharmaceutical 11. Therefore, if the task of the protein language model 15 is to estimate information about the prescription of a preservation solution for the biopharmaceutical 11, it can contribute to stably maintaining the quality of the biopharmaceutical 11. Note that the prescription information may include, instead of or in addition to the pH, the temperature of the preservation solution, the type of additive added to the preservation solution, or the concentration of the additive. Furthermore, the subject of estimation is not limited to information about the prescription of the preservation solution for the biopharmaceutical 11. For example, the degree of aggregation of antibody 16T may also be used.

[0087] As shown in Figure 13, the pre-training unit 42 performs second pre-training on masked amino acid residues. This allows for effective second pre-training. As a result, the estimation accuracy of the protein language model 15 can be further improved. Note that the second pre-training may be performed on unmasked amino acid residues instead of or in addition to the masked amino acid residues.

[0088] If a mask is set randomly without spanning the boundary between two adjacent structures, the amino acid residues on either side of the masked amino acid residue may not be masked. In such a case, the structure to which the masked amino acid residue belongs can be easily estimated based on the amino acid residues on either side, making it impossible to perform effective second pre-learning. Therefore, as shown in FIG. 10 , the generation unit 41 sets a mask across the boundary between two adjacent structures. This makes it difficult to easily estimate the structure to which the masked amino acid residue belongs, thereby enabling effective second pre-learning. As a result, the estimation accuracy of the protein language model 15 can be further improved.

[0089] 13 and 14, the pre-learning unit 42 performs the first pre-learning and the second pre-learning in parallel. This reduces the time required for the pre-learning. Note that the first pre-learning and the second pre-learning may be performed separately, such as by performing the second pre-learning after the first pre-learning.

[0090] Pharmaceuticals containing antibody 16T as a protein are called antibody drugs, and are widely used to treat chronic diseases such as cancer, diabetes, and rheumatoid arthritis, as well as rare diseases such as hemophilia and Crohn's disease. Therefore, this example, in which antibody 16T is used as the protein, can further promote the development of antibody drugs that are widely used to treat various diseases.

[0091] The fragment variable region FvR of antibody 16 is an extremely important region because it is the antigen-binding site and contains complementarity-determining regions (CDRs) that determine the characteristics of the antibody. Therefore, if the structure estimated in the second pre-learning is the structure of the fragment variable region FvR of antibody 16, it is possible to generate a protein language model 15 that better reflects the characteristics of the antibody. As a result, the estimation accuracy of protein language model 15 can be further improved.

[0092] Similarly, if the structure estimated in the second pre-learning is a structure in which atoms constituting the amino acid residues of antibody 16 are represented by positional information in a three-dimensional structure, a framework region FW having the amino acid residues of antibody 16, or complementarity determining regions CDR-H1 to CDR-H3, CDR-L1 to CDR-L3, or a chain structure, loop structure, or turn structure having the amino acid residues of antibody 16, it is possible to generate a protein language model 15 that better reflects the characteristics of the antibody. As a result, the estimation accuracy of protein language model 15 can be further improved.

[0093] The CPU 27B of the pharmaceutical development support device 13 includes an estimation unit 72. The estimation unit 72 uses a protein language model 15 that has undergone first pre-learning to estimate masked amino acid residues in reference sequence information 19R of amino acid residues that constitute antibody 16R, and second pre-learning to estimate the structure to which the amino acid residues belong. The estimation unit 72 inputs target sequence information 19T of amino acid residues that constitute antibody 16T to the protein language model 15, and causes the protein language model 15 to output a prescription estimation result 20. This makes it possible to improve the estimation accuracy of the protein language model 15.

[0094] The protein is not limited to the exemplified antibody 16T. It may also be a cytokine (interferon, interleukin, etc.), a hormone (insulin, glucagon, follicle-stimulating hormone, erythropoietin, etc.), a growth factor (IGF (insulin-like growth factor)-1, bFGF (basic fibroblast growth factor), etc.), a blood coagulation factor (factor VII, factor VIII, factor IX, etc.), an enzyme (lysosomal enzyme, DNA (deoxyribonucleic acid) degrading enzyme, etc.), an Fc (fragment crystalline) fusion protein, a receptor, albumin, or a protein vaccine. Antibodies also include bispecific antibodies, antibody-drug conjugates, low molecular weight antibodies, glycosylated antibodies, and the like.

[0095] Furthermore, substances derived from amino acids are not limited to proteins. For example, as shown in Figure 23, they may be peptides 90 or nucleic acids 91. Therefore, pharmaceuticals are not limited to biopharmaceuticals 11 that require biotechnology such as genetic recombination technology and cell culture technology, but may be peptide pharmaceuticals 92 or nucleic acid pharmaceuticals 93 that can be produced using only chemical synthesis technology without requiring biotechnology.

[0096] The protein language model 15C may continue to be trained even after being stored in the storage 25B of the drug development support device 13.

[0097] Although an example has been shown in which the learning data group 37 is generated by the generation unit 41 of the learning device 12, this is not limiting. The learning data group 37 may be generated by a device other than the learning device 12 and transmitted to the learning device 12 from that device. Also, some or all of the functions of the learning device 12 may be performed by the drug development support device 13. Similarly, some or all of the functions of the drug development support device 13 may be performed by the operator terminal 14.

[0098] The learning device 12 and the drug development support device 13 may be installed in a pharmaceutical company or a drug development contract organization, or may be installed in a data center independent of the pharmaceutical company or the drug development contract organization.

[0099] Instead of delivering screen data of the estimation result display screen 85 including the prescription estimation result 20 to the operator terminal 14, the prescription estimation result 20 itself may be delivered to the operator terminal 14. In this case, the operator terminal 14 generates the estimation result display screen 85 based on the prescription estimation result 20 under the control of the browser control unit 77.

[0100] The method of presenting the prescription estimation result 20 to the operator OP is not limited to the example of delivering screen data. The prescription estimation result 20 may be presented to the operator OP by printing it on a paper medium, or by attaching it to an email and sending it to the operator terminal 14.

[0101] The hardware configuration of the computers constituting the learning device 12 and the drug development support device 13 according to the technology of the present disclosure can be modified in various ways. For example, the learning device 12 and the drug development support device 13 can be configured as multiple computers separated as hardware to improve processing power and reliability. For example, the functions of the RW control unit 40 and the generation unit 41 and the functions of the pre-learning unit 42 and the FT unit 43 can be distributed across two computers. In this case, the learning device 12 is configured with two computers. Alternatively, the functions of the request receiving unit 70 and the RW control unit 71 and the functions of the estimation unit 72 and the screen distribution control unit 73 can be distributed across two computers. In this case, the drug development support device 13 is configured with two computers.

[0102] In this way, the hardware configuration of the computers of learning device 12 and drug development support device 13 can be changed as appropriate depending on the required performance, such as processing power, safety, and reliability. Furthermore, not only the hardware, but also APs such as operating programs 35 and 65 can be duplicated or stored in multiple storage devices in order to ensure safety and reliability.

[0103] In the above embodiment, for example, the hardware structure of the processing units that perform various processes, such as the RW control units 40 and 71, the generation unit 41, the pre-learning unit 42, the FT unit 43, the request reception unit 70, the estimation unit 72, the screen distribution control unit 73, and the browser control unit 77, can be any of the various processors shown below. As described above, the various processors include CPUs 27A, 27B, and 27C, which are general-purpose processors that execute software (operating programs 35 and 65, and estimated AP 75) and function as various processing units, as well as programmable logic devices (PLDs) that are processors whose circuit configuration can be changed after manufacture, such as FPGAs (Field Programmable Gate Arrays), and dedicated electrical circuits that are processors having a circuit configuration designed specifically for executing specific processing, such as ASICs (Application Specific Integrated Circuits).

[0104] A single processing unit may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (e.g., a combination of multiple FPGAs and / or a combination of a CPU and an FPGA).Furthermore, multiple processing units may be configured with a single processor.

[0105] Examples of configuring multiple processing units with a single processor include, first, a form in which one processor is configured with a combination of one or more CPUs and software, as typified by computers such as client and server, and this processor functions as multiple processing units. Second, a form in which a processor is used to realize the functions of the entire system including multiple processing units with a single IC (Integrated Circuit) chip, as typified by systems on chips (SoCs). In this way, various processing units are configured using one or more of the above-mentioned various processors as a hardware structure.

[0106] Furthermore, more specifically, the hardware structure of these various processors can be an electric circuit (circuitry) that combines circuit elements such as semiconductor elements.

[0107] From the above description, the technology described in the following supplementary paragraphs can be understood.

[0108] [Supplementary Item 1] A learning device for a substance language model to which sequence information of amino acid residues constituting a substance derived from amino acids contained in a pharmaceutical is applied, the learning device comprising a processor, wherein the processor performs, on the substance language model, first pre-learning to estimate masked amino acid residues in reference sequence information of amino acid residues constituting a reference substance, and second pre-learning to estimate a structure to which the amino acid residues belong. [Supplementary Item 2] The learning device according to Supplementary Item 1, wherein the processor fine-tunes the substance language model according to a desired task after completing the first pre-learning and the second pre-learning. [Supplementary Item 3] The learning device according to Supplementary Item 2, wherein the task is a task of estimating information related to a prescription for a preservative solution of the pharmaceutical. [Supplementary Item 4] The learning device according to any one of Supplementary Item 1 to Supplementary Item 3, wherein the processor performs the second pre-learning on masked amino acid residues. [Supplementary Item 5] The learning device according to Supplementary Item 4, wherein the processor performs the second pre-learning on masked amino acid residues. [Supplementary Item 6] The learning device according to Supplementary Item 4 or Supplementary Item 5, wherein the processor performs the first pre-learning and the second pre-learning in parallel. [Supplementary Item 7] The learning device according to any one of Supplementary Item 1 to Supplementary Item 6, wherein the substance is any one of a protein, a peptide, and a nucleic acid. [Supplementary Item 8] The learning device according to Supplementary Item 7, wherein the protein is an antibody. [Supplementary Item 9] The learning device according to Supplementary Item 8, wherein the structure includes a structure in a variable region of the antibody. [Supplementary Item 10] The learning device according to Supplementary Item 9, wherein the structure includes a structure in which atoms constituting the amino acid residues of the antibody are represented by positional information in a three-dimensional conformation. [Supplementary Item 11] The learning device according to Supplementary Item 9 or Supplementary Item 10, wherein the structure includes a framework region or a complementarity determining region of the antibody having the amino acid residues. [Supplementary Item 12] The learning device according to any one of Supplementary Item 9 to Supplementary Item 11, wherein the structure includes a chain structure, a loop structure, or a turn structure having the amino acid residues of the antibody.

[0109] The technology of the present disclosure can be appropriately combined with the various embodiments and / or various modified examples described above. Furthermore, it is not limited to the above embodiments, and various configurations can be adopted without departing from the spirit of the present disclosure. Furthermore, the technology of the present disclosure extends not only to programs, but also to storage media that non-temporarily store programs, and computer program products that include programs.

[0110] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[0111] In this specification, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0112] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

Claims

1. A learning device for a substance language model to which sequence information of amino acid residues constituting substances derived from amino acids contained in pharmaceuticals is applied, the learning device comprising a processor, wherein the processor performs a first pre-learning on the substance language model to infer masked amino acid residues from reference sequence information of amino acid residues constituting a reference substance, and a second pre-learning on the substance language model to infer the structure to which the amino acid residues belong.

2. The learning device according to claim 1, wherein the processor performs fine tuning of the material language model according to a desired task after the first pre-learning and the second pre-learning are completed.

3. The learning device according to claim 2, wherein the task is a task of estimating information relating to a prescription for a preservative solution of a medicine.

4. The learning device according to claim 1, wherein the processor performs the second pre-learning on masked amino acid residues.

5. The learning device according to claim 4, wherein the processor sets a mask across the boundary between two adjacent structures.

6. The learning device according to claim 4, wherein the processor performs the first pre-learning and the second pre-learning in parallel.

7. The learning device according to claim 1, wherein the substance is one of a protein, a peptide, and a nucleic acid.

8. The learning device according to claim 7, wherein the protein is an antibody.

9. The learning device according to claim 8, wherein the structure includes a structure in the variable region of the antibody.

10. A learning device according to claim 9, wherein the structure includes a structure in which atoms constituting the amino acid residues of the antibody are represented by positional information in a three-dimensional conformation.

11. The learning device according to claim 9, wherein the structure includes a framework region or a complementarity determining region having the amino acid residues of the antibody.

12. The learning device according to claim 9, wherein the structure includes a chain structure, a loop structure, or a turn structure having the amino acid residues of the antibody.

13. A method for operating a learning device for a substance language model to which sequence information of amino acid residues constituting substances derived from amino acids contained in pharmaceuticals is applied, the method comprising: performing, on the substance language model, a first pre-learning step to infer masked amino acid residues from reference sequence information of amino acid residues constituting a reference substance, and a second pre-learning step to infer the structure to which the amino acid residues belong.

14. An operating program for a learning device for a substance language model to which sequence information of amino acid residues constituting substances derived from amino acids contained in pharmaceuticals is applied, the operating program for a learning device causing a computer to execute processes including: performing, on the substance language model, a first pre-learning to infer masked amino acid residues from reference sequence information of amino acid residues constituting a reference substance; and a second pre-learning to infer the structure to which the amino acid residues belong.

15. A pharmaceutical development support device that causes a substance language model to which sequence information of amino acid residues constituting a substance derived from amino acids contained in a pharmaceutical is applied to make an inference regarding the substance, the device comprising a processor, wherein the processor uses the substance language model that has undergone first pre-training to infer masked amino acid residues in reference sequence information of amino acid residues constituting a reference substance and second pre-training to infer a structure to which the amino acid residues belong, inputs target sequence information of amino acid residues constituting a substance to be inferred to the substance language model, and causes the substance language model to output an inference result.

16. A method for operating a pharmaceutical development support device that causes a substance language model to which sequence information of amino acid residues constituting a substance derived from amino acids contained in a pharmaceutical is applied to make an inference regarding the substance, the method comprising: using the substance language model that has undergone first pre-training to infer masked amino acid residues in reference sequence information of amino acid residues constituting a reference substance, and second pre-training to infer a structure to which the amino acid residues belong; and inputting target sequence information of amino acid residues constituting a substance to be inferred into the substance language model, and outputting an inference result from the substance language model.

17. An operating program for a pharmaceutical development support device that causes a substance language model to be applied with sequence information of amino acid residues that constitute a substance derived from amino acids contained in a pharmaceutical product, to make an inference regarding said substance, wherein the operating program causes a computer to execute processes including: using the substance language model that has undergone first pre-training to infer masked amino acid residues in reference sequence information of amino acid residues that constitute a reference substance, and second pre-training to infer the structure to which the amino acid residues belong; and inputting target sequence information of amino acid residues that constitute a substance to be inferred into the substance language model, and outputting the inference results from the substance language model.

Images