Polymer molecular structure recognition system, method, and program using artificial intelligence

Abstract

A system, method, and program for recognizing polymer molecular structure formulas using artificial intelligence are disclosed. The system for recognizing polymer molecular structure formulas using artificial intelligence includes at least one processor and at least one memory for storing instructions or information for causing the at least one processor to perform operations, the operations performed by the instructions or information including: detecting an image of a polymer molecular structure formula to generate detection data including information about brackets and subscripts; inputting the detection data into a first model and a second model, respectively; outputting first cluster data from the first model; and outputting second cluster data from the second model, the second cluster data including group information about the brackets and subscripts and including information different from the first cluster data.

Description

[Technical Field]

[0001] The present invention relates to a polymer molecular structure recognition system, method, and program, and more particularly to a system, method, and program that can recognize a molecular structure from a polymer molecular structure image using artificial intelligence. [Background technology]

[0002] A structural formula is a graphical representation of a chemical or molecular structure. A structural formula can show how atoms are arranged in three-dimensional space. It can also show the chemical bonds of a molecule, either explicitly or implicitly.

[0003] A polymer is a substance with a high molecular weight, composed of many low-molecular-weight units (monomers). When expressing a polymer in molecular structure, the number of times the unit is polymerized (subscripts such as n and m) is added to the repeating units.

[0004] Meanwhile, in various documents and papers, molecular structures are provided in the form of images. Because molecular structures are provided in the form of images, it is difficult to identify or select them using a general search. In particular, polymers can be copolymers, which are made up of two or more types of units, so it is necessary to recognize images of polymer molecular structures. Summary of the Invention [Problem to be solved by the invention]

[0005] The problem to be solved by the present invention is to provide a system, method and program capable of selecting and recognizing necessary information from a polymer molecular structure image and recognizing an accurate polymer molecular structure.

[0006] The problems that the present invention aims to solve are not limited to the problems described above, and other problems not mentioned will also be clearly understood by those skilled in the art from the following description. [Means for solving the problem]

[0007] The system includes at least one or more processors and at least one or more memories storing instructions or information for causing the at least one or more processors to perform operations, wherein the operations performed by the instructions or information include detecting a polymer molecular structure image to generate detection data including information about brackets and subscripts, and inputting the detection data into a first model and a second model, respectively, to output first cluster data from the first model, and to output second cluster data from the second model, the second cluster data including group information about the brackets and subscripts and including information different from the first cluster data.

[0008] The polymer may contain two or more different monomers.

[0009] The first cluster data may include at least one of class information and coordinate information of the brackets and subscripts.

[0010] The second model may include a metric function.

[0011] In various embodiments, detecting the polymer molecular structure image to generate detection data including information about brackets and subscripts can include inputting the polymer molecular structure image to a detector and outputting the detection data, and the detector can include a Detection Transformer (DETR) or a Deformable DETR.

[0012] The detection data may include an embedding vector.

[0013] Furthermore, at least one of the first model and the second model may include a matrix that includes the embedding vector.

[0014] The group information regarding the brackets and subscripts may include information regarding which brackets and subscripts, among the detected brackets and subscripts, are included in the group corresponding to which monomer.

[0015] In various embodiments, the operations performed by the instructions or information may further include converting the cluster data into a predetermined string format, including SMILES (Simplified Molecular Input Line Entry System), and outputting the structural image.

[0016] Furthermore, the operation performed by the instruction or information may further include obtaining information on a plurality of atomic regions from the polymer molecular structure image, and obtaining information on bonding relationships between a plurality of atoms based on the information on the plurality of atomic regions.

[0017] According to another aspect of the present invention, there is provided a method for recognizing polymer molecular structure formulas using artificial intelligence, which is executed by at least one processor, and includes: detecting a polymer molecular structure image to generate detection data including information on brackets and subscripts; inputting the detection data into a first model and a second model, respectively; outputting first cluster data from the first model; and outputting second cluster data from the second model, which includes group information on the brackets and subscripts and includes information different from that of the first cluster data.

[0018] The polymer may contain two or more different monomers.

[0019] The group information regarding the brackets and subscripts may include information regarding which brackets and subscripts, among the detected brackets and subscripts, are included in the group corresponding to which monomer.

[0020] Outputting second cluster data from the second model may include generating a matrix that projects the detection data into another feature space.

[0021] A program stored in a computer-readable recording medium according to another aspect of the present invention may be stored in a computer-readable recording medium to execute a polymer molecular structure recognition method according to an embodiment of the present invention. [Effects of the Invention]

[0022] According to an embodiment of the present invention, by simultaneously including a first model that outputs brackets and subscripts in a polymer molecular structural formula as class information and coordinate information from detection data, and a second model that outputs group information regarding the brackets and subscripts, it is possible to more accurately recognize molecular structural formulas of copolymers that contain two or more different types of monomers.

[0023] The effects of the present invention are not limited to those described above, and other effects not mentioned will also be clearly understood by those skilled in the art from the following description. [Brief explanation of the drawings]

[0024] [Figure 1] FIG. 1 is a schematic diagram of a system capable of implementing a method for recognizing a polymer molecular structure according to one embodiment of the present invention. [Figure 2] FIG. 2 is a block diagram illustrating the configuration of an apparatus for performing a method for recognizing a polymer molecular structure according to an embodiment of the present invention. [Figure 3] FIG. 3 is a flowchart illustrating a method for recognizing a polymer molecular structure according to an embodiment of the present invention. [Figure 4] FIG. 4 is a block diagram illustrating a method for recognizing a polymer molecular structure according to an embodiment of the present invention. [Figure 5] FIG. 5 is a molecular structure diagram for illustratively explaining a polymer molecular structure recognition method according to an embodiment of the present invention. [Figure 6] FIG. 6 is a schematic diagram illustrating an example of metric learning that can be used in the polymer molecular structure recognition method according to an embodiment of the present invention. [Figure 7] FIG. 7 is a diagram for explaining a structural formula image according to an embodiment of the present invention. [Figure 8] FIG. 8 is a diagram for explaining an atom region recognition model according to an embodiment of the present invention. [Figure 9] FIG. 9 is a diagram for explaining a plurality of pieces of atom region information according to an embodiment of the present invention. [Figure 10] FIG. 10 is a diagram for explaining a method for acquiring connection relationship information according to an embodiment of the present invention. [Figure 11] FIG. 11 is a diagram for explaining a connection relationship recognition model according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0025] The following embodiments are provided as examples to fully convey the concept of the present invention to those skilled in the art to which the present invention pertains, and therefore the present invention is not limited to the embodiments described below and may be embodied in other forms.

[0026] The same reference numerals refer to the same elements throughout the present invention. It should be noted that the present invention does not describe all elements of the embodiments, and descriptions of content common in the technical field to which the present invention pertains or content that overlaps between embodiments will be omitted. Furthermore, terms such as "unit," "module," "component," and "block" used in this specification may be implemented by software or hardware, and, depending on the embodiment, multiple "units," "modules," "components," and "blocks" may be implemented as a single component, or one "unit," "module," "component," or "block" may include multiple components.

[0027] Throughout this specification, when a part is described as being "coupled" to another part, this includes not only direct coupling but also indirect coupling where there are other components between them, and the form of "coupling" includes all states where the parts are directly or indirectly coupled in any manner, including wired and wireless.

[0028] Furthermore, when a part is described as "comprising" a certain component, unless otherwise specified, it does not exclude other components and means that it may further include other components.

[0029] Throughout this specification, when a member is described as being "on" another member, this does not necessarily mean that the member is in contact with the other member, but also includes cases where there is another member between the two.

[0030] Terms such as "first" and "second" are used to distinguish one component from another, and the components are not limited by these terms.

[0031] Any reference to the singular shall include the plural unless the context clearly indicates otherwise.

[0032] The identification numbers in each step are used for convenience of explanation, and the identification numbers do not describe the order of the steps; the steps may be performed in an order different from that described unless the context clearly dictates a specific order.

[0033] The system for recognizing molecular structure formulas according to the present invention may include a device, and the device may include all of various devices capable of executing calculations and providing results to a user. For example, the system for recognizing molecular structure formulas according to the present invention may include at least one of a computer, a server device, a mobile terminal, etc., or may be in any one form having the same or similar functions as these.

[0034] Furthermore, the molecular structure recognition system according to the present invention may be implemented as a service provided to a user terminal by a server in the form of a web service, or in the form of a server and a user terminal linked together.

[0035] However, the method for realizing the system for predicting material properties according to the present invention is not limited to this, and all possible forms of systems can be included in the present invention.

[0036] Here, the computer may include, for example, a notebook computer, a desktop computer, a laptop computer, a tablet PC, a slate PC, or the like equipped with a web browser.

[0037] The server device may include an application server, a computing server, a database server, a file server, a game server, a mail server, a proxy server, a web server, etc., which communicate with external devices and process information.

[0038] The mobile terminal is, for example, a wireless communication device that ensures portability and mobility, and may include all kinds of handheld-based wireless communication devices such as PCS (Personal Communication System), GSM (Global System for Mobile communications), PDC (Personal Digital Cellular), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), WiBro (Wireless Broadband Internet) terminals, and smartphones, as well as wearable devices such as watches, rings, bracelets, anklets, necklaces, glasses, contact lenses, or head-mounted devices (HMDs).

[0039] The AI ​​model according to the embodiment of the present invention can be controlled, executed, trained, driven, etc. by at least one processor, and therefore, at least one of the tasks of execution, training, and driving of the AI ​​model can be performed by at least one processor. The AI ​​model can be stored in a memory.

[0040] According to embodiments of the present invention, instructions for performing operations by at least one processor may be stored in at least one memory, and the at least one processor may operate the AI ​​model and / or other components (e.g., a transform unit, a calculation unit, etc.) that implement the system.

[0041] In addition, in embodiments, the AI ​​model may include an artificial neural network (ANN), a machine learning model, etc.

[0042] An artificial neural network (ANN) is a model used in machine learning and can refer to any model with problem-solving capabilities that is composed of artificial neurons (nodes) that form a network through synaptic connections. An artificial neural network can be defined by the connection patterns between neurons in different layers, a learning process that updates model parameters, and an activation function that generates output values.

[0043] An artificial neural network may include an input layer, an output layer, and optionally one or more hidden layers. Each layer includes one or more neurons, and the artificial neural network may include synapses connecting the neurons. In an artificial neural network, each neuron can output a function value of an activation function in response to an input signal, weight, and bias input through a synapse.

[0044] Parameters can include model parameters and hyperparameters. Model parameters are parameters that are changed or determined through learning, and can include synaptic connection weights and neuron biases. Hyperparameters, on the other hand, are parameters that must be set before learning in machine learning algorithms, and can include the learning rate, number of iterations, mini-batch size, initialization function, etc.

[0045] Training an artificial neural network may involve determining model parameters that minimize a loss function, where the loss function may be used as a metric for determining optimal model parameters during the training process of the artificial neural network.

[0046] Depending on the learning method, machine learning may include, but is not limited to, supervised learning, unsupervised learning, reinforcement learning, and the like.

[0047] Among artificial neural networks, machine learning realized by deep neural networks (DNNs) that include multiple hidden layers is also called deep learning, and deep learning is included as part of machine learning.

[0048] FIG. 1 is a schematic diagram of a system capable of implementing a method for recognizing a polymer molecular structure according to one embodiment of the present invention.

[0049] As shown in FIG. 1, the system 1000 may include a device 100, a database 200, and an AI model 300.

[0050] The device 100, the database 200, and the AI ​​model 300 included in the system 1000 can communicate via a network W. Here, the network W may include a wired network and a wireless network. For example, the network may include various networks such as a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), etc.

[0051] The network W may also include the well-known World Wide Web (WWW), but is not limited to the aforementioned networks and may include, at least in part, well-known wireless data networks, well-known telephone networks, and well-known wired and wireless television networks.

[0052] The device 100 can obtain or output feature data for information about the polymer molecular structure based on the AI ​​model 300. The feature data can include a feature vector, embedding, etc.

[0053] The database 200 can store various types of learning data for training the AI ​​model 300. The database 200 can also store polymer molecular structure images, ground truth (GT) values, training sets for learning, and the like, as well as calculated data output by the AI ​​model 300 in various embodiments. However, the system 1000 may not include the database 200 if the learning of the AI ​​model 300 has been completed.

[0054] 1 illustrates a case where the database 200 is implemented externally to the device 100. In this case, the database 200 may be connected to the device 100 via a wired or wireless connection. However, this is merely an example, and the database 200 may also be implemented as a component of the device 100.

[0055] FIG. 1 illustrates a case where the AI ​​model 300 is implemented externally to the device 100 (e.g., cloud-based), but is not limited thereto, and the AI ​​model 300 may be implemented as a component of the device 100.

[0056] FIG. 2 is a block diagram illustrating the configuration of an apparatus for performing a method for recognizing a polymer molecular structure according to an embodiment of the present invention.

[0057] 2, the device 100 may include a memory 110, a communication module 120, a display 130, an input module 140, and a processor 150. However, the device 100 is not limited to this, and software and hardware configurations may be modified, added, or omitted within a scope obvious to those skilled in the art depending on the required operation. Furthermore, the device 100 may be replaced by a system, and when the device 100 includes multiple devices, each component included in the device 100 may be included in at least one of the multiple devices.

[0058] The memory 110 can store data supporting various functions of the device 100, programs for the operation of the processor 150, and input / output data. It can also store a number of application programs (applications) run by the device, and data, instructions, and AI models for the operation of the device 100. At least some of these application programs can be downloaded from an external server via wireless communication. The memory 110 can store instructions or information that cause the processor 150 to perform operations.

[0059] Such memory 110 may include at least one type of storage medium among flash memory type, hard disk type, solid state disk type (SSD type), silicon disk drive type (SDD type), multimedia card micro type, card type memory (such as SD or XD memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, and optical disk.

[0060] The memory 110 may also include a database that may be separate from the device and connected via wire or wirelessly. The database 200 shown in FIG.

[0061] The communication module 120 may include one or more components that enable communication with external devices, such as at least one of a broadcast reception module, a wired communication module, a wireless communication module, a short-range communication module, and a location information module.

[0062] The wired communication module may include various wired communication modules such as a Local Area Network (LAN) module, a Wide Area Network (WAN) module, or a Value Added Network (VAN) module, as well as various cable communication modules such as a Universal Serial Bus (USB) (registered trademark), a High Definition Multimedia Interface (HDMI) (registered trademark), a Digital Visual Interface (DVI), recommended standard 232 (RS-232), power line communication, or plain old telephone service (POTS).

[0063] The wireless communication module may include a Wi-Fi (Wifi) module, a WiBro (Wireless Broadband) module, as well as wireless communication modules that support various wireless communication methods such as GSM (global System for Mobile Communication), CDMA (Code Division Multiple Access), WCDMA (Wideband Code Division Multiple Access) (registered trademark), UMTS (universal mobile telecommunications system), TDMA (Time Division Multiple Access), LTE (Long Term Evolution), 4G, 5G, and 6G.

[0064] The display 130 displays (outputs) information and data processed by the device 100, data input or output through the AI ​​model 300, etc. The display 130 can also display execution screen information of an application program (e.g., an application) run by the device 100, and UI (User Interface) and GUI (Graphical User Interface) information based on this execution screen information.

[0065] The input module 140 is for receiving information from a user, and when information is input through the user input unit, the processor 150 can control the operation of the device 100 according to the input information.

[0066] The input module 140 may include hardware physical keys (e.g., buttons, tome switches, jog wheels, jog switches, etc. located on at least one of the front, back, and side of the device) and software touch keys. As an example, the touch keys may be virtual keys, soft keys, or visual keys displayed on the touchscreen display 130 through software processing, or may be touch keys located on a portion other than the touchscreen. Meanwhile, the virtual keys or visual keys may have various forms and be displayed on the touchscreen, and may be, for example, graphics, text, icons, videos, or a combination thereof.

[0067] Processor 150 may be implemented as a memory that stores data related to an algorithm or a program that reproduces the algorithm for controlling the operation of the components within device 100 (including learning and execution of an AI model), and at least one processor (not shown) that uses the data stored in the memory to perform the aforementioned operations. In this case, the memory and processor may be implemented as separate chips, or may be implemented as a single chip.

[0068] Processor 150 may be one or more processors and / or processing circuits for executing program code and controlling the basic operations of device 100. The processor may include a central processing unit (CPU), a graphics processing unit (GPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a complex programmable logic device (CPLD), dedicated circuitry for implementing a function, a special-purpose processor for implementing neural network-based processing, or other system.

[0069] In one embodiment, the system 1000 and device 100 according to the present invention may include at least one processor, and if multiple processors are included, these multiple processors may be included in different devices 100.

[0070] The processor 150 may also control any one or more of the aforementioned components in combination to implement on the device 100 various embodiments according to the present invention, as described below.

[0071] Fig. 3 is a flowchart illustrating a polymer molecular structure recognition method according to an embodiment of the present invention, Fig. 4 is a block diagram illustrating a polymer molecular structure recognition method according to an embodiment of the present invention, Fig. 5 is a molecular structure for illustratively explaining a polymer molecular structure recognition method according to an embodiment of the present invention, and Fig. 6 is a schematic diagram illustrating illustrative metric learning that can be used in a polymer molecular structure recognition method according to an embodiment of the present invention.

[0072] Referring to Figures 3 and 4, a method for recognizing a polymer molecular structure according to an embodiment includes preparing a polymer molecular structure image (S1010), detecting brackets and subscripts from the polymer molecular structure image to generate detection data (S1020), clustering the brackets and subscripts from the detection data to generate cluster data (S1030), extracting repeating values ​​of monomers and subscripts using the data detected from the polymer molecular structure image (S1040), and outputting the results (S1050).

[0073] The method for recognizing a polymer molecular structure according to the embodiment may be executed by at least one or more processors. Also, instructions or information for causing the at least one or more processors to perform operations may be stored in at least one or more memories. The at least one or more processors or at least one or more memories may be included in multiple devices. Also, the method for recognizing a polymer molecular structure according to the embodiment may be performed by operations performed by the instructions or information, and may include performing iterative learning.

[0074] Each step will be described in detail below.

[0075] First, a method for recognizing a polymer molecular structure according to an embodiment may include preparing a polymer molecular structure image 1210 (S1010). The polymer molecular structure image 1210 may include a polymer molecular structure including at least one monomer, and may include a copolymer molecular structure including two or more different monomers. For example, as shown in FIG. 5(a), the polymer molecular structure image 1210 may include a copolymer molecular structure including a first monomer 1315, a second monomer 1325, and a third monomer 1335.

[0076] Preparing the polymer molecular structure image 1210 may be performed by at least one processor. Instructions or information for causing the at least one processor to perform operations may be stored in at least one memory. Preparing the polymer molecular structure image 1210 may include storing the polymer molecular structure image 1210 in a memory included in the processor or in a separate volatile or non-volatile memory. The polymer molecular structure image 1210 may include an image stored in a memory, an image provided externally via a network or a communication module, or an image provided externally in real time via a network or a communication module and streamed. However, the present invention is not limited thereto and may encompass any method or scheme for recognizing the polymer molecular structure image 1210 through operations performed by at least one processor.

[0077] The polymer molecular structure image 1210 may be an image in which a structural formula is represented in a graphical form. A structural formula may be interpreted as a graphical representation of a chemical or molecular structure. A structural formula may include information about the arrangement of atoms in three-dimensional space, as well as information about the chemical bonds between atoms. The polymer molecular structure image 1210 may include annotations, such as the name of the compound.

[0078] 5(a), the polymer molecular structure image 1210 includes a first monomer 1315, a second monomer 1325, and a third monomer 1335, each of which is represented by a first bracket 1311, a second bracket 1321, and a third bracket 1331, and a first subscript 1313, a second subscript 1323, and a third subscript 1333 indicating the number of times the monomer is repeated. The polymer molecular structure image 1210 includes a first group 1310 including the first monomer 1315, the bracket 1311 separating the first monomer 1315, and the subscript 1313, and may further include a second group 1320 and a third group 1330 including the second monomer 1325 and the third monomer 1335, respectively.

[0079] Here, brackets are used to identify or separate repeating monomer units within a polymer. That is, the molecules within the brackets indicate the structure of each monomer, and the lines extending outside the brackets indicate how the monomer is connected to other atoms or molecules. The subscripts indicate the number of times the monomer is repeated, and letters such as n and m indicate that the monomer is repeated multiple times.

[0080] Next, brackets and subscripts are detected in the polymer molecular structure image 1210 (S1020).

[0081] 3 and 4, detecting brackets and subscripts in the polymer molecular structure image 1210 may include detecting the polymer molecular structure image 1210 and outputting bracket and subscript related detection data 1230. The detection data 1230 may include information related to the brackets and subscripts. The bracket and subscript related detection data 1230 may also include at least one embedding 1231, which may include an embedding vector in vector form.

[0082] Detecting the polymer molecular structure image 1210 may include inputting the image to a detector 1220 and outputting detection data 1230 related to brackets and subscripts. The detector 1220 may include a transformer-based object detector, such as a DEtection TRansformer (DETR) or a Deformable DETR, but the present invention is not limited thereto.

[0083] Referring to the example of FIG. 5, as shown in FIG. 5(b), the polymer molecular structural formula image 1210 can be detected to detect the brackets (dotted lines) and the subscripts (dashed lines).

[0084] Meanwhile, at least one embedding 1231 included in the bracket and subscript-related detection data 1230 may be an embedding containing individual information of the polymer molecular structure image 1210. The embedding 1231 may include an embedding vector containing information about the bracket and an embedding vector containing information about the subscript. The embedding vector may be, for example, a 1024-dimensional embedding vector.

[0085] The embedding 1231 may include individual embeddings containing information about the first bracket 1311, the second 1321, and the third 1331, as well as the first subscript 1313, the second 1323, and the third 1333, each of which indicates the number of repetitions of the monomer, and the individual embeddings may include embedding vectors. The embeddings 1231 may each include one embedding vector containing information about the left bracket that constitutes the bracket, and the other embedding vector containing information about the right bracket.

[0086] 5, the polymer molecule included in the polymer molecule structural formula image 1210 shown in FIG. 5(a) includes a first bracket 1311, a second bracket 1321, and a third bracket 1331, as well as a first subscript 1313, a second subscript 1323, and a third subscript 1333. Information about each bracket and subscript may correspond to an individual embedding vector.

[0087] For example, in an embodiment, the embedding 1231 may include an embedding vector 1231a containing information about the left first bracket 1311, an embedding vector 1231b containing information about the right first bracket 1311, and an embedding vector 1231c containing information about the first subscript 1313. The embedding 1231 may also include an embedding vector 1231d containing information about the left and right second brackets 1321 of the second group 1320, an embedding vector 1231e, and an embedding vector 1231f containing information about the second subscript 1323. The embedding 1231 may also include an embedding vector 1231g containing information about the left and right third brackets 1331 of the third group 1330, respectively, an embedding vector 1231h, and an embedding vector 1231i containing information about the third subscript 1333.

[0088] On the other hand, detecting brackets and subscripts in the polymer molecular structure image 1210 may include recognizing the remaining portion of the polymer molecular structure image excluding the brackets and subscripts, and it is possible to detect atoms and bonding relationships between atoms that constitute the polymer molecule included in the polymer molecular structure image 1210. Furthermore, detecting atoms and bonding relationships between atoms that constitute the polymer molecule included in the polymer molecular structure image 1210 may include detecting atoms and bonding relationships between atoms that constitute a monomer.

[0089] Methods for detecting atoms constituting a polymer molecule included in polymer molecule structural formula 1210 and bonding relationships between atoms according to various embodiments of the present invention will be described in more detail with reference to FIGS. 7 to 11. FIG. 7 is a diagram for explaining a structural formula image according to an embodiment of the present invention, and FIG. 8 is a diagram for explaining an atom region recognition model according to an embodiment of the present invention. FIG. 9 is a diagram for explaining information on a plurality of atom regions according to an embodiment of the present invention, and FIG. 10 is a diagram for explaining a method for acquiring bonding relationship information according to an embodiment of the present invention. FIG. 11 is a diagram for explaining a bonding relationship recognition model according to an embodiment of the present invention.

[0090] 7, a structural formula image 400 may include a structural formula 401 that graphically represents a molecular structure. The structural formula image 400 may also include annotations 402. When the structural formula image 400 is generated, annotations may be included in the structural formula image as an explanation or name of the molecular structure. Therefore, when attempting to recognize the structural formula image 400 and convert it into a predetermined character string format, it is necessary to exclude the annotations 402 included in the structural formula image 400 and filter only the structural formula 401 as a recognition target.

[0091] The atom region information may include at least one of atom region identification information, atom position information, and atom information in the structural formula image. The atom region identification information may mean identification information or numbers that can distinguish each of the multiple atom regions recognized from the structural formula image.

[0092] Furthermore, the atom position information may refer to coordinate information corresponding to an atom region in a structural image. For example, if an atom region is represented by a rectangle, the atom position information may include the vertex coordinates of each rectangle corresponding to the atom region and the center coordinate of the rectangle.

[0093] The atomic information may also include element symbol information of the atom corresponding to the atomic region. For example, if the image corresponding to the atomic region is represented as a vertex, carbon "C" may correspond to the atomic information. Furthermore, if the image corresponding to the atomic region includes an element symbol (e.g., oxygen "O"), the element symbol "O" information may correspond to the atomic information.

[0094] Meanwhile, the atom region information may include reliability information of the atom region information obtained from the structural formula image. For example, the processor may obtain the reliability information of the atom region information obtained from the structural formula image as a value in the range of 0.00 to 1.00, and may use only atom region information having reliability information equal to or greater than a predetermined value.

[0095] Referring to FIG. 8, a processor can input a structural formula image 501 into an atom region recognition model 502 and obtain atom region information output from the atom region recognition model 502 .

[0096] The atom region recognition model 502 may be an artificial neural network (ANN) trained to output at least one atom region information 503 contained in an input structural formula image 501. An artificial neural network (ANN) is a model used in machine learning and may refer to a general model having problem-solving capabilities, which is composed of artificial neurons (nodes) that form a network through synaptic connections. For example, the atom region recognition model 502 may be a convolutional neural network (CNN)-based artificial neural network model.

[0097] The processor can use the structural formula image and learning data on the atom region information to train the atom region recognition model 502 configured with an artificial neural network. Meanwhile, the atom region recognition model 502 can be a model trained by the processor.

[0098] The learned atomic region recognition model 502 may be stored in a memory or in a storage unit of a server, and a processor may be capable of performing operations on a model stored in a memory or the like.

[0099] 9 , the processor can acquire a plurality of pieces of atom region information 601, 602, 603, 604, 605, 606, 607, 608, 609, 610, and 611 output from the atom region recognition model 502. The plurality of pieces of atom region information may include information about vertex regions 601, 602, 603, 604, 605, 606, 608, 609, and 610 in a structural formula, an element symbol region 607 in which an element symbol is written, and an annotation region 611. For example, the atom region recognition model 502 can be trained to be able to output the element symbol region 607 in which an element symbol is written, and therefore, there may be cases where the atom region recognition model 502 outputs the annotation region 611 as atom region information. Therefore, it may be necessary to filter information about the annotation region from the plurality of pieces of atom region information. For example, since the annotation region 611 does not indicate a bonding relationship with other atom regions, the processor 180 can classify the annotation region 611 as an annotation when it is determined that no bonding relationship exists between the annotation region 611 and other atom regions. Meanwhile, the processor can obtain bonding relationship information between multiple atoms based on information about multiple atom regions.

[0100] The processor can acquire, for each atomic region, information on the bonding relationship between other atomic regions based on the plurality of pieces of atomic region information. The processor can acquire a bonding image between the first atom and the second atom based on the first atomic region information and the second atomic region information among the plurality of pieces of atomic region information, and can acquire the bonding relationship information between the first atom and the second atom based on the acquired bonding image.

[0101] 10 , a method for acquiring bond relationship information will be described. The processor can select first atom region information 601 from a plurality of atom region information. The processor can also select second atom region information 602 that is different from the first atom region information. The processor can then acquire a bond image 701 between a first atom and a second atom based on first atom position information 702 in the first atom region information 601 and second atom position information 703 in the second atom region information 602. In this case, the first atom position information 702 and the second atom position information 703 may be the center point positions of the respective atom regions, but are not limited to the center point positions of the atom regions.

[0102] The processor can obtain a combined image 701 including the center point positions of each atom region based on the first atom position information 702 of the first atom region information 601 and the second atom position information 703 of the second atom region information 602. The processor can also obtain a combined image 704 including the first atom region and the second atom region based on the first atom region information 601 and the second atom region information 602. The size and shape of the combined image can be adjusted in various ways.

[0103] Meanwhile, the processor can acquire bond images between atoms that can be combined for each of the multiple atomic regions. However, acquiring bond images between all possible atomic regions may increase the amount of calculation. Therefore, the processor can acquire only bond images between first and second atoms that exist within a predetermined distance from each other based on the multiple atomic region information.

[0104] For example, the processor can select a second atom region located within a predetermined distance from the first atom region based on first atom region information from the plurality of atom region information. Referring to Figure 10, the processor can identify second atom regions 602, 603, 608, 609, and 610 located within a predetermined distance from the first atom region 601, obtain bond images between the first atom region 601 and each of the second atom regions, and obtain bond relationship information between the first atoms and the second atoms.

[0105] Furthermore, the processor can determine that no bond relationship exists for the third atom regions 604, 605, 606, 607, and 611 that are located outside a predetermined distance from the first atom region 601. This makes it possible to reduce the amount of calculation.

[0106] Meanwhile, the processor can obtain information about bond relationships between atoms based on the obtained bond image. For example, the processor can input the bond image to a bond relationship recognition model and obtain bond relationship information output from the bond relationship recognition model.

[0107] FIG. 11 is a diagram for explaining the bond relationship recognition model. Referring to FIG. 11, a processor can input a bond image 801 into a bond relationship recognition model 802 and obtain bond relationship information output from the bond relationship recognition model 802.

[0108] The bond relationship information is information about bonds between atoms and may include information such as non-bonds, single bonds, double bonds, triple bonds, upward bonds, and downward bonds. A non-bond may refer to a case where there is no bond between atoms. An upward bond, represented by a wedge, may refer to a bond that protrudes in front of a plane. A downward bond, represented by a dotted line, may refer to a bond that recedes into the plane. The bond relationship recognition model 802 may be an artificial neural network (ANN) trained to output bond relationship information 803 for an input bond image 801. An artificial neural network (ANN) is a model used in machine learning and may refer to a general model with problem-solving capabilities that is composed of artificial neurons (nodes) that form a network through synaptic connections. For example, the bond relationship recognition model 802 may be a convolutional neural network (CNN)-based artificial neural network model.

[0109] The processor can use the learning data on the binding image and the binding relationship information to train the binding relationship recognition model 802 configured as an artificial neural network.

[0110] The learned connection relationship recognition model 802 may be stored in a memory, and the processor may utilize the connection relationship recognition model 802 stored in the memory.

[0111] On the other hand, the processor can generate an adjacency matrix based on the information on the regions of the plurality of atoms and the information on the bonding relationships between the plurality of atoms, with each of the plurality of atoms as a vertex and the information on the bonding relationships between the plurality of atoms as an edge.

[0112] FIG. 11 is a diagram illustrating an adjacency matrix. Referring to FIG. 11, the processor can generate each atom corresponding to a plurality of atom regions 601 to 611 as a vertex of the adjacency matrix. Meanwhile, the processor can generate an adjacency matrix using the bond relationship information of each of the plurality of atoms as an edge. For example, the processor can associate each bond relationship information with an arbitrary numerical value and generate the edge value of the adjacency matrix. Specifically, the processor can associate a non-bond with "0," a single bond with "1," a double bond with "2," and a triple bond with "3." Furthermore, to express the directionality of an upward bond, the processor can set the row as the starting vertex and the column as the destination vertex, and associate an upward bond in the order of row and column with "5" and an opposite direction with "0."

[0113] 11, the value in the fifth row, sixth column of the adjacency matrix may be generated as a value of "5" because the connection is upward from the first atom 605 to the second atom 606. Meanwhile, the value in the sixth row, fifth column of the adjacency matrix, which is in the reverse direction, may be generated as a value of "0." Similarly, to indicate the directionality of a downward connection, processor 180 may correspond to a value of "6" if the connection is downward from the row to the column, with the row being the starting vertex and the column being the destination vertex, and may correspond to a value of "0" if the connection is in the reverse direction.

[0114] Meanwhile, the processor can generate a predetermined character string format corresponding to the structural formula image based on the generated adjacency matrix. The processor can cycle through each atomic region corresponding to the vertices of the adjacency matrix to obtain information on bonding relationships with other atoms. The processor can identify information on each atom in the multiple atomic regions using the obtained information on bonding relationships between atoms, and generate a predetermined character string format corresponding to the structural formula image based on the information on the multiple atoms and the information on bonding relationships between the multiple atoms.

[0115] In this case, the character string format may include a mol file format, an sdf file format, etc., as a file format capable of indicating compound information (e.g., element positions, bonding relationships, etc.). On the other hand, the character string format may include information related to SMILES (Simplified Molecular Input Line Entry System).

[0116] Meanwhile, the method for detecting atoms of a polymer molecule and the bonding relationships between atoms from a polymer molecular structural formula image according to an embodiment of the present invention is not limited to those described with reference to Figures 7 to 11 above, and various structural formula image detection methods may all be used.

[0117] Furthermore, in the polymer molecular structure recognition method according to an embodiment of the present invention, detecting the atoms constituting the polymer molecule contained in the polymer molecular structure image 1210 and the bonding relationships between the atoms is not necessarily included in the step of detecting brackets and subscripts in the polymer molecular structure image 1210, and detecting the atoms constituting the polymer molecule and the bonding relationships between the atoms may be included in another step or may be realized as a separate step.

[0118] Referring again to FIGS. 3 and 4, next, brackets and subscripts are clustered from the detection data to generate cluster data (S1030).

[0119] Clustering the brackets and subscripts from the detection data to generate cluster data may include inputting the detection data to a first model 1241 and a second model 1242, respectively, and outputting first cluster data including at least one of class information and coordinate information of the brackets and subscripts from the first model 1241, and outputting second cluster data including group information regarding the brackets and subscripts from the second model 1242. The cluster data may include the first cluster data and the second cluster data.

[0120] The first model 1241 and the second model 1242 may be preset algorithms or may be pre-trained models, and at least one of the first model 1241 and the second model 1242 may be executed or trained by a processor.

[0121] The first cluster data output from the first model 1241 may contain information different from that of the second cluster data output from the second model 1242.

[0122] In an embodiment, the detection data including the embeddings 1231 may be input to a first model 1241, which may output class information and coordinate information of objects (brackets, subscripts) corresponding to each embedding 1231. For example, referring to (a) of Figure 5, a polymer molecule structural formula image 1210 may be projected onto a space having an x-axis and a y-axis, and each detection object (bracket, subscript, atom, atomic bond, etc.) of the polymer molecule may have coordinates based on the x-axis and y-axis of the space.

[0123] The first model 1241 may include a linear layer configured with matrices. In one embodiment, the first model 1241 may include a linear layer including a matrix (e.g., 1024×4) that derives class probabilities for four types, i.e., brackets, subscripts, atoms, and carbon (C), from the embedding 1231, and a linear layer including a matrix (e.g., 1024×4) that derives coordinate information of the detection object, such as brackets and subscripts, from the embedding 1231.

[0124] The class information output from the first model 1241 may include classification information of the object corresponding to each embedding 1231, i.e., information about what type of object the object represents. For example, when embeddings 1231 relating to the first bracket 1311, the second bracket 1321, and the third bracket 1331 are input to the first model 1241 and related information is output, class information indicating that the information relating to the embedding 1231 corresponds to a "bracket" may be output.

[0125] By outputting first cluster data including class information and coordinate information of the brackets and subscripts through the first model 1241, information can be output regarding which monomer the brackets and subscripts correspond to, or where the brackets and subscripts are located in the bonding relationships between other atoms.

[0126] In an embodiment, the detection data including the embeddings 1231 may be input to the second model 1242, which may output group information regarding the objects (brackets, subscripts) corresponding to each embedding 1231. The group information may include information regarding which brackets and subscripts constitute one monomer group among the brackets and subscripts.

[0127] The second model 1242 may include a layer that rearranges the input embedding 1231 into another new space. The second model 1242 may include a matrix (e.g., 1024×1024) that projects the input embedding 1231 into an embedding of another length (e.g., 1024). The second model 1242 may also include a metric function.

[0128] 6, the second model 1242, which includes a metric function, enables the embeddings of the original feature space to be clustered in the new feature space. Thus, the second model 1242 may include generating a matrix that projects the detection data into the new, other feature space.

[0129] The second model 1242 may be a model trained by various methods, for example, but not limited to, a model trained based on infoNCE loss.

[0130] For example, the second model 1242 may be executed or trained by defining positive and negative pairs using infoNCE loss and assigning losses. A positive pair corresponds to the ground truth, and for example, a first bracket 1311 and a first subscript 1313 included in a first group 1310 corresponding to a first monomer 1315 are output as a positive pair. On the other hand, if it is not correct to cluster them as one group, such as the first bracket 1311 and the second subscript 1323, a negative result is output.

[0131] Information regarding which detected brackets and subscripts are included in which groups corresponding to which monomers can be output by outputting second cluster data including group information regarding brackets and subscripts via second model 1242. For example, referring to (c) of Figure 5, second model 1242 can output second cluster data including information that first bracket 1311 and first subscript 1313 are one group included in first group 1310 corresponding to first monomer 1315, and can output second cluster data including similar group information for the remaining brackets and subscripts included in second group 1320 and third group 1330.

[0132] Thus, according to the embodiment of the present invention, the system utilizing the polymer molecular structure recognition method of the present invention simultaneously includes a first model 1241 that outputs class information and coordinate information from detection data regarding brackets and subscripts in the polymer molecular structure, and a second model 1242 that outputs group information regarding the brackets and subscripts, thereby enabling accurate recognition of copolymer molecular structure formulas containing two or more different types of monomers.

[0133] Referring again to FIGS. 3 and 4, the repeating values ​​of the detected monomers and subscripts are extracted from the polymer molecular structure image (S1040), and the results are output (S1050).

[0134] Extraction of the repeating values ​​of the monomers and subscripts detected from the polymer molecular structure image can be performed by extracting the repeating values ​​of the monomers and subscripts from information obtained through a method for detecting atoms constituting a polymer molecule and bonding relationships between atoms, but the present invention is not limited thereto, and the repeating values ​​of the monomers and subscripts can be detected through various molecular structure recognition methods.

[0135] In the embodiment, the method described with reference to FIGS. 7 to 11 can be used to extract the detected monomers and the repeating values ​​of the subscripts from the polymer molecular structure image, but the present invention is not limited thereto.

[0136] Outputting the result (S1050) can generate a predetermined character string format corresponding to the polymer molecular structure image based on the information on the plurality of atoms, the information on the bonding relationship between the plurality of atoms, and the cluster data.

[0137] In this case, the character string format may include file formats that can indicate compound information (e.g., element positions, bonding relationships, monomer information, etc.), such as the mol file format and the sdf file format. On the other hand, the character string format may include information related to SMILES (Simplified Molecular Input Line Entry System).

[0138] Meanwhile, the polymer molecular structure recognition method according to the embodiment of the present invention can be realized by the system described with reference to FIGS.

[0139] The disclosed embodiments may also be implemented in the form of a recording medium storing computer-executable instructions. These instructions may be stored in the form of program code, which, when executed by a processor, generates program modules to perform the operations of the disclosed embodiments. The recording medium may be implemented as a computer-readable medium.

[0140] Computer-readable storage media include all types of storage media that store computer-readable instructions, such as read-only memory (ROM), random access memory (RAM), magnetic tape, magnetic disks, flash memory, and optical data storage devices.

[0141] The disclosed embodiments have been described above with reference to the accompanying drawings. Those skilled in the art will understand that the present invention can be implemented in forms different from the disclosed embodiments without changing the technical idea or essential features of the present disclosure. Therefore, the disclosed embodiments are illustrative and should not be interpreted as limiting.

Claims

1. A polymer molecular structure recognition system using artificial intelligence, at least one processor; at least one or more memories for storing instructions or information for causing operations to be performed by said at least one or more processors; The operation performed by the instructions or information is detecting the polymer molecular structure image to generate detection data including information about brackets and subscripts; inputting the detection data into a first model and a second model, respectively, outputting first cluster data from the first model, and outputting second cluster data from the second model, the second cluster data including group information related to the brackets and subscripts and including information different from that of the first cluster data; A polymer molecular structure recognition system comprising:

2. The polymer molecular structure recognition system according to claim 1 , wherein the polymer comprises two or more different types of monomers.

3. The polymer molecular structure recognition system according to claim 1 , wherein the first cluster data includes at least one of class information and coordinate information of the brackets and subscripts.

4. The polymer molecular structure recognition system according to claim 1 , wherein the second model includes a metric function.

5. Detecting the polymer molecular structure image to generate detection data including information about brackets and subscripts includes inputting the polymer molecular structure image to a detector and outputting the detection data; 2. The polymer molecular structure recognition system according to claim 1, wherein the detector includes a Detection Transformer (DETR) or a Deformable DETR.

6. The polymer molecular structure recognition system according to claim 1 , wherein the detection data includes an embedding vector.

7. The polymer molecular structure recognition system according to claim 6 , wherein at least one of the first model and the second model includes a matrix including the embedding vector.

8. 3. The polymer molecular structure recognition system according to claim 2, wherein the group information regarding the brackets and subscripts includes information regarding which brackets and subscripts, among the detected brackets and subscripts, are included in a group corresponding to which monomer.

9. 2. The polymer molecular structure recognition system according to claim 1, wherein the operations performed by the instructions or information further include converting the cluster data into a predetermined character string format including SMILES (Simplified Molecular Input Line Entry System) and outputting the structural formula image.

10. 2. The polymer molecular structure recognition system according to claim 1, wherein the operations performed by the instructions or information further include obtaining information on a plurality of atom regions from the polymer molecular structure image, and obtaining information on bonding relationships between a plurality of atoms based on the information on the plurality of atom regions.

11. 1. A method for recognizing polymer molecular structure formulas using artificial intelligence, executed by at least one processor, comprising: detecting the polymer molecular structure image to generate detection data including information about brackets and subscripts; A method for recognizing a polymer molecular structure formula, comprising: inputting the detection data into a first model and a second model, respectively; outputting first cluster data from the first model; and outputting second cluster data from the second model, the second cluster data including group information regarding the brackets and subscripts and including information different from that of the first cluster data.

12. The method for recognizing a polymer molecular structure formula according to claim 11, wherein the polymer contains two or more different types of monomers.

13. The polymer molecular structure recognition method according to claim 12, wherein the group information regarding the brackets and subscripts includes information regarding which brackets and subscripts, among the detected brackets and subscripts, are included in a group corresponding to which monomer.

14. 13. The polymer molecular structure recognition method according to claim 12, wherein outputting second cluster data from the second model includes generating a matrix that projects the detection data into another feature space.

15. A program stored on a computer-readable recording medium, which is coupled with a computer to cause the computer to execute the method according to any one of claims 11 to 14.

Images