System, method, and program for recognizing polymer molecular structural formula by using artificial intelligence

Abstract

Disclosed are a system, a method, and a program for recognizing a polymer molecular structural formula by using artificial intelligence. The system for recognizing a polymer molecular structural formula by using artificial intelligence comprises: at least one processor; and at least one memory for storing instructions or information for causing the at least one processor to perform operations. The operations performed by the instructions or information comprise: detecting a polymer molecular structural formula image and generating detection data including information regarding brackets and subscripts; and inputting the detection data to a first model and a second model, respectively, thereby outputting first cluster data from the first model, and outputting second cluster data from the second model, the second cluster data including group information regarding the brackets and subscripts and including information different from the first cluster data.

Description

System, method, and program for recognizing polymer molecular structure formulas using artificial intelligence

[0001] The present invention relates to a polymer molecular structure recognition system, method, and program, and more particularly, to a system, method, and program capable of recognizing a molecular structure from a polymer molecular structure image using artificial intelligence.

[0002] A structural formula is a graphical representation of a chemical or molecular structure. A structural formula can show how atoms are arranged in three-dimensional space. It can also explicitly or implicitly display a molecule's chemical bonds.

[0003] A polymer is a substance with a high molecular weight, composed of numerous low-molecular-weight monomers. When expressing a polymer by its molecular structure, an arbitrary number of times the monomers are polymerized (e.g., n, m, subscript) is added to each repeating monomer.

[0004] Meanwhile, molecular structures are provided in image format in various literature and papers. Because molecular structures are provided in this image format, they are difficult to identify and select through standard searches. In particular, polymers are sometimes copolymers containing two or more types of monomers, requiring recognition of polymer molecular structure images.

[0005] The problem to be solved by the present invention is to provide a system, method and program capable of recognizing an accurate polymer molecular structure by selecting and recognizing necessary information from a polymer molecular structural formula image.

[0006] The problems to be solved by the present invention are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the description below.

[0007] At least one processor; and at least one memory storing instructions or information that cause the at least one processor to perform an operation, wherein the operation performed by the instructions or information includes detecting a polymer molecular structure image to generate detection data including information about brackets and subscripts; and inputting the detection data into a first model and a second model, respectively, to output first cluster data from the first model, and output second cluster data including group information about the brackets and subscripts and including information different from the first cluster data.

[0008] The above polymer may contain two or more different types of monomers.

[0009] The above first cluster data may include at least one of class information and coordinate information of the bracket and subscript.

[0010] The second model may include a metric function.

[0011] In various embodiments, detecting the polymer molecular structure image to generate detection data including information about brackets and subscripts may include inputting the polymer molecular structure image to a detector to output the detection data, wherein the detector may include a Detection TRansformer (DETR) or a Deformable DETR.

[0012] The above detection data may include an embedding vector.

[0013] Additionally, at least one of the first model and the second model may include a matrix including the embedding vector.

[0014] The group information regarding the above brackets and subscripts may include information regarding which of the detected brackets and subscripts are included in a group corresponding to which monomer.

[0015] In various embodiments, the operation performed by the command or information may further include converting and outputting the structural image into a predetermined string format including SMILES (Simplified Molecular Input Line Entry System) based on the cluster data.

[0016] In addition, the operation performed by the command or information may further include obtaining a plurality of atomic region information from the polymer molecular structure image, and obtaining bonding relationship information between a plurality of atoms based on the plurality of atomic region information.

[0017] According to another aspect of the present invention, a method for recognizing a polymer molecular structure using artificial intelligence, performed by at least one processor, includes: detecting a polymer molecular structure image to generate detection data including information about brackets and subscripts; and inputting the detection data into a first model and a second model, respectively, to output first cluster data from the first model, and outputting second cluster data including group information about the brackets and subscripts and including information different from the first cluster data from the second model.

[0018] The above polymer may contain two or more different types of monomers.

[0019] The group information regarding the above brackets and subscripts may include information regarding which of the detected brackets and subscripts are included in a group corresponding to which monomer.

[0020] Outputting the second cluster data from the second model may include generating a matrix that projects the detection data into a different feature space.

[0021] A program stored in a computer-readable recording medium according to another aspect of the present invention may be stored in a computer-readable recording medium to execute a polymer molecular structure recognition method according to embodiments of the present invention.

[0022] According to embodiments of the present invention, by simultaneously including a first model that outputs class information and coordinate information for detecting brackets and subscripts of a polymer molecular structure and a second model that outputs group information regarding brackets and subscripts, a copolymer molecular structure including two or more different types of monomers can also be more accurately recognized.

[0023] The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

[0024] FIG. 1 is a schematic diagram of a system capable of implementing a method for recognizing a polymer molecular structure according to one embodiment of the present invention.

[0025] FIG. 2 is a block diagram illustrating the configuration of a device that performs a method for recognizing a polymer molecular structure according to one embodiment of the present invention.

[0026] FIG. 3 is a flowchart illustrating a polymer molecular structure recognition method according to embodiments of the present invention.

[0027] FIG. 4 is a block diagram illustrating a polymer molecular structure recognition method according to embodiments of the present invention.

[0028] FIG. 5 is a molecular structure diagram illustrating a polymer molecular structure recognition method according to embodiments of the present invention.

[0029] FIG. 6 is a schematic diagram exemplarily illustrating metric learning that can be used in a polymer molecular structure recognition method according to embodiments of the present invention.

[0030] FIG. 7 is a drawing for explaining a structural formula image according to embodiments of the present invention.

[0031] FIG. 8 is a diagram for explaining an atomic domain recognition model according to embodiments of the present invention.

[0032] FIG. 9 is a diagram for explaining multiple atomic region information according to embodiments of the present invention.

[0033] FIG. 10 is a diagram for explaining a method for obtaining binding relationship information according to embodiments of the present invention.

[0034] FIG. 11 is a diagram for explaining a binding relationship recognition model according to embodiments of the present invention.

[0035] The following examples are provided as examples to ensure that those skilled in the art can fully grasp the spirit of the present invention. Therefore, the present invention is not limited to the embodiments described below and may be embodied in other forms.

[0036] Throughout the present invention, the same reference numerals denote the same components. The present invention does not describe all elements of the embodiments, and any content that is general in the technical field to which the present invention pertains or that overlaps between the embodiments is omitted. The terms 'part, module, element, block' used in the specification may be implemented in software or hardware, and depending on the embodiments, multiple 'parts, modules, elements, blocks' may be implemented as a single component, or a single 'part, module, element, block' may include multiple components.

[0037] Throughout the specification, when a part is said to be "connected" to another part, this includes not only direct connection but also indirect connection, and the presence of other components in between, and the form of "connection" includes all direct and indirect connections in any way, including wired and wireless.

[0038] Additionally, when a part is said to "include" a component, this does not mean that it excludes other components, but rather that it may include other components, unless otherwise specifically stated.

[0039] Throughout the specification, when we say that an element is "on" another element, this includes not only cases where the element is in contact with the other element, but also cases where another element exists between the two elements.

[0040] The terms first, second, etc. are used to distinguish one component from another, and the components are not limited by the aforementioned terms.

[0041] Singular expressions include plural expressions unless the context clearly indicates otherwise.

[0042] The identification codes for each step are used for convenience of explanation and do not describe the order of each step. Each step may be performed in a different order than specified unless the context clearly indicates a specific order.

[0043] The system for recognizing molecular structures according to the present invention may include a device, which may include any of a variety of devices capable of performing computational processing and providing results to a user. For example, the system for recognizing molecular structures according to the present invention may include at least one of a computer, a server device, and a portable terminal, or any other form having the same or similar functions as these.

[0044] In addition, the system for recognizing a molecular structure according to the present invention may include a service provided from a server to a user terminal in the form of a web service or the like, or implemented in a form in which a server and a user terminal are linked.

[0045] However, the method by which the system for predicting the characteristics of a material according to the present invention is implemented is not limited thereto, and all types of systems that can be implemented may be included in the present invention.

[0046]

[0047] *Here, the computer may include, for example, a notebook, desktop, laptop, tablet PC, slate PC, etc. equipped with a web browser.

[0048] The above server device is a server that processes information by communicating with an external device, and may include an application server, a computing server, a database server, a file server, a game server, a mail server, a proxy server, and a web server.

[0049] The above portable terminal may include, for example, a wireless communication device that ensures portability and mobility, and may include all kinds of handheld-based wireless communication devices such as a PCS (Personal Communication System), GSM (Global System for Mobile communications), PDC (Personal Digital Cellular), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), WiBro (Wireless Broadband Internet) terminal, a smart phone, and a wearable device such as a watch, ring, bracelet, anklet, necklace, glasses, contact lens, or head-mounted-device (HMD).

[0050] AI models according to embodiments of the present invention can be controlled, executed, trained, driven, etc. by at least one processor, and thus, the AI ​​models can perform at least one of the tasks of execution, training, and driving by at least one processor. The AI ​​models can be stored in memory.

[0051] Additionally, according to embodiments of the present invention, instructions that cause at least one processor to perform operations may be contained in at least one memory. At least one processor may cause an AI model to operate, and in addition to the AI ​​model, may cause other components (e.g., a conversion unit, a calculation unit, etc.) that implement the system to operate.

[0052] Additionally, in embodiments, the AI ​​model may include an artificial neural network (ANN), a machine learning model, etc.

[0053] An artificial neural network (ANN) is a model used in machine learning. It can refer to a model with problem-solving capabilities, comprised of artificial neurons (nodes) formed by the connection of synapses. An ANN can be defined by the connection patterns between neurons in different layers, the learning process that updates model parameters, and the activation function that generates output values.

[0054] An artificial neural network may include an input layer, an output layer, and optionally one or more hidden layers. Each layer contains one or more neurons, and the artificial neural network may include synapses connecting neurons. In an artificial neural network, each neuron can output a function value of an activation function based on input signals, weights, and biases received through the synapses.

[0055] Parameters can include model parameters and hyperparameters. Model parameters refer to parameters that change or are determined through learning, and may include things like synaptic connection weights and neuron biases. Hyperparameters refer to parameters that must be set before learning in machine learning algorithms, and include things like the learning rate, number of iterations, mini-batch size, and initialization functions.

[0056] Training of an artificial neural network may include determining model parameters that minimize a loss function, where the loss function may be used as an indicator for determining optimal model parameters during the training process of the artificial neural network.

[0057] Machine learning can include, but is not limited to, supervised learning, unsupervised learning, and reinforcement learning depending on the learning method.

[0058] Among artificial neural networks, machine learning implemented with a deep neural network (DNN) that includes multiple hidden layers is also called deep learning, and deep learning is included as a part of machine learning.

[0059] FIG. 1 is a schematic diagram of a system capable of implementing a method for recognizing a polymer molecular structure according to one embodiment of the present invention.

[0060] As illustrated in FIG. 1, the system (1000) may include a device (100), a database (200), and an AI model (300).

[0061] The device (100), database (200), and AI model (300) included in the system (1000) can communicate via a network (W). Here, the network (W) may include a wired network and a wireless network. For example, the network may include various networks such as a local area network (LAN), a metropolitan area network (MAN), and a wide area network (WAN).

[0062] Additionally, the network (W) may include the well-known World Wide Web (WWW). However, the network (W) according to an embodiment of the present invention is not limited to the networks listed above, and may include at least part of a well-known wireless data network, a well-known telephone network, or a well-known wired / wireless television network.

[0063] The device (100) can acquire or output feature data regarding information on a polymer molecular structure based on an AI model (300). The feature data may include a feature vector, embedding, etc.

[0064] The database (200) may store various types of learning data for training the AI ​​model (300). Furthermore, the database (200) may store polymer molecular structure images, GT values ​​(Ground-truth), training sets for learning, and the like. In various embodiments, the database may also store output data output by the AI ​​model (300). However, the system (1000) may not include the database (200) if the training of the AI ​​model (300) is complete.

[0065] FIG. 1 illustrates a case where a database (200) is implemented outside of a device (100). In this case, the database (200) may be connected to the device (100) via wired or wireless means. However, this is merely an example, and the database (200) may also be implemented as a component of the device (100).

[0066] FIG. 1 illustrates a case where the AI ​​model (300) is implemented outside the device (100) (e.g., cloud-based), but is not limited thereto, and may be implemented as a component in the device (100).

[0067] FIG. 2 is a block diagram illustrating the configuration of a device that performs a method for recognizing a polymer molecular structure according to one embodiment of the present invention.

[0068] As illustrated in FIG. 2, the device (100) may include a memory (110), a communication module (120), a display (130), an input module (140), and a processor (150). However, the present invention is not limited thereto, and the device (100) may have its software and hardware configurations modified / added / omitted within a range apparent to those skilled in the art according to a required operation. In addition, the device (100) may be replaced with a system, and the device (100) may include a plurality of devices, in which case each component included in the device (100) may be included in at least one of the plurality of devices.

[0069] The memory (110) can store data supporting various functions of the device (100), programs for the operation of the processor (150), input / output data, and a plurality of application programs (or applications) run on the device, data for the operation of the device (100), commands, and AI models. At least some of these application programs can be downloaded from an external server via wireless communication. The memory (110) can store commands or information that cause the processor (150) to perform operations.

[0070] The memory (110) may include at least one type of storage medium among a flash memory type, a hard disk type, an SSD (Solid State Disk type), an SDD (Silicon Disk Drive type), a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, and an optical disk.

[0071] Additionally, the memory (110) may be separate from the device and may include a database connected wired or wirelessly. The database (200) illustrated in FIG. 1 may be implemented as a component of the memory (110).

[0072] The communication module (120) may include one or more components that enable communication with an external device, and may include, for example, at least one of a broadcast reception module, a wired communication module, a wireless communication module, a short-range communication module, and a location information module.

[0073] The wired communication module may include various wired communication modules such as a Local Area Network (LAN) module, a Wide Area Network (WAN) module, or a Value Added Network (VAN) module, as well as various cable communication modules such as a Universal Serial Bus (USB), a High Definition Multimedia Interface (HDMI), a Digital Visual Interface (DVI), RS-232 (recommended standard 232), power line communication, or plain old telephone service (POTS).

[0074] The wireless communication module may include a wireless communication module that supports various wireless communication methods such as GSM (global System for Mobile Communication), CDMA (Code Division Multiple Access), WCDMA (Wideband Code Division Multiple Access), UMTS (universal mobile telecommunications system), TDMA (Time Division Multiple Access), LTE (Long Term Evolution), 4G, 5G, and 6G, in addition to a WiFi module and a Wireless Broadband module.

[0075] The display (130) displays (outputs) information or data processed in the device (100), data input or output through the AI ​​model (300), etc. In addition, the display (130) can display execution screen information of an application program (e.g., an application) running in the device (100), or UI (User Interface) or GUI (Graphical User Interface) information according to such execution screen information.

[0076] The input module (140) is for receiving information from a user. When information is input through the user input unit, the processor (150) can control the operation of the device (100) to correspond to the input information.

[0077] The input module (140) may include hardware physical keys (e.g., buttons, dome switches, jog wheels, jog switches, etc. located on at least one of the front, rear, and side of the device) and software touch keys. For example, the touch keys may be formed of virtual keys, soft keys, or visual keys displayed on a touchscreen type display (130) through software processing, or may be formed of touch keys placed on a part other than the touchscreen. Meanwhile, the virtual keys or visual keys may be displayed on the touchscreen in various forms, and may be formed of, for example, graphics, text, icons, videos, or a combination thereof.

[0078] The processor (150) may be implemented as a memory that stores data on an algorithm for controlling the operation of components within the device (100) (including learning or executing an AI model) or a program that reproduces the algorithm, and at least one processor (not shown) that performs the aforementioned operation using the data stored in the memory. In this case, the memory and the processor may be implemented as separate chips, or may be implemented as a single chip.

[0079] The processor (150) may be one or more processors and / or processing circuits for executing program code and controlling the basic operation of the device (100). The processor may include a system including a central processing unit (CPU), a graphics processing unit (GPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a complex programmable logic device (CPLD), a dedicated circuit for implementing a function, a special-purpose processor for implementing neural network-based processing, or other system.

[0080] In one embodiment, the system (1000) or device (100) according to the present invention may include at least one processor, and when including multiple processors, the multiple processors may be included in different devices (100).

[0081] In addition, the processor (150) can control any one or a combination of the components described above to implement various embodiments of the present invention described below on the device (100).

[0082] FIG. 3 is a flowchart for explaining a polymer molecular structure recognition method according to embodiments of the present invention, FIG. 4 is a block diagram for explaining a polymer molecular structure recognition method according to embodiments of the present invention, and FIG. 5 is molecular structures for exemplarily explaining a polymer molecular structure recognition method according to embodiments of the present invention. FIG. 6 is a schematic diagram for exemplarily explaining metric learning that can be used in a polymer molecular structure recognition method according to embodiments of the present invention.

[0083] Referring to FIGS. 3 and 4, a method for recognizing a polymer molecular structure according to embodiments includes preparing a polymer molecular structure image (S1010), detecting brackets and subscripts from the polymer molecular structure image to generate detection data (S1020), clustering brackets and subscripts from the detection data to generate cluster data (S1030), extracting repeat numbers of monomers and subscripts using data detected from the polymer molecular structure image (S1040), and outputting the result (S1050).

[0084] The method for recognizing a polymer molecular structure according to the embodiments may be performed by at least one processor. Furthermore, instructions or information that cause the at least one processor to perform operations may be contained in at least one memory. The at least one processor or at least one memory may be contained within multiple devices. Furthermore, the method for recognizing a polymer molecular structure according to the embodiments may be performed by operations performed by the instructions or information, and may include performing iterative learning.

[0085] Each step is explained in detail below.

[0086] First, a method for recognizing a polymer molecular structure according to embodiments may include preparing a polymer molecular structure image (1210) (S1010). The polymer molecular structure image (1210) may include a polymer molecular structure including at least one monomer, and may include a copolymer molecular structure including two or more different types of monomers. For example, as illustrated in (a) of FIG. 5, the polymer molecular structure image (1210) may include a copolymer molecular structure including three types of monomers (1315, 1325, 1335).

[0087] Preparing a polymer molecular structure image (1210) may be performed by at least one processor, and instructions or information that cause the at least one processor to perform an operation may be stored in at least one memory. Preparing a polymer molecular structure image (1210) may include storing the polymer molecular structure image (1210) in a memory included in the processor, a separate volatile or non-volatile memory, etc. The polymer molecular structure image (1210) may include an image stored in the memory, an image transmitted and stored from the outside through a network or communication module, an image transmitted and streamed in real time from the outside through a network or communication module, etc. However, the present invention is not limited thereto, and may include all methods or approaches for preparing a polymer molecular structure image so that the polymer molecular structure image (1210) can be recognized through an operation performed by at least one processor.

[0088] The polymer molecular structure image (1210) may be an image in which the structural formula is expressed graphically. A structural formula may refer to a graphical representation of a chemical structure or molecular structure. The structural formula may include information regarding the arrangement of atoms in three-dimensional space and information regarding chemical bonds between atoms. The polymer molecular structure image (1210) may include annotations, for example, annotations may refer to the name of a compound, etc.

[0089] Referring to the example of FIG. 5, as illustrated in (a) of FIG. 5, the polymer molecular structure image (1210) includes first to third monomers (1315, 1325, 1335), and each monomer is represented by a bracket (1311, 1321, 1331) and a subscript (1313, 1323, 1333) indicating the number of repetitions of the corresponding monomer. The polymer molecular structure image (1210) includes a first group (1310) including a first monomer (1315), a bracket (1311) distinguishing the first monomer (1315), and a subscript (1313), and may further include second and third groups (1320, 1330) including a second monomer (1325) and a third monomer (1335), respectively.

[0090] Here, brackets are symbols used to define or distinguish repeating monomer units within a polymer. That is, the molecules within the brackets represent the structure of each monomer, and the lines extending outside the brackets indicate how the monomer is bonded to other atoms or molecules. Subscripts indicate the number of times a monomer is repeated, and using letters like n or m as subscripts indicates multiple repetitions of the monomer.

[0091] Next, brackets and subscripts are detected in the polymer molecular structure image (1210) (S1020).

[0092] Referring to FIGS. 3 and 4, detecting brackets and subscripts in a polymer molecular structure image (1210) may include detecting the polymer molecular structure image (1210) and outputting detection data (1230) regarding the brackets and subscripts. The detection data (1230) may include information regarding the brackets and subscripts. In addition, the detection data (1230) regarding the brackets and subscripts may include at least one embedding (1231), and the embedding may include an embedding vector in a vector form.

[0093] Detecting a polymer molecular structure image (1210) may include inputting the image into a detector (1220) and outputting detection data (1230) regarding brackets and subscripts. The detector (1220) may include an object detector of the transformer series, and may include a DEtection TRansformer (DETR) or a Deformable DETR, but the present invention is not limited thereto.

[0094] Referring to the example of FIG. 5, brackets (dotted line) and subscripts (dashed line) can be detected by detecting a polymer molecular structure formula image (1210) as shown in (b) of FIG. 5, respectively.

[0095] Meanwhile, each of at least one embedding (1231) included in the detection data (1230) regarding brackets and subscripts may be an embedding including individual information of a polymer molecular structure image (1210). The embedding (1231) may include an embedding vector including information regarding brackets and an embedding vector including information regarding subscripts. The embedding vector may be, for example, an embedding vector having a length of 1024.

[0096] Embedding (1231) may include individual embeddings that include information about each of the brackets (1311, 1321, 1331) and subscripts (1313, 1323, 1333) indicating the number of repetitions of the corresponding monomers, and the individual embeddings may include embedding vectors. Embedding (1231) may include one embedding vector that includes information about the left bracket constituting the brackets, and another embedding vector that includes information about the right bracket.

[0097] Referring to an example of FIG. 5, the polymer molecule included in the polymer molecule structural formula image (1210) illustrated in (a) of FIG. 5 includes first to third brackets (1311, 1321, 1331) and first to third subscripts (1313, 1323, 1333), and information about each bracket and subscript may correspond to a separate embedding vector.

[0098] For example, in embodiments, the embedding (1231) may include an embedding vector (1231a) including information about the left first bracket (1311), an embedding vector (1231b) including information about the right first bracket (1311), and an embedding vector (1231c) including information about the first subscript (1313). In addition, the embedding (1231) may include an embedding vector (1231d, 1231e) including information about the second brackets (1321) on the left and right sides of the second group (1320), respectively, and an embedding vector (1231f) including information about the second subscript (1323). Additionally, the embedding (1231) may include an embedding vector (1231d, 1231e) including information about each of the second brackets (1321) on the left and right sides of the second group (1320), and an embedding vector (1231f) including information about the second subscript (1323).

[0099] Meanwhile, detecting brackets and subscripts in the polymer molecular structure image (1210) may include recognizing the remaining portions of the polymer molecular structure image excluding the brackets and subscripts, and detecting atoms constituting the polymer molecules included in the polymer molecular structure image (1210) and bonding relationships between atoms. In addition, detecting atoms constituting the polymer molecules included in the polymer molecular structure image (1210) and bonding relationships between atoms may include detecting atoms constituting a monomer and bonding relationships between atoms.

[0100] A method for detecting the bonding relationship between atoms constituting a polymer molecule included in a polymer molecular structural formula (1210) according to various embodiments of the present invention will be described in more detail with reference to FIGS. 7 to 11. FIG. 7 is a drawing for explaining a structural formula image according to embodiments of the present invention, FIG. 8 is a drawing for explaining an atomic region recognition model according to embodiments of the present invention, FIG. 9 is a drawing for explaining a plurality of atomic region information according to embodiments of the present invention, FIG. 10 is a drawing for explaining a method for obtaining bonding relationship information according to embodiments of the present invention, and FIG. 11 is a drawing for explaining a bonding relationship recognition model according to embodiments of the present invention.

[0101] First, referring to FIG. 7, the structural formula image (400) may include a structural formula (401) that graphically represents a molecular structure. In addition, the structural formula image (400) may include an annotation (402). When the structural formula image (400) is generated, an annotation may be included as an image as a description or name of the molecular structure in the structural formula image. Therefore, when recognizing the structural formula image (400) and converting it into a predetermined string format, there is a need to filter only the structural formula (401) as a recognition target, excluding the annotation (402) included in the structural formula image (400).

[0102] The atomic region information may include at least one of atomic region identification information, atomic position information, and atomic information in the structural formula image. The atomic region identification information may mean identification information (or number) that can distinguish each of a plurality of atomic regions recognized in the structural formula image.

[0103] Additionally, atomic position information may refer to coordinate information corresponding to an atomic region in a structural image. For example, if an atomic region is represented as a rectangle, the atomic position information may include the coordinates of each vertex of the rectangle corresponding to the atomic region and the coordinates of the center point of the rectangle.

[0104] Additionally, atomic information may include element symbol information of an atom corresponding to an atomic region. For example, if an image corresponding to an atomic region is represented by a vertex, carbon (C) may correspond to the atomic information. Furthermore, if an element symbol (e.g., oxygen (O)) is described in the image corresponding to an atomic region, information about the element symbol (O) may correspond to the atomic information.

[0105] Meanwhile, atomic region information may include reliability information of atomic region information obtained from a structural image. For example, the processor may obtain reliability information of atomic region information obtained from a structural image as a value between 0.00 and 1.00, and may only use atomic region information having reliability information greater than a predetermined value.

[0106] Referring to FIG. 8, the processor can input a structural formula image (501) into an atomic region recognition model (502) and obtain atomic region information output from the atomic region recognition model (502).

[0107] The atomic region recognition model (502) may be an artificial neural network (ANN) trained to output at least one atomic region information (503) included in an input structural image (501). An artificial neural network (ANN) is a model used in machine learning and may refer to a general model having problem-solving capabilities, which is composed of artificial neurons (nodes) that form a network by combining synapses. For example, the atomic region recognition model (502) may be an artificial neural network model based on CNN (Convolutional Neural Networks).

[0108] The processor can train an atomic region recognition model (502) composed of an artificial neural network using learning data regarding structural formula images and atomic region information. Meanwhile, the atomic region recognition model (502) may be a model trained by the processor.

[0109] The learned atomic domain recognition model (502) may be stored in memory or in a storage unit of a server. The processor may be enabled to perform operations on the model stored in memory, etc.

[0110] Meanwhile, referring to FIG. 9, the processor can obtain a plurality of pieces of atomic region information (601, 602, 603, 604, 605, 606, 607, 608, 609, 610, 611) output from the atomic region recognition model (502). The plurality of pieces of atomic region information can include information about vertex regions (601, 602, 603, 604, 605, 606, 608, 609, 610) in the structural formula, an element symbol region (607) in which an element symbol is described, and an annotation region (611). For example, since the atomic region recognition model (502) can be trained to also output an element symbol region (607) in which an element symbol is described, there may also occur cases in which the atomic region recognition model (502) outputs an annotation region (611) as atomic region information. Accordingly, there may be a need to filter information about annotation regions from multiple atomic region information. For example, since the annotation region (611) does not show a binding relationship with other atomic regions, the processor (180) may classify the annotation region (611) as an annotation if no binding relationship exists between the annotation region (611) and other atomic regions. Meanwhile, the processor may obtain binding relationship information between multiple atoms based on the multiple atomic region information.

[0111] The processor can obtain, based on a plurality of pieces of atomic region information, information on the bonding relationship between each atomic region and other atomic regions. The processor can obtain a bonding image between a first atom and a second atom based on first atomic region information and second atomic region information among the plurality of pieces of atomic region information, and obtain bonding relationship information between the first atom and the second atom based on the obtained bonding image.

[0112] Referring to FIG. 10, a method for obtaining bonding relationship information is described. The processor may select first atomic region information (301) from among a plurality of atomic region information. In addition, the processor may select second atomic region information (602) that is different from the first atomic region information. In addition, the processor may obtain a bonding image (701) between the first atom and the second atom based on the first atomic position information (702) of the first atomic region information (601) and the second atomic position information (703) of the second atomic region information (602). In this case, the first atomic position information (702) and the second atomic position information (703) may be the center point positions of each atomic region. However, they are not limited to the center point positions of the atomic regions.

[0113] The processor may obtain a combined image (701) including the center point position of each atomic region based on the first atomic position information (702) of the first atomic region information (601) and the second atomic position information (703) of the second atomic region information (602). In addition, the processor may obtain a combined image (704) including the first atomic region and the second atomic region based on the first atomic region information (601) and the second atomic region information (602). The size and shape of the combined image may be adjusted in various ways.

[0114] Meanwhile, the processor can obtain a bond image between combinable atoms for each of the multiple atomic regions. However, obtaining a bond image between all combinable atoms can increase the computational complexity. Therefore, based on information from the multiple atomic regions, the processor can only obtain a bond image between the first and second atoms within a predetermined distance from each other.

[0115] For example, the processor may select a second atomic region located within a predetermined distance from the first atomic region based on first atomic region information among a plurality of atomic region information. Referring to FIG. 10, the processor may specify second atomic regions (602, 603, 608, 609, 610) located within a predetermined distance from the first atomic region (601), and obtain bonding images between the first atom (601) and each of the second atoms, thereby obtaining bonding relationship information between the first atom and the second atoms.

[0116] Additionally, the processor can determine that the third atomic regions (604, 605, 606, 607, 611) located outside a predetermined distance from the first atomic region (601) have no coupling relationship. Therefore, the amount of computation can be reduced.

[0117] Meanwhile, the processor can obtain information on bonding relationships between atoms based on the acquired bonding image. For example, the processor can input the bonding image into a bonding relationship recognition model and obtain bonding relationship information output from the bonding relationship recognition model.

[0118] FIG. 11 is a drawing for explaining a combination relationship recognition model. Referring to FIG. 11, a processor can input a combination image (801) into a combination relationship recognition model (802) and obtain combination relationship information output from the combination relationship recognition model (802).

[0119] The bonding relationship information may include information about bonding between atoms, such as unbonded, single bonded, double bonded, triple bonded, up-directed bonded, down-directed bonded, etc. Unbonded may mean a case where there is no bonding between atoms. Up-directed bonding may mean a bonding that comes in front of a plane indicated by a wedge. In addition, down-directed bonding may mean a bonding that goes behind a plane indicated by a dash. The bonding relationship recognition model (802) may be an artificial neural network (ANN) trained to output bonding relationship information (803) for an input bonding image (801). An artificial neural network (ANN) is a model used in machine learning, and may mean a model having a problem-solving ability, which is composed of artificial neurons (nodes) that form a network by combining synapses. For example, the bonding relationship recognition model (802) may be an artificial neural network model based on CNN (Convolutional Neural Networks).

[0120] The processor can train a combination relationship recognition model (802) composed of an artificial neural network using learning data regarding combination images and combination relationship information.

[0121] The learned association relationship recognition model (802) can be stored in memory. The processor can use the association relationship recognition model (802) stored in the memory.

[0122] Meanwhile, the processor can generate an adjacency matrix based on information about multiple atomic regions and information about bonding relationships between multiple atoms. The processor can generate an adjacency matrix with each of the multiple atoms as a vertex and bonding relationship information for each of the multiple atoms as an edge.

[0123] FIG. 11 is a diagram for explaining an adjacency matrix. Referring to FIG. 11, the processor can generate each atom of a plurality of atomic regions (601 to 611) as a vertex of an adjacency matrix. Meanwhile, the processor can generate an adjacency matrix using bonding relationship information of each of the plurality of atoms as an edge. For example, the processor can generate each bonding relationship information as an edge value of the adjacency matrix by corresponding an arbitrary number. For example, the processor can correspond a non-bond to '0', a single bond to '1', a double bond to '2', and a triple bond to '3'. In addition, the processor can correspond an up-direction bond to '5' when the row is a starting vertex and the column is a destination vertex, and a combination in the up direction in the row and column order, and correspond it to '0' for the opposite direction.

[0124] For example, referring to FIG. 11, the value of the 6th column of the 5th row of the adjacency matrix may be generated as a corresponding value of '5' since an upward coupling is performed from the first atom (605) to the second atom (606). Meanwhile, the value of the 5th column of the 6th row of the adjacency matrix in the opposite direction may be generated as '0'. Similarly, the processor (180) may correspond the downward coupling to '6' when the row is coupled in the downward direction in the order of the row and column with the row as the starting vertex and the column as the arrival vertex to indicate directionality, and may correspond the opposite direction to '0'.

[0125] Meanwhile, the processor can generate a predetermined string format corresponding to the structural formula image based on the generated adjacency matrix. The processor can acquire information on bonding relationships with other atoms by traversing each of the atomic regions corresponding to the vertices of the adjacency matrix. Using the acquired bonding relationship information between atoms, the processor can specify atomic information for each of the plurality of atomic regions, and generate a predetermined string format corresponding to the structural formula image based on the plurality of atomic information and the bonding relationship information between the plurality of atoms.

[0126] In this case, the string format may include a file format that can represent information about compounds (e.g., element positions, bonding relationships, etc.), such as a mol file format or an sdf file format. Meanwhile, the string format may include information about SMILES (Simplified Molecular Input Line Entry System).

[0127] Meanwhile, the method for detecting atoms of polymer molecules, bonding relationships between atoms, etc. from polymer molecular structural formula images according to embodiments of the present invention is not limited to that described with reference to FIGS. 7 to 11 above, and various structural formula image detection methods can all be used.

[0128] Furthermore, in the polymer molecular structure recognition method according to embodiments of the present invention, detecting the atoms constituting the polymer molecule and the bonding relationship between the atoms included in the polymer molecular structure image (1210) is not necessarily included in the step of detecting brackets and subscripts in the polymer molecular structure image (1210), and detecting the atoms constituting the polymer molecule and the bonding relationship between the atoms may be included in another step or implemented as a separate step.

[0129] Referring again to FIGS. 3 and 4, cluster data is then generated by clustering brackets and subscripts in the detection data (S1030).

[0130] Generating cluster data by clustering brackets and subscripts from detection data may include inputting the detection data into a first model (1241) and a second model (1242), respectively, outputting first cluster data including at least one of class information and coordinate information of brackets and subscripts from the first model (1241), and outputting second cluster data including group information about brackets and subscripts from the second model (1242). The cluster data may include the first cluster data and the second cluster data.

[0131] The first model (1241) and the second model (1242) may be preset algorithms or pre-trained models. At least one of the first model (1241) and the second model (1242) may be executed or trained by the processor.

[0132] The first cluster data output from the first model (1241) may contain different information from the second cluster data output from the second model (1242).

[0133] In embodiments, detection data including embeddings (1231) may be input into a first model (1241), and class information and coordinate information of objects (brackets, subscripts) corresponding to each embedding (1231) may be output. For example, referring to (a) of FIG. 5, a polymer molecule structural formula image (1210) is projected onto a space having an x-axis and a y-axis, and each detection object (bracket, subscript, atom, atomic bond, etc.) of a polymer molecule may have coordinates along the x-axis and y-axis of the space.

[0134] The first model (1241) may include a linear layer composed of matrices. In one embodiment, the first model (1241) may include a linear layer including a matrix (e.g., 1024 x 4) that derives class probabilities for brackets, subscripts, atoms, and carbon (C) from the embedding (1231), and a linear layer including a matrix (e.g., 1024 x 4) that derives coordinate information of detection targets such as brackets and subscripts from the embedding (1231).

[0135] The class information output from the first model (1241) may include classification information of the object corresponding to each embedding (1231), i.e., information regarding what type of object the object refers to. For example, when an embedding (1231) regarding a bracket (1311, 1321, 1331) is input into the first model (1241) and related information is output, class information indicating that the information regarding the embedding (1231) corresponds to a 'bracket' may be output.

[0136] By outputting first cluster data including class information and coordinate information of brackets and subscripts through the first model (1241), information can be output regarding which monomer the brackets and subscripts correspond to, where the brackets and subscripts are located in the bonding relationship between other atoms, etc.

[0137] In embodiments, detection data including embeddings (1231) may be input to a second model (1242) to output group information of objects (brackets, subscripts) corresponding to each embedding (1231). The group information may include information regarding which brackets and subscripts form one monomer group among the brackets and subscripts.

[0138] The second model (1242) may include a layer that remaps the input embedding (1231) to a different new space. The second model (1242) may include a matrix (e.g., 1024 x 1024) that projects the input embedding (1231) to an embedding of a different length (e.g., 1024 length). The second model (1242) may include a metric function.

[0139] Referring to FIG. 6, the second model (1242) including the metric function enables the embeddings of the original feature space to be clustered (grouped) in a new feature space. Accordingly, the second model (1242) may include generating a matrix that projects the detection data into a new, different feature space.

[0140] The second model (1242) may be a model learned by various methods, for example, but is not limited to, a model learned based on infoNCE loss.

[0141] For example, the second model (1242) is executed or trained by defining positive pairs and negative pairs using infoNCE loss and assigning loss. A positive pair means something that corresponds to the ground truth, and for example, the first brackets (1311) and the first subscript (1313) included in the first group (1310) corresponding to the first monomer (1315) output a result as a positive pair. On the other hand, if the answer is not to be grouped as one group, such as the first brackets (1311) and the second subscript (1323), the corresponding negative result is output.

[0142] By outputting second cluster data including group information about brackets and subscripts through the second model (1242), information regarding which brackets and subscripts among the detected brackets and subscripts are included in which group corresponding to which monomer can be output. For example, referring to (c) of FIG. 5, the second model (1242) can output second cluster data including information that the first brackets (1311) and the first subscript (1313) are one group included in the first group (1310) corresponding to the first monomer (1315), and can also output second cluster data including similar group information about the brackets and subscripts included in the remaining second and third groups (1320, 1330).

[0143] According to embodiments of the present invention, a system using a polymer molecular structure recognition method according to the present invention can accurately recognize a copolymer molecular structure including two or more different types of monomers by simultaneously including a first model (1241) that detects brackets and subscripts of a polymer molecular structure and outputs class information and coordinate information, and a second model (1242) that outputs group information regarding brackets and subscripts.

[0144] Referring again to FIGS. 3 and 4, the repeat numbers of monomers and subscripts detected from the polymer molecular structure image are extracted (S1040), and the results are output (S1050).

[0145] Extracting the repeating numbers of monomers and subscripts detected from a polymer molecular structure image can extract the repeating numbers of monomers and subscripts from information detected through a method of detecting atoms constituting a polymer molecule and bonding relationships between atoms, but the present invention is not limited thereto, and the repeating numbers of monomers and subscripts can be detected through various molecular structure recognition methods.

[0146] In the embodiments, the method described with reference to FIGS. 7 to 11 can be used to extract the repeating numbers of monomers and subscripts detected from the polymer molecular structure image, but the present invention is not limited thereto.

[0147] Outputting the result (S1050) can generate a predetermined string format corresponding to a polymer molecular structure image based on a plurality of atomic information, bonding relationship information between a plurality of atoms, and cluster data.

[0148] In this case, the string format may include a file format that can represent information about compounds (e.g., element positions, bonding relationships, monomer information, etc.), such as a mol file format or an sdf file format. Meanwhile, the string format may include information about SMILES (Simplified Molecular Input Line Entry System).

[0149] Meanwhile, the polymer molecular structure recognition method according to embodiments of the present invention can be implemented by the system described with reference to FIGS. 1 and 2.

[0150] Additionally, the disclosed embodiments may be implemented in the form of a recording medium storing computer-executable instructions. The instructions may be stored in the form of program code, and when executed by a processor, may generate program modules to perform the operations of the disclosed embodiments. The recording medium may be implemented as a computer-readable recording medium.

[0151] Computer-readable storage media include all types of storage media that store instructions that can be deciphered by a computer. Examples include read-only memory (ROM), random access memory (RAM), magnetic tape, magnetic disks, flash memory, and optical data storage devices.

[0152] The disclosed embodiments have been described with reference to the attached drawings. Those skilled in the art will appreciate that the present invention can be implemented in forms other than the disclosed embodiments without altering the technical spirit or essential features of the present disclosure. The disclosed embodiments are illustrative and should not be construed as limiting.

Claims

1. A polymer molecular structure recognition system using artificial intelligence, at least one processor; and At least one memory storing instructions or information that cause at least one processor to perform an operation, The actions performed by the above commands or information are: Detecting a polymer molecular structure image to generate detection data including information about brackets and subscripts; and A polymer molecular structure recognition system comprising inputting the detection data into a first model and a second model, respectively, outputting first cluster data from the first model, and outputting second cluster data including group information regarding the brackets and subscripts and including information different from the first cluster data from the second model.

2. In claim 1, A polymer molecular structure recognition system, wherein the polymer comprises two or more different types of monomers.

3. In claim 1, A polymer molecular structure recognition system, wherein the first cluster data includes at least one of class information and coordinate information of the bracket and subscript.

4. In claim 1, The second model is a polymer molecular structure recognition system including a metric function.

5. In claim 1, Detecting the polymer molecular structure image and generating detection data including information about brackets and subscripts includes inputting the polymer molecular structure image into a detector and outputting the detection data. The above detector is a polymer molecular structure recognition system including a Detection TRansformer (DETR) or a Deformable DETR.

6. In claim 1, The above detection data is a polymer molecular structure recognition system including an embedding vector.

7. In claim 6, A polymer molecular structure recognition system, wherein at least one of the first model and the second model comprises a matrix including the embedding vector.

8. In claim 2, A polymer molecular structure recognition system, wherein the group information regarding the brackets and subscripts includes information regarding which brackets and subscripts among the detected brackets and subscripts are included in a group corresponding to which monomer.

9. In claim 1, A polymer molecular structure recognition system, wherein the operation performed by the above command or information further includes converting the structural formula image into a predetermined string format including SMILES (Simplified Molecular Input Line Entry System) based on cluster data and outputting the converted structural formula image.

10. In claim 1, A polymer molecular structure recognition system, wherein the operation performed by the above command or information further includes obtaining a plurality of atomic region information from the polymer molecular structure image, and obtaining bonding relationship information between a plurality of atoms based on the plurality of atomic region information.

11. A method for recognizing a polymer molecular structure using artificial intelligence performed by at least one processor, Detecting a polymer molecular structure image to generate detection data including information about brackets and subscripts; and A polymer molecular structure recognition method comprising inputting the detection data into a first model and a second model, respectively, outputting first cluster data from the first model, and outputting second cluster data including group information regarding the brackets and subscripts and including information different from the first cluster data from the second model.

12. In claim 11, A method for recognizing a polymer molecular structure, wherein the polymer comprises two or more different types of monomers.

13. In claim 12, A method for recognizing a polymer molecular structure, wherein the group information regarding the brackets and subscripts includes information regarding which brackets and subscripts among the detected brackets and subscripts are included in a group corresponding to which monomer.

14. In claim 12, A method for recognizing a polymer molecular structure, wherein outputting second cluster data from the second model includes generating a matrix that projects the detection data into another feature space.

15. A program stored in a computer-readable recording medium for executing the method of any one of claims 11 to 14, in combination with a computer.

Images