Apparatus and method for predicting line-connected objects using artificial intelligence

Abstract

The present invention is characterized in that, when recognizing a molecular structure image representing the molecular structure of a compound, the atoms that make up the molecular structure are detected as nodes and the bonds as edges, and when detecting edges for bonds, the detection information is utilized for the nodes.

Description

[Technical Field]

[0001] The present disclosure relates to a line-connected object prediction device, and more particularly to a line-connected object prediction device and method that utilizes artificial intelligence. [Background technology]

[0002] In general, a conventional line-connection type object prediction device predicts result information regarding a state in which objects in an image are connected by lines.

[0003] Such a line-connection type object prediction device predicted result information regarding the state in which objects in a map are connected with lines, result information regarding the state in which objects in a drawing are connected with lines, and result information regarding the state in which objects in a structural formula are connected with lines.

[0004] For example, a chemical formula is a pictorial representation of a molecule's structure, showing how the atoms are arranged. Such a chemical formula can explicitly or implicitly represent the chemical bonds within the molecule.

[0005] Unlike chemical formulas and chemical names, structural formulas can represent molecular structures. Therefore, chemical reactions and synthesis are usually explained using structural formulas rather than chemical names, because the changes occurring in molecules can be visualized through the structural formulas.

[0006] Although multiple compounds have different structures, the overall chemical formula exists in the same isomer. Therefore, a structural formula represents an arrangement of atoms that cannot be represented by a chemical formula.

[0007] However, conventional linear-connection type object prediction devices have limitations in reducing errors in predicting bonds as atoms when the linear-connection type object is a structural formula, and also have limitations in improving the analytical accuracy and calculation speed of the structural formula.

[0008] Therefore, recently, research has been conducted on an improved line-connected object prediction device that can reduce prediction errors and improve analysis accuracy and calculation speed when predicting objects connected by lines. [Prior art documents] [Patent documents]

[0009] [Patent Document 1] Korean Patent Publication No. 10-2022-0112692 Summary of the Invention [Problem to be solved by the invention]

[0010] The embodiments disclosed in the present disclosure aim to provide a method for improving analysis accuracy and calculation speed while reducing prediction errors when predicting a state in which objects are connected by lines.

[0011] The problems to be solved by the present disclosure are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the following description. [Means for solving the problem]

[0012] According to an embodiment of the present invention, there is provided an apparatus for predicting a linear-connection type object using artificial intelligence, the apparatus comprising: a memory; and a processor electrically connected to the memory and configured to control an operation related to the linear-connection type object prediction,

[0013] The processor:

[0014] receiving an image including the molecular structure image;

[0015] The method detects nodes corresponding to atoms of a molecular structure formula from the received image to generate node detection information, detects edges corresponding to bonds of the molecular structure formula based on the node detection information and the received image to generate edge detection information, and outputs prediction result information of the molecular structure formula image including the node detection information and the edge detection information.

[0016] In this case, the processor can output a global feature map for the received image via a backbone network.

[0017] The processor may then use a Region Proposal Network (RPN) to extract a plurality of candidate regions (region proposals) that are predicted to be nodes based on the output global feature map.

[0018] The processor may also classify and detect nodes based on the overall feature map and the extracted candidate regions using a region of interest (ROI) module, and output node detection information.

[0019] The ROI module can also determine positive samples for a node from among the plurality of candidate regions via a proposal target layer.

[0020] In addition, the ROI module can perform RoI pooling based on the determined positive samples and the global feature map, and output a feature map of nodes with a fixed size for the positive samples.

[0021] In addition, the ROI module inputs the feature map of the node into a pulley-connected layer to output a feature vector, classifies the output feature vector into classes (classifier), and performs bounding box regression to output node detection information for the molecular structure image.

[0022] In addition, the processor may detect edges based on the node detection information and the overall feature map using a Line Of Interest (LOI) module, which is a CNN-based edge detector, and output the edge detection information.

[0023] The processor:

[0024] A graph form reconstruction operator converts a graph form representation of a molecular structure into data based on object detection information including the node detection information and the edge detection information, and outputs predicted result information of the molecular structure image.

[0025] The processor can also perform an auxiliary task of feeding back edge detection information output from the LOI module to the ROI module and re-executing node detection.

[0026] The processor may also train the LOI module, ROI module, region proposal network, and backbone based on a training dataset in which node labels and edge labels are set for training images including molecular structure images.

[0027] The processor can also simultaneously apply node labels and edge labels to the molecular structure image based on a unifying loss for the nodes and edges, and train the LOI module, ROI module, region proposal network, and backbone.

[0028] The processor further classifies edges by edge class through a segmentation module and outputs edge classification information;

[0029] The output edge classification information may further be included in the object detection information.

[0030] Additionally, a computer-readable recording medium may be provided that records a computer program for executing a method for realizing the present disclosure. [Effects of the Invention]

[0031] According to the above-described problem-solving means of the present disclosure, when predicting a state in which objects are connected by lines, it is possible to reduce prediction errors while improving analysis accuracy and calculation speed.

[0032] The effects of the present disclosure are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description. [Brief explanation of the drawings]

[0033] [Figure 1] FIG. 1 illustrates a line-connection type object prediction system utilizing artificial intelligence according to the present disclosure. [Figure 2] The configuration of the line-connection type object prediction device of FIG. [Figure 3] 3 is a diagram illustrating an example of a process of outputting line-connection type object prediction result information through the processor of FIG. 2. [Figure 4]Fig. 4A is a diagram illustrating an example of a process of outputting linear-connection type object prediction result information through the processor of Fig. 2. Fig. 4B is a diagram illustrating an example of a process of outputting linear-connection type object prediction result information through the processor of Fig. 2. Fig. 4C is a diagram illustrating an example of a process of outputting linear-connection type object prediction result information through the processor of Fig. 2. [Figure 5] 1 is a flowchart illustrating a method for predicting a line-connected object using artificial intelligence according to the present disclosure. [Figure 6] 3 is a diagram showing an example of a process for detecting atoms, bonds, and lines in the processor of FIG. 2. FIG. [Figure 7] 3 is a diagram illustrating an example of a process of generating an artificial intelligence model based on connection information between atoms, bonds, and lines in the processor of FIG. 2. FIG. [Figure 8] 3 is a diagram illustrating an example of a process of generating an artificial intelligence model based on connection information between atoms, bonds, and lines in the processor of FIG. 2. FIG. [Figure 9] 3 is a diagram illustrating an example of a process of generating an artificial intelligence model based on connection information between atoms, bonds, and lines in the processor of FIG. 2. FIG. [Figure 10] 3 is a diagram illustrating an example of a process of generating an artificial intelligence model based on connection information between atoms, bonds, and lines in the processor of FIG. 2. FIG. [Figure 11] 3 is a diagram illustrating an example of a process of generating an artificial intelligence model based on connection information between atoms, bonds, and lines in the processor of FIG. 2. FIG. [Figure 12] 3 is a diagram illustrating an example of a process of generating an artificial intelligence model based on connection information between atoms, bonds, and lines in the processor of FIG. 2. FIG. [Figure 13]3 is a diagram illustrating an example of a process of generating an artificial intelligence model based on connection information between atoms, bonds, and lines in the processor of FIG. 2. FIG. DETAILED DESCRIPTION OF THE INVENTION

[0034] The same reference numerals refer to the same components throughout this disclosure. This disclosure does not describe all elements of the embodiments, etc., and general content in the technical field to which the disclosure belongs or content that overlaps between the embodiments, etc. will be omitted. The terms "unit, module, component, block" used in the specification can be realized by software or hardware, and depending on the embodiment, multiple "units, modules, components, blocks" may be realized by one component, or one "unit, module, component, block" may include multiple components.

[0035] Throughout this specification, when a part is said to be "coupled" to another part, this includes not only direct coupling but also indirect coupling, and indirect coupling includes coupling via a wireless communication network.

[0036] The terms "first," "second," etc. are used to distinguish one component from another, and the components are not limited to the terms described above.

[0037] Hereinafter, the principles of operation and embodiments of the present disclosure will be described with reference to the accompanying drawings.

[0038] In this specification, the linear-connection type object prediction device using AI according to the present disclosure includes various devices that can perform calculation processing and provide a result to a user. For example, the linear-connection type object prediction device using AI according to the present disclosure may include all or any one of a computer, a server device, and a portable terminal.

[0039] Here, the computer may include, for example, a notebook, desktop, laptop, tablet PC, slate PC, etc. equipped with a web browser.

[0040] The server device is a server that communicates with external devices and processes information, and may include an application server, a computing server, a database server, a file server, a proxy server, a web server, and the like.

[0041] The portable terminal is, for example, a wireless communication device that ensures portability and mobility, and may include all kinds of handheld-based wireless communication devices such as PCS (Personal Communication System), GSM (Global System for Mobile communications), PDC (Personal Digital Cellular), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), WiBro (Wireless Broadband Internet) terminals, smartphones, etc., as well as wearable devices such as watches, rings, bracelets, anklets, necklaces, glasses, contact lenses, or head-mounted devices (HMDs).

[0042] In this specification, the linear-connection type object prediction apparatus may be realized by including at least one of the above-mentioned computer, server device, and portable terminal. For example, the linear-connection type object prediction apparatus may be one of the computer, server device, and portable terminal, or may be realized as a system in which a server device performs a prediction method in the form of a web service and provides the service result to a computer and / or portable terminal.

[0043] The line-connected type object prediction device using artificial intelligence according to the present disclosure may receive an image including line-connected type objects, detect a feature map for a state in which objects are connected by lines from the image in order to perform AI-based learning, learn an AI model that outputs object detection information for a state in which objects are connected by lines based on the detected feature map, and output line-connected type object prediction result information for the input image via the learned AI model.

[0044] Such a line-connected object prediction device utilizing artificial intelligence can reduce prediction errors and improve analysis accuracy and calculation speed when predicting a state in which objects are connected by lines.

[0045] The following describes in detail a line-connection type object prediction device that utilizes artificial intelligence.

[0046] Fig. 1 is a diagram showing a line-connected type object prediction system utilizing artificial intelligence according to the present disclosure. Fig. 2 shows the configuration of the line-connected type object prediction device of Fig. 1. Fig. 3 and Figs. 4A to 4C are diagrams showing an example of a process of outputting line-connected type object prediction result information via the processor of Fig. 2.

[0047] As shown in FIGS. 1 to 4C, a line-connection type object prediction system utilizing artificial intelligence may include an imaging device 10 and a prediction device 100.

[0048] The imaging device 10 may acquire an image including a line-connected object and transmit the image to the prediction device 100. The prediction device 100 may predict (digitize) the line-connected object by training an artificial intelligence model that performs object detection on images displaying the bond structures of compounds in various formats. In this case, the prediction device 100 may include a communication unit 110 and a control unit 120.

[0049] In the following, the description will be limited to linearly connected objects being compounds. Therefore, the prediction device 100 can recognize the molecular structure image of a compound, predict the molecular structure information constituting the compound, and digitize it. Furthermore, based on the digitized information on the molecular structure, the prediction device 100 can convert and provide files in various compound naming formats or various structural formula images.

[0050] The communication unit 110 may communicate with the image capture device 10. Here, the communication unit 110 may receive images P1, P2, P3, ..., each including a line-connected object, acquired from the image capture device 10. Here, the line-connected object may include at least one of a map, a drawing, and a structural formula. In this case, the structural formula may include at least one of a Lewis structural formula, a simplified structural formula, a skeletal structural formula, and a projection formula. For example, the skeletal structural formula may be a stereochemical structural formula, an unspecified stereochemical structural formula, etc. As another example, the projection formula may be a Newman projection, a Haas projection, a Fischer projection, etc.

[0051] That is, when the prediction device 100 predicts a line-connected type object, it means that it recognizes an image representing the molecular structure of a compound and digitizes the molecular structural information of the compound. The prediction device 100 can convert the digitized molecular structural information into the various formats mentioned above or other formats and provide it, or can use it as a data set to be used for model learning for various physical property analyses.

[0052] Here, the digitized molecular structural information includes nodes representing atoms and / or molecules, which are objects that make up the molecular structure, positions relative to lines representing bonds, and positional relationship information, so it can include all geometric information regarding specific faces of the compound represented by the molecular structure image.

[0053] In this case, the communication unit 110 may include at least one of a wired communication module and a wireless communication module.

[0054] The wired communication module may include various wired communication modules such as a local area network (LAN) module, a wide area network (WAN) module, or a value added network (VAN) module, as well as various cable communication modules such as a universal serial bus (USB), a high definition multimedia interface (HDMI), a digital visual interface (DVI), recommended standard 232 (RS-232), power line communication, or plain old telephone service (POTS), and a processor and / or memory for driving the same.

[0055] The wireless communication module may include a Wi-Fi module, a WiBro (Wireless Broadband) module, as well as a wireless communication module that supports various wireless communication methods such as GSM (global System for Mobile Communication), CDMA (Code Division Multiple Access), WCDMA (Wideband Code Division Multiple Access), UMTS (universal mobile telecommunications system), TDMA (Time Division Multiple Access), LTE (Long Term Evolution), 4G, 5G, and 6G, and a processor and / or memory for driving the same.

[0056] The control unit 120 may include a memory 121 and a processor 122 .

[0057] The memory 121 may store data for an algorithm or a program reproducing the algorithm for controlling the operation of components within the device. The processor 122 may communicate with the memory 121 and perform the above-described operations using the data stored in the memory 121. In this case, the processor 122 may control operations related to linear-connection type object prediction. Here, the memory 121 and the processor 122 may be implemented on separate chips. Alternatively, the memory 121 and the processor 122 may be implemented on a single chip.

[0058] The memory 121 can store data supporting various functions of the device, programs for operating components within the device, input / output data, a plurality of application programs (or applications) run by the device, and data and commands for operating the device, at least some of which can be downloaded from an external server via wireless communication.

[0059] Such memory 121 may include at least one type of storage medium among flash memory type, hard disk type, solid state disk type (SSD type), silicon disk drive type (SDD type), multimedia card micro type, card type memory (e.g., SD or XD memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, and optical disk.

[0060] The memory 121 may store images P1, P2, P3, etc. including line-connected objects. The memory 121 may also store at least one of map image data, drawing image data, and structural image data, which are metadata for each line-connected object. In this case, the structural image data may include at least one of Lewis structural image data, simplified structural image data, skeletal structural image data, and projection image data. The memory 121 may store line-connected object prediction result information output through an artificial intelligence model trained based on object detection information.

[0061] The processor 122 may receive images P1, P2, P3, ... including line-connected objects via the communication unit 110. In addition, in order to perform AI-based learning, the processor 122 may detect a feature map for a state in which objects are connected by lines from the images P1, P2, P3, ... including line-connected objects, and may generate an AI model through learning that can detect object detection information regarding a state in which nodes are connected by lines from the feature map. Therefore, using the trained AI model, the processor 122 may generate object detection information regarding nodes and a state in which the nodes are connected by lines using a neural network-based object detector based on the feature map.

[0062] Here, the object detection information includes information about the objects that make up the structure of a compound, such as atoms and atomic groups (hereinafter referred to as "atoms"), and the bonds between them.

[0063] In addition, since the object detection information is information on detected atoms and bonds in a molecular structure image, it may further include position information on each object. Here, the position information may mean information on atoms connected to the detected bonds, and may further include position information on each atom.

[0064] Furthermore, the object detection information may further include information about the type of each bond. For example, the object detection information may further include information about whether the bond is single, double, or triple, and classifying it into classes such as none, up, etc., which can classify chiral and stereotypes.

[0065] For example, if the line-connected object is a structural formula, the processor 122 can receive an image including the structural formula, detect a feature map from the image including the structural formula, and generate object detection information regarding a state in which atoms and bonds are connected by lines from the feature map using an object detector. In this case, the processor 122 can first detect atoms that will become nodes and then detect bonds that will become edges.

[0066] 4A, the processor 122 may input at least one type of image data among the Lewis structure image data ID1, simplified structure image data ID2, skeletal structure image data ID3, and projected image data ID4 corresponding to structural formula image data among metadata, and train an artificial intelligence model AIM using object detection information as an output value, thereby outputting a recommended linear-connected structural formula prediction result information OD. The artificial intelligence model AIM may construct various Lewis structure image data ID1, various simplified structure image data ID2, various skeletal structure image data ID3, and various projected image data ID4 as a training dataset and perform reinforcement learning using an object detector. In this case, the memory 121 may store the linear-connected structural formula prediction result information OD trained and analyzed based on the artificial intelligence model AIM.

[0067] To predict a line-connected type object, the input unit 130 may receive an input of an image including a line-connected type object. Here, a user may directly input an image including a line-connected type object using the input unit 130. For example, a user may scan an image including a line-connected type object and directly input the scanned image including the line-connected type object using the input unit 130. Meanwhile, the input unit 130 may be any input means capable of inputting an image including a line-connected type object.

[0068] The processor 122 may output linear-connection type object prediction result information learned and analyzed based on the artificial intelligence model AIM corresponding to the input information of the input unit 130. For example, the processor 122 may output linear-connection type structural formula prediction result information OD learned and analyzed based on the artificial intelligence model AIM.

[0069] The display unit 140 can also display linear-connection type object prediction result information output by the processor 122. For example, the display unit 140 can display linear-connection type structural formula prediction result information OD. In this case, the display unit 140 can display linear-connection type Lewis structure prediction result information, linear-connection type simplified structural formula prediction result information, linear-connection type skeletal structure prediction result information, and linear-connection type projection type prediction result information.

[0070] More specifically, as shown in FIG. 4B, the artificial intelligence model according to the embodiment may include an object detector (molecule entity detector) that is trained to detect objects (e.g., nodes and edges) representing the structure of a compound in the molecular structure formula when a molecular structure image is input and output object detection information, an operator (Graph Reconstruction operator) that reconstructs the data into graph form data based on the output object detection information, and a converter (Chemical file converter) that converts the reconstructed graph form data into chemical names / structure formulas in various formats that represent the molecular structure of the compound.

[0071] In an embodiment, the object detector, the operator, and the converter may each be implemented as a separate module, i.e., the artificial intelligence model may be implemented as a module consisting of three parts with different algorithms (e.g., neural networks).

[0072] In recent years, it has become common to use NLP (nature language processing) artificial intelligence models to perform end-to-end vision testing, but this method is not suitable for cases where accurate analysis of each detailed object is required, such as the molecular structure information of a compound. For example, this method is not suitable for accurately recognizing each detailed object representing atoms and bonds, such as a molecular structure image, to digitize the molecular structure information and generate prediction result information for line-connected objects.

[0073] Furthermore, once nodes, which are objects representing atoms, and lines, which are objects representing bonds, are clearly detected in a molecular structure image, the operator that reconstructs the data into graph form data based on this and the converter that converts the graph form data into various formats that represent compounds are already developed technologies (e.g., GNN, RDKit library).

[0074] Therefore, if an object detector that accurately outputs information on object detection of molecular structures from molecular structure formula images is developed, it is possible to easily digitize molecular structure information of compounds using existing technology.

[0075] In addition, there is a large amount of chemical naming / structural formula sample data in various formats matched with molecular structure images, making it easy to build training datasets in the future.

[0076] Therefore, the object detector of this embodiment can be an artificial intelligence model optimized to detect nodes representing atoms that constitute a compound and lines representing bonds from a molecular structure image of the compound.

[0077] In one embodiment, the object detector can be an image detection model including a neural network trained on a training dataset in which object (nodes and edges) detection information is mapped to molecular structure images, where nodes are objects corresponding to atoms in a compound molecular structure and edges are objects corresponding to bonds.

[0078] Furthermore, in one embodiment, the object detector may be an image detection / construction model that is combined with the object detection model and a graph reconstruction operator to output object detection information in the form of a graph.

[0079] In this case, the graph representation of the compound mapped to the molecular structure image can be an image detection / construction model including multiple neural networks trained on a training dataset.

[0080] For example, the object detector may be a model designed based on at least one object detection model among R-CNN, SPP-net, Fast R-CNN, Faster R-CNN, SSD, and YOLO. That is, in an embodiment, the object detector may be implemented by including a neural network based on Fast R-CNN or Faster R-CNN, which has high accuracy and speed in object recognition of fine objects and accurately detects node and edge class objects.

[0081] In a compound, atoms are represented by element symbols and junctions, which are detected as nodes. Element symbols and junctions can be detected relatively accurately from a feature map. However, bonds are represented by lines (edges) between atoms. When using existing edge detectors, the detection accuracy is relatively low compared to nodes, and there are problems with not being able to properly recognize lines representing double or triple bonds or lines incorrectly represented by noise.

[0082] The concept of the invention is that, since bonds in a compound are formed between atoms, when detecting objects, if nodes are detected first and then information about the detected nodes is used as additional information to detect lines, lines can be detected accurately.

[0083] Therefore, the object detector according to the embodiment of the present invention is an object detector having a structure in which nodes are first detected and then information about the detected nodes is additionally input to detect edges.

[0084] More specifically, the object detector detects an ROI for a node, detects edge information LOI representing a bond through the ROI of the detected node and information about the molecular structure image (feature map), and then combines the ROI and LOI to generate object detection information.

[0085] Here, the object detection information may include node class information (atomic symbol, molecular formula of atomic group), node position information (e.g., coordinate information of the bounding box of each atom), edge class (e.g., single, double, triple bond, mirror image, isomer), and edge position information (e.g., edge coordinate information). Such object detection information may be in a JSON format in which information is written for each catalog.

[0086] Depending on the embodiment, such object detection information includes the class and position of the node, and information on the edges connected to each node, and therefore includes all information on atoms and bonds required for a molecular formula graph form, and can be basic information for conversion into a molecular formula graph form representation.

[0087] That is, the object detection information can be converted into a molecular structure graph Graph=N(nodes), E(edges) by a graph reconstruction operator.

[0088] As shown in FIG. 4C, an object detector according to an embodiment of the present invention may include a backbone that extracts a global feature map from an image including a molecular structure image, a Region Proposal Network (RPN) that extracts and proposes candidate regions (region proposals) in which an object may be included from the extracted global feature map, an ROI module that determines an ROI for a node based on the global feature map and the proposed candidate regions, and an LOI module that determines an LOI for an edge based on the node ROI and the global feature map.

[0089] More specifically, backbone can extract a global feature map for the entire image, including the molecular structure image.

[0090] In an embodiment, the backbone is a convolutional neural network (CNN) backbone, which can be implemented using ResNet, VGG, or the like.

[0091] A Region Proposal Network (RPN) can then extract and propose candidate regions for the object.

[0092] Specifically, the region proposal network can extract candidate regions that can contain nodes and propose multiple candidate feature maps for each candidate region.

[0093] To this end, the region proposal network can include an anchor generation layer that generates anchor boxes for the entire image, and a proposal layer that regresses node class scores and bounding boxes of candidate regions for each feature map of the generated anchor boxes.

[0094] Specifically, the anchor generation layer can generate anchor boxes that are classified into different grid cells relative to the entire image size.

[0095] The proposal layer can then extract multiple candidate regions by performing bounding box regression on the overall feature map using the class scores of each anchor box.

[0096] Specifically, the proposal layer applies non-maximum suppression to remove inappropriate objects, then extracts the top N anchor boxes with the highest class scores. Then, it applies regression coefficients to the anchor boxes to help them better detect the object's location, and finally detects candidate regions for the object.

[0097] Next, the ROI module can detect node detection information for the nodes based on the proposed candidate regions. Here, the node detection information can be a feature map for the node region corresponding to the node (atomic symbols and junctions) in the molecular structure image, or can be node detection information generated by classifying and regressing the feature map for the node region into bounding boxes.

[0098] Specifically, the ROI module can select samples for nodes among the candidate regions proposed through the proposal target layer. For example, candidate regions that match atomic symbols, which are nodes that can correspond to atoms, atomic group (e.g., functional group) symbols, and junctions can be determined as samples. That is, after calculating the likelihood of a candidate region corresponding to a node among multiple candidate regions as a score, candidate regions that meet or exceed a predetermined standard can be determined as positive samples.

[0099] Then, the ROI module can output a fixed-size node feature map for the node's positive samples through a layer that performs RoI pooling based on the determined positive samples and the overall feature map.

[0100] In addition, the ROI module (ROI: Region of Interest) inputs the feature map of the node into the pulley-connected layer to obtain a feature vector, then classifies the feature vector using a softmax function or the like, outputs each feature vector through a bounding box regressor, and finally outputs node detection information for the molecular structure image.

[0101] The LOI (Lines of Interest) module can output edge detection information representing bonds in the molecular structure to the ROI module based on the intermediate output data or / and the final output data and the overall feature map.

[0102] Here, the edge detection information refers to an edge that is a bond connecting different nodes, and may include information defining the coordinates of the edge and the nodes located at both ends of the edge.

[0103] In such an LOI module, edges for connections must be located between different nodes, so if intermediate data or / and final data of the ROI module and the entire feature map are input together, only edges for connections can be detected more accurately.

[0104] Such an LOI module is a convolution neural network (CNN)-based edge detector, and can be realized by modifying at least one edge detector structure such as Sobel, Prewitt, Scharr, and Canny Edge Detector.

[0105] Such an LOI module can also be a CNN-based deep network including an input layer, a convolution layer, a pooling layer, and a connected convolution layer.

[0106] The object detector configured in this way can input an entire image including a molecular structure image and output object detection information including ROIs for nodes and LOIs for edges.

[0107] In additional embodiments, the object detector may further include a segmentation module that classifies the class of the detected edges.

[0108] That is, the object detector can further generate a segmentation module as an additional head module, and include a segmentation module that classifies edges classified as single (single bond), double (double bond), triple (triple bond), or mirror image (UP, down, none) for the overall feature map extracted from the backbone.

[0109] Then, the object detector can add a class for each edge in the LOI by segmentation, and further include the edge class information in the object detection information.

[0110] In this case, the graph form reconstruction operator can add the added edge class information and reconstruct it into a graph form representation (Graph)=N(nodes), E(edges)=E(edges) Classification.

[0111] In addition, since the object detector may detect nodes incorrectly if the node detection information output from the ROI module does not match the edge detection information of the LOI module, the object detector can further perform an auxiliary task by feeding back the edge detection information of the LOI module to the ROI module. That is, the object detector feeds back data output from the LOI module, which is a lower layer, to the ROI module, causing the ROI module to output node detection information again.

[0112] Such an object detector can be trained based on a training dataset in which node and edge labeling operations have been performed on molecular structure images.

[0113] In one embodiment, the object detector applies a first loss function (detection loss) for node detection based on the labels for the nodes in the molecular structure image, and can train an ROI module, a region proposal network (RPN), and a backbone.

[0114] In addition, the object detector can apply a second loss function (WF loss) for edge detection based on the labels for edges in the molecular structure image to train the LOI module and backbone.

[0115] In another embodiment, the object detector can be trained by simultaneously applying node labels and edge labels to the molecular structure image based on a unifying loss for nodes and edges, and can learn the meaning of the compound structure between nodes representing atoms in the compound and edges representing bonds, thereby training to output more accurate object detection information regarding the molecular structure of the compound.

[0116] The processor 122 can then generate object prediction result information including object detection information including node detection information and edge detection information.

[0117] The processor 122 may also generate object prediction result information by converting the molecular structure into graph-type data based on the graph object detection information.

[0118] Thereafter, the processor 122 may further convert the linear-connection type object prediction result information into a Mol file (MDL Molfile). Here, the processor 122 may convert the linear-connection type structural formula prediction result information into a Mol file. For example, the processor 122 may convert the linear-connection type Lewis structure prediction result information into a Mol file, the linear-connection type simplified structural formula prediction result information into a Mol file, the linear-connection type skeletal structure prediction result information into a Mol file, or the linear-connection type projection prediction result information into a Mol file. In this case, the MOL file is in a general text format and may include molecular weight data information, atomic bond coordinates, and connectivity information.

[0119] 5 is a flowchart illustrating a method for predicting a line-connected object using artificial intelligence according to the present disclosure, and FIG. 6 is a diagram illustrating an example of a process for detecting atoms, bonds, and lines in the processor of FIG.

[0120] As shown in FIG. 5, the method for predicting linearly connected objects using artificial intelligence may include a receiving step (S510), a detecting step (S520), a generating step (S530), an outputting step (S540), and a converting step (S550).

[0121] The receiving step may receive images P1, P2, P3, ... including line-connected objects acquired from the image capturing device 10 via the communication unit 110 (S510). Here, the line-connected objects may include at least one of a map, a drawing, and a structural formula. In this case, the structural formula may include at least one of a Lewis structural formula, a simplified structural formula, a skeletal structural formula, and a projection formula. For example, the skeletal structural formula may be a stereochemical structural formula, an unspecified stereochemical structural formula, etc. As another example, the projection formula may be a Newman projection, a Haas projection, a Fischer projection, etc.

[0122] The detection step may detect a feature map for a state in which objects are connected by lines from images P1, P2, P3, ... including line-connected objects (S520) in order to perform artificial intelligence-based learning via the processor 122. Here, the processor 122 may detect a feature map for a state in which atoms and atomic bonds are connected by lines from an image including a structural formula via an object detector.

[0123] For example, as shown in FIG. 6, the processor 122 can detect an atomic symbol R1 that becomes a node on a region of interest (ROI) basis from an image including a skeletal structure, detect junctions R2 to R13 on a junction ROI basis, and detect edges L1 to L19 on a line of interest (LOI) basis.

[0124] Here, the processor 122 can simultaneously detect the atomic symbol R1 and the junctions R2 to R13 using the ROI module, and then detect the edges L1 to L19 using the LOI module.

[0125] Furthermore, the processor 122 can simultaneously detect the atomic symbol R1, junctions R2 to R13, and edges L1 to L19.

[0126] The generating step may generate object detection information regarding a state in which objects are connected by lines based on the feature map via the processor 122 (S530). Here, the processor 122 may generate object detection information regarding a state in which objects are connected by lines using an object detector based on the feature map. At this time, the processor 122 may generate object detection information regarding a state in which bonds between atoms are connected by lines using the object detector based on the feature map.

[0127] Here, the processor 122 may train the artificial intelligence model AIM using a training dataset including input values ​​of the Lewis structure image data ID1, simplified structure image data ID2, skeletal structure image data ID3, and projected image data ID4 corresponding to the structural formula image data among the metadata, and output values ​​labeling the corresponding nodes (atoms) and edges (bonds), and output the recommended linear-connection type structural formula prediction result information OD. The artificial intelligence model AIM may build and reinforce learning various Lewis structure image data ID1, various simplified structure image data ID2, various skeletal structure image data ID3, and various projected image data ID4 as a training dataset using an object detector. In this case, the memory 121 may store the linear-connection type structural formula prediction result information OD learned and analyzed based on the artificial intelligence model AIM. For example, the linear-connection type structural formula prediction result information OD may be linear-connection type Lewis structure prediction result information, linear-connection type simplified structure prediction result information, linear-connection type skeletal structure prediction result information, or linear-connection type projected prediction result information.

[0128] 7 to 13 are diagrams showing an example of a process of generating object detection information based on atomic symbols and link information between junctions and edges in the processor of FIG.

[0129] 7, in the case of a skeletal structure, the processor 122 can generate object detection information after maintaining labels for necessary junctions R15 and removing labels for unnecessary junctions R17 based on preset link information between atomic symbols, junctions, and edges, in order to improve the recognition rate of atomic symbols R14 and the calculation speed. Here, R15 and R16 may be atomic symbols or junctions. Furthermore, L21 to L23 may be edges. In this case, the link information may be atomic symbol type information, junction shape information, edge connection information, or shape information of the structure formed by the atomic symbols, junctions, and edges. Meanwhile, for convenience of explanation, the processor 122 of the present disclosure is illustrated as a skeletal structure formula in order to improve the recognition rate of the atomic symbol R14 while improving the calculation speed, but the processor 122 of the present disclosure can also be applied to a Lewis structure formula, a simplified structure formula, or a projection formula.

[0130] As shown in FIG. 8, in the case of a Lewis structure, the processor 122 may generate object detection information after maintaining labels for necessary junctions R21 to R24 and removing labels for unnecessary junction R25 based on preset link information between atomic symbols, junctions, and edges, in order to improve the recognition rate of atomic symbols R18 to R20 and junctions R21 to R24 while improving calculation speed. Here, L24 to L28 may be edges. Here, the link information may be atomic symbol type information, junction shape information, edge connectivity information, or shape information of the structure formed by the atomic symbols, junctions, and edges. Here, the junction shape information and edge connectivity information may be defect information where a connection is broken due to a junction. Meanwhile, for convenience of explanation, the processor 122 of the present disclosure is illustrated as a Lewis structure formula in order to improve the calculation speed while increasing the recognition rate of atomic symbols R18 to R20 and the recognition rate of junctions R21 to R24, but the processor 122 of the present disclosure can also be applied to a skeletal structure formula, a simplified structure formula, or a projection formula.

[0131] As shown in FIG. 9, in the case of a skeletal structure, the processor 122 predicts the shapes of the junctions R28 to R34 based on association information between preset atomic symbols R26 and R27 and edges, and labels the predicted shapes of the junctions R28 to R34 to generate object detection information, in order to improve the recognition rate of the junctions R28 to R34 and the calculation speed. Here, R26 and R27 may be atomic symbols, and L30 to L34 may be edges. Here, the association information may be atomic symbol type information, junction shape information, edge connectivity information, or shape information of the structure formed by the atomic symbols, junctions, and edges. Here, the junction shape information and edge connectivity information may be edge connectivity information forming the junction shape for each atomic symbol type. Meanwhile, for convenience of explanation, the processor 122 of the present disclosure is illustrated as a skeletal structure formula in order to improve the calculation speed while increasing the recognition rate of the junctions R28 to R34, but the processor 122 of the present disclosure can also be applied to a Lewis structure formula, a simplified structure formula, or a projection formula.

[0132] 10 and 11, in the case of a skeleton structure, in order to improve the recognition rate of junction R35 and improve the calculation speed, processor 122 may predict the shape of junction R35 based on graph loss information, label the predicted shape of junction R35, and then generate object detection information using an object detector. In this case, L35 may be an edge when the shape of junction R35 is normally predicted, and as shown in FIG. 11, processor 122 may generate object detection information corresponding to the case when the shape of junction R35 is normally predicted using an object detector. Furthermore, L36-1 and L36-2 may be edges when the shape of junction R35 is predicted abnormally, and as shown in FIG. 11, processor 122 may generate object detection information corresponding to the case when the shape of junction R35 is predicted abnormally using an object detector.

[0133] To improve this, the present disclosure can analyze the connectivity between atomic symbols and junctions and edges based on graph loss information between atomic symbols and junctions and edges.

[0134] 12 and 13, in the case of a skeleton structure, the processor 122 may analyze the connectivity between the atomic symbol R40 and the junctions R41, R43, and R44 and the edges L38, L39-1, L39-2, L40, L41-1, and L41-2 based on graph loss information. Here, the graph loss information may include at least one of the atom shape, bond shape, angle between lines, line length, line shape, and graph outline.

[0135] 12, the processor 122 can generate object detection information after retaining labels for necessary junctions R40 and R41 and removing labels for unnecessary junction R42 based on at least one of the atomic shape, bond shape, angle between lines, line length, line shape, and graph outline between atomic symbol R40 and junction R41 and edges L38, L39-1, and L39-2. In this case, L38 may be a loss calculated only from true positive samples, and L39-1 and L39-2 may be losses incorrectly calculated due to false positive samples.

[0136] 13, the processor 122 can generate object detection information by retaining labels for necessary junctions R43 and R44 and removing labels for unnecessary junction R45 based on at least one of the shape of the connections between junctions R43 and R44 and edges L40, L41-1, and L41-2, the angle between the lines, the length of the lines, the shape of the lines, and the overall shape of the graph. In this case, L40 may be a loss calculated only from true positive samples, and L41-1 and L41-2 may be losses incorrectly calculated due to false positive samples.

[0137] Meanwhile, as shown in Figures 12 and 13, the processor 122 of the present disclosure is illustrated as a skeletal structure formula for convenience of explanation, but the processor 122 of the present disclosure can also be applied to a Lewis structure formula, a simplified structure formula, or a projection formula.

[0138] The output step may output line-connected type object prediction result information learned and analyzed based on the object detection information via the processor 122 (S540). At this time, an image including line-connected type objects may be input to the input unit 130 in order to predict line-connected type objects. Here, a user may directly input an image including line-connected type objects using the input unit 130. For example, a user may scan an image including line-connected type objects and directly input the scanned image including line-connected type objects using the input unit 130.

[0139] The processor 122 can output linear-connection type object prediction result information learned and analyzed based on the artificial intelligence model AIM corresponding to the input information of the input unit 130. In this case, even when input information including error information is received, the processor 122 can learn and analyze based on the artificial intelligence model AIM and output linear-connection type object prediction result information corrected based on the analysis result. For example, the processor 122 can output linear-connection type structural formula prediction result information OD learned and analyzed based on the artificial intelligence model AIM.

[0140] Here, the display unit 140 may also display the linear-connection type object prediction result information output by the processor 122. In this case, the display unit 140 may display the linear-connection type structural formula prediction result information OD. For example, the display unit 140 may display the linear-connection type Lewis structure prediction result information, the linear-connection type simplified structural formula prediction result information, the linear-connection type skeletal structure prediction result information, and the linear-connection type projection prediction result information. The display unit 140 may display reliable linear-connection type object prediction result information to the user based on the results accurately analyzed by the processor 122.

[0141] Meanwhile, the conversion step may further convert the linear-linked object prediction result information into a Mol file (MDL Molfile) via the processor 122 (S550). Here, the processor 122 may convert the linear-linked structural formula prediction result information into a Mol. For example, the processor 122 may convert the linear-linked Lewis structure prediction result information into a Mol, the linear-linked simplified structural formula prediction result information into a Mol, the linear-linked skeletal structural formula prediction result information into a Mol, and the linear-linked projection formula prediction result information into a Mol.

[0142] When a line-connected object is a structural formula, the present disclosure can reduce errors in predicting junctions as atomic symbols and improve the analysis speed of the structural formula. In addition, the present disclosure can improve the analysis accuracy and calculation speed by using a junction junction algorithm and an atomic symbol detection algorithm together.

[0143] Meanwhile, the present disclosure can be applied to the structural formula of an inorganic compound that does not have eight outermost electrons, and can also output linear-connection type structural formula prediction result information for the structural formula of the inorganic compound based on object detection information.

[0144] Meanwhile, for convenience of explanation, the present disclosure has illustrated outputting line-connected type structural formula prediction result information based on object detection information, but the present disclosure is not limited to this, and may further output at least one of line-connected type map prediction result information and line-connected type drawing prediction result information, which are necessary to distinguish between lines and objects.

[0145] 1 to 4C, at least one component may be added or removed depending on the performance of the components, etc. Furthermore, it will be readily understood by those skilled in the art that the relative positions of the components, etc. may be changed depending on the performance or structure of the system.

[0146] Although FIG. 5 describes multiple steps as being executed sequentially, this is merely an illustrative example of the technical concept of this embodiment, and a person having ordinary knowledge in the technical field to which this embodiment pertains would be able to modify and alter the order described in FIG. 5 or execute one or more of the multiple steps in parallel, and apply various modifications and variations, without departing from the essential characteristics of this embodiment, and therefore FIG. 5 is not limited to a chronological order.

[0147] Meanwhile, the disclosed embodiments may be implemented in the form of a recording medium storing computer-executable instructions. The instructions may be stored in the form of program code, and when executed by a processor, may generate program modules to perform the operations of the disclosed embodiments. The recording medium may be implemented as a computer-readable recording medium.

[0148] Computer-readable recording media include all types of recording media that store instructions that can be decoded by a computer, such as ROM (Read Only Memory), RAM (Random Access Memory), magnetic tape, magnetic disk, flash memory, and optical data storage devices.

[0149] The disclosed embodiments have been described above with reference to the accompanying drawings. Those skilled in the art will understand that the present disclosure may be embodied in forms different from the disclosed embodiments without changing the technical concept or essential features of the present disclosure. The disclosed embodiments are illustrative and should not be construed as limiting. [Industrial Applicability]

[0150] The present invention relates to a prediction device and method for converting molecular structure formulas of compounds in molecular structure formula images into data using an artificial intelligence model including a deep network, and therefore has industrial applicability.

Claims

1. A line-connection type object prediction device utilizing artificial intelligence, Memory and a processor electrically connected to the memory and configured to control operations related to the linear-connection type object prediction; Equipped with The processor: Receive an image containing a molecular structure image, Detecting nodes corresponding to atoms of a molecular structure from the received image to generate node detection information; detecting edges corresponding to bonds in the molecular structure based on the node detection information and the received image to generate edge detection information; The line-connection type object prediction device outputs prediction result information of the molecular structure image including the node detection information and edge detection information.

2. The processor: The linear-connection type object prediction device of claim 1 , wherein the linear-connection type object prediction device outputs a global feature map for the received image via a backbone network.

3. The processor:

3. The linear-connection type object prediction device according to claim 2, wherein a region proposal network (RPN) is used to extract a plurality of candidate regions predicted to be nodes based on the output global feature map.

4. The processor:

4. The line-connection type object prediction device according to claim 3, wherein a ROI (Region of Interest) module is used to classify and detect nodes based on the overall feature map and the extracted plurality of candidate regions, and output node detection information.

5. The ROI module: The linear-connection type object prediction device of claim 4 , wherein a positive sample for a node is determined from the plurality of candidate regions through a proposal target layer.

6. The ROI module: The linear-connection type object prediction device of claim 5 , further comprising: performing Route of Inference (ROI) pooling based on the determined positive samples and the overall feature map; and outputting a feature map of nodes with a fixed size for the positive samples.

7. The ROI module: The feature map of the node is input to a pulley-connected layer to output a feature vector; 7. The linear-connection type object prediction device of claim 6, wherein the output feature vector is subjected to classifying and bounding box regressing to output node detection information for the molecular structure image.

8. The processor:

10. The line-connected object prediction device of claim 7, further comprising: detecting edges based on the node detection information and the overall feature map using a line of interest (LOI) module that is a CNN-based edge detector; and outputting the edge detection information.

9. The processor:

9. The line-connection type object prediction device according to claim 8, wherein the graph representation of the molecular structure is converted into data based on object detection information including the node detection information and the edge detection information through a graph reconstruction operator, and prediction result information of the molecular structure image is output.

10. The processor:

9. The line-connection type object prediction apparatus according to claim 8, wherein edge detection information output from the LOI module is fed back to the ROI module, and an auxiliary task of re-executing node detection is performed.

11. The processor:

9. The line-connected object prediction device of claim 8, wherein the LOI module, ROI module, region proposal network, and backbone are trained based on a training dataset in which node labels and edge labels are set for training images including molecular structure images.

12. The processor:

12. The line-connected object prediction device of claim 11, wherein the node labels and edge labels for the molecular structure image are simultaneously applied based on a unifying loss for the nodes and edges, and the LOI module, ROI module, region proposal network, and backbone are trained.

13. The processor: The segmentation module classifies the edges by edge class and further outputs edge classification information; The line-connection type object prediction device of claim 1 , wherein the output edge classification information is further included in the object detection information.

14. A method for predicting a molecular structure image using artificial intelligence, which is carried out by a device, receiving an image including the molecular structure image; detecting a feature map for connecting objects with lines from the image; detecting nodes corresponding to atoms of a molecular structure from the received image to generate node detection information; detecting edges corresponding to bonds in the molecular structure based on the node detection information and the received image to generate edge detection information; outputting prediction result information of the molecular structure image, including the node detection information and edge detection information; A method comprising:

Images